Offline FCC Memory Issues

Hello! I work for a company as an instructor where we teach residents inside Correctional Facilities how to code. I apologize if this forum post steps over a line about information requesting on how FCC handles deployments or something similar. We have used an older version of FreeCodeCamp with custom auth procedures for many years now and it has been outdated for a long time. I have stepped up to the plate to make a new version of it using current repository information.

I have everything working so far aside from one issue. There is a memory overflow when running pnpm run start-ci, (gatsby exits with EMFILE: too many open files, open ‘\client\public\404.html’) or even pnpm run develop (which we originally thought was due to verbose logging, but appears to be the same problem, especially since we have 20 concurrent users).

The 404 is being called from transformers.js on line 273: frame.contentDocument.documentElement.innerHTML = contents;. When the challenge is a multifile challenge and the link to styles.css is present, the client will try asking gatsby for the file in four different ways (as viewed through procmon.exe) before returning a 404.html file. However, due to the concurrency and the frequency of students updating their iframes as they type, we are reaching memory heap issues within about 3 hours or so (two times a class). I have tried adding a redirect in serve.json, and I have not noticed any relief of requests (in fact, page-data and app-data started being called at the end of challenge paths as well). I also added a ServiceWorker to handle requests to styles.css and force cancel them, but the classroom environment is on VLAN, so there is no https protocol implemented and the Service Workers will not run because of security protocols. As a last resort, I tried kubernetes to do load balancing, but I’m hitting a dead end with it as it is out of my expertise (I have not worked with it much in the past, sadly).

This issue is not just restricted to our runtime, analyzing the network requests across the production version of FreeCodeCamp shows the same attempts. I am wondering what is done on your end to prevent the overflow, especially as you scale up?

We are working in Windows 11 with everything else installed according to the contribution guide for setting up locally, no challenge or user storage logic has been touched, just CSS, components, and authentication/session logic. We can move over to a cloud provider once items work as expected, we also have docker available to us.

Thank you in advance for any advice you can offer, and I greatly appreciate you taking the time to read it! If anything doesn’t make sense, let me know and I will try my best to append those changes.

TL;DR: Sorry if in the wrong category, but I am having memory heap crashes on 20 user environments and am looking for possible solutions to this problem.

Hello there,

I am not adept with what you are trying to do, and have never seen that error, but will mention some things that come to mind:

We do not even do this, as the server instances are quite simple. For 20-ish concurrent users, I would be very surprised if multiple instances of the server were necessary.

At most, we handle the traffic through a load balancer: Contribution Guidelines | freeCodeCamp.org

We use pm2 to run multiple instances of the server: Contribution Guidelines | freeCodeCamp.org

The only other thing I can think to mention is it sounds odd you are using the CI script we have to handle this. Personally, I would just follow the devops flight manual to the T (almost).

I do not know how you would handle all the dynamically loaded content, but assume you have already crossed this bridge.

Some gotchas when using Windows:

  1. We do not support it
  2. File names - often cause issues, because the file names are “too long”
  • I doubt this would have an effect on a production build, though :thinking:

Question:

  • Does each user have their own hardware? Or, is the vm running the server also handling the client browsers?

Maybe @raisedadead can provide assistance?

Hey @zdubois

Thanks for reaching out.

Sorry if in the wrong category, but I am having memory heap crashes on 20 user environments and am looking for possible solutions to this problem.

Can you confirm if this means you are literally running 20 instances of freeCodeCamp locally on 20 machines? OR you have 20 users connecting to one instance of the server.

Just to give you some insights:

  1. Our codebase absolutely needs a linux-like environment.

    If you have Windows 11, you will be needing to use WSL with a distro like Ubuntu on it to get it working.

  2. The deployment (production-like) is supposed to be run so that:

    • You have one or more servers/machines running the API (code in /api-server) with PM2.
    • You have one or more servers/machines running the client (code in the client).
    • We have devised scripts and templates for this. You are welcome to read more about this in our devops handbook: https://contribute.freecodecamp.org/#/devops
    • Some assets need internet access because we load them from our CDN.
    • When run in production, we can handle 500-600 requests/sec from thousands of users on machines with 2 vCPUs and 4 GiB of memory.
    • The databases are not local and run on other dedicated providers. We configure them via the connection strings.
  3. We have CI/CD processes that build and deploy the codebase in production mode to independent machines for both the API and the Clients.

Now, your situation is probably completely different because I am assuming you will not have access to the internet (for following a setup similar to us)?

Since you mentioned K8s, just so that you know we do not have OCI images for the services yet, we are working on a such a setup but we do not have a timeline. Once we do you should be able to tweak things and run it on docker or other container solutions of your choice.

You can in theory run freeCodeCamp’s codebase in a local only single server setup with some hacks. That said even that needs a decent machine with at least 8-12 GB memory because of all the cruft in the tooling we have at the moment.

My advice is if you have a few machines that you can cobble together in a LAN then you should be able to provision them like we do, by building a production binary (devops handbooks may have some details on this) and then swap out the parts in code that need internet access namely:

  • Authentication
  • CDN for loading static assets like stylesheets and images, etc.

If you have specific questions, please get in touch with us and we are happy to advice the best we can.

Hi! Sorry for the long time before responding, things were busy on my end.

The instance was being run from a local server and available to our users on LAN.

After my previous posting I managed to ask another individual within the company to start up a droplet on Digital Ocean. I already was aware that Windows would not work, but it was all I had available at the time and it appears the heap overflow was just because Windows does not like all of that file movement.

Using the droplet we have not had a single crash that has not been caused by our own hands (aka me trying to push changes as we are still working on a pipeline to do so). I switched the database over to a mongoDB Atlas cluster and we are definitely seeing an increase in responsiveness overall. The reason why we cannot use FreeCodeCamp itself is due to outside concerns that may jeopardize our relationship with grantmakers.

This was one of the first requirements before we actually started letting students actively use it. This was actually quite an interesting fix however, we managed to change ajax.ts and authentication.js as well as some of the passport configuration before it gets to @freeCodeCamp/loopback-component-passport and we haven’t had an issue since.

This was resolved as of last night actually! I had made a custom file to go through a returned search of VSCode using regex with about 95% accuracy and then pipe the data into a file. After that the CDN links were changed over to the other endpoint.

Here is that regex by the way, just because it is humorous: (/^)?(https:)((\/\/)|(//))((cdn)|(unpkg)|(fonts)).*?((?=("|`))|(.+$/i)).

Thank you for your time and I’m sorry that it was pretty much resolved after taking the time to understand documentation and move items over to their appropriate environments. I’ll have to ask if I can provide more in depth information about this overall. Please let me know if you have questions about the process we took to make it work.

1 Like

The reason why we cannot use FreeCodeCamp itself is due to outside concerns that may jeopardize our relationship with grantmakers.

That’s fine.

We are licensed as BSD-3-Clause - so you are more than welcome to hack on it, we appreciate the attribution though :slight_smile:.

Please let me know if you have questions about the process we took to make it work.

Yes - Please consider writing about it (here in this thread or a blog) so that others like you may benefit from your work.

Good luck and happy coding!