Hi, this is my first post here. I’m hoping someone with good knowledge in HTTP requests would be able to find the root of the problem.
I’m trying to adapt this example antoinevastel .com/nodejs/2022/02/26/nodejs-optimized-https-proxy-no-dependencies.html to tunnel HTTPS requests through a HTTP tunnel. I need to throttle the requests from several different processes running in parallel to avoid hitting a remote server’s limits and I use my local proxy server for that.
I published the Class with the proxy and the request logic in a gist here gist.github .com/Funecio-Agrante/a01f367cff6a0acbaf1ce90558abf5bd
I was able to adapt the code and make it work. I made a small change where instead of checking if the variable for the Agent object is null I check directly if the socket that Agent is using is open.
Some processes make recurring requests at a regular intervals that usually don’t last more than a few minutes. Those do not seem to find any problems. But some other processes create tasks that can run for days or weeks, constantly making requests and processing data. While running 2 of these persistent tasks at the same time they get interrupted at around 4000 seconds after the script is initialized without throwing any error (every time). If I run just one, it gets interrupted in little over 2000 seconds, so about half the time. I can’t figure out why this is happening or how to make the code show me an error when this happens.
The only reporting I get is that the Agent socket closes without errors, but I don’t know why. The Agent socket closing also happens due to inactivity when running the other shorter processes, but the code just creates a new Agent when a new request is initiated. The problem is in this situations where the execution stops around 2000 / 4000 seconds the processes are making several requests per second and there is no reason for timeouts. This happens both when the shorter processes are running and when they are stopped, so they don’t seem to have any influence.
Before switching to my custom proxy, I was making these requests directly with node-fetch and there was no such problems. I also tried using node-fetch and fetch with the Agent to my proxy in the options. I do get a connection, but I get data even if the local proxy is not running and the code to throttle the requests is not executed.
What is causing this behavior and how can I catch it? I set up all the event emitters that made sense to me. Any help is appreciated. Thanks.