Im attempting to run a Fast-Api server in [python](advertisment redacted) that handles a process that takes around 30-40 seconds to complete. The Second script is using the Selenium Module to gather images that get processed so I don’t think it can be handled with normal async operation. I’m looking to have this second script function as a sort of background worker that doesn’t stall other requests from being ran.
At the moment i have the server exectute the program and return its output but the problem is it stalls the server and goes 1 by 1. Ideally this process would run multiple at once and return them as completed.
example of my code: FastApi App:
from fastapi import FastAPI
import sys
import subprocess
app = FastAPI()
def run_program():
s2_out = s2_out = subprocess.check_output([sys.executable, "script2.py"])
return s2_out
@app.get("/")
async def read_root():
return run_program()
script2.py Code (What The Server is getting output from):
import sys,time,random
counter = 0
def main():
global counter
timeValue = random.randint(0,10)
time.sleep(timeValue)
print(f'Run Number:{counter} | Finished after:{timeValue} Seconds')
counter += 1
if __name__ == "__main__":
main()
ServerTest.py Code (What Im testing the server with.)
from concurrent.futures import ThreadPoolExecutor
import requests
Counter = 0
def main():
global Counter
PreviousNum = Counter
print(f'Sending Request Number:{PreviousNum}')
r = requests.get('http://127.0.0.1:8000')
print(f'Request Number:{PreviousNum} Finished With Response:{r.content.decode()}')
with ThreadPoolExecutor() as executor:
for x in range(10):
executor.submit(main)
Counter += 1
ServerTest.py returns
Sending Request Number:0
Sending Request Number:1
Sending Request Number:2
Sending Request Number:3
Sending Request Number:4
Sending Request Number:5
Sending Request Number:6
Sending Request Number:7
Sending Request Number:8
Sending Request Number:9
Request Number:3 Finished With Response:"Run Number:0 | Finished after:10 Seconds\r\n"
Request Number:6 Finished With Response:"Run Number:0 | Finished after:10 Seconds\r\n"
Request Number:2 Finished With Response:"Run Number:0 | Finished after:0 Seconds\r\n"
Request Number:5 Finished With Response:"Run Number:0 | Finished after:10 Seconds\r\n"
Request Number:9 Finished With Response:"Run Number:0 | Finished after:6 Seconds\r\n"
Request Number:4 Finished With Response:"Run Number:0 | Finished after:5 Seconds\r\n"
Request Number:7 Finished With Response:"Run Number:0 | Finished after:8 Seconds\r\n"
Request Number:8 Finished With Response:"Run Number:0 | Finished after:1 Seconds\r\n"
Request Number:0 Finished With Response:"Run Number:0 | Finished after:2 Seconds\r\n"
Request Number:1 Finished With Response:"Run Number:0 | Finished after:5 Seconds\r\n"
I know that Run Number is off. Its something I should’ve passed as an argument instead of putting inside the script. But all of the items went 1 by 1. That cause it to take 67 seconds.
The Ideal way and way I am looking for this server to run is the total time taking around 10-11 seconds. Since 10 seconds was the highest sleep value.
Is there a way I can have the server run script2.py through subprocess in a new thread and return each value as they complete instead of waiting 1 by 1?.
I don’t think subprocess supports using returning values through async.
Any help at all Is appreciated. Thank you