Run external program without hanging server fast-api

Im attempting to run a Fast-Api server in [python](advertisment redacted) that handles a process that takes around 30-40 seconds to complete. The Second script is using the Selenium Module to gather images that get processed so I don’t think it can be handled with normal async operation. I’m looking to have this second script function as a sort of background worker that doesn’t stall other requests from being ran.

At the moment i have the server exectute the program and return its output but the problem is it stalls the server and goes 1 by 1. Ideally this process would run multiple at once and return them as completed.

example of my code: FastApi App:

from fastapi import FastAPI
import sys
import subprocess
app = FastAPI()

def run_program():
    s2_out = s2_out = subprocess.check_output([sys.executable, "script2.py"])
    return s2_out

@app.get("/")
async def read_root():
    return run_program()

script2.py Code (What The Server is getting output from):

import sys,time,random
counter = 0

def main():
    global counter
    timeValue = random.randint(0,10)
    time.sleep(timeValue)
    print(f'Run Number:{counter} | Finished after:{timeValue} Seconds')
    counter += 1

if __name__ == "__main__":
    main()

ServerTest.py Code (What Im testing the server with.)

from concurrent.futures import ThreadPoolExecutor
import requests
Counter = 0
def main():
    global Counter
    PreviousNum = Counter
    print(f'Sending Request Number:{PreviousNum}')
    r = requests.get('http://127.0.0.1:8000')
    print(f'Request Number:{PreviousNum} Finished With Response:{r.content.decode()}')
with ThreadPoolExecutor() as executor:
    for x in range(10):
        executor.submit(main)
        Counter += 1

ServerTest.py returns

Sending Request Number:0
Sending Request Number:1
Sending Request Number:2
Sending Request Number:3
Sending Request Number:4
Sending Request Number:5
Sending Request Number:6
Sending Request Number:7
Sending Request Number:8
Sending Request Number:9
Request Number:3 Finished With Response:"Run Number:0 | Finished after:10 Seconds\r\n"
Request Number:6 Finished With Response:"Run Number:0 | Finished after:10 Seconds\r\n"
Request Number:2 Finished With Response:"Run Number:0 | Finished after:0 Seconds\r\n"
Request Number:5 Finished With Response:"Run Number:0 | Finished after:10 Seconds\r\n"
Request Number:9 Finished With Response:"Run Number:0 | Finished after:6 Seconds\r\n"
Request Number:4 Finished With Response:"Run Number:0 | Finished after:5 Seconds\r\n"
Request Number:7 Finished With Response:"Run Number:0 | Finished after:8 Seconds\r\n"
Request Number:8 Finished With Response:"Run Number:0 | Finished after:1 Seconds\r\n"
Request Number:0 Finished With Response:"Run Number:0 | Finished after:2 Seconds\r\n"
Request Number:1 Finished With Response:"Run Number:0 | Finished after:5 Seconds\r\n"

I know that Run Number is off. Its something I should’ve passed as an argument instead of putting inside the script. But all of the items went 1 by 1. That cause it to take 67 seconds.

The Ideal way and way I am looking for this server to run is the total time taking around 10-11 seconds. Since 10 seconds was the highest sleep value.

Is there a way I can have the server run script2.py through subprocess in a new thread and return each value as they complete instead of waiting 1 by 1?.

I don’t think subprocess supports using returning values through async.

Any help at all Is appreciated. Thank you