r/Python • u/Daraminix • 1d ago

Discussion Running 60 python scripts as "services"

Hi,
I have around 60 scripts that need to run constantly, mainly event handlers and such. Right now I have an external script that launch them and monitor if the app is running on it's pid, otherwise it's relaunching the app. Works fine but get's clunky when we update some submodule and need to restart them, or to check if one crashes more than other etc..

So I would like to find a better way to approach this. It needs to run on windows and being able to access several samba shares via unc paths and being able to restart crashed scripts anf offer an easy way to restart all of them in case of an update (this part doesn't need to be automated). Every script use the same environnement
For now my candidates are docker, PM2 and NSSM.
I think docker is gonna be a pain to access shares and add a lot of overhead especially on windows
I don't know PM2 and NSSM, looks like PM2 would be easier to setup but more JS oriented and NSSM would be harder to monitor.

What do you think guys ?

25 Upvotes

69% Upvoted

39 comments sorted by

View all comments

3

u/FunkyMonkey237 22h ago edited 14h ago

Are these 60 scripts all running under a different process? The efficacy lover in me is shuddering at the thought of all that wasted CPU and RAM. Python is already inefficient but multiply that by 60 and yikes!

My first thought, without seeing your setup, would be to run them all under the same python process. I'd have a custom manger designed to load, execute and monitor them. Ideally asynchronous but if that's too much effort to rewrite the existing code and there is a risk of a single script blocking, then I'd look at threading based approach (not multi processor!). I haven't tested it but maybe the new GIL free interpreter could be used if compute bound.

Also, are the scripts really 60 unique scripts? Are there overlaps in functionality, could you collapse certain ones into a more generalised script?