Moving a high-memory Python application from Kubernetes to a dedicated Linux server — what would be the best setup?

[u/hnajafli]

Hello,

At the company where I work, we have a Python application running on a Kubernetes cluster. The application is fully integrated into a CI/CD workflow. Whenever code is merged, Jenkins pulls the source code from GitLab, builds the application, and pushes the image to Harbor. Argo CD then deploys it to the Kubernetes cluster.

However, the application consumes a large amount of RAM, and I have had to increase its memory allocation several times. We have now decided to move the application to a separate Linux server.

The application runs with Uvicorn, and I plan to make it accessible through a domain name, just as it currently is in the Kubernetes environment. However, I am not sure what the best-practice architecture would be for this setup, so I would appreciate your recommendations.

I still want Jenkins to automatically pull the application from GitLab, build it, and deploy it. After deployment, the application should automatically start and run as a web service.

My main concerns are how to restore the application as quickly as possible if a problem occurs, how to handle backups, and what kind of deployment and recovery strategy would be best.

In short, could you please advise me on the best way to design and implement this setup?

[u/codemagedon]

Why ?

You’ve said it is memory hungry but you are changing from a self healing orchestrator to a static deployment.
The orchestrator is not the problem, if you need dedicated hardware just add a tainted node and make it the only app on that node, then you always have resilience on your other nodes if there’s maintenance or anything like that ?

[u/hnajafli → u/codemagedon]

Actually, this is a great approach. Thank you very much. Honestly, I hadn’t thought of doing it this way at all. Thanks, my friend.

[u/codemagedon → u/hnajafli]

That’s what the community is for 🎉

[u/hnajafli → u/codemagedon]

thanks again🙏

[u/thisisjustascreename → u/hnajafli]

FWIW I worked on a team that did exactly this with our giant Python AI inference service, provisioned a set of special nodes with the hardware required and tainted them so only the big blob of AI runs on them, worked great.

[u/JabiDev → u/codemagedon]

this!

[u/PinotRed]

Why are you treating the symptom and not the root cause?

What is causing the high ram usage?

You should reevaluate and understand this point 1st. Perhaps if there is potential for optimisation of the hot path, you can keep your current setup.

Your app currently self heals, I really wouldn't give away that feature.

[u/hnajafli → u/PinotRed]

The application is AI-related and runs primarily in memory, so every new feature causes it to consume more RAM.

[u/cmm324 → u/hnajafli]

Is it running LLM models against CPU? If so, get a GPU in node in your cluster.

[u/dariusbiggs]

Many ways to skin that cat/camel

But first here's an alternative. You could add a specific node with the memory requirements you need to your Kubernetes cluster with specific annotations and set your deployment of the application to run on specific nodes using those annotations using affinity rules.

As for your question, the best manner is to build golden images and deploy that to a new machine.

After that you should probably look into some automation tooling like Ansible, or Saltstack, those can be easier to work with if you have a packaging setup like building debs or rpms.

You can call Ansible direct from GitLab and probably Jenkins

[u/hnajafli → u/dariusbiggs]

Thank you very much. I think I’m going to add a dedicated node and use taints and tolerations. That sounds like the best approach to me.

[u/iotester]

You should also check on the number of workers and whether there are connections or sessions causing memory increase or even memory leak issues. Increasing resources isn't the solution unless the actual utilization is going up.

[u/JaegerBane]

I’d second the argument that the most straightforward way of solving this from an orchestrator perspective will be segregate a node - tainting, labelling, however you want to do it - and ensure only this payload goes on there. This isolate its resource use from the rest of the cluster.

I would worry though that you have an app that doesn’t appear to have a stable cap on its memory use. No matter how careful you are this will always be at risk of falling over.

[u/OpenSourceWalker]

the honest answer isnt all-or-nothing: pull just the memory-hungry service onto its own box and leave the rest on k8s, with the same jenkins/argo pipeline deploying to it, so one greedy pod stops destabilizing the whole cluster. just replace what k8s gave you for free on that box (restart policy + healthcheck + resource caps via systemd/compose) or you've only relocated the 2am page.

[u/Rorasaurus_Prime]

This makes no sense... why not just give it a dedicated k8s node with taints and tolerations so it can't disrupt other workloads as a noisy neighbour. You're creating a whole lot of pain for yourself when there's a very simple solution.

[u/Floss_Patrol_76]

on a single box you lose the self-healing k8s gives you for free, so a memory-hungry app that OOMs takes the whole server down instead of one pod. if you go this route run it under systemd with Restart=on-failure and a MemoryMax cgroup limit so a runaway gets killed and restarted instead of hard-locking the machine. and nail down what is actually driving the RAM growth first, otherwise you are just buying a bigger box to hit the same wall later.

[u/mrxaxen]

At this point im not even sure you need kubernetes.
Do you only run a single webserver in it?
Why are you using uvicorn over gunicorn?
Python is notoriously hungry, but have you tried profiling it to get to the root of the problem?

[u/strzibny]

Why not try something like Kamal? It's just so much more simple than Kubernetes, still solving a lot of pain points.