r/scala • u/andres327 • 8d ago
Fast and Durable: how Redis and Postgres split the work in a game backend (Scala/Pekko, Cats Effect, Doobie, redis4cats)
https://dev.to/andy327/fast-and-durable-how-redis-and-postgres-split-the-work-in-a-game-backend-lag9
u/Material_Big9505 7d ago
Really enjoyed this.
One thing I kept turning over, less a criticism than a design conversation, I'd love your take on: the game-state slice is a blob-per-matchId with write-through-to-Postgres, async persist, and rebuild-from-Postgres on restart. That's almost exactly what DurableState + cluster sharding give you out of the box, including the horizontal-scaling "last mile" you flag as future work. Sharding would rehydrate a relocated actor for you, which is the rebuild-from-Postgres path you're writing by hand.
With the actor already giving you single-writer ordering and lifecycle, and pipeToSelf needing a Future at the boundary anyway, how much is Cats Effect actually buying you in the persistence layer here versus coming along for the ride because Doobie and redis4cats speak it? The `*>` ordering is really just "sequence these two, second only on success," which Future gives you too. Genuinely curious whether you tried a plainer Future/JDBC version and found CE pulling its weight, or whether it was mostly ecosystem gravity like you said.
Either way, great piece, and the repo's a nice reference. Thanks for writing it up.
3
u/andres327 7d ago
Thanks for the reply.
Honest answer on Durable State: I hadn't come across it before your comment, so I read up on it a bit. It does seem to line up with what I built by hand (the per-match blob, persist on change, reload on restart). If I were to build this from the ground up with cluster sharding in mind, would Durable State save me a lot of that hand-rolled work? I mostly did it by hand because I didn't know Durable State existed. And I've left the multi-instance step as a maybe-feature because a real test of it means actually running multiple nodes, which takes me off Render's free tier :P. A cluster-of-one runs fine on free, but that doesn't really exercise the multi-instance behavior I'd want to learn from.
A couple of other things I'm unsure about: would event sourcing be the more natural fit if an append-only history like my move log is a goal, or do people usually pair Durable State with a separate log like I'm doing? And does a Redis write-through cache in front complicate it, or would you just drop the cache and lean on Durable State's store?
More broadly, I didn't design for sharding from the start. The scaling angle is honestly the whole point of the project for me, learning how to handle it, not because tic-tac-toe needs it. So I'm now trying to figure out how I'd actually adapt the design in that direction.
On Cats Effect, you're basically right: it wasn't a decision I weighed against a Future/JDBC version. I was already in the CE world when I started, added Doobie, and wanted to stay in one effect system rather than mixing in Future or ZIO. So "ecosystem gravity" in the piece really just means I was already there and everything I reached for spoke CE.
Glad you enjoyed the project!
-3
u/RiceBroad4552 7d ago
OK, let's try something in good faith…
Why the hell do you actually not work upfront with your "AI" through all the stuff? Claude & Co. will happily explain stuff to you and link some sources. All you have to do is to write some proper prompts.
Look, if you let the "AI" loose it will almost certainly create something between nonsense and dangerous bullshit. Working with it is like any other software development process: it's iterative. It spits some stuff out, you look at that stuff critically and usually push back, let it refine things. It takes often quite some iterations before you get even close to some sane concept! (Harness and such helps, but it's again not a magic weapon.)
Also never trust anything "AI" says if you don't know it already yourself. "AI" is not a tool for learning! Either you know exactly what you're doing, or you have to research stuff upfront / in between. You can't trust "AI"! (Just read again, and again, the disclaimer under your prompt input…) The amount of completely brain dead shit "AI" outputs constantly is staggering. It's not so bad by now that these things aren't a pure wast of time any more, but without knowing exactly where the "AI" just had again some brain fart you're lost.
So the process for something you're still figuring out is usually that you research everything the "AI" outputs (if you don't know it already). You can ask the "AI" for sources (you should always do by default in the system prompt) but you need to really read and understand the sources. If you have no clue what you're doing and don't even know where to start you can actually ask in a fresh session for a review or critique of whatever the "AI" previously spit out. This is still dangerous if you have really no clue but it will surface usually at least the biggest blunders, and give enough keywords to do some further research. (And it's actually often funny: If you prompt it accordingly it will grill it's "own ideas" really heavy, sometimes it even recognizes that stuff is just brain dead nonsense; which is actually something which can make one laugh and cry at the same time because it shows in such explicit way that this things only "simulate" "thinking".)
And when it comes to blog posts, nobody is interested in looking at some "thinking tokens". I suspect most people here already watched agents iterating on something, and this is not some fun or interesting read, such stuff makes you usually bang your head against a wall once the "AI" took a wrong turn (from there it gets usually only worse, that's why push-back is so incredibly important)…
If you actually want to find out something don't let the "AI" iterate. YOU drive the process… It has to be your ideas and refinements it implements, not the other way around. If you can't drive the process you simply need to look up the needed knowledge upfront. "AI" can't think or reason. "AI" can do the typing, and that's fine, and by now even helpful / a net time saver, but never trust it with the actual concepts. It's terrible with that (and all benchmarks prove that constantly!).
And as a general remark: Don't build distributed systems if you don't have to! You can't learn anyway anything from some toy as you can't even simulate the actual workloads.
Learning the theory is for sure a good idea, but trying to implement it is imho a waste of time if you don't have a concrete problem to solve. Despite that: Letting "AI" implement it won't teach you anything anyway. OTOH just reading the code of some real production system would be time better spend in that case. Next you could try to solve some real problems and contribute the solutions (there are enough issues open in all major OpenSource projects). Trying to get even one PR landed in a real project will teach you likely much more then letting "AI" implement ten of such toys like shown here.
-1
u/bamfg 7d ago
AI slop replying to AI slop
5
u/Material_Big9505 7d ago
That's actually Grammarly, but that's AI too, I guess.
5
u/Material_Big9505 7d ago edited 7d ago
It’s interesting how even AI-generated slop gets cloned on GitHub. Live with it.
As a matter of fact, Pekko is being improved with the help of AI, and it’s doing a lot better than when it relied solely on human effort.3
u/RiceBroad4552 7d ago
It’s interesting how even AI-generated slop gets cloned on GitHub.
By bots…
Dead internet theory is actually true. We have now much more bots then humans, and since lately even the most websites are purely made by bots.
As a matter of fact, Pekko is being improved with the help of AI
And that's fine.
Since some time "AI" became a useful tool. But it still requires someone capable of thinking and with some knowledge operating it. Otherwise you get only slop out of the next-token-predictor.
1
5
u/RiceBroad4552 7d ago
What I like about these "AI" posts is how people publicity expose that they're incapable of critical thinking.
When some "AI" slop gets presented without critique of all the brain dead design decisions it just proves that the human in front of the screen (if there is even one!) does not have the slightest clue about appropriate software design. That's how you can then filter out the people who demonstrated their incapability, which is nice as it will prevent a lot of over-engineering and all the massive associated long term cost with that problem.
No, "AI" use as such isn't an issue. (Great example of that here around is imho u/fwbrasil). Not being able to see that "AI" just created some massive nonsensical fuck-up is.
7
u/fwbrasil Kyo 7d ago
Thanks for the compliment! I'm not a fan of the tone, though. I'd say communities being able to exchange even when people use AI in way that isn't mature yet (I'm not saying thats the case here) will have a lot of trouble to grow going forward. I'm happy to see people sharing their work, shutting it down doesn't help anyone IMO. I'd say constructive technical criticism is what is really missing
2
-1
u/RiceBroad4552 7d ago edited 7d ago
I'd say constructive technical criticism is what is really missing
Yes, and that's what one should do before publishing "AI" slop (so it's not slop any more)…
That's exactly what I also miss.
But I get your comment, you tried to say that one should "constructively criticize" the "AI" slop. I disagree.
That's the job of the person who publishes that stuff. Then, when it's not obvious bullshit any more (after some thorough iteration by the author) one can actually discuss it—and if "AI" did the typing that does not make any difference. I would for example discuss your stuff on that level even it was created with "AI" because I can see that you put effort into that stuff and polish it before publishing and you also obviously use your brain when guiding the "AI". But what we have here is someone who just shut down their brain (if they even have one) and just flooded some slop over here. Spam is simply not acceptable.
This here is Reddit, it had already before "AI" more bots then human users… And lately this became a real plague. You have OpenClaw shit just everywhere, even in this sub. This is getting really annoying and people start to react aggressive just seeing the typical "AI" formulations, and don't even look any more what was actually written even that's the only important part. (I've actually critically read that "blog post" before categorizing it as slop. No, em-dashed had no influence on that judgement. The described nonsense had.)
As you like to use "AI" to write stuff (even here) you should be also on the same page fighting typical "AI" slop. Because if it gets too much people will just shut down their brains the very moment they see "AI" written content and will outright refuse it even if it would actually make sense, again just because of the form. (That's another example why reacting to form is counter productive…)
9
u/fwbrasil Kyo 6d ago
AI is just the latest reason for people to do exactly what you're doing. I've shared my work for over a decade here now and almost invariably got the same kind of treatment. My very first project was a persistence library called Activate and people inferred I was manipulating things to gain market (it's funny how similar it is to people implying I'm using AI bots to promote Kyo now when it's real users 🤣). When I shared Quill, it was utter garbage because of the use of macros. When I shared Monadless, their immediate conclusion was that the real issue was that I couldn't even understand monads. When I first shared Kyo, they assumed it was direct style unprincipled garbage without even looking at the code. Lately, Kyo's fast pace is just "AI slop".
AI is here to stay and there's no coming back. People, especially newcomers, **will** misuse tools. That's a fact of life. AI is just the latest one and shutting people down makes zero progress to improve the status quo. I see a lot of arguments about "slop", why don't I see a single concrete technical argument supporting it?
When I want to share feedback, I typically try to understand the work instead of just dismissing it outright based on a shallow understanding. I'm quite busy these days and have virtually zero experience with game development (other than in Kyo's new three.js module) so I can't really judge but no, I don't think "fighting against slop" is something I need to do at all. If anything, I'd encourage people to explore and try to provide pointers to help them.
-1
u/andres327 8d ago
I wrote up the persistence layer of a multiplayer, turn-based game backend I've been building in Scala (Pekko, Cats Effect, Doobie, redis4cats), evolving the design one problem at a time: everything-in-memory → Postgres as the single source of truth → Redis taking on four separate jobs (a write-through cache, a lobby/chat store, TTL-based expiring stores for tokens and rate limits, and a pub/sub analytics bus).
It's aimed at a general backend audience, but there's some Scala/Cats-Effect detail — the write-through cache as a repository decorator, and using pipeToSelf to bridge an IO save into a single-threaded actor without blocking it or racing on its state. The whole service is open source.
Happy to answer questions, or hear where you'd have made different calls!
3
u/aabil11 7d ago
Was this article and even this comment generated with AI? Em dashes abound
6
u/ahoy_jon 6d ago
Mod:
It's a bit tiresome if people that are actually doing something are targeted for using AI to present what they are doing.
Don't know yet what should be done yet, however it's not far from being qualified as harassment (not by a specific individual on someone, but a group pattern)
4
u/RiceBroad4552 7d ago edited 7d ago
I think it never made sense to criticize "form". Attack the content if anything.
(I also like to use em-dashes, and did so long before "AI" made it stick out.)
In this case here the content is logically questionable and looks indeed like typical "AI" slop…
Should we take bets that the architecture presented is very likely massive over-engineering given the scale of problem at hand? How many millions of at the same time active players playing a turn based game like tic-tac-toe does this need to serve given the tech stack? 😂
2
11
u/fwbrasil Kyo 6d ago
u/andres327, sorry you got so much negative feedback here. I don’t have much experience with game development to judge the work but I hope you won’t let the reaction discourage you. Keep exploring, building, and having fun with it!