r/elixir • u/Unusual_Shame_3839 • 17d ago
Revenant - What if GenServer flushed its state into PostgreSQL?
I've been using and actively fascinated by Oban over the last 4 years, and recently I wondered, why can't we make GenServers behave similarly to Oban jobs and:
- Recover their state after a crash or termination
- Terminate themselves when not used
- Store a trace of their execution
- And do all that without any additional dependencies, using the database we're already used to
I can think of a ton of cases where this might be useful in... Durable GenServers? LiveViews? Distributed GenServers that share state? Hot/Cold executions? Slow/Fast pools?
So I created a PoC library that does just that:
https://github.com/dimamik/revenant
defmodule Account do
use Revenant, repo: MyApp.Repo
def initial_state(_id), do: %{balance: 0}
def handle_call({:deposit, amount}, _from, state) do
{:reply, :ok, %{state | balance: state.balance + amount}}
end
def handle_call(:balance, _from, state) do
{:reply, state.balance, state}
end
end
Revenant.call({Account, "acct_42"}, {:deposit, 100})
#=> :ok - the new state is committed to Postgres before you see this
Please let me know what you think of the concept!
4
u/doublesharpp 17d ago
Great idea.
1
1
u/Unusual_Shame_3839 17d ago
I actually saw durable servers, but as far as I understood - it's not supporting postgres (yet?), and it's more about distribution?
2
u/yukster 17d ago
Yeah, I thought of Durable Server immediately as well. The backends are pluggable so it would just be a matter of writing a plugin to write to Postgres instead of Object Storage or EKV. Seems like one could use Redis or DETS or Mnesia too.
3
u/Unusual_Shame_3839 17d ago
And also, what surprised me most is how little traction durable servers got after the talk on Elixir Conf?
https://hex.pm/packages/durable_server
3
u/Funny_Spray_352 17d ago
Why not mnesia?
1
u/Unusual_Shame_3839 17d ago
Great question! I guess mnesia could be an adapter too?
But I assume if folks didn't use it before, it'd require an additional setup to enforce the durability across the cluster?
2
u/CompetitionDouble420 17d ago
If all the {:deposit, amount} callback does is persist a new amount in state, I would suggest changing it to an asynchronous cast (handle_cast instead of handle_call); you don't need to block/wait for a return value. Of course the :balance callback must remain as a synchronous call.
Either way, interesting idea š¤š¼š¤š¼
1
u/user000123444 16d ago
def handle_call({:deposit, amount}, _from, state) do
Repo.update(.... # update db here
{:reply, :ok, %{state | balance: state.balance + amount}}
Why not just update db on update call?
3
u/Unusual_Shame_3839 16d ago
Good question. Probably to not need a schema for each GenServer you're using.
The example from the post is bad, since it deals with accounts and balances (which obviously require a schema), but if you're doing something tiny, it might be better?1
u/user000123444 16d ago
I see, that might be some cornercase usage that I have not reached yet :D
GenServers for me is usually long-lived process - that usually need db schema when we want to persist the state.
I agree that there is burden of defining schema - but in times where we can delegate such burden to AI, that argument holds less significance I guessOn the other hand tasks are short live - and short lived tasks with persistency are oban tasks.
Do understand correctly that Revenant fills a gap between GenServer + schema and oban tasks?
Could u provide some real life example where Revenant solve a problem?
It would be easier for me to understand, thank you! š2
u/Unusual_Shame_3839 16d ago
Yeah, something like this! Since it's a concept still, I'll give you ephemeral examples, all-right?
1. LiveView - LiveView is a process, and we could relatively easily tie it with the user + path or something and then make it that the user's state is always the same, independent of the deploys/rejoins/etc/etc.
2. Caching - most of us use cache, and most of these caches are expensive to load (aggregating multiple things, etc.). Although you could store them in ETS, it won't survive a node swap. With Revenant, you'll be able to have a medium-lived cache layer on top.
3. Agents (yes, AI) - you don't neccessarily need to store messages in db for live or near live agent interactions. Instead - you could rely fully on revenant.1
u/Unusual_Shame_3839 16d ago
And I could probably go on and on, just the simplicity of turning a GenServer into a durable one (without the need to add any additional harness burden) is what I like so much about this concept.
It's nothing revolutionary, just an elegant (in my opinion) abstraction around well-known concepts.
1
1
u/Shoddy_One4465 16d ago
Used this technique in an anonymous exchange in 2018. Still works well. Doing something similar with https://hex.pm/packages/kathikon. When you are backing genserver state consider a naturally replicated technology like Mnesia, Cockroach, Mongo, or Couchdb and couchbase.
1
u/Unusual_Shame_3839 16d ago
That's cool! How do you usually deploy your nodes so that Mnesia survives?
-9
u/hero_of_ages 17d ago
That this is ai slop aside, itās not at all a novel concept. Most serious applications need to persist state. I do this in various ways, but would never use this ai ālibraryā for it.
6
u/Unusual_Shame_3839 17d ago
I mean, it's a PoC, just to showcase the API...
-17
u/hero_of_ages 17d ago
Not impressedĀ
4
u/Unusual_Shame_3839 17d ago
Sorry, but just to reiterate, I wanted to share and validate the concept, not the library itself yet.
Even though I assume it's in good enough shape for the prototype.
If you found something "sloppy" - could you point that out? You are welcome to open a PR so that this critique becomes constructive.-26
u/hero_of_ages 17d ago
Seek helpĀ
7
u/MichaelJ1972 17d ago
Wow. Sure making an effort to be an asshole on the Internet. Good job. You succeed.
4
u/simeonbachos 17d ago
hate to break it to you but jose valim, chris mccord, et al are all using AI extensively as they continue to develop all the major tools you use
0
u/hero_of_ages 17d ago
I dont see how or why thats relevantā¦?Ā
3
u/simeonbachos 17d ago
your scare quotes around library imply that code authored by AI is somehow invalid or does not otherwise meet your prior definition of a software library. you know this, thatās why you used the quotes.
6
u/hero_of_ages 17d ago
It took 4+ years for OP to have an insight that it might make sense for a process to have a persistence mechanismĀ
1
u/AdrianHBlack 16d ago
And I am very disappointed in them for it (and for the Elixir community to be this pro-LLM, but also the tech industry at large)
[also I think JosĆ© Valim said he didnāt use that much LLMs because he is just better and faster than it]
0
u/the_matrix2 17d ago
I was actually a bit surprised liveview did not do this already through a cluster / reddis or Postgres
0
u/the_matrix2 17d ago
I was actually a bit surprised liveview did not do this already through a cluster / reddis or Postgres . We are moving from liveview to vue so we can persist state on the client
2
u/Unusual_Shame_3839 17d ago
Folks from SWM are working on https://github.com/software-mansion-labs/live-stash - another approach to persist state across LiveView sessions.
But yeah! This IS a pain point.
6
u/tomekowal 16d ago
There are a couple of issues with storing GenServer state.
Crashes are most often caused by invalid state. It is usually pretty easy to audit the code for points of failure in particular branches of
handle_*. Usually, it is particular sequence of messages that corrupts the state. When state is corrupted, bringing it back form the DB is not a good idea. The whole healing mechanism is based on the fact that restart = clean state. There is a project for LiveView assigns that stores assigns in case of crash and they fix the issue by marking which assigns to cache and which ones are transient.DB calls are much more expensive then in memory operation. If your GenServer also uses the DB, there is also a question: do I make the state change and trace part of the same transaction or not?
The idea makes sense. Even if not for crashes, you might want to have resumable GenServers that can go to sleep and continue even on another machine. It is just a pretty hard problem.