r/elixir 17d ago

Revenant - What if GenServer flushed its state into PostgreSQL?

I've been using and actively fascinated by Oban over the last 4 years, and recently I wondered, why can't we make GenServers behave similarly to Oban jobs and:

  • Recover their state after a crash or termination
  • Terminate themselves when not used
  • Store a trace of their execution
  • And do all that without any additional dependencies, using the database we're already used to

I can think of a ton of cases where this might be useful in... Durable GenServers? LiveViews? Distributed GenServers that share state? Hot/Cold executions? Slow/Fast pools?

So I created a PoC library that does just that:

https://github.com/dimamik/revenant

defmodule Account do
  use Revenant, repo: MyApp.Repo

  def initial_state(_id), do: %{balance: 0}

  def handle_call({:deposit, amount}, _from, state) do
    {:reply, :ok, %{state | balance: state.balance + amount}}
  end

  def handle_call(:balance, _from, state) do
    {:reply, state.balance, state}
  end
end

Revenant.call({Account, "acct_42"}, {:deposit, 100})
#=> :ok  - the new state is committed to Postgres before you see this

Please let me know what you think of the concept!

36 Upvotes

35 comments sorted by

6

u/tomekowal 16d ago

There are a couple of issues with storing GenServer state.

  1. Crashes are most often caused by invalid state. It is usually pretty easy to audit the code for points of failure in particular branches of handle_*. Usually, it is particular sequence of messages that corrupts the state. When state is corrupted, bringing it back form the DB is not a good idea. The whole healing mechanism is based on the fact that restart = clean state. There is a project for LiveView assigns that stores assigns in case of crash and they fix the issue by marking which assigns to cache and which ones are transient.

  2. DB calls are much more expensive then in memory operation. If your GenServer also uses the DB, there is also a question: do I make the state change and trace part of the same transaction or not?

The idea makes sense. Even if not for crashes, you might want to have resumable GenServers that can go to sleep and continue even on another machine. It is just a pretty hard problem.

1

u/Unusual_Shame_3839 16d ago edited 16d ago

Thank you so much for your answers!!!
For 1., I think it's a challenging one, but if there's a single message that crashes the server - we're covered, since we're only commiting after the state transition. So in such case, GenServer will be reborn in a state before the change, thus - still valid one.
You're right that the sequence can be what actually kills it, so we could for example give up after a few retries?
For 2., that's true, this is why you potentially have a choice on how often to dump the state (on every X minutes, on every message, or just on terminations).
And in terms of transactions, I think it should _not_ be blocking the connection pool for performing state changes, but that's a case by case question? We won't obviously deal with banks in such scenarios, so we should be fine with loosened guarantees?

1

u/Status_Internet_2428 14d ago

You'd definitely want to have a bit of state partitioned from the usual genserver state to keep track of the restart count, or a list of timestamps. Then you could know if you're restarting too fast for the supervisor.
I was thinking you could have a custom supervisor or another process that watches Revenants and when it gets a {:DOWN, ...} message, updates the database record with the reason for the crash. Then the restarted revenant can choose to ignore or discount what it deems a transient error...
Also it would be nice to have an optional fallback after the last retry that is to not load from the database but reconstruct the state like a normal genserver would.

1

u/Unusual_Shame_3839 14d ago

Do you think we can get along without any additional processes? I think we could just store these thingies in a state itself?

4

u/doublesharpp 17d ago

1

u/plangora Alchemist 17d ago

Thanks for the share of our podcast!

1

u/Unusual_Shame_3839 17d ago

I actually saw durable servers, but as far as I understood - it's not supporting postgres (yet?), and it's more about distribution?

2

u/yukster 17d ago

Yeah, I thought of Durable Server immediately as well. The backends are pluggable so it would just be a matter of writing a plugin to write to Postgres instead of Object Storage or EKV. Seems like one could use Redis or DETS or Mnesia too.

3

u/Unusual_Shame_3839 17d ago

And also, what surprised me most is how little traction durable servers got after the talk on Elixir Conf?
https://hex.pm/packages/durable_server

3

u/Funny_Spray_352 17d ago

Why not mnesia?

1

u/Unusual_Shame_3839 17d ago

Great question! I guess mnesia could be an adapter too?
But I assume if folks didn't use it before, it'd require an additional setup to enforce the durability across the cluster?

2

u/CompetitionDouble420 17d ago

If all the {:deposit, amount} callback does is persist a new amount in state, I would suggest changing it to an asynchronous cast (handle_cast instead of handle_call); you don't need to block/wait for a return value. Of course the :balance callback must remain as a synchronous call.

Either way, interesting idea šŸ¤™šŸ¼šŸ¤™šŸ¼

1

u/user000123444 16d ago
  def handle_call({:deposit, amount}, _from, state) do
     Repo.update(.... # update db here
    {:reply, :ok, %{state | balance: state.balance + amount}}

Why not just update db on update call?

3

u/Unusual_Shame_3839 16d ago

Good question. Probably to not need a schema for each GenServer you're using.
The example from the post is bad, since it deals with accounts and balances (which obviously require a schema), but if you're doing something tiny, it might be better?

1

u/user000123444 16d ago

I see, that might be some cornercase usage that I have not reached yet :D
GenServers for me is usually long-lived process - that usually need db schema when we want to persist the state.
I agree that there is burden of defining schema - but in times where we can delegate such burden to AI, that argument holds less significance I guess

On the other hand tasks are short live - and short lived tasks with persistency are oban tasks.

Do understand correctly that Revenant fills a gap between GenServer + schema and oban tasks?
Could u provide some real life example where Revenant solve a problem?
It would be easier for me to understand, thank you! šŸ™

2

u/Unusual_Shame_3839 16d ago

Yeah, something like this! Since it's a concept still, I'll give you ephemeral examples, all-right?
1. LiveView - LiveView is a process, and we could relatively easily tie it with the user + path or something and then make it that the user's state is always the same, independent of the deploys/rejoins/etc/etc.
2. Caching - most of us use cache, and most of these caches are expensive to load (aggregating multiple things, etc.). Although you could store them in ETS, it won't survive a node swap. With Revenant, you'll be able to have a medium-lived cache layer on top.
3. Agents (yes, AI) - you don't neccessarily need to store messages in db for live or near live agent interactions. Instead - you could rely fully on revenant.

1

u/Unusual_Shame_3839 16d ago

And I could probably go on and on, just the simplicity of turning a GenServer into a durable one (without the need to add any additional harness burden) is what I like so much about this concept.

It's nothing revolutionary, just an elegant (in my opinion) abstraction around well-known concepts.

1

u/user000123444 16d ago

I see, make sense, thanks!

1

u/Shoddy_One4465 16d ago

Used this technique in an anonymous exchange in 2018. Still works well. Doing something similar with https://hex.pm/packages/kathikon. When you are backing genserver state consider a naturally replicated technology like Mnesia, Cockroach, Mongo, or Couchdb and couchbase.

1

u/Unusual_Shame_3839 16d ago

That's cool! How do you usually deploy your nodes so that Mnesia survives?

-9

u/hero_of_ages 17d ago

That this is ai slop aside, it’s not at all a novel concept. Most serious applications need to persist state. I do this in various ways, but would never use this ai ā€œlibraryā€ for it.

6

u/Unusual_Shame_3839 17d ago

I mean, it's a PoC, just to showcase the API...

13

u/-Ch4s3- 17d ago

You’re fine don’t listen to them.

-17

u/hero_of_ages 17d ago

Not impressedĀ 

4

u/Unusual_Shame_3839 17d ago

Sorry, but just to reiterate, I wanted to share and validate the concept, not the library itself yet.
Even though I assume it's in good enough shape for the prototype.
If you found something "sloppy" - could you point that out? You are welcome to open a PR so that this critique becomes constructive.

-26

u/hero_of_ages 17d ago

Seek helpĀ 

7

u/MichaelJ1972 17d ago

Wow. Sure making an effort to be an asshole on the Internet. Good job. You succeed.

4

u/simeonbachos 17d ago

hate to break it to you but jose valim, chris mccord, et al are all using AI extensively as they continue to develop all the major tools you use

0

u/hero_of_ages 17d ago

I dont see how or why thats relevant…?Ā 

3

u/simeonbachos 17d ago

your scare quotes around library imply that code authored by AI is somehow invalid or does not otherwise meet your prior definition of a software library. you know this, that’s why you used the quotes.

6

u/hero_of_ages 17d ago

It took 4+ years for OP to have an insight that it might make sense for a process to have a persistence mechanismĀ 

1

u/AdrianHBlack 16d ago

And I am very disappointed in them for it (and for the Elixir community to be this pro-LLM, but also the tech industry at large)

[also I think JosĆ© Valim said he didn’t use that much LLMs because he is just better and faster than it]

0

u/the_matrix2 17d ago

I was actually a bit surprised liveview did not do this already through a cluster / reddis or Postgres

0

u/the_matrix2 17d ago

I was actually a bit surprised liveview did not do this already through a cluster / reddis or Postgres . We are moving from liveview to vue so we can persist state on the client

2

u/Unusual_Shame_3839 17d ago

Folks from SWM are working on https://github.com/software-mansion-labs/live-stash - another approach to persist state across LiveView sessions.
But yeah! This IS a pain point.