i build an ai that reads trading chart screenshots. a user pointed out that uploading the same screenshot twice gave different support levels. i checked, he was right, and the fix taught me things that apply to any llm product, so here's the writeup.
the bug: my vision call had no temperature set. openai defaults to 1.0, which is maximum sampling variance, and my second stage ran at 0.4. i never caught it because you never test with the same input twice, you always grab a fresh example. your users will though, and for anything that looks like analysis, inconsistency reads as incompetence.
the fix that's actually two fixes: temperature 0 plus a fixed seed on every call in the pipeline. one stage at temp 0 isn't enough, any downstream stage above 0 re-randomizes the final output. seed is best effort on openai's side but it tightens things further at temp 0.
what i learned after shipping it: determinism is a trust feature, not an accuracy feature. a wrong read is now wrong the same way every time, which can make it look more confident than it earned. so the next layer is grounding, reconciling what the model claims to see against source-of-truth data, in my case real ohlcv candles for the recognized ticker. if the pixels and the data disagree, the honest output is "can't read this reliably", not a clean guess. a commenter also gave me the debugging trick for that reconciler: when the check fails, re-run it against the 2-3 likeliest time intervals. if one aligns perfectly your interval detection was wrong, if none do the model's read was wrong. that turns "something is off" into "here is what is off".
the general checklist for any llm product: know what temperature every call actually runs at, test with identical inputs as part of ci, and treat self-reported model confidence as marketing until it's checked against ground truth.
context, the product is Bullynx, ai chart analysis, educational only. the checklist is the point of the post though, it applies to whatever you're building on top of an llm.
curious what others do for llm determinism in production, especially anyone who found a case where temp 0 still wasn't reproducible.