Confidence carries no signal.
A model writes with the same assurance when it's right, nearly right, and flat wrong. Good prose can dress up bad reasoning. And the failures don't land where you'd expect: it can handle a hard task, then miss an easy one in the same run.
Trust doesn't transfer.
The same model that summarizes a dense filing correctly will miss a plain constraint in the next step. Getting one task right tells you very little about the next one. Andrej Karpathy named it jagged intelligence. We met the same shape in Office: help that was sharp inside the situations we'd modeled and lost one step outside them. Covering the gaps was most of the work. Whatever you call it, it's the thing to design around.
Proper testing is a job.
The honest answer is that reliability has to be measured. Doing this properly means building a test set, holding examples back, and scoring the results - which is exactly what nobody will ever do inside a notes app. You can't ask a user to run evals. So the real design question: how much rigor can the product smuggle in without asking?
What to build.
The user's history is a test set they never had to label: they already know what last month's material meant. Make it one click to run on work where they know the answer. Small trial batches before anything runs on a schedule. Output displayed next to its source, so checking takes a glance. Put the edit control right next to the output, because tuning happens while the miss is fresh. Let the model help fix itself: a user who can hand the instructions, the input, and the bad output back to a chat is one question away from a better version. And don't decorate output with confidence the model doesn't have - no checkmarks or verified badges unless something actually verified it.