Old vs New: designing "I don't know" into a financial report
On the Adhoc SCF portfolio-analysis system, the most important UI affordance is not the number. It is the row that says "we cannot tell, please confirm".
Italian financial advisors who use the Adhoc SCF system are licensed professionals. They are personally liable for the recommendations they send to their clients. A portfolio-analysis tool that produces confident-but-wrong numbers does not just embarrass them — it can end their career. The product brief for the AI pipeline I built at Adhoc SCF was therefore unusual: the system's number-one job is to refuse, clearly, when it does not know.
The pipeline ingests a client's current portfolio — a stack of KID documents, an Excel from a previous advisor, sometimes scanned statements — and produces a benchmarked "Old vs New" report. Old is the client's current allocation. New is the advisor's proposed alternative. The report compares costs, expected returns and risk profile across the two, line by line.
The naïve implementation extracts every field, computes every number, and prints the report. The advisor reads it, finds three rows that smell wrong, opens the source PDFs, and spends an hour fact-checking. That is not a 30-minute review; that is a 90-minute review with a worse mood. The first version of the system did exactly this. Advisors hated it.
The fix was structured uncertainty. Every extracted field comes with a confidence score from the Document AI extraction step. We expose that score directly in the report. A row with high confidence renders normally — number, source citation, done. A row with medium confidence renders in a softer colour, with a "click to verify" affordance that opens the source PDF region. A row with low confidence does not render the number at all. It renders a "needs review" pill with the source snippet and a one-click confirm-or-edit form.
The insight is that an advisor will happily handle five "needs review" rows in a 50-row portfolio. What they will not happily handle is one wrong number buried in 49 right ones, because they have to check all fifty to find it. Structured "I don't know" turns a quality-control problem into a workflow problem, and workflow problems are easy: you just put the work next to the affordance that does it.
A second design decision: the report has a "snapshot" panel at the top that shows the AI's overall confidence in the document set. Eighteen high-confidence fields, four medium, two need-review. The advisor sees the system's self-assessment before reading a single number. By the time they get to the rows, they already know how much to trust the body of the report. We log every advisor decision on the "needs review" cards; that log is the training data for the next iteration.
The lesson is not specific to finance. Whenever a model produces output that a domain expert will spend their afternoon reviewing, the highest-value UI feature is not the answer. It is the calibrated refusal — the visible distinction between a high-confidence claim, a medium-confidence one, and a "we cannot tell, please confirm". Build that distinction into the data model, surface it in the UI, and you turn a tool from "I have to double-check everything" into "I trust the green rows, I verify the yellow ones, I fix the red ones". That is the moment a tool becomes infrastructure.