Finance AI programmes fail on data far more often than on models. That claim is not controversial among people who have run one. What is less understood is why the failure arrives so late — typically after a successful pilot, when the programme has momentum, a budget and visibility.
The reason is that finance teams are extremely good at compensating. Years of manual reconciliation, informal adjustment and tribal knowledge have made fragile data produce reliable reports. From the top of the organisation the data looks fine, because the output is fine. The compensation is invisible precisely because it works.
A pilot does not disturb that. It runs on an extract, curated by the person who knows where the problems are, over a period chosen because it was clean. Production does disturb it, all at once, at period end.
This guide makes the fragility visible before you spend money on it. Seven tests, runnable in about a week, requiring no tooling and no vendor.
How to run it
Take one process — the close, AP, or reconciliation. Not "finance." Take a recent, ordinary period, not a clean one. Involve the people who do the work, and make it explicit that finding problems is the objective, because the natural instinct is to present well.
Each test is scored pass / partial / fail. Record the evidence, not the impression.
| # | Test | What it reveals | What failing it costs in production |
|---|---|---|---|
| 1 | Trace a figure | Lineage | Outputs nobody can defend to an auditor |
| 2 | Two people, one number | Definitional consistency | A model trained on contradictory labels |
| 3 | The unopened extract | Hidden manual steps | Automation that stops at an invisible human step |
| 4 | The exception log | Whether variation is known | Models confidently mishandling the cases that matter |
| 5 | The master data check | Reference integrity | Mis-coding at scale, silently |
| 6 | The timing test | Data availability | Models running before their inputs exist |
| 7 | The absent expert | Tribal dependency | A process no one can supervise |
Test 1 · Trace a figure
Pick one line in the management accounts. Trace it back through every transformation to the source transactions — without asking the person who prepared it.
Pass: you can do it from documentation and system records. Partial: you can do it, but only by opening a spreadsheet whose logic is undocumented. Fail: you need the preparer.
Why it matters: if a human cannot trace a figure, no one can explain a model's output on it either. This is the test that most directly predicts whether you can put AI anywhere near the record.
Test 2 · Two people, one number
Ask two people independently to produce the same figure — a cost centre total, a category of spend, an ageing bucket — without conferring.
Pass: identical. Partial: same number, different method. Fail: different numbers, both defensible.
Why it matters: a fail means the definition lives in people, not in the data. A model trained or configured against inconsistent definitions learns the inconsistency and applies it confidently, which is worse than applying it inconsistently, because it stops looking like an error.
Test 3 · The unopened extract
Take one routine report and list every step between source system and final output. Count the steps where a human opens a file and does something.
Pass: none. Partial: one or two, documented. Fail: several, and at least one that nobody had listed before starting the exercise.
Why it matters: every manual step is a place automation stops, and undocumented ones are where implementations discover scope they did not budget. This test almost always finds something, and the surprise is the finding.
Test 4 · The exception log
Ask for the list of exception types handled in this process last quarter, with volumes.
Pass: it exists, with owners and rules. Partial: it can be reconstructed from memory. Fail: there is no list, and the answer is "we just deal with them."
Why it matters: exceptions are where models are least reliable and where the material value sits. An organisation that cannot enumerate its exceptions cannot specify what a model must handle, cannot test whether it does, and will discover the gap in production.
Test 5 · The master data check
Take the reference data the process depends on — chart of accounts, cost centres, suppliers, customers. Count duplicates, inactive-but-used records, and entries with no owner.
Pass: governed, with a maintenance process. Partial: clean but maintained informally by one person. Fail: duplicates in active use, and disagreement about which is correct.
Why it matters: master data problems produce mis-coding at scale and near-silently. Automated coding against a duplicated supplier record is wrong every time, at machine speed, and looks entirely normal in aggregate.
Test 6 · The timing test
For each input the process needs, record when it is actually available versus when the process assumes it is.
Pass: inputs arrive before they are needed, reliably. Partial: they arrive in time, but the schedule is informal. Fail: the process routinely starts on incomplete data and corrects later.
Why it matters: the correction step is human judgement absorbing a timing problem. Automate the process and the correction disappears while the timing problem does not — this is one of the most common ways an automated process becomes less accurate than the manual one.
Test 7 · The absent expert
Identify the person whose absence would most disrupt this process. Then establish what specifically they know that is not written down.
Pass: nothing material. Partial: they are faster, but others can complete it correctly. Fail: specific decisions only they can make.
Why it matters: that undocumented knowledge is exactly what a model would need to be specified with, and exactly what nobody will be able to supervise it against. It is also the single best predictor of the other six results.
Reading the results
| Result | What it means | What to do |
|---|---|---|
| 6–7 pass | Genuinely ready. Rare. | Proceed. Choose the use case on value, not on data availability. |
| 4–5 pass | Ready for a contained deployment | Deploy where the failing tests do not apply, with human review retained |
| 2–3 pass | Not ready. A pilot will succeed and mislead. | Remediate in the sequence below before committing to a programme |
| 0–1 pass | The data problem is your current problem | AI is not the next decision. This is costing you money today, without AI. |
That last row is the important one. A finance function failing five or more of these tests is already paying for it — in close days, in restatements, in the effort of the people compensating. AI does not create that cost; it makes it visible and expensive.
Remediation sequence
Fix in dependency order. Each stage makes the next cheaper.
- Master data (test 5). Everything downstream inherits it. Also the most mechanical and
least contested — a good place to start because it builds momentum.
- Definitions (test 2). Agree what each figure means and record it. Organisational work,
not technical.
- Exceptions (test 4). Enumerate, assign owners, write the rules. This is also the input
any AI specification will require.
- Timing (test 6). Fix availability before automating anything that depends on it.
- Manual steps (test 3). Now removable, because 1–4 have removed the reasons for most
of them.
- Lineage (test 1). Largely follows from the above.
- Tribal knowledge (test 7). The residue. Whatever remains after 1–6 is the genuinely
hard part, and it is now small enough to attack.
The common mistake is starting at 5 or 6 because they are the visible symptoms. Removing a manual step that exists to compensate for a master data problem removes the compensation, not the problem.
What you can do meanwhile
Failing this diagnostic does not mean doing nothing with AI for a year. It means being precise about where the output goes.
Safe while remediating — work where the output does not enter the record and a human rewrites it: drafting commentary and narrative, summarising contracts and policies, first-draft analysis, research, interrogating documents. None of this requires lineage, because none of it produces a number anyone books.
Not safe while remediating — anything producing a journal entry, a coding decision, a reconciliation certification or a disclosed figure. These require the data foundations the tests examine, and deploying them early is how a programme acquires a control finding.
That distinction lets a finance function build genuine capability and confidence during remediation, rather than waiting — which matters, because capability is itself one of the dimensions that will otherwise block deployment later.
- The data problem surfaces after the pilot, not before it — because pilots run on
curated extracts over clean periods, and production does not.
- **Run seven tests on one process, in an ordinary period, with the people who do the
work.** Test 7 (the absent expert) predicts the other six.
- Remediate in dependency order, starting with master data, and meanwhile use AI freely
wherever the output does not enter the record.
Where this leaves you. Run the seven tests on a single process this month. If you fail more than three, the honest conclusion is that the data work is your current programme — and it was already overdue before AI made it urgent.