Sheffield Bancorp batch report (`gemini-3.5-flash`)

3 data-science tasks · one model · 5 rollouts each (15 rollouts total), scored by a weighted per-criterion rubric graded by an agent-judge. Model: gemini-3.5-flash (engine gemini-cli), served on Vertex. Environment: a native-filesystem synthetic retail-bank data room — a 2.16 GB bank.db SQLite database (a transaction ledger of several million rows) plus a Slack export, an email .mbox, and workspace docs, dashboards, and reports; the model works through the engine's native shell (sqlite3, python3) and file tools. Grading: an agent-judge scores each rollout under a detailed grading specification — a weighted per-criterion rubric (49–63 criteria per task; each criterion carries a weight and the weights sum to 100) together with the task's specific grading guidance — comparing the delivered artifact against that guidance and a reference solution; a task's score is the mean across its 5 rollouts. Per-rollout artifacts accompany this report.

What this is

The setup, in plain terms. The model plays a data analyst working inside a synthetic retail bank's data room. Its material is a large transaction database — a 2.16 GB SQLite file holding a ledger of several million deposits, transfers, and payments — together with supporting policy documents, dashboards, a Slack export, and an email archive. It answers a realistic banking-analytics question by querying that material with SQL and Python, and delivers a structured work product: a script plus a JSON file, a CSV, or a memo.
What the three tasks ask, at a glance.
retail-deposits-flash — spot and quantify an unusual jump in the retail-deposit trend and decide whether it is safe to build the quarter's liquidity report on.
related-party-exposure — restate an analysis of exposure between the bank's own employees and the customers they are personally connected to (related-party exposure), as a set of quantified KPIs in a single JSON file.
aml-structuring-routing — build an anti-money-laundering routing packet that decides which accounts are safe to auto-route and which must be escalated for manual review, where the pattern of concern is structuring (many sub-threshold deposits engineered to stay under a reporting limit).
Why it's hard. Each answer hinges on one subtle, domain-specific judgment — which transactions to include or exclude, how to count them, or which of several competing rules governs — and the raw ledger must be trusted over management's already-summarized figures.

1. Executive summary

gemini-3.5-flash reaches a domain mean of 55.8%, and the shortfall is one consistent model failure: on every task the score turns on a single analytical decision, and the model characteristically has the right facts already in view and then makes the wrong call on that one deciding question — and never catches it. It stands up a genuine analytics floor on each task (runnable sqlite3/python pipelines, the whole data room mined rather than the database alone, the buried conventions applied correctly — New-York UTC-5 date bucketing, the $9,500 high-value threshold, entity_id-only matching between employees and customers — and zero fabricated numbers), so the structural and method criteria are earned reliably. But the one interpretation that gates each score — a root-cause diagnosis, a transaction-counting grain, or a two-part filter — is decided wrong and survives to the deliverable. The model's process has a build stage and a self-check stage but no real verification stage: every run re-reads its own output for structure — that the file parses, the keys are present, the row count is right — never re-deriving the one number the task actually turns on against what was asked, so the deciding error is never caught and the model repeatedly declares the work done over an answer that misses the bar.

Task	Description	`gemini-3.5-flash`
`retail-deposits-flash`	Q1 retail-deposits flash sanity check — decide whether the trend is safe for the liquidity story and quantify the correction	62.8%
`related-party-exposure`	Restate the Q1 related-party (employee↔customer) exposure analysis as a structured JSON of quantified KPIs	57.6%
`aml-structuring-routing`	Build the Q1 AML structuring routing packet — identify which accounts are safe to auto-route vs. escalate	47.1%
Mean		55.8%

Genuine failures observed across the 15 rollouts:

The one deciding call is wrong, task by task — and the model computed the right answer's ingredients first. On related-party-exposure all five rollouts (54.6–63.6%) count transactions at the owned-account grain, double-counting any transaction between two accounts of the same employee-customer match — 20,158 transactions / $7,486,913.69 against the intended 19,910 / $7,463,451.12, an inflation of exactly 248 transactions — and rollout 1 (55.4%) even reported both bracketing grains (20,158 and 19,288) with the intended figure sitting between them, never landing the middle. On retail-deposits-flash three of five (rollout 1, 56.8%; rollout 2, 53.0%; rollout 4, 46.4%) diagnose the deposit spike as a synthetic onboarding cohort and correct only two days, though the intended root cause — bank-internal book-entry salary flows sourced from the bank's own internal suspense account (bank-internal postings, not real customer money) — was identified in every run. On aml-structuring-routing, landing the exact 2-account population needs a rolling-24-hour window and a sub-threshold-only filter composed together, and no run applies both (rollout 1, 63.6%, routes 15; rollout 5, 62.5%, routes 1; rollouts 3 and 4, 56.5% / 52.8%, route 14). Each correct action was one step from what the model already did.
Where the analysis is right, the model still under-packages — it computes the operand and declines the one-line rollup. On retail-deposits-flash all five rollouts leave the planning-safe correction, the ratio, the retained non-internal figure, and the branch driver unstated though every operand was in the stream — rollout 5 (84.3%) computed both totals (51,963,251.51 and 50,262,286.41) whose difference is the planning-safe number (1,700,965.10) and whose quotient is the ratio (30.55×), and reported neither; rollout 3 (73.5%) printed the six clean-customer components that sum to the retained figure (110,805.13) and never summed them. On related-party-exposure all five compute the top same-branch conflict ($57,662.07, computed in rollout 4's session, 56.2%) and the 21/22 same-branch employee/customer split, and never emit them under the expected keys. The answer was in reach; it was not delivered in the form the task asked for.
A hollow verification stage lets every wrong call survive uncaught. All 15 rollouts do genuine multi-step work and real computational self-checks (schema probes, per-day reconciliations to the cent), but every run re-reads its own output only for structure. On aml-structuring-routing rollout 1 (63.6%) re-reads all three of its output files yet never notices that routing 15 accounts flatly contradicts its own "packet is overbroad / not safe to route" conclusion; on related-party-exposure rollout 5 (54.6%) prints the JSON key list and never re-derives its counting grain; on retail-deposits-flash every run reconciles intermediate queries to the cent but no run re-reads its final packet against the task's explicit asks. The added steps refine the machinery; none audits the deciding quantity.

2. What was run

Tasks: 3 Sheffield Bancorp data-science tasks that each require a client-ready analytical deliverable — a runnable script plus a memo, or a structured JSON/CSV — computed against the shared retail-bank data room and satisfying a specific set of quantified requirements.
Rollouts: 5 independent rollouts per task — 15 rollouts total — so consistency (not just a single sample) is measured.
Scoring: each rollout is graded by an agent-judge running a detailed grading specification — a weighted per-criterion rubric (49–63 criteria per task, each criterion carrying a weight, the weights summing to 100) together with the task's specific grading guidance — comparing the delivered artifact against that guidance and a reference solution. Reported here as a percentage; a task's score is the mean across its 5 rollouts.
How patterns were derived: for every rollout we read the model's full solver trajectory (raw execution stream) first, distilled its behavior, then folded in the grader's per-criterion evidence and score as corroboration. Patterns are cited to specific rollouts and verbatim behavior.
Config: a native-filesystem environment — the synthetic retail-bank data room (the 2.16 GB bank.db SQLite database with its several-million-row transaction ledger, plus a Slack export, an email .mbox, and workspace docs, dashboards, and reports); the model works through the engine's native shell (sqlite3, python3) and file tools across all three tasks. gemini-3.5-flash served on Vertex.

3. Delivery

All 15 rollouts ran to completion and were scored; none refused, and the 2.16 GB warehouse was probed and queried successfully in every rollout. One rollout (aml-structuring-routing rollout 2, 0.0%) produced no deliverable: after an extended research phase the model entered a planning mode that disabled its write and shell tools, never resumed, and wrote no files.

4. Per-task analysis

4.1 `retail-deposits-flash` — Q1 retail-deposits flash sanity check (62.8%)

Task: build deposit_flash_sanity.py plus a packet that decides whether the Q1 retail-deposit trend is safe for the ALCO liquidity-buffer story, quantifies any correction, names the dates and drivers that matter, and gives committee-ready bullets. (ALCO — the bank's asset-liability committee, which sets the liquidity buffer.)

Per-rollout (%): 56.8 · 53.0 · 73.5 · 46.4 · 84.3.

Three of five rollouts diagnose the deposit spike wrong. Rollouts 1 (56.8%), 2 (53.0%), and 4 (46.4%) attribute the spike to a synthetic onboarding cohort — 1,156 accounts opened 2026-03-03 with leet-spelled names — and correct only the two days those accounts touch (a ~$1.4–2.8M correction), then declare the remaining ~$8M "stable and safe." The intended diagnosis is that the spikes are internal book-entry salary flows sourced from the bank's internal suspense account (bank-internal postings, not real customer money), and the correct fix removes all internal-originated postings. The model identified that suspense account in every run; the three weak runs simply applied the removal to the wrong subset — one join predicate from the right answer. Rollout 4 (46.4%) goes furthest off: it rejects the required New-York time-bucketing as a "rendering bug," reverts to raw UTC, and endorses the internal-laden ~$7.97M figure as a "highly reliable baseline" — the opposite of the required recommendation.
Even the two correct diagnoses leave the headline aggregates unstated. Rollouts 3 (73.5%) and 5 (84.3%) both read the source-side story right, but no run reports the planning-safe correction, the ratio, the retained non-internal deposits, or the branch driver — although every operand was computed. Rollout 5 (84.3%) computed the two totals (51,963,251.51 and 50,262,286.41) whose difference is the planning-safe figure (1,700,965.10) and whose quotient is the ratio (30.55×), then over-corrected — ruling the residual $1.70M of customer transfers "closed-loop… net external cash inflow exactly $0.00" — so it never surfaced the number it was one step from stating. Rollout 3 (73.5%) printed the six clean-customer components that sum to the retained figure (110,805.13) and never summed them, and no run computed the branch driver (Chicago, the largest exposure at ~$5,323,207.06).
Verification reconciles the machinery, never the deliverable. Every run cross-checks intermediate queries to the cent, but no run re-reads its final packet against the task's explicit asks — the corrected total, the ratio, the retained figure, the branch driver — so the aggregation gap survives in all five.
What the model built: all five stand up a runnable script and packet, isolate the six spike dates, match the non-spike regression anchor (2026-02-18 = 18,614.24), and issue an explicit "not safe" verdict. The score is set by the diagnosis fork and the unreported rollups.

4.2 `related-party-exposure` — Q1 related-party exposure restatement (57.6%)

Task: restate the Q1 2026 related-party exposure analysis over the data room and emit one JSON file with 12 mandated top-level keys — matching employees to customers by entity_id, applying the $9,500 high-value threshold and the New-York UTC-5 date convention, and quantifying transaction counts, volumes, and same-branch conflicts.

Per-rollout (%): 55.4 · 63.6 · 58.3 · 56.2 · 54.6.

A single transaction-counting-grain bug caps every rollout. All five count transactions at the owned-account grain, so a transaction between two accounts of the same employee-customer match is counted twice — 20,158 transactions / $7,486,913.69 against the intended 19,910 / $7,463,451.12, an inflation of exactly 248 transactions. This one error zeroes the whole computed-dollar cluster — the load-bearing criteria — in every run. Rollout 1 (55.4%) reported both bracketing grains (20,158 and the global-unique 19,288) with the intended 19,910 between them, and never identified the middle resolution: dedup within a match while staying independent across matches.
A diagnostic tell confirms the mechanism: the same-branch dollar aggregate ($278,480.05) is correct in all five while the total is wrong, because the double-counted transactions live in the non-same-branch bucket.
Sub-metrics the model computed go unreported. The 21 same-branch employees / 22 same-branch customers, and the top same-branch conflict ($57,662.07, computed in rollout 4's session, 56.2%), are absent from the expected keys in every run — computed but not surfaced.
Rollout 3 (58.3%) layers two further defects on the grain bug: its closed-branch set unions employee-branch with account-branch relationships, inflating that figure to $984,618.71 (against ~$709,814.42 in the other runs); and its high-value verdict returns is_same_branch = false by comparing only the direct transaction account, ignoring its own computed same-branch aggregation.
Verification never touches the deciding number. Every run reads the JSON back for key presence and validity; none re-derives the grain or reconciles its total against the same-branch subset it also computed.
What the model built: 25 criteria pass every run — file and key structure, the New-York UTC-5 convention, the $9,500 threshold, entity_id-only matching, the 471 employee count, the same-branch aggregate, Bristol as the closed branch, and evidence citations — with zero fabricated numbers. The cap is the counting grain it never re-examines.

4.3 `aml-structuring-routing` — Q1 AML structuring routing packet (47.1%)

Task: build the Q1 AML structuring routing packet — decide which accounts are safe to auto-route versus escalate, applying the governing $9,500-floor / group-by-destination structuring rule from the Slack/CRO policy (over the deprecated runbook and single-transaction dashboard rules), and produce five deliverables including a route CSV, a source-conflict CSV, and a memo. (Structuring — many sub-threshold deposits engineered to stay under a reporting limit.)

Per-rollout (%): 63.6 · 0.0 · 56.5 · 52.8 · 62.5.

The exact structuring population is 2 accounts, and it requires two constraints composed at once — no run composes them. Landing exactly 2 accounts needs a rolling-24-hour window (to catch a cross-midnight 03-30→03-31 pair) and a sub-threshold-only filter (excluding individual ≥$9,500 transactions, so single large transfers do not inflate the set). Rollout 1 (63.6%) implements the rolling window but not the sub-threshold filter and routes 15 accounts; rollout 5 (62.5%) applies the sub-threshold filter on a calendar-day window and routes 1; rollouts 3 (56.5%) and 4 (52.8%) apply neither and route 14. Both correct pieces existed in the run population — rollout 1's rolling window and rollout 5's sub-threshold filter — but were never combined in a single run; one composition from the exact answer.
Rollout 2 (0.0%) produced no deliverable: after an extended research phase the model entered a planning mode that disabled its write and shell tools, never resumed, and wrote no files, ending by asking the user to confirm its strategy.
Reporting-granularity gaps zero criteria the model had the data for. Where a target account is present, the route CSV reports account-summary fields (days_flagged, cumulative_total_usd, average-per-day) instead of the structuring evidence the task wants — the max-window amount, per-window count, window span, and triggering transaction IDs. Rollout 1 (63.6%) computed the max-window amount (10693.57) in scratch, then reported the cumulative figure (21027.68) instead.
Verification is format-only. Rollout 1 (63.6%) re-reads all three of its output files but validates existence and format, never catching that routing 15 accounts flatly contradicts its own "packet is overbroad / not safe to route" conclusion.
What the model built: every productive run delivers all five files, correctly concludes the owner packet is not safe to route as-is, applies the governing $9,500-floor / group-by-destination rule, holds out the manual-escalation conversation for review, rejects the deprecated runbook rules, and cites four or more artifact types. The cap is the two-part computation and the routing evidence it does not surface.

5. Appendix — artifacts

Per-rollout artifacts (solver trajectories, reasoning traces, and the grading records) accompany this report, one set per task.