Ask your data: natural-language reporting without letting a model write SQL

A model that writes SQL against your database has two failure modes, and the dangerous one is silent. The loud failure is a query that errors — wrong column, wrong table, permission denied — and you see it immediately. The quiet failure is a query that runs, returns a number in roughly the right range, and is wrong: a join that fans out and doubles a count, a filter on the wrong date column, a rate averaged across groups instead of computed over a total. Nobody catches the second kind, because a number that looks right is treated as right.

Text-to-SQL demos well for the same reason it fails in production. On a small schema with well-named tables and one customer's data in it, a model gets the query right often enough to feel like magic. On a real multi-tenant system — sub-organisations, branches, roles that cascade down a tree, thirty tables carrying an organisation id and two that do not — the schema is the hard part of the question, and the model is inferring it from column names.

The failure you see, and the one you do not

A syntax error is a good outcome: something is broken, you know it is broken, and nobody puts it in a slide. The problem is the class of queries that are valid SQL, execute happily, and answer a question nobody asked.

How many learners completed the compliance module last quarter? The obvious query joins enrolments to completions to the module table. If a learner holds two enrolment records — a transfer between branches, a re-enrolment after a policy change — the join produces two rows, and the count reports two people where there was one. The SQL is valid, the number is inflated, and the only person who could catch it already knew the answer.

The other reliable ways a generated query is wrong while running perfectly:

• A join that fans out, so a count of people silently becomes a count of rows.
• A rate computed as the average of per-group rates rather than a total over a total, which weights a group of six the same as a group of six hundred.
• A date filter on the wrong column: `created_at` where the question meant `completed_at`.
• A timezone assumption, so a Dubai administrator's Monday is bucketed by a Milan clock.
• Soft-deleted rows nobody mentioned, because the flag is `archived` on one table and `status` on the next.

None of these produce an error. Each produces a figure, which the model then explains confidently, because explaining is a separate act from computing and draws on the same guess.

The permission problem you cannot prompt your way out of

A text-to-SQL feature has to answer one question before any other: as whom does the query run? Both available answers are bad. Run it as a service account with broad read access and the question box becomes the widest read path in the building — a branch manager types a sentence, the model omits the branch filter, and another branch's rows come back. Run it under the asking user's own restricted role and it breaks constantly, because the model must reconstruct the tenancy model from column names to return any rows at all.

The usual patch is a paragraph in the system prompt: only query this user's organisation, never select from tables holding personal data. A prompt is not an access boundary. It is a request, evaluated by the same probabilistic process that wrote the bad join, and it degrades exactly where it matters — an unusual phrasing, a long conversation, a question that half-matches the instructions.

The boundary that holds is the one the database enforces. Row-level security evaluates a policy on every query, whoever wrote it, so a query missing a tenant filter returns nothing rather than everything. That only helps if the query runs under the reader's identity — so the way out is to stop asking the model for queries.

A specification is a much smaller thing to get right

Change what the model produces. Not SQL — a report specification: which measures, which dimensions to group by, which filters, which chart. A short, closed structure, expressed as JSON, addressed to a registry of metrics a human defined in advance.

{
  "measures": ["completion_rate"],
  "dimensions": ["branch", "month"],
  "filters": [
    { "field": "programme", "op": "eq", "value": "compliance-2026" },
    { "field": "completed_at", "op": "between", "value": ["2026-04-01", "2026-06-30"] }
  ],
  "chart": "bar"
}

Every part of that object is checkable before a row is read. `completion_rate` is in the registry or it is not. `branch` is a dimension that metric may be grouped by or the specification is refused. The operators are a closed list. No free text reaches the database, so an invented metric fails validation instead of returning a plausible number.

The compiler does the part that has to be right every time: which view each metric reads, which aggregation it uses, that a rate is a numerator over a denominator on one view rather than an average of averages, and which grain each column lives at. That logic is written once and covered by tests.

The compiled query then goes through the same door an ordinary report uses: same validator, same compiler, same row-level permissions as the person who asked. A question typed in English is not a privileged path — it reaches exactly what its author could already open by clicking through the standard reports.

Certified metrics: same question, different words, same number

Validation stops a wrong query. It does not stop a wrong definition, and definitions are where most reporting arguments live. Ask three colleagues what an active learner is. One says signed in this month. One says opened a lesson. One says submitted something for grading. All three are defensible, and they produce visibly different numbers from the same database on the same afternoon.

If a model picks the definition per question, the definition moves with the wording. "How many active learners this month" and "how many learners were active in September" can resolve to two different computations, and the reader cannot see that the questions were read differently. Two credible, conflicting figures in one week end a reporting layer's usefulness.

With a registry in place, phrasing stops mattering: both versions of the active-learner question resolve to the same metric id, the same view, the same aggregation. The model's job shrinks to mapping messy wording onto a known name — a translation task with a fixed vocabulary, and the part of this problem a language model is genuinely suited to. It also gives the system somewhere to stand when it does not know: ask for something nobody has modelled and the honest answer is a refusal naming the nearest metric it holds.

Label the answer, and log every run

A validated specification guarantees the number was computed the way the definition says. It does not guarantee the specification matches the question asked. A model can pick a defensible-but-wrong metric, group by a dimension you did not mean, or read "last quarter" as calendar where your organisation runs on a fiscal year. The arithmetic is right; the interpretation is a guess.

So the output needs a different badge from a certified report — not a disclaimer in a tooltip, but a visible label on the tile and on the export. Certified means a human wrote the definition and a human built the report. AI-generated means a human wrote the definition and a model chose it. An AI answer can be promoted, but never promotes itself: a person reads the proposed specification, agrees it is the question they meant, and saves it as a report.

Every run should also leave a record, and the record has to hold the specification rather than only the sentence:

• The question as typed, and the specification the model proposed from it.
• The metric ids it resolved to, so you can see which definition produced the figure.
• Who asked, in which organisation, and the permissions the query ran under.
• The model version and the prompt version, so a change in behaviour traces to a change somebody made.
• The outcome — rows, a refusal, or a validation failure. Refusals map the gap between what people ask for and what the registry models.
• A request id shown on screen as well as stored, so somebody holding a screenshot can be matched to a run.

"The model produced it" is not an answer to how a figure in a regulatory return was derived; a run record carrying a specification, a metric id and an identity is. Models and prompts also get updated, and without a log you cannot tell whether last month's answers would still come out the same today. Keep it append-only: a log that can be edited afterwards is a convenience, not evidence.

How this works in Lurno

Lurno's reporting is built to the shape described above. Metric definitions live in one reviewed file — 32 of them today, each naming its source view, its aggregation, the dimensions it may be grouped by, and a privacy classification. Thirteen certified reports ship on top of them, and adding a metric is a pull request rather than an admin form.

The question bar reads a description of that registry and returns a report specification. It does not write SQL and does not hold a database connection: the AI service does not import the query compiler at all, and a build test fails if anyone adds the import. The proposal is validated exactly like a hand-built report and run by the same compiler, as the person who asked. Tenant isolation sits in the database rather than in application code — 868 row-level security policies across 676 migrations — so a query that lost its filter returns nothing.

Answers from the question bar are badged Ungoverned (AI), never Certified, and every run lands in an append-only log with its own request id. Metrics that could re-identify a person group only above a minimum size; below that the cell is suppressed rather than rounded. The same rule holds everywhere a model touches the platform, described on how AI works across Lurno: the model drafts, a person decides, and the boundary is enforced in code.

Six questions for a vendor selling this

Six questions separate the two architectures, and none require you to be an engineer.

1. Does the model write SQL? If yes, everything below matters more.
2. As whom does the query run — the person asking, or a service account?
3. Where is tenant isolation enforced: in the database, or in the code that builds the query?
4. Where are metric definitions kept, who can change one, and what happens to existing reports when somebody does?
5. Ask the same question two different ways, in front of me. Do the numbers match?
6. Ask it something you know it cannot answer. Does it refuse and say why, or does it approximate?

The last one tells you the most. A system built on a registry knows the edge of what it models and can name it. A system that writes SQL has no edge — every question is answerable, which is another way of saying no answer carries information about whether it should be believed.