Back to All News

The foreign enterprise market spent two years teaching AI to answer questions about its own data. The result: 21% correct answers

Article date

10 14 2026

Article Author

Reading Time

10 minutes

The first wave of enterprise AI data assistants ended predictably. Companies connected a language model directly to their data warehouse, let executives ask about margins, and received answers that looked authoritative and were quietly wrong. This is not an opinion. It is a documented result.

The Spider 2.0 benchmark, published at ICLR 2025, was built from 632 real enterprise workflow tasks: BigQuery and Snowflake estates often exceeding 1,000 columns, with solutions requiring multi-step SQL beyond 100 lines. An agent framework based on o1-preview solved 21.3% of those tasks, against 91.2% on the cleaner Spider 1.0. For GPT-4o the collapse is steeper: 10.1% against 86.6%.

Vendor promises sit separately from that reality. Marketing material advertises 85–90% text-to-SQL accuracy; enterprise deployments report 10–31% on production schemas.
The model is not at fault. Undefined meaning is
Why does accuracy fall that far? On a tidy schema, the word "margin" maps to one specific column. Inside a real company, it maps to an agreement between people. If three divisions calculate margin differently, the model will honestly return three different numbers, and none of them is an error. The error is a decision made on any of them without verification.

The lesson for an executive assistant is narrower than it first appears. The hard part is not generating SQL. The hard part is agreeing what the company calls each metric, and being able to show where a number came from.
A citation is not an interface flourish. It is the condition of use
An answer you cannot verify cannot be the basis of a decision. That is the only reason worth showing the reasoning behind a conclusion rather than just the conclusion.

PlanExpert builds this into the response structure itself: a written conclusion, charts, tables, intermediate analytics as step validation, and a reference to a document or a specific page with a link to the original. In practice, asked why margin fell in Q1, the assistant answers: margin −4.2% driven by a 12% rise in cost of goods, with logistics as the main driver. It shows the source: the Q1 financial report file, with a link into 1C Cost Analytics.

The difference is structural. The first version is "the AI produced a number, believe it or not." The second is "here is the number, here is where it came from, verify it in two minutes."
Business ontology is the step the foreign market skipped
The PlanExpert pipeline has four stages: data, business ontology, agents, dialogue.

The first stage is a single data source. Connectors reach ERP, CRM, and customer APIs; ClickHouse; PostgreSQL and Postgres Pro. Files arrive by drag and drop and are indexed automatically.

The second stage is automatic ontology: linking scattered systems into one model of the business. This is the step missing from the first wave of foreign deployments, and its absence is precisely what produced confident, wrong numbers. The gap Spider 2.0 revealed does not open at the model level. It opens at the level of meaning: a tidy schema versus a thousand columns and rules that were never written down.

We did not design this logic after watching someone else fail. The platform was built around it from the start.
Data never leaves your perimeter
The platform is deployed entirely on the customer's infrastructure. Language models run on your GPUs. No query reaches an external service. There are no third-party APIs and no third-party clouds.

The stack is built on open source, without vendor lock-in. The product is listed in the Russian software registry, which matters for critical information infrastructure and regulated industries. For those customers, on-premise deployment is not an advantage. It is the entry requirement: cloud assistants are eliminated during approval, and their feature lists become irrelevant.
A pilot that tests your data, not the model
The PlanExpert pilot runs for 2 months at a fixed price of RUB 5 million. It includes platform deployment on the customer's servers, connection of up to two corporate systems, adaptation to business logic, and testing against your real data.

The timeline has five stages. Analysis of IT infrastructure and data sources takes 1–2 weeks. Integration and connector configuration take 2–3 weeks. The pilot itself, verifying metric accuracy and answer quality, takes 2–3 weeks. Launch with user training takes from 1 week. Support, SLA, and source expansion run continuously.

The third stage is the one that matters. What gets tested is not whether the AI works, but whether the system's answers match what your specialists consider correct. If they diverge, that is a conversation about metrics, not about the model.
When it will not work
An honest conversation starts with conditions on the customer's side. There are four, and each one surfaces during the first stage of the pilot.

If metrics are undefined. The platform adapts to your business logic and your metric dictionary, but it can only adapt to something that exists. If your company has no agreed answer to "what do we call margin, and over what period," the system cannot give an unambiguous answer. The right first step is an internal conversation about definitions, not a pilot. The same cause sits behind the 21.3% figure in the benchmark: where the meaning of a metric was never fixed explicitly, any system answers by guesswork.

If there is no machine access to the sources. Connectors exist for PostgreSQL, Postgres Pro, ClickHouse, ERP systems including 1C, CRM platforms, and customer REST APIs. If the required data sits in a system without such access, and can only be exported by hand once a month, answer quality is limited by the export frequency, not by the platform.

If someone expects an answer where none can exist. The assistant assembles and explains what is present in your sources. It does not replace the executive's decision or remove responsibility for it. Where a question requires a management choice rather than a calculation, the system will supply the data for the choice and not the choice itself.

If no one owns the process. Five implementation stages include audit, configuration, user training, and handover of roles and permissions. All of that requires a person inside the company who is accountable for the outcome. Without one, the pilot will finish technically successful and stay unused.
Questions and answers
IT director: how is this different from the BI system we already deployed and nobody uses?

BI answers the question an analyst predefined, through a dashboard built in advance. Here the question is asked in words, and the answer is assembled from sources at the moment of the request, with intermediate steps shown and a link to the underlying data. Dashboards are not replaced. The queue to the analytics team for every new slice is.

CFO: can I trust a number the system produced?

You can trust an answer you can check. Every conclusion is backed by a source: a document and a page, or a specific query in the system. Verifiability is not a feature here. It is the condition without which the product makes no sense in a management context.

Chief digital officer: what happens to our data?

Nothing leaves your infrastructure. The platform is deployed on your servers; models run on your GPUs. Compliance with critical information infrastructure requirements is part of the product, and it is listed in the Russian software registry.

COO: how long does this take and what does it cost?

Deployment, connection of up to two corporate systems, adaptation to business logic, and testing on real data: 2 months, RUB 5 million fixed, with no hidden payments.

Owner: what if the pilot shows our data is a mess?

Then you learn that in two months and five million rubles, rather than after eighteen months of implementation at a multiple of the cost. That is what a pilot is for. It tests not only the platform, but your readiness for it.
What to do first
Before discussing deployment, answer one question honestly: how much time currently passes between a manager's question and a finished answer? If the honest answer is "between two days and two weeks," you have the exact gap that companies worldwide tried to close by connecting a language model to their warehouse, and got 21% correct answers.

The next step is a 30-minute meeting to discuss your case, a platform demo, and a pilot statement of work within one week. The conversation starts with the question that currently takes you two weeks to answer, not with a technical specification.