This is a follow-up to our post AI-Ready Data Is Still Analytics-Ready Data. Here I want to dive a little deeper into the impedance mismatch between large language models and structured data.
Large language models are trained to take unstructured text as input and generate unstructured text as output. That makes them very good at working with large volumes of documents. It also makes them surprisingly good at generating computer code, which is itself just another form of unstructured text. In both cases, the model is operating in its native medium. Tokens in, tokens out. Pattern completion over language.
Structured data lives in a very different world. Tables have schemas. Columns have types. Relationships matter. Cardinality matters. Measures have business meaning. Joins encode assumptions about the world. None of this is native to how large language models were trained to reason. When we ask a language model to “work with data,” what we are really doing is forcing it to simulate how a human analyst would interact with tools that operate over structured data. The model is not reasoning over the table. It is reasoning about the language we use to talk about the table.
That gap shows up in practice. Large language models are good at writing SQL, but they do not know whether the SQL is semantically correct. They can generate plausible queries that reference columns that do not exist, join tables in ways that double count metrics, or produce aggregations that violate business logic. We call these hallucinations, but that framing is misleading. The model is not hallucinating in some pathological way. It is doing exactly what it was trained to do: produce likely looking text given prior examples. The problem is that correctness in data work is not a property of text. It is a property of how that text maps onto real schemas, real relationships, and real semantics in a live system.
This is part of why I keep coming back to the idea of AI-ready data foundations. If you want to use LLMs for data work today, you have to compensate for their impedance mismatch with structured data. That means putting guardrails and context around them. Explicit schemas. Governed metrics. Defined join paths. Rich semantic layers. Tooling that constrains what the model can ask for and how results are interpreted. The model becomes useful not because it “understands” your data, but because it is embedded in an environment that does.
There is a parallel research track that is worth paying attention to here. A growing body of work is exploring what are sometimes called tabular foundation models. These are machine learning models pretrained on large and diverse structured datasets so that they can perform zero-shot or in-context learning on new tabular problems with minimal fine-tuning. Instead of being trained primarily on text, they are trained to model distributions over rows, columns, and relationships in tables.
Early examples include models like TabPFN, Mitra, and TabICL. The promise of this line of work is that, in the future, we may have foundation models whose native medium is structured data rather than language. Models that reason directly over tables, not just over text about tables. If that pans out, the impedance mismatch problem may shrink. The model would no longer need to be mediated entirely through SQL and tool calls. It could operate closer to the underlying data representation itself.
But those models are still over the horizon for most practical, production use. Even if they mature, they will not eliminate the need for well-modeled, well-governed data. A model trained on messy schemas, ambiguous metrics, and inconsistent joins will faithfully learn and reproduce that mess. Foundation models do not rescue you from foundational problems. They amplify whatever foundations you give them.
So in the near term, the practical path forward is not to pretend that LLMs “understand” your data. They do not. The path forward is to design your data stack so that tools can safely mediate between language models and structured systems. That means treating schemas, semantics, and data quality as first-class infrastructure. If and when tabular foundation models become viable, they will sit on top of the same foundations. They will not replace the need to build them.
