Agentic Data LineageADL

Column-level lineage

How to extract column-level data lineage from application code

Data lineage records how data moves across systems. Column-level lineage records it field by field: which source columns fed a target column, and through which transformations — the level at which “where did this number come from” can be answered.

It is asked for in four situations: proving to an auditor that a reported figure traces back to its sources; working out what breaks before a column changes; migrating a platform without silently dropping a field; and establishing, quickly, what a bad upstream load contaminated.

The tooling for part of this is mature. Practice is still poor. Ten years after BCBS 239 was published, the Basel Committee assessed all 31 global systemically important banks and reported that additional work was required at every one of them to attain or sustain full compliance. Basel Committee, November 2023 ↗

Lineage is one requirement among many there, and the one where the tooling story splits cleanly in two: it is two problems wearing a single name, and only one of them has an automated answer.

Where the logic declares itself, lineage is already solved

A query is self-contained: it names its source columns, its target, and the expressions between them. The lineage is in the text. The same holds for ETL tool metadata, declarative mapping files, and warehouse query history — the artefact states the mapping, and reading it is a parsing exercise.

Mature open tooling — SQLGlot’s lineage module, sqllineage — returns a column graph from a statement, and catalogs assemble the rest from logs and tool metadata.

If a parser can solve it, it is already solved. Buy it, do not build it, and read no further for that half of the estate.

Where the logic is application code, nothing declares the mapping

Application code is different in kind, not in degree. Nothing obliges it to record the mapping between a source field and a target field — that mapping is the consequence of what the code does, and you know it only by working out what the code does.

Three properties make that hard.

Dynamic
Names are assembled while the program runs, from a parameter, an environment value, a date, a lookup. The name of the table being written may appear nowhere in the repository. Static reading can describe the shape of such a name; it cannot tell you the name.
Modular
One transformation is spread across parts of a codebase that never reference each other directly — derived in one place, called from another, configured in a third. No single location holds the answer.
Data-dependent
Which branch executes, which columns are selected, which target is written: decided by a control table, a properties file, or a value produced by an earlier step. That information is not in the repository at all.

Why a parser cannot finish the job

Two limits. The first is the one above: where a value only exists at run time, it is not in the syntax tree to be found.

The second costs more. Coverage is bought idiom by idiom. A parser encodes rules about how code is written, so every unfamiliar library, wrapper or house style needs new rules. Coverage tracks the effort spent extending it, which means the tool never stops being a project. The remainder is handed to a person — and in lineage the remainder is what matters, because an unexplained column is exactly the one an auditor asks about.

Star expansion is the obvious failure. The cases below are harder to dismiss: naming every column does not save you.

Running the code answers only for the code that ran

There is a third route. Instrument the job, let it execute, and capture lineage after the names have resolved — no static analysis required, because nothing is left to infer. Where you can do this, fidelity is excellent and you should do it first.

Its ceiling is inherent rather than a matter of tooling maturity. You get lineage for code that ran, on data that flowed, in an environment you could instrument. A branch not taken this quarter gives you nothing. Jobs you cannot run give you nothing. And it cannot answer a question before a change ships, because it only describes what already happened.

The same problem in four languages

Each section makes the same point in a different language, and they are independent. Pick the one you work in.

Extracting data lineage from Python code

Much of Python resolves statically, and serious tools go well beyond a naive tree walk: dataflow analysis, constant propagation, alias tracking, inlining helpers across modules. They will follow an import, unwrap a decorator, and can sometimes reduce a runtime-built name to a partial pattern — useful, even when it is not an answer.

The difficulty starts where the language’s flexibility does. Dynamic: table names assembled from environment values and dates. Modular: the transformation applied to a column chosen by dictionary dispatch, so the function bound to a field is not known until run time and usually lives in another module. Data-dependent: the configuration that decides what to read frequently lives in a control table in the warehouse, not in the repository.

Two more that catch tools out: SQL arrives as a string, so a Python analyser needs a full SQL parser embedded in it, per dialect; and most enterprises do not call data libraries directly, but through an in-house wrapper whose semantics no off-the-shelf model of a dataframe library knows anything about.

Example — every column named, still unresolvable
def load_dimension(conn, entity: str, run_date: date) -> None:
    cfg = DIM_CONFIG[entity]                  # config keyed by a caller's argument
    src = f"{os.environ['RAW_SCHEMA']}.{cfg['source_table']}_{run_date:%Y%m}"

    # columns are named explicitly — no SELECT * to blame
    df = pd.read_sql(
        f"SELECT customer_id, gross_amt, fee_amt FROM {src} WHERE load_dt = %(dt)s",
        conn, params={"dt": run_date},
    )

    df = df.rename(columns=cfg["column_map"])  # mapping built elsewhere
    for col, fn in TRANSFORMS[entity].items(): # which function? decided at run time
        df[col] = fn(df)

    df.to_sql(cfg["target_table"], conn, schema=os.environ["MART_SCHEMA"])

A parser sees three source columns and a write. The two schema names are environment values and are not recoverable at any price. The rest — which table was read, which was written, what the columns were renamed to, which inputs fed each derived column — is scattered across DIM_CONFIG and TRANSFORMS, and is reachable only by an analyser that can pin entity at every call site.

What a parser cannot reach

Whichever section you read, the shape is the same: the column names were in the source, and the missing piece was not.

It is worth splitting that missing piece in two, because the halves have different fates.

Most of it is a comprehension problem. A transformation spread across four files. A helper that computes a net amount whatever it is named. A name assembled from parts that are all present somewhere in the repository. Nothing is absent here — the information is there and scattered, and what a parser lacks is not access but the ability to read for intent.

Some of it is an information problem. A column list held in a control table, a branch decided by a properties file, a name that exists only while the job runs. That is not in the repository at any price, and no system whose input is source code recovers it. The correct output there is an explicit unresolved edge, not a guess.

The first half is what gets handed to a person, who reads the code and decides. That person is the bottleneck: slow, expensive, finite, and their answer goes stale on the next release. They are also the only step in the process that reads for intent.

So the thing worth automating is not the parsing. It is the reading.

If not a parser, then a model — and a model brings its own constraints

Language models read unfamiliar code and work out what it does, without being told in advance how it was written. A new house style is not a new project; it is just more code.

A model does not behave the way a parser behaves, though. It has properties that no prompt and no larger model removes, and anyone building this way builds inside them.

Attention degrades with size, and faster with complexity
Volume is the smaller half of it: nested logic, many interacting names and long chains of indirection cost attention faster than length does. So the unit of work must be small and simple. Splitting a file at an arbitrary line count buys the first at the cost of the second — each piece gets harder to understand, because the part that explained it was cut away. Units have to be bounded by meaning rather than by text, which is awkward precisely because the code was modular to begin with.
Decomposition then makes omission silent
A run returning a third of the writes produces a graph that looks complete and is not, and the gap is invisible without reading the code you were trying to avoid reading. Under-recall, not invention, is the failure mode to design against: bounded units so nothing is skipped by accident, an explicit account of what was examined, and a reviewer pointed at what was left unresolved rather than left to notice its absence.
The context window is finite, so what you supply is a decision
Anything loaded is attention spent, which makes selection the design problem: a small working context, everything else retrieved on demand, and the system judging what is worth supplying for the question in front of it.
A model answers from training when the input is thin
Asked about code it cannot fully see, it will often produce something plausible rather than report that it cannot tell. For lineage that is the worst available failure, because a plausible edge is indistinguishable from a real one after the fact. Every conclusion has to trace to something actually supplied, and “not supplied” has to be an outcome the system can return.
A model is not served as a function, so reproducibility has to be engineered
The same input can yield a different answer on a second run unless something prevents it, and for an artefact a regulator may read, that is disqualifying. It is achievable, and it is a property of the system rather than of the model: fix the decomposition, fix what is retrieved and in what order, ground every conclusion in supplied input rather than recollection. Be exact about the bar, though — byte-identical output is not the promise. A note may be worded differently, an ordering may vary. What has to hold is the substance: the same edges, the same fields marked unresolved. Treat the model version as part of the input.

None of these is fixed by a better model. They are properties of the medium, and they press harder as the codebase grows. What answers them is everything around the model — how work is divided, what is retrieved and when, what each conclusion is grounded in, what is held fixed between runs. The difficulty of this problem has moved: it is a systems problem now, not a prompting one.

Which approach should you use

Queries, ETL metadata, declarative mappings, warehouse history
A catalog and an established parser. Solved — buy, don’t build.
Jobs you can run and instrument
Runtime capture, for high-fidelity lineage on the paths that execute.
Application code
The gap. A parser resolves the well-behaved part and hands you the rest. A reading system trades a higher cost per run for independence from house style, with under-recall as the risk to manage.

Under all three, an unresolved edge is a result. Knowing which columns you cannot explain is a different position from not knowing, and it is the one that survives questioning.

Frequently asked questions

What is column-level data lineage?

The field-by-field record of how data moved: for a given target column, the source columns that fed it and the transformations applied on the way. Table-level lineage stops at "this table feeds that one".

Column-level or table-level — when does the difference matter?

Table-level is enough for dependency maps and scheduling. It is not enough for an audit question about one figure, for impact analysis before a field changes, or for finding what a bad load contaminated downstream. Those are all questions about a column, and a table-level graph cannot answer them.

Why is lineage missing from my data catalog?

Catalogs read artefacts that declare their mappings — query logs, ETL metadata, warehouse history. Application code declares nothing: the mapping is a consequence of what the code does, not a record of it. The gap in a catalog is usually the jobs written in Python, SAS, Scala or Java.

Can you extract data lineage from Python code?

Partly, and the boundary is sharp. Straightforward assignment chains resolve statically. Names built from environment values or configuration, transformations chosen by dispatch at run time, and column sets that live in a database do not — that information is not in the source.

How do you get lineage from SAS macros?

Not reliably from source alone. A DATA step can write a macro variable that later code resolves, so a table or column name may arrive from a row of data at run time. Source analysis narrows the candidates; the control data decides between them.

Does OpenLineage give column-level lineage for Spark?

Yes. Its Spark integration walks the resolved logical plan and emits column-level lineage without code changes. The limit is that it only covers code that actually ran.

Why not just use a parser?

Use one wherever it works — for queries it is the right answer. Parsers need extending for every new idiom and house style, and cannot resolve values that only exist at run time. That leaves a remainder needing manual work, and the remainder refills on every release.

Can an LLM alone extract data lineage from code?

For one short, self-contained script, often. But a method is judged by its worst case, the way complexity is — and the worst case is thousands of lines across many scripts that only mean anything together, with names that resolve at run time. There a model alone cannot: attention degrades with size and structure, the code does not fit in a context window, and part of the answer was never in the text. Closing that gap takes an agentic system around the model — dividing the work, retrieving what each unit needs, and marking what the code never says.

What does BCBS 239 require for data lineage?

It requires banks to aggregate risk data accurately and to reconcile what they report back to its sources, which in practice means being able to demonstrate the path from a reported figure to its origin. Ten years after publication the Basel Committee reported that further work was needed at every one of the 31 global systemically important banks. Basel Committee progress report, November 2023 ↗

Where we are

Those are the constraints we build inside. Agentic Data Lineage extracts column-level lineage from application source code. We are close on Python, and are extending the same machinery to the others, which is the real test.

The claim is narrow and checkable: given the same codebase the system returns the same edges and the same unresolved set, it accounts for what it examined, and it tells you what it could not resolve rather than filling the gap with something plausible.