Rubble / Blog
How Does Software Engineering Work in Practice in the AI Era?
A module has not changed a single line of source code today.
But one of its dependencies was upgraded, the compiler version changed, and an environment variable that affects the build result changed as well.
Can continuous integration safely reuse yesterday's cache?
If we look only at the source revision, the answer can easily be "yes." But what we ultimately deliver is not a source directory. It is an artifact produced from that source under a specific set of dependencies, toolchains, configurations, and environmental conditions. Once any condition that actually contributes to the result changes, the old cache may no longer represent the software we have now.
This problem is not limited to build caches.
The same change immediately raises more questions: Which downstream components need to be rebuilt? Which test results are no longer valid? What should go into the container image? Which tasks does the pipeline actually need to run? Which code should a coding agent read first? When a vulnerability is found in an open-source component, which products actually contain it? Why did two versions produce different artifacts?
The first two articles approached this from Build Engineering and software engineering as a whole, and discussed why, as implementation costs fall, the center of engineering work increasingly shifts toward change handling.
But knowing that an engineering system should identify impact, establish evidence, control boundaries, and reach explicit dispositions such as accepting, adjusting, splitting, deferring, rejecting, or redesigning a change still leaves a more practical question unanswered:
How do these capabilities actually become part of everyday engineering? How do we put them into practice?
This article focuses mainly on how to establish the technical facts and verifiable execution that change handling depends on. Whether a particular impact is ultimately acceptable, and who has the authority to accept the residual risk, still belongs to the decision authority discussed in the first two articles and will not be expanded here.
If the implementation still consists of manually maintaining more dependency lists, pipelines, artifact descriptions, supply chain inventories, and agent prompts, then we are only copying the same implicit knowledge into more places.
What really needs to change is what objects the engineering system uses to describe software, and how the relationships between those objects participate in actual execution.
What Kind of Engineering Facts Should Describe Software?
Start with Engineering Units
Source files, directories, repositories, and modules are among the easiest things to treat as basic objects in software engineering.
They matter, but they are not natural engineering boundaries.
A compiler contains no application source code, yet it can directly change a large number of artifacts. A system library may come from the operating system but still be part of the software's runtime. An environment variable may look like nothing more than a string, but if it affects compiler options or feature selection, it can change the final result as well.
So an engineering unit cannot simply be another name for a software module.
I prefer to think of an engineering unit as:
Something for which the engineering system needs an independent identity, clear inputs and outputs, and a boundary of change, and whose relationships with other objects the system needs to understand.
An engineering unit can be large.
A product family can be one engineering unit. So can a specific product, subsystem, or platform.
It can also become progressively finer-grained: a component, module, toolchain, system library, or file. When necessary, a configuration item that can independently change a result, or even a single environment variable, can be its own engineering unit.
An engineering unit does not necessarily have to compile into a binary. A patch, document, test report, verification result, or set of environmental settings can all be outputs of an engineering unit. What matters is that every engineering unit produces one or more outputs with clear semantics, so that its direct downstream users know what they are actually consuming.
Granularity is not determined by lines of code.
What really determines granularity is whether an object needs to be independently identified, reused, verified, replaced, and tracked as it changes, and whether it has an engineering boundary worth expressing separately.
"Engineering unit" here is not intended to replace established terms such as configuration item, build target, artifact, or test task. It is simply an umbrella term used in this article to discuss how objects at these different levels can be identified, composed, and traced within the same change. Once we enter a specific domain, the more precise professional term should still be used.
This is really just a straightforward form of divide and conquer. It reflects a familiar engineering approach: break a complex system into units with clear boundaries that can be understood and verified independently, then organize them again through explicit relationships.
A product-level engineering unit should not need to care directly about thousands of source files. It only needs to know which product configurations, platforms, or subsystems it directly depends on, and what definite engineering outputs those direct upstream units provide.
Each subsystem then deals with its own direct upstream units, while modules manage finer-grained dependencies.
The complete product still forms a full graph of engineering relationships, but each engineering unit only needs to understand the part directly connected to it. Product-level reasoning does not need to keep drilling down to files, and file-level changes do not need to expose every internal detail to the product level.
Details can be encapsulated. Real relationships cannot disappear because of that encapsulation.
This is not fundamentally different from code modularity. The difference is that we extend the same divide-and-conquer approach beyond source interfaces into builds, toolchains, environments, artifacts, products, and runtime conditions.
Relationships Describe How Outputs Are Produced
Once engineering units exist, the next step is to establish the relationships between them.
These relationships cannot be understood only as imports, function calls, or repository dependencies in source code.
Suppose a binary is compiled from source A. The compilation uses toolchain B, links against system library C, selects a feature through configuration D, and has one build option determined by environmental input E.
A, B, C, D, and E all participate in producing that artifact.
A source-level call graph alone would miss relationships that can actually change the result.
The core of the relationship model should express the causal and derivation relationships by which engineering units produce outputs: which direct upstream units a unit depends on, what those upstream units provide to it, and under what conditions it produces its own result.
A module's source code remaining unchanged does not mean that nothing has changed from an engineering perspective. If its toolchain, system library, or another upstream engineering unit changes, then the statement "what artifact does this source produce under these conditions?" is no longer the same as before.
The reverse is also true: we should not simply treat everything on a machine as an input.
If an environment variable is never read, it should not invalidate the entire downstream graph merely because its value happened to change. The relationship model should express actual causality, not mechanically add everything observable in the environment.
That is the kind of precision an engineering model needs:
Changes that should propagate must not be missed, and changes that do not participate in the result must not be amplified for no reason.
For a moment, imagine software as a LEGO castle. Source code, toolchains, system libraries, and configurations can all be treated as bricks used to build it.
Knowing which bricks were used still does not tell us what castle will result.
Without assembly instructions, the same set of bricks given to different people, at different times and places, can produce different results. What determines the castle is not just the inventory of bricks, but also how they are assembled.
The relationship graph is like a set of assembly instructions that a machine can understand.
It records not only "what exists," but also "how these things combine to produce the final result."
For a build-oriented engineering unit that is expected to be reproducible, the same inputs and the same assembly instructions, under controlled execution, should produce the same result. Outputs that genuinely include time, randomness, signatures, or other side effects need to express that nondeterminism explicitly. They cannot continue to use the identity and caching rules of ordinary reproducible outputs.
Nor does a "complete set of engineering relationships" mean that the engineering model claims to know everything about the real system.
It means that, for a specific output, every factor that the current model allows to affect that output must be inside the corresponding causal scope. Real relationships that still exist outside the model have to be discovered later through execution, failure, and runtime feedback.
From a systems engineering perspective, work around a system's technical identity, composition, and change can be understood as part of Configuration Management (CM).
But in modern software, the technical state that needs to be maintained cannot stop at source revisions and configuration files. Dependencies, toolchains, environments, artifacts, and the derivation relationships between them all need to enter a traceable and continuously revisable engineering model.
A single repository can provide a shared source of truth for source code. It cannot by itself explain why the final artifact has the form it does.
Engineering State Describes What a Unit Actually Is Right Now
Modern software engineering relies heavily on human-readable names.
A module is called foo. A library is version 3.2.1. An image is called latest. A build corresponds to a particular Git commit.
These names are useful for communication, but they are not enough to determine whether two engineering results are actually the same.
At least three levels need to be distinguished here.
An engineering unit answers: "Which object is this?" The same compiler, module, or product can have many engineering states over time.
An engineering state answers: "Under which source, dependency, toolchain, configuration, and environmental conditions was this object formed?"
The output content answers: "What did this execution actually produce?"
The same engineering unit can have different engineering states. Two different engineering states may even happen to produce byte-for-byte identical artifacts in one execution, but the causes and provenance of those results are still different. Evidence already established for one state cannot automatically be transferred to the other.
So an identity used for build and provenance decisions needs to include every input that affects the current engineering state.
Mutable source references should resolve to definite content when a specific engineering state is formed. The selected toolchain and system libraries need definite identities. Configuration and environmental inputs that actually affect the result need to become part of that engineering state as well.
These facts should not depend on engineers manually keeping another table in sync.
The engineering system should record whatever it can determine automatically during actual resolution and execution.
Secrets and sensitive information obviously do not need to be stored in plaintext as a result. What matters is preserving a controlled representation sufficient to determine identity and change, rather than replacing the original security boundary with exhaustive recording.
Once the complete engineering state changes, the old state can no longer directly represent the current one. Even if the old and new states eventually produce identical content, their causal provenance is still different.
Only when a downstream unit explicitly depends on a defined and verified output boundary can it reuse that output by content without inheriting every internal derivation difference from the upstream engineering unit.
Encapsulation reduces the detail that downstream units need to understand and inherit. It does not erase the provenance of the output.
How a Change Lands on the Same Engineering Object
If the relationship model stores only "what things look like now," we still cannot easily answer one of the most common questions in long-term software development:
Why are these two versions different?
When a formal output is produced, the engineering system should also freeze a corresponding snapshot of the relationships that formed it.
That snapshot contains more than the source revision.
It also includes the direct and transitive relationships that actually participated in producing the output: dependency states, toolchains, relevant system libraries, build configuration, target platform, and environmental inputs that affect the result.
An artifact is then no longer an isolated file. It is bound to the facts that explain why it became what it is.
When a new change arrives, we can first compare the old and new engineering states.
If a module's source code is unchanged but the compiler changed, that difference appears directly. If a system library was upgraded, the relationships can show which downstream units now need to be reconsidered. If an environmental input that affects an artifact changed, it no longer remains hidden on some CI machine.
Before we execute a build, we can therefore already know:
Which engineering units have changed derivation conditions, why their old results no longer directly represent the present state, and along which known relationships the change may continue to propagate.
At this point, we have the reason for the change and the impact scope that the current model can compute.
After actual execution, the engineering system gains another kind of fact:
What artifacts were actually produced, which tests and verifications were completed, and which relationships or assumptions were shown by execution not to hold.
Tests need to enter the same engineering model.
Build and test can belong to the same engineering unit, or they can be split into different units. The right split depends on whether they need independent execution, independent reuse, and independent evidence. Test reports and verification conclusions can themselves be outputs with explicit semantics.
If a product relationship graph completely describes how the software is produced but no engineering unit produces any test or verification result, then the graph has established the causal scope of the build, but it has not established an evidence scope for whether the software behaves as expected.
The difference between two engineering states can therefore form a richer engineering change than a Git diff.
Code changes are only one part of it.
Toolchain changes, system library changes, dependency-state changes, environmental changes, the currently computable impact scope, newly produced artifacts, completed verification, and unresolved gaps in the model can all belong to the same engineering change.
This is where two questions from the first two articles finally land on the same engineering object:
Before execution, the engineering state tells us why the old result is invalid and what may be affected.
After execution, the same engineering state remains bound to the actual results and verification evidence.
The engineering system no longer has to guess impact from a code diff first and then search for evidence in a separate pipeline and another set of reports.
The cause, propagation, and result of a change begin to belong to the same set of engineering facts.
The Engineering Model Is Formed and Revised Through Execution
The most practical difficulty in building such a model is that many projects have never expressed these relationships completely.
Some of them live in build scripts, some in CI, some in container definitions, and some only on developer machines or in people's experience.
AI is valuable here.
It can read project structures, build scripts, pipelines, dependency declarations, and existing documentation to help identify possible engineering units and propose candidate relationships. After a build fails, it can also help analyze what the current model may have missed.
For large legacy systems, this can significantly reduce the cost of recovering an initial engineering model from scripts, documents, and human experience.
But there is one boundary that cannot be crossed:
AI can help establish and extend the relationship model. A relationship does not become an engineering fact simply because AI says, "I think this is how it works."
The same applies to people.
A senior engineer saying, "This module should depend only on these things," can be an excellent starting point for modeling. It cannot replace objective verification.
AI inference, human declarations, and relationships observed during execution can all participate in the modeling process, but they should not carry the same level of confidence until they have been tested through actual execution or other independent verification.
A model only remains under pressure from reality when it participates in real decisions.
Build order is derived from the relationships. Cache reuse is decided from engineering state. The scope that needs reverification comes from relationship propagation. The actual source revisions, toolchains, dependencies, and environmental conditions used during execution are automatically recorded into the engineering state.
At that point, the engineering model stops being an asset inventory that engineers must periodically synchronize. It becomes a set of facts naturally produced by the ordinary process of building, verifying, and releasing software.
Verification itself must also be objective.
For a build-oriented engineering unit, if the declared inputs are complete, the same engineering state should independently produce the same result. If actual execution still reads tools, files, environmental state, or network resources outside the model, the execution process itself should expose the inconsistency.
People and AI propose a description of the world. Actual execution tests whether that description is sufficient to explain the result.
But one successful execution cannot establish that the model contains every real relationship.
If the model misses a downstream relationship, it may simply fail to schedule that downstream unit, so no failure appears yet.
Making the engineering model participate in actual execution therefore only makes it easier for reality to refute the model. It does not make the model automatically complete.
For everyday development, builds, tests, and continued use can keep exposing missing relationships. Where stronger assurance is required, teams can also run periodic full builds or perform cross-checks at different times, under different levels of isolation, and in meaningfully different environments. This reduces the risk that the model sees only the part of reality that it chose to execute.
A build suddenly failing after isolation, an apparently unrelated modification affecting downstream software, an agent repeatedly requesting tools outside its declaration, or a runtime result that the model cannot explain—these are not merely execution failures.
They are also evidence that the engineering model needs to be revised.
The goal is therefore not to create a global engineering graph that claims to be permanently complete.
What we actually need is a revisable engineering model:
People and AI propose candidate relationships.
Actual execution turns what can be determined into engineering facts.
Hermetic execution and independent reproduction across environments keep testing those facts.
When reality shows that the model cannot explain the result, new relationships enter the model and change the next decision.
Configuration Management can only manage technical facts that have already entered the engineering model.
The engineering system must also be able to revise the model itself.
How a Three-Stage Bootstrap Can Test the Engineering Model
The first complex task we asked AI to perform under this engineering model was to build a complete GCC source bootstrap from an empty directory.
This was not a matter of asking an agent to change a few lines in an already prepared project.
It had to find the GCC source itself, determine how to obtain it, continue discovering the dependencies, tools, and system conditions required to build GCC, and gradually turn all of those into engineering units and candidate relationships.
The process took more than a day.
From the outside, the AI's work did not look elegant.
It looked more like exploring a maze.
It would go a few steps and discover a missing condition. After adding that, it would continue and find that the original route no longer worked. Sometimes it backed up a little. Sometimes it returned to a much earlier point and chose a different path entirely.
The relationship model was filled in continuously during this process.
New dependencies were discovered. Tools that had previously been hidden in the environment became explicit. Some boundaries that initially looked correct had to be changed after real execution.
The final result was a three-stage GCC bootstrap, not merely getting GCC to "compile successfully once."
Nor was the important part AI finally telling us:
"I have successfully established GCC's build relationships."
If we stopped there, the engineering model would still be only AI's interpretation.
We took the same relationship model into other environments and executed the complete three-stage bootstrap again.
It completed successfully again using the same relationships.
This still cannot logically establish that the engineering model contains every possible relationship. But it provides evidence that can be checked repeatedly in other environments without depending on the original agent or on subjective human interpretation.
Reproduction can also happen at different times and in different environments, and it can be extended to different isolation levels or more substantially different hardware. The greater the environmental differences, the less likely it is that a hidden dependency happens to remain identical everywhere, and the stronger the evidence becomes.
This process shows an important boundary:
AI is very useful for helping us explore an engineering model, but the model ultimately needs to be independently verifiable without the original agent and without subjective human judgment.
Otherwise, we have only replaced implicit experience held by people with implicit experience held by AI.
Do Not Start with the Entire Organization
At this point, it is easy to arrive at a seemingly reasonable but actually dangerous implementation plan:
First model every codebase, toolchain, platform, environment, artifact, and service relationship in the company.
Projects like this are usually very difficult to finish.
The larger the scope, the easier it is to build a metadata system detached from actual execution. Every team starts filling in fields while the real cache decisions, CI pipelines, and release systems continue using the old logic.
A better starting point is a problem that already creates recurring real cost.
You can start with caching.
Choose a build unit whose cache is frequently invalidated incorrectly or reused when it should not be. Bring its source, direct upstream dependencies, toolchain, configuration, and environment into its engineering state. If the model is correct, it should directly change cache decisions.
You can also start with continuous integration.
Choose a long pipeline that reruns many unrelated tasks after every modification. First establish enough relationships to compute the affected engineering units, then let the next pipeline be generated directly from those relationships.
Or start with helping agents navigate a large codebase.
Establish sufficiently reliable engineering boundaries for one product or subsystem, so that an agent begins from a candidate engineering subgraph rather than searching the entire repository.
Or start with the software supply chain.
Choose a group of third-party dependencies whose impact has historically been difficult to determine. Make every release artifact able to answer what it actually uses and through which relationships a risky component enters which products.
The important thing is not which use case comes first.
The important thing is that the first engineering model must control a real decision.
If the model is only there for people to look at, sooner or later it becomes another outdated architecture diagram.
If caches, CI, artifact production, component navigation, or agent working boundaries actually depend on it, errors and omissions will keep being exposed through use.
Once local relationships stabilize, expand gradually along direct upstream and downstream connections.
After the relationships inside a product become stable, expand into cross-product, cross-platform, and system-of-systems technical-state relationships.
This is a path that grows outward from real engineering facts rather than beginning with an idealized model of the entire organization.
How the Same Engineering Facts Enter Everyday Development
By this point, the foundation we need is fairly clear:
Engineering units represent objects. Relationships represent the causal structure by which outputs are produced. Engineering state fixes one specific set of conditions. State snapshots preserve historical facts. Before execution, the difference between old and new states gives us the reason for change and the impact that the current model can compute. After execution, actual results and verification evidence remain bound to the new state.
The next question is no longer how to keep expanding the model, but how existing engineering systems should use the same set of facts.
Continuous integration, artifacts and supply chains, and coding agents ask different questions. They therefore need different views and cannot all use the same decision rule based only on reachability in the graph.
But they should no longer maintain separate copies of the facts describing "what this software is and how it was formed."
Build and Continuous Integration
The basic object in many CI systems today is a long-lived pipeline.
Engineers maintain a Jenkinsfile, GitHub Actions workflow, or another task definition and keep adding conditions:
Run A when this directory changes. Run B on another platform. Run C only when a shared component changes.
As the number of products, platforms, and tests grows, these conditions eventually become another dependency system of their own.
It has to be maintained indefinitely and can easily drift away from the actual structure of the software.
If the relationship model can already answer:
Which engineering units had their engineering states invalidated by this change?
then the pipeline no longer needs to carry the responsibility for dependency modeling.
For each candidate change, the engineering system can compare the old and new engineering states, determine which build and test units need their results re-established, and generate an execution plan that exists only for this change.
Tasks in that plan correspond to engineering units. Dependencies between tasks come directly from the actual relationships between those units.
Tasks that can run in parallel can run in parallel. A task waits only when it genuinely requires the output of a direct upstream unit. The pipeline no longer depends on human experience to bundle several modules into one task or on large numbers of conditions that only approximate the real dependencies.
In an ordinary incremental pipeline, an engineering unit does not need to enter the execution plan if this change does not affect it, its existing result is still valid, and no independent verification rule requires it to run again.
A shared toolchain change may still produce something close to a full pipeline. A well-bounded local modification may require only a small part of the full engineering graph.
Every build task that enters the pipeline now has an explicit reason:
The engineering state it corresponds to was invalidated by this change, so its result needs to be established again.
There is no longer a need to put a large set of tasks into a fixed pipeline first and then use if statements, path checks, and environmental guesses to decide "which ones should actually run today."
Once the pipeline instance generated for this change has finished, it can even be deleted.
What needs to persist is the relationship model that produced it, the rules used to generate the execution plan, the corresponding engineering states, and the actual execution records—not a fixed task topology that people must maintain forever.
CI changes from a long-lived, manually maintained task graph into an execution view of one specific change over the current engineering relationships.
This changes several costs at once.
First, the maintenance cost of the pipeline itself falls.
Second, each change schedules only the engineering units that are actually affected, so the execution scope shrinks.
More importantly, real parallelism increases.
Configuring a pipeline for a certain amount of concurrency does not mean that amount of parallelism is actually achieved. Manually designed tasks usually contain team boundaries, historical compatibility decisions, and earlier performance optimizations. Modules with no real dependency may have been placed in one large task and therefore execute serially. Other pieces of work that do have an ordering dependency may have been made parallel simply because one is slow and the other is fast, producing intermittent and unstable build failures.
When task boundaries and dependency order come directly from the actual relationships between engineering units, the pipeline can get much closer to the maximum parallelism that is safely available from the engineering structure itself.
The build environment changes as well.
Traditional pipelines often bind products to environments. Product A uses one build image, product B maintains another, and a platform that depends on a special compiler keeps a dedicated group of execution nodes indefinitely.
If every engineering unit already knows exactly which toolchain, system libraries, configuration, and platform conditions it requires, the execution infrastructure no longer needs to permanently "pretend to be a machine for a particular product."
The same general-purpose build infrastructure can execute more engineering units that previously required different environments. Each unit receives its explicitly declared conditions when it runs.
Container images no longer need to serve as large, manually maintained environment templates either. The environment required for a build, and the dependencies that a runtime artifact needs to contain, can both be derived from the corresponding relationships.
This also means that local builds and CI should not have different software semantics.
CI may run on different machines, and the local host environment may be completely different. But as long as the inputs that actually affect the result are the same and execution stays within the declared boundary, a build-oriented engineering unit should have the same engineering state and result.
Problems caused by engineering inputs in CI should be reproducible locally.
If the same engineering state repeatedly produces unstable failures, the next step should be to look for undeclared relationships or attribute the problem to lower-level infrastructure such as hardware, storage, or scheduling—not simply say that "the CI environment is different."
Conversely, a build result already produced locally can also be safely reused by CI when the engineering state is the same, the result has been objectively verified, and it satisfies the current trust policy.
Some high-trust scenarios may still require rebuilding in an independent controlled environment. But that is done to create additional independent evidence, not because local development and CI inherently represent two different pieces of software.
A Product with More Than 900 Modules
These changes became very visible in a large product with more than 900 modules.
The product had long maintained a completely hand-orchestrated continuous integration pipeline with more than 200 task nodes and complex dependencies between them.
Some build tasks handled only one module. Others built several modules at once.
This partitioning was not derived directly from the real engineering structure of the product. It had gradually emerged from years of team organization and performance optimization.
Pipeline maintainers used experience to combine several modules into a task and to adjust which tasks could run in parallel.
But that orchestration was not the same thing as the product's real engineering relationships.
Some modules that could have been built in parallel were grouped into one large task and had to run sequentially. Some tasks with real ordering dependencies were deliberately run in parallel because one took much longer than the other, occasionally creating build failures that were difficult to reproduce reliably.
A deeper problem was that global structure and local detail rarely existed in the same person's head.
Only a very small number of engineers had a reasonably complete understanding of the overall pipeline, but they could not also understand exactly how every task performed its build. Engineers in individual development teams often understood their own local tasks very well but did not know the relationships across the entire product.
Every substantial pipeline optimization therefore required those two kinds of knowledge to be coordinated again. If a feature branch changed the pipeline, merging it eventually meant merging not only the code but also another independently maintained layer of pipeline logic.
We later rebuilt the entire product relationship model around independent engineering units.
The first full pipeline generated directly from those relationships had more than 900 tasks.
That full pipeline was actually slower than the old hand-written pipeline with a little over 200 tasks.
The reason was straightforward: finer engineering-unit granularity increased the number of scheduling nodes from a little over 200 to more than 900, adding overhead in the scheduler, execution resources, and task startup.
This was useful evidence in itself. Simply splitting modules more finely does not automatically produce better performance.
The real change happened after the first complete run.
Once those engineering units had produced results that could be safely reused, later changes no longer needed to generate more than 900 tasks every time.
Comparing the old and new engineering states directly showed which units needed their results rebuilt and how the change would propagate through downstream relationships.
In actual development, most everyday changes ended up affecting only about 30 engineering units.
A change to a toolchain or shared foundational module could still affect almost the entire pipeline. That is expected. The original team also deliberately ran full builds for this kind of change.
The new system does not make every change local. It stops treating every ordinary change as if it were global.
As a result, an everyday pipeline that previously contained more than 200 manually maintained tasks would usually shrink to around 30 tasks whose results genuinely needed to be rebuilt.
The new tasks were usually smaller as well.
A task in the old pipeline often built several modules together. In the new pipeline, tasks correspond directly to declared engineering units. Dependency order between tasks also corresponds directly to the actual dependencies between those units.
Work that can run in parallel is no longer forced into serial execution by a large task. Work that cannot safely run in parallel is no longer executed concurrently just because someone manually balanced task durations.
The "tens of times" improvement here is not a precise ratio derived from 200 ÷ 30.
It describes the result developers experienced directly. After code was pushed, the wait from the start of integration to the end of the build fell to a small fraction of what it had been. For most everyday changes, the improvement was typically by a factor measured in the tens. Build-resource consumption during execution likewise fell to a small fraction of its previous level.
The difference did not come from one isolated performance trick.
Fewer tasks needed to run. Individual tasks were smaller. Task dependencies were more accurate. Safe parallelism was higher. Unstable failures caused by incorrect parallel execution also declined.
Once a change finished using a set of build resources, those resources quickly became available to the next change or another project, substantially improving resource turnover.
The work of the continuous integration team changed as well.
They no longer spent most of their time maintaining a manually defined task graph with more than 200 nodes and repeatedly deciding "which modules does this task actually own?"
What was worth maintaining was the boundary of each engineering unit: which inputs determine its engineering state, which direct upstream units it depends on, and which direct downstream units consume its results.
The pipeline became a generated consequence of those relationships.
A Controlled Comparison That Changed Only CMake
This performance difference did not come only from incremental builds and caching.
In another CMake-based project, we refactored only the CMake files. We did not change the source code, toolchain, or set of modules being built.
Both the old and new builds ran on the same 32-core machine, both were limited to eight concurrent tasks, and both built the same modules under otherwise identical conditions.
The new build framework took only one quarter of the time required by the old one.
The same externally configured concurrency does not mean the same actual parallelism.
The old framework contained manually created task boundaries and ordering constraints. There were often not enough ready tasks to keep all eight execution slots busy, or a large task still executed much of its work serially internally.
After reorganizing execution around the actual engineering relationships, more genuinely independent tasks could run at the same time and the critical path became substantially shorter.
This case largely removed incremental caching, change scope, and machine differences from the comparison. What remained was mainly the structure of the task relationships themselves.
It shows that in complex engineering systems, the parallelism achievable through manual orchestration is often lower than the parallelism reachable from the real engineering relationships.
Artifacts and the Software Supply Chain
On the artifact side, a relationship snapshot first solves a provenance problem.
If a release artifact is bound to the engineering state that produced it, we know not only "what file is this?" but also which source it came from, which toolchain was used, and which dependencies and configurations participated in producing it.
Comparing two software versions therefore no longer has to mean comparing only their Git diffs.
We can compare the relationship snapshots bound to the two artifacts.
If the compiler changed while the source did not, the difference appears directly. If a system library was upgraded, we can follow the relationships to see which artifacts it entered. If an environmental condition that affects the result changed, it no longer remains hidden on the build machine.
The same facts can also become an important source of software supply chain information.
Which third-party components were used, which exact versions they were, through which relationships they entered a particular artifact and product, and which toolchain produced them are facts that already exist in the engineering state that formed the artifact.
When a vulnerability is found in an open-source component, we no longer need to rely only on repository searches to guess which projects may have used it.
Starting from that engineering unit, we can follow downstream relationships to determine which artifacts contain or depend on it, which products still use those artifacts, and which historical engineering states included the component.
The first question being answered is:
Where exactly is this component, and through which relationships did it enter which outputs?
Whether it needs to be removed immediately, whether products need to be rereleased, or whether some temporary mitigation is acceptable is a later risk decision.
The same engineering facts can also support provenance evidence, third-party license analysis, and artifact composition analysis.
The condition is still that the relationship graph must be sufficiently faithful to reality.
If the model misses a dynamic library quietly supplied by the host system, the supply chain view will miss it as well.
The accuracy of software supply chain information therefore returns to the original question: did the relationships that actually formed the artifact enter the engineering model, and can they be objectively verified?
From Trust in Source Code to Trust in Artifacts
For software used by telecom operators, governments, the military, aerospace, automotive systems, medical devices, and other environments with high security and audit requirements, delivering source code for review is usually not enough.
Even if reviewers have inspected the source and found no known vulnerability or malicious code, another question still has to be answered:
Was the software actually deployed in the product really built from the source that was reviewed?
If this relationship cannot be established, there is still a gap between source review and the final product.
And even if we can establish that the final artifact came from that source, the build process itself may introduce further uncertainty.
Which compiler was used? Which system libraries were linked? Which code generators and patches participated? Which environment variables were read during the build? Did it obtain undeclared files from the host machine? Did it temporarily use another tool, or fetch additional input from the network?
As long as some of these factors still depend on "whatever happened to be on the build machine at the time," it remains difficult to explain exactly how the final artifact was produced.
The relationship graph can extend source review all the way to the final artifact.
Source code is one engineering unit in the graph. Toolchains, system libraries, build configurations, and other conditions that can affect the result enter the graph as well. The output of each step becomes an explicit input to the next until the artifact actually delivered into the product is produced.
We can then answer more than:
This source code was reviewed.
We can also answer:
Which source produced this final artifact, which engineering units it passed through, what inputs each step used, and what output each step produced.
The build process itself therefore becomes auditable.
But the more important step is still independent reproduction.
If the relationship model can be executed again on another machine, at another time, or even in a substantially different isolated environment and produce the same specified artifact, then a third party no longer has to accept only the supplier's statement that "this binary was built from that source."
It can execute the relationship independently and compare the result.
The strength of verification can be increased further.
For ordinary development, reliable rebuilding in a controlled environment may already be enough. For higher-security environments, verification can be repeated at different times, under different isolation levels, or even in heterogeneous environments. Each additional independent check reduces the part of the system that still has to be trusted without being verified.
None of this means that building a relationship graph automatically makes software "absolutely trustworthy."
Source review can still miss problems. Toolchains can contain defects. Tests and verification have their own boundaries.
What changes is that several links that previously could only be connected through organizational reputation, process declarations, and human experience gradually become engineering facts that can be checked independently.
Instead of:
This source code is trustworthy. Please believe that the final software was built from it.
we move toward:
You can inspect the source, trace how it produced the final artifact, audit each intermediate step, and reproduce the same result in an independent environment.
Only then do we begin moving from trust in the source, to trust in the build process and artifact provenance, and toward end-to-end trustworthiness.
Real trust does not come from asking others to accept your conclusion.
It comes from letting them verify that conclusion independently.
Coding Agents
The first direct value of the relationship model for a coding agent is that it gives large projects a more structured starting point than global search.
In a small repository, starting from keywords in a requirement, searching for them, and then reading a few files along call relationships often works well enough.
In a large project, this approach can break down quickly.
Sourcegraph's analysis of 1,281 agent runs repeatedly observed failures such as wandering through the codebase, selecting the wrong files or symbols, making only partial changes, repeatedly searching and backtracking, and consuming effective context with large amounts of irrelevant code.
CodeScaleBench separately evaluates code modification in large repositories and the discovery and understanding of files, symbols, and dependencies. Being able to generate code and being able to find the complete engineering context are not the same capability.
The relationship model does not provide another form of full-text search.
It is more like a map.
If we ask an agent to find a particular building in an unfamiliar city, the traditional approach is somewhat like telling it the name of the city and asking it to search every street from the beginning.
A complete relationship structure gives it another route.
The task first maps to a product or product family, then to the relevant subsystem, and then follows the relationships down toward a platform, module, component, directory, file, and specific symbol.
Instead of loading every detail of the city into the agent at once, the search narrows progressively like this:
city → district → street → neighborhood → building
This is why both coarse-grained and fine-grained engineering units need to exist.
If the graph contains only files, the agent still has to understand the global structure starting from hundreds of thousands of files. If it contains only product-level nodes, it cannot continue down to a specific implementation.
Once the scope is small enough, semantic search, symbol references, call analysis, and code reading can take over.
The relationship model answers:
Which direction should I go first?
Code intelligence then answers:
Now that I am here, which implementation is the relevant one?
Only then does the agent decide how to modify it.
Once candidate locations have been found, the relationship model can also show structural differences between possible modification points before the implementation is written.
Changing a shared component may require very little code but affect a large downstream scope. Changing an adapter layer may require more code but keep propagation much smaller. Adding a new engineering unit creates a new long-term relationship of its own.
The system does not need to automatically choose the solution with the "smallest impact." Its job is to make these structural consequences visible before implementation, rather than discovering the propagation only after the code has been written.
Once engineering-unit boundaries are explicit, the same boundaries can also define an agent's operating scope.
The agent can read the source required by that unit and the outputs of its direct upstream units, use declared tools, and continue searching, modifying, building, and verifying within that scope.
If the task needs a new tool, additional network access, or a dependency not declared in the model, the agent should not complete the task by temporarily installing software, modifying PATH, or directly changing machine state.
It should propose a new engineering change.
The value of these boundaries is therefore not merely to "restrict the agent."
Precisely because the boundary is clear, the agent can be given more autonomy inside it. A small number of changes that genuinely require expanding the boundary can be separated from a much larger body of work that can be completed autonomously.
We have already seen this feedback loop in a recent real development task.
After an agent completed one version of a modification, the new engineering state showed that the change affected engineering units that should not have been affected. The propagation scope was substantially larger than the task was expected to touch.
The agent reconsidered its implementation based on that change, withdrew its previous commit, and took another implementation path.
By the time we noticed what had happened, it had already corrected the change.
What matters in this example is not that "AI found its own mistake."
It did not discover the problem by rereading the code it had just written and using the same interpretation again to decide whether that code was correct.
The relationship model gave it feedback from a source independent of the code text:
The engineering propagation caused by this modification does not match the scope that the task was expected to touch.
Implementation judgment at the code level and impact judgment at the engineering level were no longer coming from the same source of information.
Search helps AI find the code.
The relationship model can also tell it afterward:
What did you actually change?
The same boundaries can also change part of the way people collaborate.
At the level where engineering state is propagated, an upstream team does not need to broadcast all of its internal details indiscriminately to everyone. It only needs to provide a definite engineering state to the downstream units that actually depend on it.
This does not eliminate cross-team design work, product decisions, or the management of non-local quality attributes. But it can reduce a great deal of state propagation that previously depended on meetings, chat groups, and manual synchronization.
These Capabilities Did Not Begin with the AI Era
At this point, it is easy to interpret these practices as a new engineering method for the AI era.
They are not.
Engineering units have always existed. In the past, we may have called them products, components, targets, toolchains, artifacts, or environments.
The relationships have always existed too.
A build engineer knows which compilers affect which products. A release engineer knows which versions need reverification after a system library upgrade. A platform engineer knows which pipelines become invalid after a dependency changes. An engineer who has maintained a system for many years can often tell immediately which historical relationships a seemingly small modification will actually touch.
The old pipeline for the product with more than 900 modules was itself evidence that these relationships existed.
They had simply been compressed into a little over 200 manually defined tasks, and then distributed further between the small number of maintainers who understood the global pipeline and the local experience held by individual teams.
Much of this knowledge used to live in scripts, processes, documents, and people's experience.
It was always participating in software development, but it was not always expressed as one coherent set of engineering facts.
It is a little like dark matter in software engineering.
We can infer its existence from many effects: why a cache becomes invalid, why one change requires full testing, why a product can only be built in one particular environment, or why an experienced engineer takes one look at a seemingly harmless area and knows it must not be changed.
But the relationships themselves are often invisible.
When implementation moved more slowly, human experience still had time to fill the gaps between these partial models.
As AI increases implementation speed, implicit knowledge becomes harder to keep supplying through ad hoc human judgment. An agent does not naturally possess the project intuition that one engineer accumulated over ten years, and CI does not suddenly understand the real relationships of an entire product simply because code generation got faster.
What the AI era exposes, then, is not a new engineering requirement that did not exist before.
It makes it harder to continue ignoring engineering facts that were always there but were never fully expressed.
At the same time, AI significantly reduces the cost of recovering an initial engineering model from legacy scripts, documentation, and human experience.
Not More Process, but a Different Way of Producing Engineering Facts
This approach does not require every team to use the same tools, nor does it require all state to live in one system.
What needs to be unified is engineering identity, relationship semantics, and verifiable interfaces—not physical storage locations or tool implementations.
What we need to establish is a structure like this:
Software is composed of engineering units that can be independently identified and combined. Relationships between units express the causality by which outputs are produced. The engineering state of each unit is determined by the inputs that actually participate in the result. Formal outputs are bound to the corresponding relationship snapshot. Differences between old and new states form an engineering change. Actual execution then produces results and verification evidence, which in turn revise the engineering model.
Builds, caching, continuous integration, artifact management, software supply chain systems, and agents do not need to become one system as a result.
They only need to share the same set of engineering facts that can already be verified, and then use those facts according to their own semantics.
This is not about creating a more elaborate metadata world.
It does the opposite. It reduces the number of times the same reality has to be separately described in build scripts, CI, container images, artifact repositories, security systems, agents, and people's memory.
The first two articles discussed how a change is understood, verified, and handled.
What becomes concrete here is the foundation underneath that process:
First let the engineering system preserve an accurate account of why the software is what it is now. Then let every change be discovered, executed, verified, and revised along those same facts.
An implementation can be regenerated. A pipeline instance can be deleted at any time. A build environment can be recreated. A container image can be derived again.
What needs to remain over the long term are the engineering units that determine the result, the verifiable relationships between them, and the evidence through which reality keeps forcing us to revise those relationships.
About the author
Chai Feng is a Senior Technical Consultant and Team Coach at Odd-e, with over 20 years of experience in software and hardware R&D and engineering management. His work focuses on modern software development, software architecture, large-scale software engineering, build and CI/CD systems, and AI-assisted development.