As my teammates and I continue building a large multi-agent system and adding new participants, a recurring question arises: should we allow this agent to operate in production?
Consider an agent capable of changing the status of work in a mission-critical tax system and another capable of messaging leadership when a task is complete. Each capability may seem reasonable alone. Combined, the system could incorrectly update the system, interpret that update as successful, and immediately communicate incorrect information.
Before production, we need to ask what actions the agent can take, how reversible they are, what downstream agents or tools it can trigger, what evidence is required before it proceeds, and when a human should be involved.
Autonomy needs evaluation across the full chain of actions, dependencies, and consequences. The decision depends on the specific agent, tools, workflow, and operating conditions.