failure-recovery

Failure Recovery

Agents fail. Networks time out, models hallucinate, tools error, and edge cases surprise. Failure recovery design determines whether a failure becomes a dead end or a graceful detour.

Failure Types in Multi-Agent Systems

• Agent failure: A single agent crashes, times out, or produces invalid output
• Handoff failure: Context is lost or corrupted during transfer between agents
• Coordination failure: Agents conflict, deadlock, or produce inconsistent results
• Resource failure: External tools, APIs, or data sources are unavailable
• Cascading failure: One agent's failure causes downstream agents to fail

Recovery Strategies

• Retry: Try the same operation again. Works for transient errors (network timeouts, rate limits). Set a retry limit to avoid infinite loops.
• Fallback: Switch to an alternative approach. A different agent, a simpler method, or a cached result.
• Escalation: Pass the problem to a more capable agent or to a human. Used when the failure is beyond the current agent's ability to resolve.
• Graceful degradation: Deliver a partial result rather than nothing. Tell the user what worked and what didn't.
• Compensation: Undo the effects of a partially completed workflow before retrying or escalating.

Designing Recovery Paths

For each point in the workflow where failure is possible:

• What could fail? List the failure modes
• What's the first recovery strategy? Usually retry for transient errors
• What's the fallback? If retry fails, what's the alternative?
• When do you escalate? After how many retries or what type of failure?
• What does the user see? Transparent about the failure or silently recovered?
• What's the worst case? If all recovery fails, what's the graceful degradation?

User Experience of Failures

• Invisible recovery: The system retries or falls back without the user noticing. Best for minor, quickly resolved failures.
• Transparent recovery: The system tells the user something went wrong and how it's handling it. "This is taking longer than usual — trying an alternative approach."
• Participatory recovery: The system asks the user to help. "I couldn't access your calendar. Can you check the connection?"
• Honest failure: The system tells the user it can't complete the task and explains why. Offers alternatives.

Design Artefacts

• Failure mode inventory per agent and per handoff
• Recovery strategy specifications (retry limits, fallback paths, escalation triggers)
• Cascading failure analysis
• User experience specifications for each failure scenario
• Recovery testing protocols