Agent Failure Cascade Prevention and Rollback

3.1.1. Failure Domain Taxonomy

Failures in agent ecosystems fall into the following categories:¶

Agent-Local Failure:

A failure confined to a single agent instance (e.g., out-of-memory, logic error). The blast radius is limited to the agent itself and its immediate callers.¶

Service Failure:

A failure affecting all instances of a particular agent service (e.g., model endpoint unavailable). The blast radius includes all agents that depend on the failing service.¶

Infrastructure Failure:

A failure in shared infrastructure (e.g., network partition, certificate authority unavailable). The blast radius may span multiple failure domains.¶

Semantic Failure:

An agent produces incorrect output without raising an error (e.g., misconfiguration, wrong decision). This is the hardest category to detect and may propagate silently through the DAG.¶

3.1.2. Propagation Vectors in Agent Ecosystems

Failures propagate through the following vectors:¶

1. Synchronous request chains: An agent blocks waiting for a failing downstream agent, causing its own callers to time out.¶

2. Shared state corruption: An agent writes incorrect data to a shared store, causing other agents reading that data to fail or make incorrect decisions.¶

3. Resource exhaustion: A failing agent consumes excessive resources (connections, memory, compute), starving healthy agents.¶

4. Retry amplification: Multiple agents retry requests to a failing agent simultaneously, overwhelming it further.¶

3.2.1. States

The circuit breaker has four states:¶

CLOSED (normal):

Requests flow through normally. The agent tracks the error rate over a sliding window (default: 60 seconds).¶

OPEN (failure detected):

When the error rate exceeds the configured threshold (default: 50% over the window), the breaker opens. All requests to the downstream agent are immediately rejected locally. The agent MUST emit an ECT with exec_act value "circuit_breaker_open".¶

HALF_OPEN (recovery probe):

After a cooldown period (default: 30 seconds), the breaker transitions to HALF_OPEN and allows a single probe request. If the probe succeeds, the breaker returns to CLOSED. If the probe fails, the breaker returns to OPEN with doubled cooldown (exponential backoff, maximum 300 seconds).¶

CLOSED (recovered):

When a probe succeeds in the HALF_OPEN state, the breaker returns to CLOSED and the agent MUST emit an ECT with exec_act value "circuit_breaker_close".¶

3.2.2. State Transition Rules

                error_rate > threshold
  CLOSED ────────────────────────────────► OPEN
    ▲                                        │
    │  probe succeeds                        │ cooldown expires
    │                                        ▼
    └──────────────────────────────── HALF_OPEN
                                       │
                            probe fails │
                                        ▼
                                      OPEN
                                (cooldown *= 2,
                                 max 300s)

The following rules govern state transitions:¶

1. CLOSED to OPEN: The error rate over the sliding window exceeds the configured threshold. The agent MUST emit a "circuit_breaker_open" ECT and reject all subsequent requests to the downstream agent.¶

2. OPEN to HALF_OPEN: The cooldown timer expires. The agent MUST allow exactly one probe request through.¶

3. HALF_OPEN to CLOSED: The probe request succeeds. The agent MUST emit a "circuit_breaker_close" ECT and resume normal operation. The error rate counters MUST be reset.¶

4. HALF_OPEN to OPEN: The probe request fails. The cooldown period MUST be doubled (up to a maximum of 300 seconds).¶

3.2.3. Circuit Breaker Registration and Discovery

Agents MUST expose circuit breaker state at a well-known endpoint:¶

GET /.well-known/cascade/circuits HTTP/1.1

Response:¶

{
  "circuits": [
    {
      "downstream_agent": "spiffe://example.com/agent/router-mgr",
      "state": "open",
      "error_rate": 0.75,
      "window_s": 60,
      "last_failure_ect": "550e8400-e29b-41d4-a716-446655440099",
      "cooldown_remaining_s": 22
    }
  ]
}

3.2.4. ECT Integration

Each circuit breaker state change MUST produce an ECT node:¶

{
  "jti": "cb-open-uuid",
  "exec_act": "circuit_breaker_open",
  "par": ["error-ect-uuid"],
  "ext": {
    "cascade.downstream_agent":
      "spiffe://example.com/agent/router-mgr",
    "cascade.error_rate": 0.75,
    "cascade.window_s": 60,
    "cascade.cooldown_s": 30
  }
}

{
  "jti": "cb-close-uuid",
  "exec_act": "circuit_breaker_close",
  "par": ["cb-open-uuid"],
  "ext": {
    "cascade.downstream_agent":
      "spiffe://example.com/agent/router-mgr",
    "cascade.total_cooldown_s": 30
  }
}

3.3.1. Blast Radius Containment Strategies

Agents MUST implement the following containment strategies:¶

1. Request rejection at the boundary: When a circuit breaker opens, the agent MUST return a structured error to its callers indicating that the downstream dependency is unavailable, rather than propagating the failure.¶

2. Timeout enforcement: Agents MUST enforce timeouts on all downstream requests. The timeout MUST be shorter than the caller's timeout to prevent timeout cascades.¶

3. Graceful degradation: When a non-critical downstream agent is unavailable, agents SHOULD continue operating with reduced functionality rather than failing entirely.¶

3.3.2. Domain Boundary Enforcement

Failure domains are defined by the workflow topology in the ECT DAG. Each workflow (identified by the wid claim) constitutes a failure domain. Cross-workflow failures MUST be escalated through the HITL mechanism [I-D.nennemann-agent-dag-hitl-safety] rather than propagating automatically.¶

Agents at domain boundaries MUST:¶

1. Validate all incoming requests against the circuit breaker state of their downstream dependencies before accepting work.¶

2. Emit a "circuit_breaker_open" ECT when rejecting work due to downstream unavailability.¶

3. Report domain health status via the circuits endpoint.¶

3.3.3. Bulkhead Patterns for Agent Pools

When multiple workflows share a common agent pool, the pool MUST implement bulkhead isolation:¶

1. Connection limits: Each workflow MUST have a maximum number of concurrent connections to the shared agent pool.¶

2. Queue isolation: Each workflow's requests MUST be queued independently, preventing one workflow's backlog from blocking others.¶

3. Resource quotas: Shared agent pools SHOULD enforce per-workflow resource quotas (CPU, memory, request rate).¶

3.4.1. Detection Signals

Agents MUST monitor the following signals for cascade detection:¶

Error Rate:

The ratio of failed requests to total requests over a sliding window. An error rate exceeding the circuit breaker threshold indicates a potential cascade.¶

Latency Spike:

A sudden increase in response latency (e.g., p99 latency exceeding 3x the baseline) indicates downstream congestion or failure. Agents SHOULD track latency baselines using exponentially weighted moving averages.¶

Resource Exhaustion:

Thread pool saturation, connection pool exhaustion, or memory pressure above configured thresholds indicates that a cascade is consuming resources.¶

3.4.2. Propagation Tracking via ECT DAG Analysis

Orchestrators SHOULD analyze the ECT DAG to detect cascading patterns:¶

1. Error clustering: Multiple "circuit_breaker_open" ECTs referencing the same downstream agent within a short window indicate a shared dependency failure.¶

2. Depth-first propagation: Errors propagating along par chains in the DAG indicate a synchronous cascade.¶

3. Breadth-first propagation: Multiple sibling nodes in the DAG failing concurrently indicate a shared infrastructure failure.¶

3.4.3. Alert Format and Escalation

When cascade detection identifies a propagating failure, the detecting agent MUST emit a cascade alert ECT:¶

{
  "exec_act": "cascade_detected",
  "ext": {
    "cascade.pattern": "depth_first",
    "cascade.affected_agents": 4,
    "cascade.root_cause_ect": "error-ect-uuid",
    "cascade.blast_radius": [
      "spiffe://example.com/agent/a",
      "spiffe://example.com/agent/b",
      "spiffe://example.com/agent/c"
    ]
  }
}

Cascade alerts with more than 3 affected agents SHOULD trigger HITL escalation per [I-D.nennemann-agent-dag-hitl-safety].¶

4.1.1. Walking the ECT DAG Backwards

The rollback process follows par references in reverse:¶

1. Identify the failing ECT node.¶

2. Find the checkpoint ECT associated with the failing action (referenced via par).¶

3. Follow par references backwards to identify all downstream actions that were caused by the checkpointed action.¶

4. Issue rollback requests to each affected agent in reverse topological order.¶

  Checkpoint A ──► Action A1 ──► Checkpoint B ──► Action B1
                                      │
                                      └──► Action B2

  Rollback order: B2, B1, B, A1, A (reverse topological)

4.1.2. Compensating Actions vs State Restoration

Rollback can be performed through two mechanisms:¶

State Restoration:

The agent restores its state from the checkpoint snapshot. This is the preferred mechanism when the checkpoint contains a complete state snapshot (verified via out_hash).¶

Compensating Action:

When state restoration is not possible (e.g., the action involved an external API call), the agent executes a compensating action that semantically reverses the original action. Compensating actions MUST be recorded as ECT nodes with exec_act value "compensate".¶

4.1.3. Rollback Scope

Rollback can be scoped to three levels:¶

Single Agent:

Only the specified agent's checkpoint is rolled back. No downstream propagation occurs.¶

Sub-DAG:

The checkpoint and all downstream checkpoints in the sub-DAG are rolled back. This is the default when cascade is true.¶

Full Workflow:

All checkpoints in the workflow are rolled back and the workflow is terminated. This requires Rollback Coordinator authorization.¶

4.2.1. Checkpoint Creation

An agent MUST create a checkpoint ECT before any consequential action. An action is consequential if it modifies external state (network configuration, database records, API calls with side effects).¶

A checkpoint is an ECT with:¶

• exec_act: "checkpoint"¶

• par: the ECT of the action being checkpointed¶

• out_hash: SHA-256 hash of the agent's state snapshot¶

{
  "jti": "ckpt-uuid",
  "exec_act": "checkpoint",
  "par": ["action-ect-uuid"],
  "out_hash": "sha256:...",
  "ext": {
    "cascade.reversible": true,
    "cascade.rollback_uri":
      "https://agent-b.example.com/.well-known/cascade/rollback",
    "cascade.target": "router-07.example.com",
    "cascade.description": "Update BGP peer configuration",
    "cascade.ttl": 86400
  }
}

The cascade.reversible field MUST be present. If false, the agent declares that this action cannot be automatically undone and rollback requests MUST be escalated to a human operator via the HITL mechanism [I-D.nennemann-agent-dag-hitl-safety].¶

4.2.2. Checkpoint Storage and Retrieval

Checkpoint ECTs MUST be stored for at least the duration specified by cascade.ttl. Agents MUST store checkpoints in durable storage that survives agent restarts.¶

Agents MUST expose a checkpoint retrieval endpoint:¶

GET /.well-known/cascade/checkpoints/{jti} HTTP/1.1

The response MUST include the checkpoint ECT and its verification status (whether out_hash matches the current stored state snapshot).¶

4.2.3. Checkpoint Verification

Before executing a rollback, the agent MUST verify the checkpoint integrity:¶

1. Retrieve the checkpoint ECT.¶

2. Verify the ECT signature chain (L2/L3).¶

3. Verify that the stored state snapshot matches out_hash.¶

4. Verify that the checkpoint has not expired (cascade.ttl).¶

If verification fails, the agent MUST reject the rollback request and emit an error ECT.¶

4.3.1. Rollback Coordinator Role

For rollbacks spanning multiple agents (sub-DAG or full workflow scope), a Rollback Coordinator MUST be designated. The coordinator is typically the orchestrator or the agent that initiated the workflow.¶

The coordinator is responsible for:¶

1. Computing the blast radius by traversing the ECT DAG.¶

2. Determining rollback order (reverse topological sort).¶

3. Issuing rollback requests to each affected agent.¶

4. Tracking rollback progress and handling failures.¶

5. Emitting the final rollback completion ECT.¶

4.3.2. Two-Phase Rollback Protocol

Distributed rollback follows a two-phase protocol:¶

Phase 1: Prepare¶

The coordinator sends a prepare request to each affected agent:¶

POST /.well-known/cascade/rollback/prepare HTTP/1.1
Content-Type: application/json
Execution-Context: <prepare-ect>

{
  "rollback_id": "urn:uuid:...",
  "checkpoint_id": "ckpt-uuid",
  "scope": "sub_dag"
}

Each agent MUST respond with either:¶

• "prepared": The agent has verified its checkpoint and is ready to roll back.¶

• "cannot_prepare": The agent cannot roll back (e.g., checkpoint expired, irreversible action).¶

Phase 2: Execute¶

If all agents respond "prepared", the coordinator sends execute requests in reverse topological order:¶

POST /.well-known/cascade/rollback HTTP/1.1
Content-Type: application/json
Execution-Context: <rollback-ect>

{
  "rollback_id": "urn:uuid:...",
  "checkpoint_id": "ckpt-uuid",
  "phase": "execute"
}

If any agent responds "cannot_prepare" in Phase 1, the coordinator MUST either:¶

• Proceed with partial rollback (if the unprepared agent is not on the critical path), or¶

• Abort the rollback and escalate to HITL.¶

4.3.3. Partial Rollback Handling

When a distributed rollback cannot be completed fully, the coordinator MUST:¶

1. Roll back all agents that responded "prepared".¶

2. Record the partial rollback result in the ECT DAG.¶

3. Emit an ECT with exec_act value "rollback_complete" and cascade.status set to "partial".¶

4. Include the list of agents that could not be rolled back in the cascade.failed_agents extension claim.¶

4.3.4. Conflict Resolution During Concurrent Rollbacks

When multiple rollback requests target overlapping portions of the ECT DAG:¶

1. The rollback with the broader scope takes precedence (full workflow > sub-DAG > single agent).¶

2. If scopes are equal, the earlier rollback request (by timestamp) takes precedence.¶

3. The losing rollback request MUST be rejected with an error indicating the conflicting rollback ID.¶

Agents MUST implement idempotent rollback: receiving the same rollback_id twice MUST return the same result without re-executing the rollback.¶

4.4.1. ECT Nodes for Rollback Actions

Each rollback action MUST produce ECT nodes for audit:¶

Rollback Start:

exec_act: "rollback_start", par references the error ECT that triggered the rollback.¶

{
  "jti": "rb-start-uuid",
  "exec_act": "rollback_start",
  "par": ["error-ect-uuid"],
  "ext": {
    "cascade.rollback_id": "urn:uuid:...",
    "cascade.checkpoint_id": "ckpt-uuid",
    "cascade.scope": "sub_dag",
    "cascade.reason": "Upstream cascading failure"
  }
}

Rollback Complete:

exec_act: "rollback_complete", par references the rollback start ECT.¶

{
  "jti": "rb-complete-uuid",
  "exec_act": "rollback_complete",
  "par": ["rb-start-uuid"],
  "out_hash": "sha256:...",
  "ext": {
    "cascade.rollback_id": "urn:uuid:...",
    "cascade.status": "completed",
    "cascade.state_hash_before": "sha256:...",
    "cascade.state_hash_after": "sha256:...",
    "cascade.cascaded": [
      {
        "agent": "spiffe://example.com/agent/monitor",
        "status": "completed"
      },
      {
        "agent": "spiffe://example.com/agent/classify",
        "status": "escalated"
      }
    ]
  }
}

4.4.2. Rollback Audit Trail

The complete rollback audit trail is captured in the ECT DAG:¶

  error ECT
     │
     ▼
  rollback_start ECT
     │
     ├──► agent-A rollback_complete ECT
     │
     ├──► agent-B rollback_complete ECT
     │
     └──► agent-C compensate ECT

Status values for individual agent rollbacks: completed, partial, escalated, failed.¶