Resilience

Failing while appearing healthy

Conventional systems fail visibly. They stop responding, return errors, or exceed a latency threshold, and every one of those states is detected by monitoring that most organizations already operate.

AI systems fail differently. A model whose inputs have shifted away from the population it was built for continues to respond, within latency, without error, at normal volume. Every availability measure reports health. The answers are simply worse, and they become worse gradually.

This is why RES-2 requires monitoring of output quality alongside availability. Without quality monitoring, an AI failure can continue unnoticed while the system appears healthy. People affected may include anyone processed since the failure began, rather than only those processed during a visible outage.

The question about turning it off

The most useful question in this domain is not whether an incident plan exists. It is whether anyone can stop the system, and whether they ever have.

The answer given in the meeting is almost always that the system would be turned off. The answer that emerges on examination is frequently different. Suspension may require a vendor to act, a configuration change that follows the normal release process, a change advisory board that meets weekly, or a decision from a committee whose members are unavailable outside business hours. Each is reasonable as a control over routine change. Each is an obstacle when the system is actively producing wrong outcomes.

RES-3 and RES-4 therefore separate two things that are usually conflated. Authority is the question of who may decide. Mechanism is the question of what happens after they do, and how long it takes. An organization can have a clear answer on authority and discover during an incident that the mechanism takes four days.

Fallback determines whether suspension is real

A control that the organization cannot afford to use is not available to it.

Where an AI system was deployed and the manual process it replaced was dismantled, suspension means the business process stops. The decision to suspend then becomes a decision to halt operations, which is a materially different decision from the one the runbook describes, and which will be resisted for as long as the failure remains arguable.

RES-6 exists to surface this before deployment rather than during an incident. The question is not whether a fallback procedure is documented. It is whether the capacity to operate it still exists, which is a funding decision taken at the point of deployment and rarely revisited.

Remediating harm

RES-7 is the obligation that most commonly cannot be met, and the reason is architectural rather than procedural.

Remediating the people affected by a failure requires knowing who they were, what the system decided about them, and on what basis. That requires the system to have retained its decisions and their inputs. Many do not, because retaining them was not a requirement at build time and because the retention has a privacy cost of its own under Foundations.

The consequence is an organization that can detect a failure, stop the system, and correct the fault, but cannot identify the people who were affected in the interim. From a regulatory and reputational standpoint, that is the part that matters, and it is determined months earlier by a design decision nobody escalated.