Assurance

The pivot

The six domains preceding this one describe what an organization says about its AI. There is an inventory, an owner, a benefit case, a risk classification, a data lineage, and an obligation map. Each is an assertion.

This domain tests whether the claims made in the earlier domains can be relied on. Without independent testing, the board has no sound basis for distinguishing a supported claim from one that merely sounds complete.

This is also the domain where the language in use is least reliable. The word assurance is applied to activities that are not assurance at all: testing performed by the build team, monitoring dashboards, vendor certifications, and user acceptance. Each of those has a purpose. None establishes that a stated claim is true, and describing them as assurance produces a board that believes its systems have been examined when they have not.

What independence requires

Independence is a property of the relationship, not of the job title.

A function is independent of a system where it did not build it, does not operate it, did not select it, and where its own objectives are unaffected by the finding. The third condition is the one most often missed. A risk function reporting to the executive accountable for delivering the system may be organizationally separate and still unable to produce an inconvenient conclusion.

The practical test is historical rather than structural. A challenge function with genuine authority will usually have used it. Where it has never delayed, blocked, or materially changed a deployment, either every deployment was sound or the authority is nominal, and the question is worth asking.

Testing against the right conditions

A model evaluated on the data used to develop it has been tested on the conditions it was constructed for. Production is where it encounters the conditions nobody anticipated.

ASR-3 therefore requires validation before release on data representative of the population the system will serve, including the groups it materially affects, under conditions as close to production as can be arranged. That is more demanding than it appears: the data must be available for testing, lawful to use for that purpose under Foundations, and representative of the population the system will actually encounter rather than the one it was designed for. It also requires a dated plan for validating the system against actual outcomes once it is live, which RES-2 carries out, and which shows whether the two populations differ.

Where either is not possible, the limitation is itself a finding. A system that cannot be validated under representative conditions carries an exposure that belongs in the register, not a gap that is left unstated.

Vendor assurance

A substantial proportion of AI capability arrives inside products the organization did not build and cannot inspect. Assurance in those cases is obtained rather than performed, and what is obtained is frequently misunderstood.

Security and operating certifications address the environment in which a system runs: access control, change management, availability, incident response. They are valuable and they are not model assurance. A certification of this kind establishes that the vendor operates a controlled environment. It establishes nothing about whether the model produces correct output for the organization's population, on the organization's data, for the organization's purpose.

ASR-7 therefore requires the organization to assess what a vendor's evidence covers and to record the gap. The gap is not a criticism of the vendor. It is the part of the assurance obligation that has not transferred, and which consequently remains with the organization.