Which parts matter in a procurement review?

The known weaknesses and the testing account, well ahead of the capability description. Weaknesses give you a checklist to verify yourself, and the testing section shows whether the vendor found anything uncomfortable and said so. Capability claims are the part you were going to evaluate anyway.

What if a vendor does not publish one?

It lengthens the review rather than ending it. You end up asking directly for what the document would have contained: known failure modes, what adversarial testing was done, and what was accepted as residual risk. Vendors that publish shorten their own sales cycle, which is increasingly why they do it.

Can we rely on it being current?

Only if it is dated and the product has not moved since, which is worth checking rather than assuming. These systems change on the vendor's schedule, and a card describing behaviour from two releases ago is a historical document. The date is the first thing to look at and frequently the least prominent thing on the page.

Glossary

System card

A system card is a vendor's published account of how a whole product behaves rather than of one model inside it, which makes it the document that answers the questions a buyer actually has.

In plain terms

A published account of how the whole product behaves: what it will do, where it is weak, what testing it was put through and what was found. It covers the finished thing rather than one piece inside it, which is why it answers the questions somebody buying it would actually ask.

Why it matters

Because buyers do not purchase components. What reaches a customer is a product with instructions, filters and refusals wrapped around a model, and its behaviour is a property of that whole arrangement, so a document describing only the model inside answers a question nobody in a procurement conversation asked.

How it works

It describes the assembled product, which is where behaviour actually lives. Filters, standing instructions, refusal rules and whatever surrounds the model all shape what a user experiences, so an account at this level can say what the thing does while an account of the model alone can only say what it might do. That difference is the whole reason the format exists.

The testing section is the part with real content, and it is the part to read closely. Adversarial testing before release, what was found, what was fixed and what was accepted as a residual risk together tell you how seriously the vendor takes this, and a card that describes testing without reporting anything uncomfortable is describing a process rather than a result.

Known weaknesses are stated deliberately, which is unusual enough to be worth using. Vendors publishing these tend to name specific failure modes, and those names are the most useful lines in the document because they let you test the same things yourself rather than discovering them in production. Treat the list as a starting checklist.

It is becoming a procurement artefact rather than a research one. Assurance platforms and frameworks now expect documentation of this kind for AI systems, so a vendor with a current card shortens a review and a vendor without one lengthens it, independent of how good the underlying product is.

What each document can honestly claim

The reason this distinction has become load-bearing is that behaviour turned out to be mostly a property of the wrapping. Two products built on the same model can refuse different things, format differently, decline in different tones and fail in different places, because the instructions, filters and routing around the model are doing an enormous amount of the work. A document describing the model alone is therefore accurate and beside the point for anybody deciding whether to deploy the product, which is exactly the frustration procurement teams have been reporting. The product-level account exists to close that gap, and the section that makes it worth reading is the one describing what testing was done and what it found. A vendor willing to publish something uncomfortable there is telling you more about how the product will behave in a year than any capability claim can.

Seen in the wild

A compliance platform whose framework coverage now includes AI management standards alongside the traditional security ones, so this documentation has a place to land.
Vanta
A trust platform that collects evidence continuously across frameworks and publishes an organisation's posture through a trust centre.
Drata
The hub for open machine learning, where the published account travels alongside the weights and the licence.
Hugging Face

Common misconceptions

People assume

It is the same thing as a model card with a different name.

In fact

One describes a component and the other a product. Behaviour comes from the assembly, the instructions and the filters as much as from the model, so a product-level account can state what users will experience while a component-level one can only describe what the part is capable of.

People assume

The testing section is boilerplate.

In fact

It is the section that separates a serious document from a marketing one. What was tested, what was found and what was accepted as residual are all specific claims, and a card describing a testing process while reporting no uncomfortable findings has told you about its own candour rather than about the product.

Telling them apart

System card vs Model card

System card

The product, as assembled and shipped.

Model card

One model, as built.

Buyers need the first and open-weight users need the second, which is why the two formats grew up in different parts of the field.

Questions

Which parts matter in a procurement review?: The known weaknesses and the testing account, well ahead of the capability description. Weaknesses give you a checklist to verify yourself, and the testing section shows whether the vendor found anything uncomfortable and said so. Capability claims are the part you were going to evaluate anyway.
What if a vendor does not publish one?: It lengthens the review rather than ending it. You end up asking directly for what the document would have contained: known failure modes, what adversarial testing was done, and what was accepted as residual risk. Vendors that publish shorten their own sales cycle, which is increasingly why they do it.
Can we rely on it being current?: Only if it is dated and the product has not moved since, which is worth checking rather than assuming. These systems change on the vendor's schedule, and a card describing behaviour from two releases ago is a historical document. The date is the first thing to look at and frequently the least prominent thing on the page.