Somewhere in your organization there is a JSON file named something like golden_set_v3_final.json, and it contains four hundred of your customers’ real conversations.
It was assembled with the best intentions — you can’t test an assistant on synthetic pleasantries; you need the real distribution, the real typos, the real “my husband passed and I need to close his account.” It lives in a repo, or a shared bucket, or a data-science workspace. It gets copied to laptops when someone debugs a regression. A contractor labeled half of it last quarter. Nobody has ever listed it in a security review, because it isn’t production. It’s test data.
Here is the uncomfortable reframe this article exists to make: in a regulated business, that file is a regulated asset. Not metaphorically — jurisdictionally. The Gramm-Leach-Bliley Act’s Safeguards Rule binds a nonbank financial company to a written information-security program covering customer information wherever it is — access controls, encryption, monitoring, disposal. The rule does not contain an exemption for data that is being used to test things. If consumer-report data is in the mix, the Fair Credit Reporting Act’s permissible-purpose and disposal duties ride along. The evaluation infrastructure is inside the boundary. It just isn’t inside anyone’s diagram.
The census of embarrassing places
This series’ rule is that governance starts with an inventory, so start by counting what evaluation actually accumulates. Four asset classes, each with a habit of escaping notice:
Golden datasets. Curated real interactions with expected outcomes — the blocking scenarios Part 1 demanded. By construction they concentrate the sensitive cases: disputes, hardship, bereavement, servicemember status. A golden set is a distillate of your most protected customer moments, version-controlled and widely copied, which is exactly backwards from how the rest of the program treats such data.
Traces. Every observability pipeline for an LLM system stores prompts and completions — meaning customer utterances, retrieved context, and everything the model said back. Traces are the most complete record of customer interaction your company holds, more candid than the CRM, and they default to whatever retention your tracing vendor shipped with.
Labeling queues. Humans — often contractors, sometimes offshore — read customer data to produce ground truth. That’s a data disclosure, recurring, at scale, through tooling procurement never reviewed as a data-sharing arrangement.
Red-team corpora. Attack transcripts and jailbreak logs, which necessarily include the sensitive data the attack tried to extract — plus a how-to library for attacking your own system. Doubly worth protecting; usually in a wiki.
If your AI inventory has one row per system, these assets belong on the rows — the AI bill of materials was always supposed to list the corpora. This article is the enforcement of that line item.
The manifest the build refuses to violate
The fix is not a policy about test data. Policies are wishes. The fix is the same move as everywhere else in this series: bind the rule to a control point that makes it physically true.
The control point for datasets is a manifest — a versioned, machine-read declaration that travels with every eval asset: what classes of data it contains (PII fields, account data, consumer-report elements), where it may live, who may read it, what it may be used for, and when it dies. The manifest is not documentation about the dataset; it is an input to the machinery that uses the dataset. The ingestion job reads it and refuses to load a corpus into an environment its manifest doesn’t permit. The eval runner reads it and refuses to write outputs containing manifest-flagged fields into unprotected result stores. CI reads it and fails the build on a dataset with no manifest at all — the paved road makes an unmanifested dataset more annoying to use than a manifested one, which is how the goat path loses.
Minimization then becomes an engineering dial instead of an argument. Some scenarios genuinely need raw real data; most don’t. The honest hierarchy: synthetic where behavior is what’s tested; masked or tokenized real data where distribution matters but identity doesn’t; raw real data only where fidelity is the point — carried under the strictest manifest, in the fewest copies, with named readers. The manifest encodes which tier each asset sits in, so “why does this dataset contain unmasked account numbers” has an answer in the file, not in someone’s memory.
Retention is a clock, and clocks need watchmen
Every manifest carries an expiry, because both statutes care about disposal — Safeguards expects it, FCRA demands it for report data. And an expiry nobody enforces is a belief, so the disposal is a scheduled job: sweep the manifests, delete or re-certify what’s past date, and — the part that matters — fail loudly on silence, dead-man style. The job that proves deletion happened is the disposal evidence; the job’s heartbeat is the proof the control is alive. Traces get the same treatment with a shorter clock: retention tiered by risk classification, aggressive redaction at the pipeline (secrets and PII scrubbed before storage, not at query time), and sampling raw only where a live investigation justifies it.
The labeling queue gets the disclosure treatment: labelers see the minimum field set for the labeling task (the manifest, again, defines the projection), access is individually granted and logged, and the vendor relationship is reviewed as what it is — a data-sharing arrangement under your Safeguards program, with the flow-down obligations that implies.
The examiner asks about test data now
For years, evaluation tooling flew under the audit horizon; examiners asked about production systems, and the eval stack was scenery. That era is ending for a simple reason: the eval stack is where the evidence comes from. Part 1 argued that your blocking evals are your compliance testing evidence. The moment you present eval results to an examiner as proof of control, the datasets behind those results become part of the record — their provenance, their handling, their disposal. You cannot simultaneously claim “our golden set proves the system never misstates terms” and “our golden set is nobody’s problem.”
Which closes the loop the way this series keeps closing it: build the control so the evidence is exhaust. The manifest is the data-classification evidence. The refusing build is the access-control evidence. The dead-man-checked disposal job is the retention evidence. When the question comes — and it is starting to come — the answer is a query, not a scramble.
The quiet takeaway is about status, not code. The eval suite spent this whole series as the enforcement layer’s proof — the thing that keeps every other control honest. Anything that important to the integrity of a regulated system was never going to stay “just test data.” Promote it formally: the eval stack is production, with production’s obligations. It only ever pretended to be tooling.
Last in this series: jurisdiction as a runtime parameter — why the same answer is legal in one state and a violation in the next, and why the fix is policy as data, not policy as prompt.