What a holdout corpus told us

The acceptance criteria for this project include a line that looks simple: the entity resolution should map at least 95% of vendor strings without manual review.

Measuring that turns out to be the entire difficulty.

The number that means nothing

Our first measurement replayed every hand-curated component edge through the parser and asked whether the rules reproduced it. It scored 100%.

That number is worthless as evidence and we say so on the repository README. We wrote both the rules and the strings they were tested against. Of course they match. It is a regression harness: useful for catching a rule someone broke last week, and no evidence at all about strings we have not seen.

The number that means something

So we built a corpus of realistic spec strings for systems deliberately not in the seed, froze it, and measured once.

61.7%.

Then we wrote rules against its failures, which improved it to 83.3%, and at that point the corpus was burned: it had become a tuning set. We renamed the file to make that explicit and wrote a new one, drawn from vendor and architecture families the rules were not authored from: 1990s vector and MPP systems, Hitachi, Huawei, Cerebras, Ampere, DEC Alpha, SGI NUMAlink.

45.6%, on the first and only measurement.

We could have reported 96.7% off the burned corpus. It would have been a real number computed from real data and it would have been meaningless, because we had written rules against exactly those failures.

What the holdout caught that we would not have

Two genuine bugs, both in rules that claimed a family and then failed inside it:

Infiniband NDR200 did not match. Our pattern ended with a word boundary after NDR, and a word boundary cannot sit between R and 2. Every string carrying a link-rate suffix (NDR200, NDR400, HDR100) silently failed to resolve as InfiniBand.

EPYC 7H12 did not match. Our pattern was EPYC\s+(\d{4}). AMD’s high-frequency parts embed a letter (7H12, 7F72), so the entire high-frequency line was invisible to the parser.

Neither would have surfaced from the strings we wrote ourselves, because we wrote them in the format our rules expected. That is what a holdout is for.

We fixed the second one after freezing, which moved the corpus from 45.6% to 47.1%, and we report both numbers for that reason. A later pass adding SKU-level rules for a seed expansion (written from the new systems’ spec strings, never from the holdout) moved it to 51.5%. That rise is genuine generalisation and the corpus stays valid.

How to read 51.5%

It is a floor rather than an expectation, and the holdout is deliberately adversarial. Most of the vendors in it are not in our graph at all. A real published edition is dominated by mainstream Lenovo, Dell, HPE, Intel and NVIDIA strings, where coverage would be substantially higher.

But we do not know how much higher, and we will not until we ingest one. So the honest statement is: the 95% target is not met on unseen input, and the figure on a real edition is unknown.

Why publish a failing number at all

Because the alternative is worse. This site’s entire proposition is that a number quoted from it can be checked. A project that publishes only its flattering measurements has no standing to ask anyone to trust its unflattering ones.

There is also a practical argument. The three fragments in the burned corpus that will never resolve are Custom, Custom Interconnect and 6D-Torus: the first two carry no vendor information at all, and the third describes both Fujitsu’s Tofu and Sugon’s TC8600 fabric. The parser refuses all three, and 6D-Torus carries a deliberately sub-floor rule so that it surfaces as a weak candidate in the review queue, and can never become an edge.

A parser that guessed on those would score higher. It would also be wrong roughly half the time, on rows nobody would ever check.