SEMPITE Research · Methods · September 2026 · By Jacob Gerrish
We Have Published Numbers That Were Wrong. Here Is How We Find Them.
Every SEMPITE study ships the raw records and the code that produced them, and every published figure is re-derived from those records by a script that refuses to build the package if a single number fails to match. This page exists because that process has caught four real errors, one of them months after publication. The corrections are more useful to you than a claim that we do not make mistakes.
The rule: if the data cannot ship, the study does not
A study becomes a published dataset when we can hand a stranger the per-unit records and the script that produced them. Some SEMPITE studies fail that test and are not deposited. That is deliberate, and it is why four datasets exist rather than twenty-three.
Every package is assembled by a build script that recomputes each headline from the raw rows and compares it against what the write-up claims. If any figure fails to match, the script exits and no package is produced. Verification is a gate, not a report that someone reads afterwards.
Why that matters more than it sounds: the usual failure is not a fabricated number. It is a number that was true of an earlier version of the data, or true under a definition that quietly changed. A gate catches those. A human re-reading their own work does not.
Correction one: the rule that thought a city was a business
In July and August 2026 we published that 32% and then 43% of the businesses Google ranks first go unnamed by ChatGPT. Both figures were wrong.
The rule deciding whether an AI answer mentioned a business counted any distinctive-looking word from the business's Google listing title. Those titles are keyword-stuffed, so the words included the city and the trade. Asked "I need a plumber in Miami", almost every answer contains the words plumber and Miami, and the rule read that as a match.
We rewrote the matcher in September, re-scored the answers we had stored, and republished. The August figure is 96.0% invisible, not 43%. The new rule errs toward counting a mention wherever a call is ambiguous, so the published invisibility rate can understate the problem but not inflate it.
What made the fix possible: we had stored every AI answer in full, not just the verdicts. Had we kept only "mentioned: true", the error would have been permanent and undetectable. Storing the raw response is what lets a study be corrected instead of retracted.
Correction two: arithmetically perfect, built on almost nothing
A verifier that checks arithmetic is not a check on data quality. During the Shopify tagging study, a run passed every arithmetic test we had while resting on 47 of 1,075 stores. Every percentage was correctly computed. The collection underneath had silently failed.
Collection health is now gated separately from analytical correctness, because the two fail independently and only one of them is visible in the output. A run must prove it collected what it claims before its numbers are computed at all.
Correction three: finding wishlist code is not finding a wishlist
Detecting wishlist code in a page is easy. Confirming that a shopper can actually save something is not. Across 1,028 Shopify storefronts, 46.0% carry wishlist code and 28.1% have a save control we could confirm works.
Anyone measuring this by pattern-matching page source alone overstates working wishlists by roughly 18 percentage points. We now run a second verification pass that re-fetches each candidate and records the evidence behind every verdict, and that evidence ships in the dataset so any individual call can be disputed.
Correction four, found while writing this page
Preparing the AI Visibility Index for deposit, we found that the scoring code was computing published rates over answers that had been stored truncated. All 150 August Google AI Mode observations were affected. The package README said those answers were excluded; the summary file published a rate for them anyway.
A truncated answer can only undercount mentions, so that figure was an upper bound on invisibility rather than a measurement of it. The block now reports the exclusion and its reason instead of a number. Every other figure was unaffected, which we established with a before-and-after diff showing exactly one value changed.
What ships with every study
Four things, in every deposit, with no exceptions:
The limitations sections name the things that would embarrass us: sampling frames that do not generalise, confidence intervals that are optimistic because observations are clustered, and at least one figure in one dataset that cannot be reproduced from the files provided, stated plainly rather than omitted.
Check it yourself
The four datasets below carry permanent DOIs, open data and open code under CC BY 4.0 and MIT. One of them has a runnable notebook that recomputes every headline figure from the shipped files in about twenty seconds, on someone else's hardware, with no input from us.
- The llms.txt corpus, two observations six weeks apart
- The SEMPITE AI Visibility Index, 50 local markets, quarterly
- Product tagging and saved intent, 1,028 Shopify storefronts and 700 Steam titles
- Who Google's AI cites for book discovery, 95 citations
What we do not claim
That our numbers are right. That is not something a methods page can establish, and a vendor asserting it should not be believed. What we claim is narrower and checkable: the data behind every published figure is available, the code that produced it is available, the limitations are written down, and when we get something wrong we say so in public with the corrected data attached.
Three of the four corrections on this page were found by our own process before anyone else saw them. One was found after publication and republished. We expect there to be a fifth.
Frequently asked questions
Why publish your own mistakes?
Because the alternative is asking you to take a vendor's numbers on trust. Every marketing agency claims its research is rigorous. Almost none of them publish the data, and none of them tell you about the time they got a figure wrong by 53 percentage points. A correction with the corrected data attached is evidence. A claim of rigour is not.
How do I check a SEMPITE figure myself?
Open the dataset behind the study, which is linked from every research page, and recompute the number from the raw rows. The collection and analysis code ships with it. For the Shopify tagging study there is a notebook on Kaggle that does this in about twenty seconds without you installing anything.
Do you correct figures after publication?
Yes, in public. The July and August 2026 AI visibility figures were republished in September after we found the matching rule was wrong. The research page carries a dated correction notice, the dataset documents what changed and why the earlier figure was wrong, and the old number is not quietly edited out.
Why are only four studies deposited when you publish more than twenty?
A study becomes a deposit only when its raw records and collection code can ship with it. Some of our published studies cannot meet that test, usually because the underlying data was not retained in a reusable form. Those stay as write-ups and are not presented as open datasets. We would rather have four studies you can check than twenty you have to trust.
What does it mean that your confidence intervals are optimistic?
In the AI Visibility Index, three businesses in a market are scored against the same AI answer, so those three observations are not independent of each other. The Wilson intervals we publish treat them as though they are, which makes the intervals narrower than they should be. A cluster-robust interval would be wider. We state this in the limitations rather than leaving you to discover it.