Coverage and quality report

Headline figures and what they mean

One snapshot, built 27 Oct 2026, used on every page of the site and in the dataset.

Universe and survivorship

The history covers S&P 500 companies from 2016. Current members are read for the whole period. Companies that left the index since then, through acquisition, merger, bankruptcy or falling market value, are included too, and read for the years they were members. Without them, any back-test would favour survivors.

• Core 100 (the S&P 100) is read first and in full: 43 of 43 quarters done, completed 2026-10-10.
• Rest of the S&P 500 is read company by company rather than quarter by quarter: 400 of 400 current members have had their 8-K press releases from 2016 read.
• Former members: 214 matched to their SEC filer records, 214 read so far.
• Every record carries sp500_member_at_announcement and former_sp500_member, so a study can use the index exactly as it stood on each date.
• 91,408 historical SEC filings read so far.

Accuracy

First accuracy check, September 2026. A random sample of 100 published records (53 open, 47 with an outcome) was re-checked by a second, independent AI model that read each SEC source document without seeing the recorded answer. Every one of the 51 disagreements, and 10 randomly chosen agreements, was then decided against the sources: 37 by the founder and 24 by a further AI review delegated by the founder. Figures below count the final decisions; one record left undecided is excluded.

• Real commitments: 80 of 99 (80.8%) were dated, checkable commitments as defined in the methodology.
• Deadlines: 77 of 80 (96.2%) matched the time the company stated.
• Outcomes: 32 of 36 (88.9%) were right on the evidence.

Quotes: 95 of 100 appear word for word in the SEC document.

What changed as a result. The 19 records found not to be commitments (regulator decisions, events already completed, financial targets, forecast assumptions, conference attendance and one programme belonging to another company) were withdrawn and logged, and 4 outcomes and 3 deadlines were corrected. The extraction rules were tightened for each kind of error, deadlines now follow each company's fiscal calendar and the date of the statement, joint filings are recorded under the listed parent, and every published record is being re-screened against the tightened rules.

How to read these figures. The samples are small, so each rate has a wide margin: the 95% intervals are roughly 72%–87% for real commitments, 90%–99% for deadlines and 75%–96% for outcomes. The decisions were made by the founder and by further AI review, not by a reviewer independent of the pipeline, and the pipeline has changed since (a tie-break and a third check on negative findings were added in methodology 1.3). These results describe the September 2026 version, not the current one.

Next audit, before any Kept Score is published. A frozen snapshot; a stratified random sample of 300 records labelled by a reviewer independent of extraction, oversampling negative findings; two reviewers on every disputed record; a separate read of random source filings to measure commitments the pipeline missed. Results will be published separately, never as one blanket accuracy figure, for five measures: valid commitments extracted, commitments missed, restatements matched to the right commitment, deadlines read correctly, and outcomes decided correctly, each with error categories and confidence intervals. The sample deliberately includes filings that discuss several projects at once (for example different aircraft programmes in one release) and reports that state several targets (water and fleet electrification side by side), where matching errors are most likely. Agreement between AI checks is reported as consistency, not as independent confirmation. Our internal release threshold is at least 98% outcome correctness in the evaluation set, with every featured negative finding approved by a person; it is a threshold, not a certification.

The sample is drawn afresh for each check and every error found is corrected in the record.

Point-in-time rules

• announced_date is the SEC filing date of the press release, the first day the commitment was public.
• evidence_date is the filing date of the decisive evidence. Use it, not resolved_date, to avoid look-ahead, and exclude records where it is blank (knowledge_date_known = false).
• A company may have made a fact public in a press release before the SEC filing date; the filing date is the conservative, verifiable knowledge date we use.
• Restatements are added as new versions. The original commitment is never overwritten.
• Historical outcomes are established only from SEC filings made after the deadline, never from today's web pages.
• Corrections are logged with a date and reason on the corrections page.

Known limitations

• Only commitments made in SEC filings are covered: 8-K press releases from U.S. companies and, since October 2026, 6-K press releases from non-U.S. companies listed in the U.S. Earnings calls, investor days and filings made only outside the U.S. are not read yet.
• Only commitments with a stated date or period are recorded. Undated ambitions are out of scope by design.
• Extraction and outcome checks are done by AI models under fixed rules, with quote checks and human review of flagged cases. Errors remain possible. Agreement between models measures consistency, not truth: two models reading the same source can share a mistake. The accuracy figures above measure errors on a small sample.
• Rates describe the commitments captured and resolved, not everything a company said: undated ambitions are excluded and outcomes with no public record are left out.
• Event type and materiality are indicative labels from published word rules (v0.1). They are not used in the Kept Score.
• The Kept Score is published only for companies with at least 15 counted outcomes in the last 36 months.

Field specification KCS-0.1. Methodology · Dataset and pilots