Method

Every experiment follows the same protocol, so results can be compared and reproduced on any machine.

What is measured

One main result, three supporting measures.

Main result
Check time
Time spent by the type-checker, the main result. Reported as the median, with the fastest and slowest runs.
Supporting
Total time
Parse, bind and check together.
Supporting
Memory
Peak resident memory (RSS) of the largest compiler process, from /usr/bin/time. The compiler's own "Memory used" line is a different metric and is always labeled separately.
Supporting
Files, types, instantiations
The compiler's counters. They must be identical across runs of the same condition, which proves every run checked the same program.

Protocol

A number is only worth publishing if anyone can get it again.

Rule 1
Pin everything
TypeScript versions are pinned in package.json, external projects are cloned at a pinned commit, and the machine context is saved with every result.
Rule 2
Measure the same way
Each condition runs tsc -p <project> --extendedDiagnostics several times. In protocol mode, one warm-up run per condition is discarded, conditions are interleaved (A B C A B C), and the machine idles 2 minutes first.
Rule 3
Change one thing at a time
A fix is minimal and keeps the same type safety.
Rule 4
Prove type safety
When a fix changes types, a safety.ts file lists usages that must still fail to compile. The experiment stops if it fails.
Rule 5
Publish the verdict, whatever it is
A hypothesis can be confirmed, rejected or stay open.

Timings vary between machines: compare conditions measured on the same machine.
Above 5% coefficient of variation, the cause is investigated and 5 more runs are added. If it stays above 5%, the limits say so.
Gains are ratios of medians, shown with a conservative range: fastest run of one condition over slowest of the other, both ways.
TypeScript 6 and 7 may not count types and instantiations the same way: do not compare these counters across compilers.
Close other heavy applications while measuring.