A Few Per Cent of the Web, Held Properly
We do not try to hold the whole web. We hold the small share of it that originates what the rest repeats: the people and organisations who publish first, including the long tail of working practitioners that a popularity-ranked index buries. It is a deliberate slice, not a sample, which is why it still answers the kind of question you would otherwise take to a search engine.
The size of that slice is the point. A full-web crawler has to decide how often to revisit a billion pages, and the arithmetic forces a revisit interval measured in days. Our surface is orders of magnitude smaller and we know whose it is, so the accounts that matter can be watched continuously instead of swept occasionally. Less to recrawl is not a cost saving we pass on; it is the reason the index can be current at all. It is also why this runs in Europe on our own hardware: an index that tried to hold everything would make EU-only infrastructure prohibitive.
Holding that slice properly is the part that took years. We have had continuous, compliant API access to the major platforms for long enough to have real historical depth, and every year compounds: a profile enriched over years carries signal quality that one assembled last month cannot match, and each new source creates fresh cross-references across every profile already collected. Recreating it today means navigating API restrictions, policy changes and historical gaps that did not exist when we started, whatever the compute budget.