{"measured_at":1792303200,"stale":false,"rows":108679001950,"sources":[{"key":"reddit_comments","label":"Reddit comments","family":"Social","relation":"reddit.comments","within":null,"description":"Near-complete Reddit comment archive, 2005 to present: comments removed before capture are absent, archive months land with lag, and the live tail captures a fraction of current volume, so a zero here is absence in indexed data, not at the source. A reading that still carries a body or an author name another reading shows Reddit has deleted or removed is not served here. A comment id can arrive more than once, so count comments with uniqExact(id) and keep the newest row per id with ORDER BY state_observed_at DESC, retrieved_on DESC LIMIT 1 BY id. Temporal coverage is published on this schema entry: extent (computed from indexed data, with computed_at) and known_holes (declared gaps)."},{"key":"reddit_posts","label":"Reddit posts","family":"Social","relation":"reddit.posts","within":null,"description":"Near-complete Reddit submission (post) archive, 2005 to present: title, selftext, score, comment count, and outbound domain/url per post; a post's comment tree is in reddit.comments via link_id = concat('t3_', id). Posts removed before capture are absent, archive months land with lag, and the live tail captures a fraction of current volume, so a zero here is absence in indexed data, not at the source. A reading that still carries a text, a title or an author name another reading shows Reddit has deleted or removed is not served here. Temporal coverage is published on this schema entry: extent (computed from indexed data, with computed_at) and known_holes (declared gaps)."},{"key":"bluesky","label":"Bluesky","family":"Social","relation":"bluesky.posts","within":null,"description":"Temporal coverage is published on this schema entry: extent (computed from indexed data, with computed_at) and known_holes (declared gaps)."},{"key":"mastodon","label":"Mastodon","family":"Social","relation":"mastodon.posts","within":null,"description":"Mastodon statuses, one row per observation of a status until ReplacingMergeTree(engagement) ORDER BY (created_day, uri) merges them (the caveat gives the dedupe)."},{"key":"hackernews","label":"Hacker News","family":"Social","relation":"hackernews.items","within":null,"description":"Source-native Hacker News items. upvotes and comment_count follow HN since 2026-09-12 (changed items are re-loaded newest-wins; the view reads FINAL). A story dated from 2025-12-01 up to that day has its hackernews.story_scores observation as of 2026-09-17 (each story whose count read lower took the observation as a newer version that day); a story with no score record keeps its count at first sight, and rows loaded through 2025-11 have the count as of their load. hackernews.story_scores is the score record since 2025-12-01 (latest per hn_id via argMax over observed_hour) — rank recent stories and read a trajectory through that relation. Temporal coverage is published on this schema entry: extent (computed from indexed data, with computed_at) and known_holes (declared gaps)."},{"key":"stackexchange","label":"Stack Exchange","family":"Social","relation":"stackexchange.posts","within":null,"description":"Stack Exchange network: questions and answers across every indexed site. Comments are not served: comment_count is the source's count of a post's comments, and their bodies are behind the post's uri, not in any relation here. Temporal coverage is published on this schema entry, never assumed from this prose: extent (computed from indexed data, with computed_at) and known_holes (declared gaps)."},{"key":"forums","label":"Forums","family":"Social","relation":"forums.posts","within":null,"description":"Primary forum corpus: source-native posts and comments across thousands of forum sites (LessWrong, DataSecretsLox, DEV, EA Forum, 4chan boards, crypto governance, plus a long tail); the live row and site counts are the census (scry.io/sources, /v1/stats/wire). site_key is the community identity, spelled as the data spells it (lesswrong, eaforum, devto, datasecretslox, the 4chan boards as 4chan_g, 4chan_pol, ...): the source column identifies only the curated sources, and a large share of rows have source = 'manual', so enumerate site_key — not source alone — before scoping a community question. Archived base refreshed in place by live source-native forum tails."},{"key":"lesswrong","label":"LessWrong · in Forums","family":"Social","relation":"forums.posts","within":"forums","description":"Primary forum corpus: source-native posts and comments across thousands of forum sites (LessWrong, DataSecretsLox, DEV, EA Forum, 4chan boards, crypto governance, plus a long tail); the live row and site counts are the census (scry.io/sources, /v1/stats/wire). site_key is the community identity, spelled as the data spells it (lesswrong, eaforum, devto, datasecretslox, the 4chan boards as 4chan_g, 4chan_pol, ...): the source column identifies only the curated sources, and a large share of rows have source = 'manual', so enumerate site_key — not source alone — before scoping a community question. Archived base refreshed in place by live source-native forum tails."},{"key":"eaforum","label":"EA Forum · in Forums","family":"Social","relation":"forums.posts","within":"forums","description":"Primary forum corpus: source-native posts and comments across thousands of forum sites (LessWrong, DataSecretsLox, DEV, EA Forum, 4chan boards, crypto governance, plus a long tail); the live row and site counts are the census (scry.io/sources, /v1/stats/wire). site_key is the community identity, spelled as the data spells it (lesswrong, eaforum, devto, datasecretslox, the 4chan boards as 4chan_g, 4chan_pol, ...): the source column identifies only the curated sources, and a large share of rows have source = 'manual', so enumerate site_key — not source alone — before scoping a community question. Archived base refreshed in place by live source-native forum tails."},{"key":"substack","label":"Newsletter posts","family":"Writing & web","relation":"substack.posts","within":null,"description":"Substack posts across publications on *.substack.com and custom domains alike: one row per (publication_host, post_id) with title, subtitle, byline, published time, free-or-paid visibility, the body as plain text where read (text_quality_state says how far), Substack's comment and reaction counts, and metadata with the post's outlinks. Comments are substack.comments; publications are substack.publications."},{"key":"substack_comments","label":"Newsletter comments","family":"Writing & web","relation":"substack.comments","within":null,"description":"Substack comments beneath the posts of substack.posts: one row per (publication_host, post_id, comment_id) with the commenter's display name, handle and user id, the reply parent, Substack's status and moderation flags, like and reply counts, and the text. Threads are present for a minority of posts; comment_count on the post is Substack's own count."},{"key":"substack_publications","label":"Newsletter publications","family":"Writing & web","relation":"substack.publications","within":null,"description":"Substack publications: one row per publication_host with the root URL, Substack's publication id, the custom domain or subdomain where known, the title where known, and first and last days seen. The join key for substack.posts and substack.comments."},{"key":"mailing_lists","label":"Mailing lists","family":"Writing & web","relation":"mailing_lists.messages","within":null,"description":"Public mailing-list and Usenet archive messages (extropians, SL4, and every list mailing_lists.catalog names); payload is the message text, threaded by parent/root keys."},{"key":"web_crawl","label":"Web crawl pages","family":"Writing & web","relation":"crawl.pages","within":null,"description":"Text extractions of observed web pages across every observed host — the live open-web corpus. (The /search crawled_url family still reads the earlier static archive in internet.documents; this relation is the fresh one.)"},{"key":"commoncrawl_pages","label":"Common Crawl pages","family":"Writing & web","relation":"commoncrawl.pages","within":null,"description":"Whole-page extracted text — boilerplate (nav, cookie banners, footers) included; the same URL recurs across epochs by design (one row per capture). The recall layer: for indexed clean text grep commoncrawl.distillate first and hydrate here."},{"key":"commoncrawl_index","label":"Common Crawl index","family":"Writing & web","relation":"commoncrawl.index","within":null,"description":"The Common Crawl per-capture URL index (CDX): one row per WARC capture — full URL decomposition, fetch status, MIME, CLD2 languages, content SHA-1 digest, and the exact WARC (filename, offset, length) triple that re-derives the raw record from CC's public hosting. The existence layer: what the web had at a URL or domain, without text cost; content_digest joins duplicate content across captures."},{"key":"commoncrawl_distillate","label":"Common Crawl distillate","family":"Writing & web","relation":"commoncrawl.distillate","within":null,"description":"The cleaned reading layer of Common Crawl: genre-classified pages (article / forum / mailing_list) selected by link-graph centrality and URL-shape predicates, with boilerplate-free extracted text (~4.3KB avg vs ~8.2KB raw WET). Every row has the WARC triple; tier A adds html and text, tier B text, tier C neither. Token-indexed (hasToken over lower(text))."},{"key":"internet_documents","label":"Internet documents","family":"Writing & web","relation":"internet.documents","within":null,"description":"Internet text documents across 85 source families (source keys e.g. crawled_url, lesswrong, mailing_list): one row per document, with content_text, uri, source, and quality fields. A frozen federated snapshot (nothing after ~2026-04) — contents overlap but do not match the live per-source tables; the live-forum sources' current rows are in forums.posts beside this archive. A record_ref '<source>:<uuid>' resolves here by id (non-UUID refs resolve in forums.posts). The crawled_url family is the earlier static web archive; the live corpus is crawl.pages."},{"key":"wikipedia","label":"Wikipedia","family":"Writing & web","relation":"wikipedia.articles","within":null,"description":"English Wikipedia articles: one row per main-namespace page with title, full article text (payload), word count, categories (JSON), and quality flags — the full 6.2M-page set loaded 2026-03 plus new articles from the recentchanges feed. embeddings.wikipedia_articles is its semantic companion (page_id)."},{"key":"openalex","label":"OpenAlex works","family":"Scholarship","relation":"openalex.works","within":null,"description":"OpenAlex scholarly-work metadata snapshot (the 2026-06-26 public release; nothing changes between releases, so freshness_lag is the release's age and extent.max is a publisher-assigned date, not an arrival date): title, venue, year, DOI, authorships (author id, name, ORCID, institutions, corresponding flag), topics, concepts, keywords, funders, open-access locations, citation counts, and the citation graph (referenced_works). One row per work version in a ReplacingMergeTree (count works with uniq(id), an estimate within about 1%). doi_norm (lowercase bare DOI) joins academic.papers full text via doi_norm = decodeURLComponent(paper_key); paper_key keeps the first slash literal (10.48550/arxiv.2211.09527) and percent-encodes later ones (10.18653/v1%2f2024.acl-long.331). An arXiv DOI (10.48550/arxiv.<id>) identifies the preprint work, type = 'preprint', with its own cited_by_count; the published version is a separate work under the journal DOI (2404.18213: 15 citations as the preprint, 41 as its 2025 journal article), and academic.catalog's doi contains that journal DOI when one exists. counts_by_year is Array(Tuple(year, cited_by_count)) with Nullable members: citations in a window are arraySum(arrayMap(t -> coalesce(tupleElement(t, 2), 0), arrayFilter(t -> coalesce(tupleElement(t, 1), 0) BETWEEN 2012 AND 2015, counts_by_year))) — tupleElement, never t.1, and coalesce before the sum (0.8 s over the table beside a token anchor)."},{"key":"openalex_citations","label":"OpenAlex citations","family":"Scholarship","relation":"openalex.cited_by","within":null,"description":"Reverse citation edges of the OpenAlex graph: one row per (cited work, citing work) with the citing work's publication date — who cites this paper, newest first, in milliseconds. Follow references forward through works.referenced_works, citations backward here; hydrate either end via openalex.works."},{"key":"academic_papers","label":"Academic papers (full text)","family":"Scholarship","relation":"academic.papers","within":null,"description":"Filter quality_label = 'good' for the strictly faithful core. Rows whose quality_flags contain 'tex_pandoc' or 'tex_clean' (the arXiv TeX route) have 'good' by construction, not by measurement; in 'tex_pandoc' rows LaTeX macros pandoc could not expand are dropped silently, and 'tex_clean' rows keep raw unexpanded TeX, so counts and other macro-valued numbers may be missing — verify against another rendering. pdf_bytes > 0 marks a PDF-backed row (the source PDF's size); converter gives the route the text took (PDF text layer, OCR, arXiv TeX or ar5iv HTML, PMC XML, a preprint abstract). How many rows, papers, PDF-backed or TeX-derived texts, and tokens are on hand is served measured on this entry's corpus_status (with computed_at), never quoted here."},{"key":"paper_extractions","label":"Paper extractions","family":"Scholarship","relation":"academic.extractions","within":null,"description":"Every academic-paper text-extraction attempt and revision: status, error class, text-quality measures, and the text itself."},{"key":"arxiv","label":"arXiv","family":"Scholarship","relation":"academic.catalog","within":null,"description":"The canonical bibliographic catalog: one merged row per paper across arXiv, PMC, PubMed, Europe PMC (incl. bioRxiv/medRxiv preprints), INSPIRE-HEP, HuggingFace papers, and a DOI-keyed journal full-text corpus, enriched from OpenAlex — title, authors, abstract, venue, year, source identifiers, MeSH terms, citedness, open-access status, license, retraction flag, which corpora have the paper, and whether full text is on hand. Corpus size is never quoted here: this entry's corpus_status gives the measured row, full-text, identifier and open-access counts with computed_at."},{"key":"pubmed","label":"PubMed","family":"Scholarship","relation":"academic.catalog","within":null,"description":"The canonical bibliographic catalog: one merged row per paper across arXiv, PMC, PubMed, Europe PMC (incl. bioRxiv/medRxiv preprints), INSPIRE-HEP, HuggingFace papers, and a DOI-keyed journal full-text corpus, enriched from OpenAlex — title, authors, abstract, venue, year, source identifiers, MeSH terms, citedness, open-access status, license, retraction flag, which corpora have the paper, and whether full text is on hand. Corpus size is never quoted here: this entry's corpus_status gives the measured row, full-text, identifier and open-access counts with computed_at."},{"key":"crossref","label":"Crossref","family":"Scholarship","relation":"academic.catalog","within":null,"description":"The canonical bibliographic catalog: one merged row per paper across arXiv, PMC, PubMed, Europe PMC (incl. bioRxiv/medRxiv preprints), INSPIRE-HEP, HuggingFace papers, and a DOI-keyed journal full-text corpus, enriched from OpenAlex — title, authors, abstract, venue, year, source identifiers, MeSH terms, citedness, open-access status, license, retraction flag, which corpora have the paper, and whether full text is on hand. Corpus size is never quoted here: this entry's corpus_status gives the measured row, full-text, identifier and open-access counts with computed_at."},{"key":"library","label":"Library catalog records","family":"Scholarship","relation":"books.catalog","within":null,"description":"Unified bibliographic catalog: one row per source record (a book listed in several catalogues appears once per id_prefix) across the file-backed book index (keyed md5:*), the DOI-keyed journal index, and metadata-only library records (OCLC/WorldCat, OpenLibrary, ISBNdb, Google Books, EBSCO eds). Title, author, publisher, year, language, content type, file availability flags, and the bibliographic identifier/classification maps (ISBN-13/10, DOI, OCLC, OpenLibrary, ASIN, DDC, LCC, subjects). Identifier arrays are cluster evidence from record merges, not verified per-object identity — cross-check title/author before trusting an identifier hit. Bibliographic metadata about works only — never file content and no download paths."},{"key":"books","label":"Book passages","family":"Scholarship","relation":"books.passages","within":null,"description":"Book passages with books-norm-2 normalized text, packed to at most 1,024 cl100k tokens with a section heading, exact token counts, and character offsets."},{"key":"book_texts","label":"Book & magazine full text","family":"Scholarship","relation":"books.texts","within":null,"description":"Full text of books, magazines, and standards documents: one row per extracted file with format (epub, pdf, djvu, fb2, html, txt), converter, quality label, and the complete extracted text. md5 joins books.catalog (record_id = concat('md5:', md5)) for title, author, year, and language. Grows continuously as new collections are loaded — fiction and non-fiction, Russian fiction, world magazines, standards documents, English non-fiction epubs, and a large scanned-book collection."},{"key":"patents","label":"Patents","family":"Scholarship","relation":"patents.publications","within":null,"description":"Global patent full text, one row per (country_code, publication_number, lang): title, abstract, description and the claims (flattened here; claim-level rows in patents.claims), kind_code, yyyymmdd dates, inventors, applicants, assignees, CPC and IPC codes, citations, priority claims and legal_status, each row naming its source (google_patents_bigquery:<snapshot> is the CC BY 4.0 Google Patents Public Datasets snapshot, google_patents_page a patents.google.com /patent/<number>/<lang> page, office bulk as it lands), source_url, source_observed_at, precedence (higher wins: primary-office original 40, open corpus with text 30, Google source-language page 20, mirror 15, machine translation 10, metadata-only rows 5), is_machine_translation and sections (which of abstract, description, claims the row carries). An original and a translation are separate rows, never merged."},{"key":"kalshi","label":"Kalshi markets","family":"Markets","relation":"markets.catalog","within":null,"description":"The canonical prediction-market catalog: one folded row per market across Kalshi (largely auto-minted financial-range series), Polymarket, and Manifold — title, category, status, resolution, open/close/settle times, volume, liquidity, and per-venue extras. market_key = '<source>/<native id>' is the spine identity; event_native_id groups markets under their venue event."},{"key":"polymarket","label":"Polymarket markets","family":"Markets","relation":"markets.catalog","within":null,"description":"The canonical prediction-market catalog: one folded row per market across Kalshi (largely auto-minted financial-range series), Polymarket, and Manifold — title, category, status, resolution, open/close/settle times, volume, liquidity, and per-venue extras. market_key = '<source>/<native id>' is the spine identity; event_native_id groups markets under their venue event."},{"key":"sec","label":"SEC filings","family":"Records","relation":"sec.edgar_documents","within":null,"description":"The verbatim substrate; for token-efficient reading of narrative filings use sec.edgar_markdown (the same documents rendered to markdown at ~0.15x the bytes) and join back here by (accession_number, sequence) only when the raw HTML itself is the object of study. Join a filing's documents by accession_number, a company's filings by has(ciks, '<cik>')."},{"key":"registries","label":"Package registries","family":"Code","relation":"packages.catalog","within":null,"description":"The canonical software-package catalog: one merged row per package across the six native registry corpora (npm ~4.2M, Go modules ~2.0M, PyPI ~829K, NuGet ~768K, Packagist ~457K, crates.io ~245K) and the Libraries.io meta-registry snapshot (~30 further platforms: Maven, RubyGems, CocoaPods, ...) — description, latest version, license, repository, downloads, keywords, language, dependent-project counts, and which stores have it. package_key = '<ecosystem>/<name>' is the spine identity."}]}