FineWeb

Overview

FineWeb is a 15-trillion-token English web corpus released by Hugging Face, distilled from 96 Common Crawl snapshots spanning 2013–2024. Every document passed URL filtering, language identification, MinHash deduplication and a battery of C4/Gopher-style quality heuristics, with each design decision validated by ablation training runs. Models pretrained on FineWeb consistently outperform those trained on C4, RefinedWeb, Dolma or The Pile at equal compute.

Contents

Roughly 23.5 billion documents in Parquet shards, partitioned by crawl dump. Each row carries the cleaned text, source URL, crawl timestamp, language score and token count. Official subsets (sample-10BT, sample-100BT, sample-350BT) make scaled-down experiments and data ablations cheap.

Why it matters for neuronet training & evaluation

Pretraining data quality is the dominant cost lever for LLM neuronets. FineWeb gives creators a reproducible, permissively licensed (ODC-By) base corpus: pin a revision, stream the shards to miners, and train base models whose data lineage stays fully auditable on-chain.