The Common Pile

dataset model
Toronto & Vector Institute

Colin Raffel / r-three open-data effort — an EleutherAI + Hugging Face + University of Toronto collaboration releasing a large openly-licensed corpus and the LLMs trained on it.

Paper (arXiv)GitHub

Outputs 2

The Common Pile v0.1

dataset

<p>An 8TB corpus of openly-licensed and public-domain text, curated to train competitive LLMs without copyrighted data.</p>

Paper (arXiv)GitHub
Size 8TB
License Openly licensed / public domain
HuggingFace

Comma v0.1

model

<p>7B LLMs trained solely on The Common Pile, competitive with Llama-1/Llama-2-era models trained on unlicensed web data.</p>

Paper (arXiv)
Architecture DENSE
Parameters 7B
License Apache 2.0
datasetopen-weightresearch