infini-gram
libraryAn unbounded n-gram engine over trillion-token corpora that answers arbitrary-length n-gram queries in ~20ms via a suffix-array index (COLM 2024). It exposes a public API and package with real adoption for interpolated decoding, training-data analysis, and contamination checks; infini-gram mini (2025) extends the engine. UW-branded (Jiacheng Liu, Sewon Min, Zettlemoyer, Choi, Hajishirzi) even where it indexes AI2 corpora.