SEDD (GPU deduplication)

GPU-accelerated MinHash-LSH dataset deduplication (originally released as FED): up to 58.3x faster than SlimPajama's CPU pipeline and 8.6x faster than NVIDIA NeMo Curator's GPU implementation, deduplicating 1.2T tokens in 5.1 hours on 16 GPUs. The data-curation infrastructure underneath the Thunder group's Thunder-LLM pipeline, released for reuse.