Nate@nate_512

Follow

Hugging Face DataTrove under $1

A Hugging Face engineer processed 1.2 million CommonCrawl pages for under $1 using DataTrove, now running natively on Hugging Face Jobs. No Slurm cluster needed. The same pipeline code works with just a simple executor swap, cloud tasks fan out automatically, and CPU filtering plus GPU work together.

1mo

Nate@nate_512

Follow

Hugging Face DataTrove under $1

A Hugging Face engineer processed 1.2 million CommonCrawl pages for under $1 using DataTrove, now running natively on Hugging Face Jobs. No Slurm cluster needed. The same pipeline code works with just a simple executor swap, cloud tasks fan out automatically, and CPU filtering plus GPU work together.

1mo