Refactor the download and publication of the datasets and convert it into CLI script (#95)

Co-authored-by: Remi <re.cadene@gmail.com>
This commit is contained in:
Adil Zouitine
2024-04-29 00:08:17 +02:00
committed by GitHub
parent 81e490d46f
commit 55dc9f7f51
15 changed files with 1410 additions and 827 deletions

View File

@@ -342,7 +342,6 @@ def compute_stats(hf_dataset, batch_size=32, max_num_samples=None):
"max": max[key],
"min": min[key],
}
return stats