Datasets are repos too

Half the Hub consists of datasets, and most beginners never see them. Yet that is exactly where the trick lives that separates datasets from models: you can use them without having them.

The viewer - looking inside before you load anything

Open any dataset, for example stanfordnlp/imdbExternal - Opens in a new tab. Below the description you see a table with real rows, plus filters, a full-text search and per-column distributions.

That is the dataset viewer, and it is more than a preview. In thirty seconds it answers the questions you would otherwise download 20 GB and open a notebook for: which columns are there? Are the labels balanced? How long are the texts really? Is there junk in it?

At the top sit the splits - usually train, test, sometimes validation. They are separate sets of files in the same repo, not filters.

Step 1 - Using without downloading

pip install -U datasets

The usual way loads everything:

from datasets import load_dataset

ds = load_dataset("stanfordnlp/imdb", split="train")   # downloads and caches all of it
print(ds[0])

The other way loads nothing:

from datasets import load_dataset

ds = load_dataset("stanfordnlp/imdb", split="train", streaming=True)
for example in ds.take(5):
    print(example["label"], example["text"][:80])

streaming=True gives you not a Dataset but an IterableDataset: it fetches blocks over HTTP while you iterate and puts nothing on your disk. You can no longer write ds[42] and there is no len(ds) - in exchange you can work with a dataset larger than your hard drive.

Both ways accept .map(), .filter() and .shuffle() - with streaming these steps are applied lazily, that is, only while iterating:

short = ds.filter(lambda b: len(b["text"]) < 500).map(
    lambda b: {"length": len(b["text"])}
)
print(next(iter(short)))

Step 2 - Your own dataset

Now the other direction. Creating a dataset repo is not a big deal - push_to_hub() handles repo creation, conversion to Parquet and upload in one call.

Take something small and your own. For instance a handful of sentences with a label:

# my_dataset.py
from datasets import Dataset

rows = {
    "text": [
        "The delivery arrived three weeks late.",
        "Thank you, everything worked perfectly.",
        "The device was already damaged when I unpacked it.",
        "Very friendly support, would order again.",
    ],
    "label": [1, 0, 1, 0],   # 1 = complaint
}

ds = Dataset.from_dict(rows)
ds.push_to_hub("yourusername/complaints-mini")

python my_dataset.py

This needs a token with write access - the moment from chapter 02 where the second, more powerful token comes into play. If you do not want a public repo: push_to_hub("…", private=True).

After a minute or two the dataset sits at huggingface.co/datasets/yourusername/complaints-mini - with a viewer, because push_to_hub writes Parquet by itself.

Step 3 - Write the card

A dataset without a description is worthless to everyone else, and to you in six months as well. Edit the repo's README.md directly in the browser and answer four questions:

1. Where does the data come from? Written by you, collected, derived from another source?
2. What do the columns mean? label: 1 = complaint is one line and saves every reader ten minutes.
3. Under which licence is it published? The license: field in the YAML header.
4. What is it not suitable for? Four hand-written sentences are not a benchmark. Say so.

Step 4 - Fetching it back

The circle closes, visibly:

from datasets import load_dataset

mine = load_dataset("yourusername/complaints-mini", split="train")
print(mine)
print(mine[0])

Your dataset now behaves exactly like stanfordnlp/imdb. That is the real point of the Hub: there is no second class. Same tools, same calls, same viewer - whether the repo comes from a research lab or out of your terminal five minutes ago.

The milestone

Two pieces of evidence:

• A script that prints five rows from a large public dataset with streaming=True - and du -sh ~/.cache/huggingface has not grown noticeably afterwards.
• A URL to your own dataset where the viewer shows rows and the README explains what the columns mean.