Build a Reproducible Portuguese NLP Pipeline in Colab with sri
Notebook projects often fail at the first handoff. One person has a dataset in an undocumented folder, another has a different tokenizer version, and a third cannot tell whether a CSV was created from the same input. The model is not always the hard part. Reproducing the data boundary is.
The open-source sri repository provides a practical notebook sequence for Portuguese text work. It starts by converting a JSON dataset into a small, explicit CSV contract, then moves through cleaning, part-of-speech tagging, named entity recognition, embeddings, and analysis. The repository is MIT licensed and its README documents Google Colab as the main environment.
This tutorial focuses on the first reproducible path: prepare a dataset with the repository's first notebook, verify the generated CSV, and understand how the later notebooks consume it. You can run the same steps in Colab or a local Jupyter environment.
TL;DR
You will:
- Clone or open the repository.
- Create
data/train.jsonwith document titles. - Install the pinned tokenizer dependency used by the preparation notebook.
- Generate
data/documentos.csvwith two columns:idanddocumento. - Check the documented row and token limits before moving to NER or embeddings.
The result is a small, inspectable input contract for the rest of the notebook sequence. It is not a trained model and it does not claim to represent every Portuguese text domain.
Prerequisites
You need:
- A Google account if you want to run the notebooks in Colab.
- Git if you want a local checkout.
- Python 3 and Jupyter for local execution.
- A JSON dataset whose records contain a
titlefield. - Enough memory to install the notebook dependencies and load a Portuguese BERT tokenizer.
The repository's current main branch has no package release that pins the whole workflow. The preparation notebook itself specifies transformers==4.49.0. Treat the current branch and that dependency line as the reproducibility boundary, and recheck them if the repository changes.
1. Get the notebooks
The repository README recommends putting the project under a Google Drive folder named Colab Notebooks/SRI when using Colab. You can also open the first notebook directly from GitHub:
https://colab.research.google.com/github/paladini/sri/blob/main/0_0_PrepararDocumentosCSV_trainjson_v1.ipynb
For a local checkout, use:
git clone https://github.com/paladini/sri.git
cd sri
The repository is a collection of notebooks rather than an installable Python package. That distinction matters: the notebook files and their input paths are the source of truth for this workflow.
2. Create the input contract
The current data guide says that data/train.json can be a list of objects with at least a title field. The notebook also accepts a dictionary with a train list or other dictionary-shaped records. Keep the input local if it contains material you are not authorized to publish.
Create data/train.json with a small test fixture first:
[
{
"date": "2024-01-01",
"title": "Brasil inicia um novo programa de dados abertos",
"text": "Texto completo opcional para uma análise posterior"
},
{
"date": "2024-01-02",
"title": "Pesquisadores avaliam modelos de linguagem em português"
}
]
The preparation step reads title, normalizes whitespace, and writes that value to the documento column. The optional date and text fields are not preserved by this first conversion. If later analysis needs them, keep the original JSON beside the generated CSV and design a separate transformation instead of assuming the first notebook carries every field forward.
3. Run the preparation notebook
The notebook installs the tokenizer dependency with this exact line:
%pip install transformers==4.49.0 -q
Its implementation uses the neuralmind/bert-base-portuguese-cased tokenizer, sets MAX_TOKENS = 512, and sets MAX_ROWS = 500. It searches for data/train.json locally and also checks the common Colab and Drive locations when running inside Colab.
Run the notebook cells in order. The important output is data/documentos.csv, written as UTF-8 with a semicolon separator. The generated file has this shape:
id;documento
1;Brasil inicia um novo programa de dados abertos
2;Pesquisadores avaliam modelos de linguagem em português
The IDs are assigned from one in the order of accepted records. Records without a usable title are skipped. The notebook stops after 500 accepted rows, and the tokenizer truncates each title to at most 512 tokens.
4. Verify before using another notebook
Do not jump directly to embeddings because a cell produced output. Verify the artifact you are about to pass to the next stage.
In a Python cell, check the file contract:
from pathlib import Path
import pandas as pd
output_path = Path("data/documentos.csv")
df = pd.read_csv(output_path, sep=";")
assert list(df.columns) == ["id", "documento"]
assert len(df) <= 500
assert df["id"].is_unique
assert df["documento"].notna().all()
print(f"rows={len(df)}")
print(df.head(3).to_string(index=False))
This check does not prove that the text is correct. It proves that the next notebook receives the expected columns, a bounded number of rows, and non-empty document values. Compare a few rows with the original JSON as a human review step.
5. Continue through the notebook sequence
The README groups the notebooks into stages:
- Preparation creates the CSV input.
- Pre-processing includes segmentation, cleaning, and spaCy part-of-speech tagging.
- Analysis creates distributions and tables.
- Embeddings includes Word2Vec and BERT-based experiments.
- NER identifies named entities and can render them with spaCy's displaCy.
The order is useful because it makes the data boundary visible. A later notebook should consume the generated documentos.csv, not an undocumented copy in a different folder. When adapting a notebook, record the source commit, the input path, the dependency versions, and any model name that downloads from an external hub.
For an NER experiment, the repository includes a notebook named 1_3_NER_spaCy_v1.ipynb. For BERT embeddings, it includes 3_2_Embeddings_BERT_Huggingface_v1.ipynb. Those notebooks are experiments and teaching material, not a general Portuguese NLP service with an accuracy guarantee.
Why the first conversion matters
The preparation notebook creates a narrow interface between data collection and language processing. It makes four decisions inspectable:
- Which field becomes the document text.
- How missing titles are handled.
- How many records enter the sequence.
- How long each document can be for the selected tokenizer.
That interface is more valuable than silently passing a large JSON object between cells. A reviewer can inspect the CSV, compare it with the source fixture, and rerun the same preparation step after changing the input.
The tradeoff is information loss. The first notebook keeps only an assigned ID and normalized title text. If your research question depends on dates, authors, labels, or full article bodies, preserve those fields separately and document the join key you will use later.
Failure modes and security boundaries
The notebook cannot find train.json
Place the file at data/train.json relative to the notebook, or use the Colab Drive layout documented in the notebook. Do not silently upload a different fixture just to make the cell continue. Print the resolved input path before processing.
The CSV has fewer rows than expected
The notebook skips records without a usable title and limits output to 500 accepted rows. Inspect the source records and the normalization step before increasing the limit. A smaller output can be the correct result.
Tokenizer downloads fail
The workflow needs the transformers package and downloads the Portuguese tokenizer model. Network restrictions, cache permissions, or a changed model identifier can stop execution. Do not treat a cached tokenizer as proof that another machine can reproduce the run.
Sensitive data enters the notebook
Colab notebooks can contain uploaded files, outputs, and credentials from the runtime. Keep private datasets out of public repositories, remove secrets from cells and outputs, and review the generated CSV before sharing it. The MIT license covers the repository code and notebooks; it does not authorize you to redistribute someone else's dataset.
FAQ
Does sri train a Portuguese language model?
No. It provides practical notebooks for preprocessing, NER, embeddings, and related experiments. The first notebook prepares data for those experiments.
Why is the output separated with semicolons?
The notebook writes documentos.csv with sep=';'. Read it with the same separator or the columns will not be parsed as intended.
Can I use a different JSON field?
The documented preparation path reads title. Adapt the notebook only after deciding how the new field should be normalized and how its token limit should be checked.
Is 512 tokens a quality threshold?
No. It is the notebook's maximum for this preparation step. Truncation can remove useful context, so inspect the data and choose a limit appropriate for the downstream model and research question.
Takeaway
The most reusable part of sri is its explicit handoff from JSON records to a bounded CSV. Start with a tiny fixture, run the preparation notebook, inspect documentos.csv, and only then move to NER, embeddings, or analysis. That habit makes a notebook sequence easier to review and easier to reproduce.
If you were extending this workflow, which field would you preserve next in the intermediate contract: dates, source labels, or full document text?
AI assistance disclosure: AI assistance was used to organize and edit this tutorial. The repository README, data guide, preparation notebook, NER notebook, BERT notebook, license, and current
mainbranch were checked against the linked primary sources. The local repository clone was also used to run the documented build and verification path where available.
Top comments (0)