DEV Community: Hakim

Scraping de legifrance ? Considérez plutôt l'API

Hakim — Sun, 17 Mar 2024 23:16:32 +0000

Si vous envisagez le scraping pour récupérer des articles sur le site legifrance vous devriez considérer l'API publique disponible dans le catalogue d'API Piste.

Obtention des identifiants OAuth

Commencez par créer un compte sur Piste.

Ensuite, rendez-vous sous "API > Consentement CGU API" pour accépter les CGU de Légifrance

Créez une application dans l'onglet application.

Une fois créée, modifiez votre application pour y ajouter l'API Légifrance

De retour dans l'onglet applications, cliquez sur votre application puis récupérer votre client id et votre client secret.

Obtention d'un token

L'obtention d'un token se fait via une requête POST sur l'url : https://oauth.piste.gouv.fr/api/oauth/token

Exemple de requête avec cURL :



curl  -X POST \
  'https://oauth.piste.gouv.fr/api/oauth/token' \
  --header 'Accept-Encoding: gzip,deflate' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --header 'Content-Length: 140' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode 'client_id=xxx' \
  --data-urlencode 'client_secret=yyy' \
  --data-urlencode 'scope=openid'

Voici le même exemple sous python :



import requests
from requests_oauthlib import OAuth2Session

client_id = "xxx"
client_secret = "yyy"

res = requests.post(
  "https://oauth.piste.gouv.fr/api/oauth/token",
  data={
    "grant_type": "client_credentials",
    "client_id": client_id,
    "client_secret": client_secret,
    "scope": "openid"
  }
)

res.json()
# {'access_token': 'WxkeizeaK7vqh1A3swGD14RASpM4tTf2PXzC32SV1mzBOebmx1asYA',
#  'token_type': 'Bearer',
#  'expires_in': 3600,
#  'scope': 'openid resource.READ'}

Récupération d'un article à l'aide du token

Le jeton obtenu ci-dessus permet de requête l'API Piste Légifrance.

Voici un exemple avec cURL :



curl  -X POST \
  'https://api.piste.gouv.fr/dila/legifrance/lf-engine-app/consult/getArticle' \
  --header 'Authorization: Bearer k0ESUrVPNCywGOm4Ef9Qehgxj8Ktr4gMhtPNc3DRilz5v11y3JKEDP' \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "id": "LEGIARTI000006307920"
}'

Le même exemple en python :



import requests

response = requests.post(
    "https://api.piste.gouv.fr/dila/legifrance/lf-engine-app/consult/getArticle",
    json={"id": "LEGIARTI000006307920"},
    headers={
        "Authorization": "Bearer k0ESUrVPNCywGOP4Ef9Qehgxj8Ktr4gMhtPNc3DRilz5v11y3JKEDP",
        "Content-Type": "application/json",
    },
)

response.json()
# {
#   "executionTime": 0,
#   "dereferenced": false,
#   "article": {
#     "id": "LEGIARTI000006307920",
#     "idTexte": null,
#     "type": "AUTONOME",
#     "texte": "L'impôt sur le revenu est établi d'après le montant total du revenu net annuel dont dispose chaque foyer fiscal. ..."
#   }
# }

La documentation de l'API est disponible dans le rubrique "API > Mes API" puis en cliquant sur Légifrance.

pylegifrance

Je suis tombé par hasard sur la librairie pylegifrance dont l'objectif est de faciliter la récupération des articles legifrance via l'API de Piste.

Il s'agit plus d'un protoype de librairie puisqu'elle n'est pas encore publiée sur pypi mais a l'air prometteuse donc à surveiller.

Train a Sentence-CamemBERT

Hakim — Sat, 09 Mar 2024 11:59:30 +0000

The CamemBERT model is a state-of-the-art language model for modeling the
French language.

It is a RoBERTa model that has been trained on a large number of French texts and can be easily adapted to a large number of tasks thanks to finetuning.

Here we're going to to finetune the model for sentence embedding.

Sentence-BERT

The output of a BERT model is an embedding vector for each token. To obtain an embedding of the text as a whole, we need to define a transformation strategy to go from individual token embeddings to an embedding vector for the sentence as a whole.

The simplest and most effective strategy is simply to take the average of the token embeddings.
This strategy is known as mean pooling.

If you'd like to find out more about the strategies that have been considered, take a look at this paper: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.

Finetuning a BERT model into a Sentence-BERT model

The authors of the paper mentioned above have built a Python library called sentence-transformers, for manipulating Sentence-BERT models.

We'll use it to obtain a Sentence-CamemBERT model from a CamemBERT model available on huggingface.

Prerequisites

We will be using the following packages:

datasets
sentence-transformers

Training data

First of all, we're going to retrieve the training data.
We will use the French part of the dataset STSb Multi MT.
This is a dataset containing pairs of sentences and a score between 0 and 5 representing the similarity between the two sentences.

from datasets import load_dataset

sts_train_dataset = load_dataset("stsb_multi_mt", name="fr", split="train")
sts_dev_dataset = load_dataset("stsb_multi_mt", name="fr", split="dev")
sts_test_dataset = load_dataset("stsb_multi_mt", name="fr", split="test")

We'll then convert the retrieved data into InputExample objects that can be used for training.

from typing import List
from sentence_transformers import InputExample

def dataset_to_input_examples(dataset) -> List[InputExample]:
    return [
    InputExample(
        texts=[example["sentence1"], example["sentence2"]],
        label=example["similarity_score"] / 5.0,
    )
    for example in dataset
]

sts_train_examples = dataset_to_input_examples(sts_train_dataset)
sts_dev_examples = dataset_to_input_examples(sts_dev_dataset)
sts_test_examples = dataset_to_input_examples(sts_test_dataset)

We will use the CamemBERT model named almanach/camembert-base for finetuning:

from sentence_transformers import evaluation, losses, SentenceTransformer
from torch.utils.data import DataLoader

batch_size = 32

model = SentenceTransformer("almanach/camembert-base")

train_dataloader = DataLoader(sts_train_examples, shuffle=True, batch_size=batch_size)
train_loss = losses.CosineSimilarityLoss(model=model)

We use cosine-similarity loss objective to train the model.

Finally, an evaluator is built to monitor the model's performance on the dev dataset during training.

sts_dev_evaluator = evaluation.EmbeddingSimilarityEvaluator.from_input_examples(
    sts_dev_examples, name="sts-dev"
)

We can now start training the model:

model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    evaluator=sts_dev_evaluator,
    epochs=10,
    warmup_steps=500,
    save_best_model=True,
)

Model evaluation

Once training is complete, you can measure the model's performance on the test data set you've kept
away:

sts_test_evaluator = evaluation.EmbeddingSimilarityEvaluator.from_input_examples(
    sts_test_examples, name="sts-test"
)

sts_test_evaluator(model, ".")

I get a Pearson correlation of 0.837 which is at the same level as the Sentence-CamemBERT I found on huggingface :

Model	Pearson Correlation	Parameters
`h4c5/sts-camembert-base`	0.837	110M
`Lajavaness/sentence-camembert-base`	0.835	110M
`inokufu/flaubert-base-uncased-xnli-sts`	0.828	137M
`h4c5/sts-distilcamembert-base`	0.817	68M
`sentence-transformers/distiluse-base-multilingual-cased-v2`	0.786	135M

Sentence-BERT model distilled

As you may have noticed in the table above, I've also trained a Sentence-CamemBERT model that's about half the size (68M parameters vs. 110M) and yet performs very well: h4c5/sts-distilcamembert-base.

This is in fact a model obtained by following the above procedure but starting from the distilled CamemBERT model: cmarkea/distilcamembert-base.

This so-called "distilled" model was obtained by removing half of the layer of the CamemBERT base model and training it to maintain its performance.

To find out more about the distillation process, please consult the following papers:

Et voilà. You can find my two Sentence-CamemBERT models on huggingface :

An introduction to packaging in Python

Hakim — Fri, 08 Mar 2024 21:54:43 +0000

Learning how to package your code is very useful for any python developer. It gives you a better understanding of how Python works and, above all, enables you to share your code with others or simply deploy it in a runtime environment.

So how does it work? Why not just share your code via git? Actually, it's more complex than it sounds. There's even a whole working group, the Python Packaging Authority (aka pypa), which has been working on this subject since since 2011 and it's a constantly evolving field.

In this article, we'll take a quick look at packaging in Python and present a simple method for packaging your code in 2023.

TL;DR

Use the pyproject.toml file following the setuptools guide.

You can simply use pip install build && python -m build to build your package.

Adopt src-layout.

Packages in Python

First, a few definitions:

Python module

A module python un is a text file with the extension .py containing python code.

It's called a module because it "modularizes" the python source code into several .py files. We then use python's import functionality to import elements from another module.

For example, if you create a count_lines.py file containing a count_lines_file function, you can then import this function:

# count_lines.py
def count_lines_file(filepath: str) -> int:
    """Count the number of lines in a file"""
    return sum(1 for _ in open(filepath))

from count_lines import count_lines_file
count_lines_file("count_lines.py")
3

When you execute the instruction from count_lines import count_lines_file, python looks for the module in the current directory (where python was launched); then in the python installation directory /usr/lib/python3.11; then in the directory where python installs default packages: /usr/lib/python3.11/site-packages.

As soon as the module is found, it is executed in the python environment and the elements defined in it become available.

The list of directories in which python searches for modules is provided by the sys package:

import sys
sys.path
["", "/usr/lib/python3.11/python311.zip", "/usr/lib/python3.11/python311", "/usr/lib/python3.11/site-packages"]

Package python

A package is a folder that groups together a set of python modules and facilitates access to them by creating a namespace: from numpy.linalg import norm.

To create a package, simply create a __init__.py file (which can be empty) in a folder.
The folder is then considered by python as a package.

Let's create a package for the count_lines.py module:

count_package
├── __init__.py
└── count_lines.py

We can now use "pointed" notation to import the count_lines module:

from count_package.count_lines import count_lines_file
count_lines_file("count_lines.py")
3

When executing the instruction from count_package.count_lines import count_lines_file, python looks for a count_package folder containing a __init__.py file in the sys.path directories (the current directory and the default directories seen above).

If the package is found, the modules present in it can be accessed using the "pointed" notation.

Distribute your package

To distribute your package, create a distribution. This is an archive containing the package to be distributed, which can then be installed using the pip package manager.

There are two main distribution formats:

The Source Distribution (sdist) format: this is an archive containing all source code and metadata.
The Built Distribution format: this is a distribution format in which a number of things have been pre-compiled to facilitate installation on other environments. This is particularly useful for modules written in C / C++.

The wheel format is the reference Built Distribution format. It is the format developed by the Python Packaging Authority and is widely used to distribute packages.

There are many tools available for creating distributions, but here we'll focus mainly on the tools created by the Python Packaging Authority, which have become indispensable: setuptools, build and twine.

setuptools

setuptools is the tool used by the vast majority of projects to build their distributions.

Let's take our count_package example from earlier and see how to create a distribution with setuptools.

Our project tree might look something like this:

projet_genial
├── count_package
│   ├── __init__.py
│   └── count_lines.py
├── tests
│   └── test_count_lines.py
├── .gitignore
├── LICENSE.md
└── README.md

setuptools needs a configuration file to know what to include in the distribution and all the project metadata.

Unfortunatly there is three configuration files standards currently used (and you can use them all at the same time...) :

The setup.py file, the oldest and most popular (3.7 million results on github)
The setup.cfg file, the second most popular file (0.4 million hits on github)
The pyproject.toml file, the latest arrival and now the official standard for all python python packaging tools. (0.2 million hits on github)

pyproject.toml is the official standard, but is still in the minority compared to setup.py, so we'll be looking at all three file formats.

setup.py

The setup.py file, as its extension suggests, is a python file. It has the following form:

# setup.py
from setuptools import setup

setup(
    name='count_package',
    author='me',
    description='Package for counting the number of lines in files.'
    version='0.0.1',
    python_requires='>=3.7, <4',
    install_requires=[
        'pandas',
        'importlib-metadata; python_version >= "3.8"',
    ],
)

The very fact that it's a python file is both its strength and its weakness: it's possible to build the configuration dynamically in the code, but this makes it difficult to parse and interface with other external tools.

In addition, since this format is specific to setuptools, distributions of the sdist type can only be installed if setuptools has been installed on the target environment and in a compatible version.

Sadly, since the vast majority of projects started using setuptools and setup.py, it became difficult to propose alternatives, so projects like flit had to be built "on top" of setuptools, so the setup.py doesn't encourage innovation.

Moreover, its use is often problematic.
To take the example from this article, you might be tempted to introduce a if/else condition in your setup.py to manage a dependency needed in python 2.7 based on sys.version, but in doing so you would be introducing a vicious bug: the dependency will be included or not depending on the environment that compiles the distribution and not depending on the environment that is installing it.

It's also tempting to import your own package from setup.py to manage the version. But by doing so, sdist distributions will crash at installation because the package you are trying to import is not yet present in the python environnment.

In short : please do not use setup.py anymore

And if you do, use it in a declarative way.

Using the file as a script: python setup.py is depreciated, as the documentation clearly explained in the setuptools documentation :

It is important to remember, however, that running this file as a script (e.g. python setup.py sdist) is strongly
discouraged, and that the majority of the command line interfaces are (or will be) deprecated (e.g. python setup.py
install, python setup.py bdist_wininst, ...).

We also recommend users to expose as much as possible configuration in a more declarative way via the pyproject.toml
or setup.cfg, and keep the setup.py minimal with only the dynamic parts (or even omit it completely if applicable).

See Why you shouldn't invoke setup.py directly
for more background.

setup.cfg

To address the issues mentioned above and make configuration more declarative, in 2016 pypa created the setup.cfg file format.

The example setup.py file above is equivalent to the following setup.cfg file:

# setup.cfg
[metadata]
name = count_package
version = 0.0.1
author = me
description = Package for counting the number of lines in files.

[options]
python_requires = >=3.7,<4
install_requires =
    pandas
    importlib-metadata; python_version >= "3.8"

This format has had many fans but has recently been superseded by the pyproject.toml format, which is now the official way of declaring python package configuration.

pyproject.toml

In addition to adopting the declarative approach of setup.cfg, the pyproject.toml format introduces a number of new features.

It is now possible (and even mandatory) to specify the package builder.
It is also a means of centralizing the configuration of numerous development tools in an agnostic way, rather than multiplying configuration files such as tox.ini, .coveragerc, etc.

The pyproject.toml format includes a mandatory section to define the builder to be used to build the package :

# pyproject.toml
[build-system]
requires = [
  "setuptools>=60",
  "wheel>=0.30.0",
  "cython>=0.29.4",
]
build-backend = "setuptools.build_meta"

With pyproject.toml it is now possible to declare to pip the dependencies needed for the build!

It is then perfectly possible to specify the use of a builder other than setuptools, such as flit :

# pyproject.toml
[build-system]
requires = ["flit"]
build-backend = "flit.api:main"

This format is gradually becoming the preferred way of centralizing package configuration.
It is the format preferred by setuptools and a number of third-party tools use it to store their configuration: black, pytest, isort, etc.

Here's a sample pyproject.toml file for our count_package example package:

# pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "count_package"
version = "0.0.1"
description = "Package for counting the number of lines in files."
name = "my_package"
authors = [
    {name = "me", email = "email@me.fr"},
]
requires-python = ">=3.8,<4"
dependencies = [
    "pandas",
    'importlib-metadata; python_version >= "3.8"',
]
dynamic = ["version"]

The setuptools documentation gives more details on configuration in pyproject.toml format.

Build a distribution of your package

Once you've written your configuration file (preferably pyproject.toml), all that's left to do is build your package.

The modern way to do this is to use the build package developed by pypa:

pip install --upgrade build
python -m build

build will first install the builder specified in your pyproject.toml file, then use it to build an sdist distribution and a wheel.

You can then use the twine package to publish it on the official pypi repository.

For backward compatibility with older versions of packaging libraries, you can create a minimal setup.py file in addition to the pyproject.toml file:
# setup.py
from setuptools import setup
setup()

Version management

In the pyproject.toml example seen above, the package version is set manually. You therefore need to change it each time you want to publish a new version of your package.

I find the tool setuptools-scm very useful for managing package versions using git or mercurial.

This is done very simply by adding the pyproject.toml dependency and specifying that the version is dynamic:

# pyproject.toml
[build-system]
requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2"]

[project]
# version = "0.0.1"  # Remove any existing version parameter.
dynamic = ["version"]

[tool.setuptools_scm]
write_to = "src/pkg/_version.py"

When building a package, setuptools-scm will search for the last tag with a valid version number and then deduce the package version number. By default, the version is built from three elements:

The last tag with a valid version number (example: v1.2.3)
The distance to this tag (number of revisions since this tag)
Working directory status (if there are any uncommitted changes)

Once the version number has been deduced, a _version.py file will be created inside the distribution at the specified location (e.g. src/pkg/_version.py), allowing the package version to be known from the distribution without the git history being present on the target environment.

If you are in the habit of entering the version number of your packages yourself, please note that valid version formats are governed by PEP 440.
If you don't comply with these specifications, you're likely to run into problems when publishing or installing your packages.
In particular, versions v1.2.3-local or v1.2.3-dev are invalid.

The layout

When configuring your package, regardless of the method used (setup.py, setup.cfg or pyproject.toml), you must specify the packages and subpackages you wish to include in your distribution:

# pyproject.toml
[tool.setuptools]
packages = ["mypkg", "mypkg.subpkg1", "mypkg.subpkg2"]

Fortunately, setuptools has an automatic discovery feature for your packages and subpackages. This is compatible with two classic project layouts:

flat-layout:

count_package
├── count_package
│   ├── __init__.py
│   └── count_lines.py
├── tests
│   └── test_count_lines.py
├── .gitignore
├── LICENSE.md
└── README.md

and layout with a src-layout folder:

projet_genial
├── src
|   ├── count_package
│   ├── __init__.py
│   └── count_lines.py
├── tests
│   └── test_count_lines.py
├── .gitignore
├── LICENSE.md
└── README.md

The difference may seem minimal, but personally I have a strong preference for src-layout because it prevents bad habits and forces you to understand how the package import and installation system
system works in Python.

In fact, when you develop your package, you test the functionalities you add to it as you go along.

To do this, we are very tempted to simply import our package from our :

# test_file.py
from count_package import count_lines

This will work if you use a flat-layout and run your test module from the directory containing the count_package folder, because as we saw above python includes the current directory in the list of directories where it searches for modules.

However, this is a bad habit for two reasons:

Firstly, if you use setup.py, which is located at the root of your project, it is able to import the count_package package it is supposed to install on a client in sdist mode, which can cause bugs if you're not careful.
Secondly, you're not really testing the package as it will be installed on others! For example, you may not have thought to include data files in your configuration file, and your tests should crash as a result. But since these files are present in your working directory, you won't notice a thing.

For these reasons, I think it's best to opt for a src-layout and use an editable installation with the command: pip install -e . for local development. This allows you to install the package by making a symbolic link with your code, so that any changes you make are immediately reflected in the installed package.

For an in-depth analysis of the benefits of src-layout, I refer you to this article (which dates back to 2014).

Going further

If you want to learn more about the packaging eco-system I found the resources listed in this article very usefull : 🐍 Best resources on Python packaging 📖

In the meantime here is a non-exhaustive list of alternatives to setuptools you should consider :

Pipenv: allows you to jointly manage your project's virtual environment and its dependencies. dependencies. Adds a valuable feature: generation of Pipfile.lock files, which reference reference exact versions of dependencies to enable identical reproduction of the development environment. development environment.
Poetry: a powerful tool for managing virtual environments environments, dependencies (and dependencies on dependencies), generate a poetry.lock file similar to Pipfile.lock, publish your package, etc. However, it does not comply with certain PEP standards.
PDM: next-generation package manager for python. Unlike PerformanceEntry, it respects PEP standards.
Hatch: Pypa's new tool for managing python projects. It has many interesting features.
uv: pip written in rust to make it 10 to 100 times faster.

Bonus

Try a next-gen manager Hatch, PDM or Poetry.
I'm sharing my simple template cookie-cutter for your python projects.

Let's be happy

Everything I've told you here may seem like a lot, and yet I've only skimmed the surface. In any case, I think we can count ourselves lucky when we see what the Sam & Max site wrote in 2018:

First we have distutils, setuptools, distribute, and distribute2 which were all at one time the "standards recommended for packaging a lib.
Then came the days of eggs, exe, and other stuff that easy_install would go and find anywhere in the wild, blindly following links on PyPi.
Not to mention the stuff that had to be compile at every turn.
Besides, nothing was encrypted when downloaded, and pip wasn't packaged with Python.
Python, it was dying on stupid errors like badly managed encoding...
On top of that, virtualenv was a separate thing, with lots of competitors, and linked system packages by default.
Not to mention that we didn't have python -m.

In short, Python packaging was a real mess. Not to mention a shitty documentation.

It's a lot simpler these days.

References

pypa's guide to python packaging: An Overview of Packaging for Python
An article on wheels: What Are Python Wheels and Why Should You Care?
A series of three very enlightening articles on how Python packaging works, written by Bernát Gábor in 2019 :
- The state of Python Packaging
- Python Packaging - Past, Present, Future](https://bernat.tech/posts/pep-517-518/)
- Python packaging - Growing Pains
An article in praise of setup.cfg on the late Sam & Max site: about setup.cfg
Stackoverflow question: What is pyproject.toml file for

JSON in data science projects: tips & tricks

Hakim — Thu, 07 Mar 2024 00:20:00 +0000

Some useful tips and libraries for manipulating json in your data science projects.

The standard `json` module

Python has a standard module called json that lets you quickly manipulate JSON files.

Loading

import json

with open("data/example.json", "r") as f:
    data = json.load(f)

data
# [{'id': 0, 'content': [0.0, 0.0, 1.0]}, {'id': 1, 'content': [0.0, 1.0, 0.0]}]

Backup

First tip: when working with textual data, the ensure_ascii=False option is very useful
to preserve, among other things, accents when saving

with open("data/example.json", "w") as f:
    json.dump(data, f, ensure_ascii=False)

Second tip: the indent option in the dump method indents the data in the backup file.

with open("data/example.json", "w") as f:
    json.dump(data, f, ensure_ascii=False, indent=2)

Issues related to numpy

Data science projects often use numpy.
However, numpy objects are not JSON-serializable and therefore require conversion to standard python objects in order to be saved:

import numpy as np

data = np.array([[0., 0., 1.], [0., 1., 0.]])

with open("data/numpy.json", "w") as f:
    json.dump(data, f, ensure_ascii=False)

---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[12], line 6
      3 data = np.array([[0., 0., 1.], [0., 1., 0.]])
      5 with open("data/numpy.json", "w") as f:
----> 6 json.dump(data, f, ensure_ascii=False)

TypeError: Object of type ndarray is not JSON serializable

By converting the array into a list, the data object can be saved:

with open("data/numpy.json", "w") as f:
    json.dump(data.tolist(), f, ensure_ascii=False)

But that's not very practical...

One solution is to create a custom JSONEncoder
which converts the numpy.ndarray using its tolist method at save time:

class NumpyJSONEncoder(json.JSONEncoder):
    """JSONEncoder to store python dict or list containing numpy arrays"""

    def default(self, obj):
        """Transform numpy arrays into JSON serializable object such as list
        see : https://docs.python.org/3/library/json.html#json.JSONEncoder.default
        """
        if isinstance(obj, np.ndarray):
            return obj.tolist()

        return json.JSONEncoder.default(self, obj)

with open("data/numpy.json", "w") as f:
    json.dump(data, f, ensure_ascii=False, cls=NumpyJSONEncoder)

The `orjson` library

orjson is the fastest JSON library available for python. It natively manages dataclass objects,
datetime, numpy
and UUID objects.

A few things to remember when working with orjson :

There is no load or dump method, you have to use loads and dumps instead.
You must use flags to use certain functionalities, such as orjson.OPT_SERIALIZE_NUMPY to serialize serialize numpy objects

import orjson

with open("data/example.json", "rb") as f:
    data = orjson.loads(f.read())

with open("data/example.json", "wb") as f:
    f.write(orjson.dumps(data, option=orjson.OPT_SERIALIZE_NUMPY))

Note that the json file is written in binary (hence rb and wb).

Performance

orjson claims to serialize numpy.ndarray 4 to 12 times faster than the standard library. This can be
by comparing the two methods described above:

data = {i: np.random.randn(100) for i in range(100)}

def save_json():
    with open("data/fast.json", "w") as f:
        json.dump(data, f, ensure_ascii=False, cls=NumpyJSONEncoder)

def save_orjson():
    with open("data/orfast.json", "wb") as f:
        f.write(orjson.dumps(data, option=orjson.OPT_SERIALIZE_NUMPY|orjson.OPT_NON_STR_KEYS))

%timeit save_json()
%timeit save_orjson()

# 15.5 ms ± 251 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# 1.15 ms ± 66.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In this example, orjson is more than 10 times faster. Note that the OPT_NON_STR_KEYS option is used to enable
allow orjson to save non-string keys.

(Tests run with python 3.11)

FastAPI and orjson

The FastAPI documentation contains a guide to using orjson](https://fastapi.tiangolo.com/advanced/custom-response/?h=orjson#use-orjsonresponse) to serialize JSON responses.

This is particularly useful for APIs that expose machine learning models whose outputs are often numpy.ndarray

import numpy as np

from fastapi import FastAPI
from fastapi.responses import ORJSONResponse

app = FastAPI()

@app.get("/random-vector", response_class=ORJSONResponse)
async def get_random_vector():
    return ORJSONResponse(np.random.randn(100))

JSON Lines

When working with the JSON format, it's not uncommon to manipulate collections of objects.

[
    {"id": 0, "name": "toto"},
    {"id": 1, "name": "titi"},
]

To be valid, objects must be contained in a JSON list, hence the square brackets around the objects in the collection. However, this is not at all practical for reading large volumes of data, as you have to parse the entire file the entire file and load everything into memory.

This can be remedied by using the [JSON Lines] format (https://jsonlines.org/). This involves nothing more and nothing less than placing one JSON object per line, so that you can browse the objects without having to parse the entire
collection all at once.

{"id": 0, "name": "toto"}
{"id": 1, "name": "titi"}

The [jsonlines] library (https://jsonlines.readthedocs.io/en/latest/index.html) is very useful for manipulating
such files. It can also be combined with orjson.

import jsonlines
import orjson

with jsonlines.open("data/many_examples.jsonl", "r", loads=orjson.loads) as reader:
    for obj in reader:
        print(obj)

# {'id': 0, 'name': 'toto'}
# {'id': 1, 'name': 'titi'}

TL;DR

A brief summary of the tips seen here:

Use ensure_ascii=False when working with the standard json library.
Consider the orjson library for the performance and functionality it offers.
Consider the JSON Lines format for collections of JSON objects.

JSON dans les projets data science : Trucs & Astuces

Hakim — Wed, 06 Mar 2024 23:51:39 +0000

Quelques astuces et librairies utiles pour manipuler du json dans vos projets de data science.

Le module standard `json`

Python possède un module standard nommé json qui permet de manipuler rapidement des fichiers JSON

Chargement

import json

with open("data/example.json", "r") as f:
    data = json.load(f)

data
# [{'id': 0, 'content': [0.0, 0.0, 1.0]}, {'id': 1, 'content': [0.0, 1.0, 0.0]}]

Sauvegarde

Première astuce : lorsque l'on travaille avec des données textuelles l'option ensure_ascii=False est très utile
pour préserver, entre autres, les accents lors de la sauvegarde

with open("data/example.json", "w") as f:
    json.dump(data, f, ensure_ascii=False)

Deuxième astuce : l'option indent de la méthode dump permet d'indenter les données dans le fichier de sauvegarde

with open("data/example.json", "w") as f:
    json.dump(data, f, ensure_ascii=False, indent=2)

Les problématiques liées à numpy

On utilise très souvent numpy dans les projets de data science.
Or les objets numpy ne sont pas JSON-sérialisable et nécessite donc une conversion en objets standards python pour
être sauvegardés :

import numpy as np

data = np.array([[0., 0., 1.], [0., 1., 0.]])

with open("data/numpy.json", "w") as f:
    json.dump(data, f, ensure_ascii=False)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[12], line 6
      3 data = np.array([[0., 0., 1.], [0., 1., 0.]])
      5 with open("data/numpy.json", "w") as f:
----> 6     json.dump(data, f, ensure_ascii=False)

TypeError: Object of type ndarray is not JSON serializable

En convertissant le tableau en liste, on parvient à sauvegarder l'objet data :

with open("data/numpy.json", "w") as f:
    json.dump(data.tolist(), f, ensure_ascii=False)

Mais ce n'est pas très pratique...

Une solution est de créer un JSONEncoder personnalisé qui convertit les numpy.ndarray grâce à leur méthode tolist au moment de la sauvegarde :

class NumpyJSONEncoder(json.JSONEncoder):
    """JSONEncoder to store python dict or list containing numpy arrays"""

    def default(self, obj):
        """Transform numpy arrays into JSON serializable object such as list
        see : https://docs.python.org/3/library/json.html#json.JSONEncoder.default
        """
        if isinstance(obj, np.ndarray):
            return obj.tolist()

        return json.JSONEncoder.default(self, obj)

with open("data/numpy.json", "w") as f:
    json.dump(data, f, ensure_ascii=False, cls=NumpyJSONEncoder)

La librairie `orjson`

orjson est la librairie JSON la plus rapide disponible pour python. De
plus elle gère nativement les objets dataclass,
datetime, numpy
et UUID ce qui est très pratique.

Quelques éléments à retenir lorsque l'on travaille avec orjson :

Il n'y a pas de méthode load ni dump, il faut utiliser les méthodes loads et dumps à la place.
Il faut utiliser des flags pour utiliser certaines fonctionnalités tels que orjson.OPT_SERIALIZE_NUMPY pour sérialiser les objets numpy

import orjson

with open("data/example.json", "rb") as f:
    data = orjson.loads(f.read())

with open("data/example.json", "wb") as f:
    f.write(orjson.dumps(data, option=orjson.OPT_SERIALIZE_NUMPY))

A noter que l'on écrit dans le fichier json en binaire (d'où rb et wb).

Performances

orjson annonce une sérialisation des numpy.ndarray 4 à 12 fois plus rapide qu'avec la librairie standard. On peut le
vérifier sur un exemple en comparant les deux méthodes présentées plus haut :

data = {i: np.random.randn(100) for i in range(100)}

def save_json():
    with open("data/fast.json", "w") as f:
        json.dump(data, f, ensure_ascii=False, cls=NumpyJSONEncoder)

def save_orjson():
    with open("data/orfast.json", "wb") as f:
        f.write(orjson.dumps(data, option=orjson.OPT_SERIALIZE_NUMPY|orjson.OPT_NON_STR_KEYS))

%timeit save_json()
%timeit save_orjson()


# 15.5 ms ± 251 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# 1.15 ms ± 66.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Dans cet exemple orjson est plus de 10 fois plus rapide. A noter que l'option OPT_NON_STR_KEYS est utilisé pour
permettre à orjson de sauvegarder des clés qui ne sont pas des chaines de caractères.

(Tests exécutés avec python 3.11)

FastAPI et orjson

La documentation de FastAPI contient un guide pour utiliser orjson
pour sérialiser les réponses JSON.

C'est particulièrement pratique pour les API qui exposent des modèles de machine learning dont les sorties sont souvent
des numpy.ndarray

import numpy as np

from fastapi import FastAPI
from fastapi.responses import ORJSONResponse


app = FastAPI()


@app.get("/random-vector", response_class=ORJSONResponse)
async def get_random_vector():
    return ORJSONResponse(np.random.randn(100))

JSON Lines

Il arrive très souvent que l'on manipule des collections d'objets lorsque l'on travaille avec le format JSON.

[
    {"id": 0, "name": "toto"},
    {"id": 1, "name": "titi"},
]

Pour être valide, le objets doivent être contenus dans une liste JSON, d'où les crochets encadrant les objets de la
collection. Cependant cela n'est pas du tout pratique pour lire de gros volume de données puisqu'il faut parser
l'intégraliter du fichier et tout charger en mémoire.

Pour remédier à cela, on peut utilise le format JSON Lines. Il ne s'agit ni plus ni moins que
de placer un objet JSON par ligne de façon à pouvoir parcourir les objets sans devoir parser l'intégralité de la
collection d'un seul coup.

{"id": 0, "name": "toto"}
{"id": 1, "name": "titi"}

La librairie jsonlines est très pratique pour manipuler
de tels fichiers. De plus on peut la combiner avec orjson.

import jsonlines
import orjson

with jsonlines.open("data/many_examples.jsonl", "r", loads=orjson.loads) as reader:
    for obj in reader:
        print(obj)

# {'id': 0, 'name': 'toto'}
# {'id': 1, 'name': 'titi'}

TL;DR

Petit résumé des astuces vues ici :

Utilier ensure_ascii=False lorsqu'on travaille avec la librairie standard json
Considérer la librairie orjson pour les performances et les fonctionnalités qu'elle apporte
Penser au format JSON Lines pour les collections d'objets JSON

DEV Community: Hakim

Scraping de legifrance ? Considérez plutôt l'API

Obtention des identifiants OAuth

Obtention d'un token

Récupération d'un article à l'aide du token

pylegifrance

Train a Sentence-CamemBERT

Sentence-BERT

Finetuning a BERT model into a Sentence-BERT model

Prerequisites

Training data

Model evaluation

Sentence-BERT model distilled

An introduction to packaging in Python

Packages in Python

Python module

Package python

Distribute your package

setuptools

setup.py

setup.cfg

pyproject.toml

Build a distribution of your package

Version management

The layout

Going further

Bonus

Let's be happy

References

JSON in data science projects: tips & tricks

The standard json module

Loading

Backup

Issues related to numpy

The orjson library

Performance

FastAPI and orjson

JSON Lines

TL;DR

JSON dans les projets data science : Trucs & Astuces

Le module standard json

Chargement

Sauvegarde

Les problématiques liées à numpy

La librairie orjson

Performances

FastAPI et orjson

JSON Lines

TL;DR

The standard `json` module

The `orjson` library

Le module standard `json`

La librairie `orjson`