DEV Community: Líkið Geimfari

Generating secrets in Clojure

Líkið Geimfari — Tue, 04 Oct 2022 06:10:51 +0000

Generating secrets is a very important part of any security-sensitive application. In this article, I'm going to tell you about a low-level library that helps to generate secrets in Clojure.

If you're familiar with Python, you might have heard of secrets module from Python's standard library.

Well, the secrets.clj is just like Python's secrets, but for Clojure — it's a library designed to generate cryptographically strong random numbers suitable for managing data such as passwords, account authentication, security tokens, and related secrets.

Installation

Add secrets.clj to your project.clj file:

[likid_geimfari/secrets "1.1.1"]

then run lein deps to install it.

That's it, you're ready to go:

(ns example.core
  (:require [secrets.core]
            [secrets.tools]
            [secrets.constants]))

Usage

Typical use cases are:

Generating random numbers
Creating passwords, SMS-codes and OTP
Generating random tokens
Generating password recovery URLs and session keys

secrets.core/randbelow(n)

This function generates a secure random integer in the range [0, n), where n is the exclusive upper bound.

Example:

user=> (secrets.core/randbelow 9999)
7612

secrets.core/choice(seq)

This function returns a random element from a non-empty sequence or throws an exception if the sequence is empty.

user=> (secrets.core/choice ["bob" "alice" "eve"])
"eve"

secrets.core/choices(seq)

Just like secrets.core/choice, but this function returns a list of random elements picked from the sequence:

(secrets.core/choices ["bob" "alice" "eve"] 2)
("eve" "alice")

secrets.core/token-hex(nbytes)

Generates a secure random string in hexadecimal format. The string has nbytes random bytes, and each byte is converted to two hex digits.
If n-bytes are not supplied, a reasonable default gets used, which is 32.

user=> (secrets.core/token-hex(64))
"3a3e8e6636000dd3b7d39aa4316935f27c2f013d768f0c00f309efb453f34dbc673060db2cd8af288494892848"

secrets.core/token-urlsafe(nbytes)

Generates a secure random string in URL-safe format.

(defn generate-password-recovery-url [n]
  (str "https://mydomain.com/reset=" (secrets.core/token-urlsafe n)))
(generate-password-recovery-url 64)
"TItm04q8by00MRMcNBt7I3Yx-wSxyUa79isRLNyQJCd8K75RnqUahwcWA_rURBt1clknJiRGrubapGaUrEUnSw"

secrets.core/token-bytes(nbytes)

Generates a secure random string in bytes format.

(secrets.core/token-urlsafe(16))
#object["[B" 0x3b2454e9 "[B@3b2454e9"]

How many bytes should tokens use?

To be secure against brute-force attacks, tokens need to have sufficient randomness.
The number of random bits needed for a token depends on the application, but 256 bits
is considered to be cryptographically strong.

Personally, I would recommend using 64 bytes (512 bits).

Links

Original post: https://isaak.dev/2022/10/generating-secrets-in-clojure
Telegram Channel: https://t.me/software_dev_channel

Python libraries to make your code readable, reliable and maintainable

Líkið Geimfari — Mon, 24 Aug 2020 13:40:35 +0000

Experienced programmers understand perfectly well that in development they spend most of the time reading code and therefore they treat the process of writing code with the deepest trepidation (and sometimes with fanaticism). To write quality and maintainable code, you need to take the time to write tests and integrate QA tools. There is a whole technique aimed at test-driven development (TDD) and I will not devote this article to the topic of testing as such. Tests are absolutely necessary and there is nothing to discuss. In this article, we are going to talk about tools that help you write quality Python code.

Testing Frameworks

pytest is a framework that makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries.

Features

Detailed info on failing assert statements (no need to remember self.assert* names);
Auto-discovery of test modules and functions;
Modular fixtures for managing small or parametrized long-lived test resources;
Can run unittest (including trial) and nose test suites out of the box;
Python 3.5+ and PyPy 3;
Rich plugin architecture, with over 315+ external plugins and thriving community;

Hypothesis is a family of testing libraries that let you write tests parametrized by a source of examples. A Hypothesis implementation then generates simple and comprehensible examples that make your tests fail. This simplifies writing your tests and makes them more powerful at the same time,
by letting software automate the boring bits and do them to a higher standard than a human would, freeing you to focus on the higher-level test logic.

Robot Framework is a generic open-source automation framework for acceptance testing, acceptance test-driven development (ATDD), and robotic process automation (RPA). It has simple plain text syntax and it can be extended easily with libraries implemented using Python or Java.

Test Runners

tox is a command-line driven CI frontend and development task automation tool.

tox creates virtual environments for all configured so-called testenvs, it then installs the project and other necessary dependencies and runs the configured set of commands:

E2E Testing (GUI / Frontend)

Selenium is an umbrella project encapsulating a variety of tools and libraries enabling web browser automation. Selenium specifically provides an infrastructure for the W3C WebDriver specification — a platform and language-neutral coding interface compatible with all major web browsers.

Locust is an easy to use, scriptable, and scalable performance testing tool. You define the behavior of your users in regular Python code, instead of using a clunky UI or domain-specific language. This makes Locust infinitely expandable and very developer-friendly.

TestCafe is a Node.js tool to automate end-to-end web testing. Write
tests in JS or TypeScript, run them and view results.

Works on all popular environments: TestCafe runs on Windows, MacOS, and Linux. It supports desktop, mobile, remote and cloud browsers (UI or headless).
1 minute to set up: You do not need WebDriver or any other testing software. Install TestCafe with one command, and you are ready to test: npm install -g testcafe
Free and open source: TestCafe is free to use under the MIT license. Plugins provide custom reports, integration with other tools, launching tests from IDE, etc. You can use the plugins made by the GitHub community or make your own.

Usage example:

PyAutoGUI is a cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard.

Fake Data

Mimesis is a high-performance fake data generator for Python, which
provides data for a variety of purposes in a variety of languages. The fake data could be used to populate a testing database, create fake API endpoints, create JSON and XML files of arbitrary structure, anonymize data taken from production and etc.

The key features are:

Performance: The fastest data generator available for Python.
Extensibility: You can create your own data providers and use them with Mimesis.
Generic data provider: The simplified access to all the providers from a single object.
Multilingual: Supports data for a lot of languages.
Data variety: Supports a lot of data providers for a variety of purposes.
Schema-based generators: Provides an easy mechanism to generate data by the schema of any complexity.
Country-specific data providers: Provides data specific only for some countries.

Usage:

>>> from mimesis import Person
>>> person = Person('en')

>>> person.full_name()
'Brande Sears'

>>> person.email(domains=['mimesis.name'])
'roccelline1878@mimesis.name'

>>> person.email(domains=['mimesis.name'], unique=True)
'f272a05d39ec46fdac5be4ac7be45f3f@mimesis.name'

>>> person.telephone(mask='1-4##-8##-5##3')
'1-436-896-5213'

Generating data using schemas:

from mimesis.schema import Field, Schema
from mimesis.enums import Gender

_ = Field('en')

schema = Schema(schema=lambda: {
    'id': _('uuid'),
    'name': _('text.word'),
    'version': _('version', pre_release=True),
    'timestamp': _('timestamp', posix=False),
    'owner': {
        'email': _('person.email', domains=['test.com'], key=str.lower),
        'token': _('token_hex'),
        'creator': _('full_name', gender=Gender.FEMALE)},
})
data = schema.create(iterations=1)

The result:

[   {   'id': 'bd69b0b1-ac7f-42a7-a6b7-0fe04e8847d3',
        'name': 'certified',
        'owner': {   'creator': 'Dorthey Ramsey',
                     'email': 'birretta1894@test.com',
                     'token': '9cddc4363819109adf4a3cc0065bfdb8c38c5ce8244af4da68e75a45cf5bdc12'},
        'timestamp': '2005-04-18T13:09:37Z',
        'version': '9.9.0-rc.7'}]

Mocking

FreezeGun is a library that allows your Python tests to travel through time by mocking the datetime module.

Once the decorator or context manager have been invoked, all calls to datetime.datetime.now(), datetime.datetime.utcnow(), datetime.date.today(), time.time(), time.localtime(), time.gmtime(), and time.strftime() will return the time that has been frozen.

HTTPretty is an HTTP client mocking tool for Python - inspired by Fakeweb for Ruby.

Common use cases:

Test-driven development of API integrations
Fake responses of external APIs
Record and playback HTTP requests

responses is a utility library for mocking out the requests Python library.

Example of usage:

import responses
import requests

@responses.activate
def test_simple():
    responses.add(responses.GET, 'http://twitter.com/api/1/foobar',
                  json={'error': 'not found'}, status=404)

    resp = requests.get('http://twitter.com/api/1/foobar')

    assert resp.json() == {"error": "not found"}

    assert len(responses.calls) == 1
    assert responses.calls[0].request.url == 'http://twitter.com/api/1/foobar'
    assert responses.calls[0].response.text == '{"error": "not found"}'

Code coverage

Coverage.py measures code coverage, typically during test execution. It uses the code analysis tools and tracing hooks provided in the Python standard library to determine which lines are executable, and which have been executed.

Object Factories

factory_boy is a fixtures replacement based on thoughtbot's factory_bot.

As a fixtures replacement tool, it aims to replace static, hard to maintain fixtures with easy-to-use factories for complex objects.

Code Style

wemake-python-styleguide is actually a flake8 plugin with some other plugins as dependencies.

pycodestyle is a tool to check your Python code against some of the style conventions in PEP8.

Features:

Plugin architecture: Adding new checks is easy.
Parseable output: Jump to error location in your editor.
Small: Just one Python file, requires only stdlib. You can use just the pycodestyle.py file for this purpose.
Comes with a comprehensive test suite.

Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.

Typing

mypy is an optional static type checker for Python. You can add type hints (PEP 484) to your Python programs, and use mypy to type check them statically. Find bugs in your programs without even running them!

Here is a small example to whet your appetite (Python 3):

from typing import Iterator

def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a + b

Pyre is a performant type checker for Python compliant with PEP 484.
Pyre can analyze codebases with millions of lines of code incrementally – providing instantaneous feedback to developers as they write code.

Pyre ships with Pysa, a security-focused static analysis tool we've built on top of Pyre that reasons about data flows in Python applications. Please refer to our documentation to get started with our security analysis.

Repository: https://github.com/facebook/pyre-check

Typeshed contains external type annotations for the Python standard library
and Python builtins, as well as third party packages as contributed by people external to those projects.

django-stubs contains type stubs and a custom mypy plugin to provide more precise static types and type inference for Django framework. Django uses some Python "magic"
that makes having precise types for some code patterns problematic. This is why we need this project. The final goal is to be able to get precise types for most common patterns.

returns gonna make your functions return something meaningful, typed, and safe!

Features:

Brings functional programming to Python land
Provides a bunch of primitives to write declarative business logic
Enforces better architecture
Fully typed with annotations and checked with mypy, PEP561 compatible
Adds emulated Higher Kinded Types support
Has a bunch of helpers for better composition
Pythonic and pleasant to write and to read 🐍
Support functions and coroutines, framework agnostic
Easy to start: has lots of docs, tests, and tutorials

That's it for now!

You can read more in my personal blog: https://isaak.dev

Twitter: https://twitter.com/likid_geimfari
GitHub: https://github.com/lk-geimfari
Telegram: @the_art_of_development

Thoughts about naming variables and methods

Líkið Geimfari — Sat, 11 Jul 2020 12:12:21 +0000

The proper naming of variables, functions, methods, and classes is one of the most important attributes of elegant and clean code, that clearly reflects the intentions of the programmer, without assumptions about what was meant.

In this article, we will talk about code that is the exact opposite of the one described above - about code that was written carelessly or thoughtlessly.
This article is a small confession, because I, like any other programmer, have also written such code in the past (in fact, I still write the bad code sometimes, though refactoring makes it much better). This is nothing terrible as long as we understand that we need to work on it.

Let's start.

Variables

One of the most annoying kinds of variables is variables that give a false impression of the nature of the data they store.

The requests library is extremely popular among Python developers, and if you have ever been looking for something related to requests, you must have come across with something like this:

import requests

req = requests.get('https://api.example.org/endpoint')
req.json()

Whenever I see this, I feel annoyed and even not because of the shortened name, but because the name of the variable does not match what is stored in this variable.

When you make a request (requests.Request), you get a response (requests.Response), so reflect this in your code:

response = requests.get('https://api.example.org/endpoint')
response.json()

Not r, not res, not resp and certainly not req, exactly response. res, r, resp – these are all variables whose contents can be understood only by looking at their definitions, and why jump to the definitions
when you can initially give a suitable name?

Let's look at another example, but now from Django:

users_list = User.objects.filter(age__gte=22)

When you see users_list somewhere in the code, you quite rightly expect that you can do this:

users_list.append(User.objects.get(pk=3))

but no, you can't do this, since .filter() returns a QuerySet:

Traceback (most recent call last):
# ...
# ...
AttributeError: 'QuerySet' object has no attribute 'append'

If it is very important for you to specify a suffix, then specify at least one that reflects the real situation:

users_queryset = User.objects.all()
users_queryset.order_by('-age')

also okay, because such abbreviations (_qs) are usual for Django:

users_qs = User.objects.all()

If you really want to write exactly _list, then take care that the list really gets into the variable:

users_list = list(User.objects.all())

Indicating a type of data that a variable contains is often a bad idea, especially when you deal with dynamic languages, such as Python. In cases when it is very necessary to note that the object is a container data type, it is enough to simply indicate the name of the variable in the plural:

users = User.objects.all()

Consider another example:

info_dict = {'name': 'Isaak', 'age': 25}
# ...
# ... 
info_dict = list(info_dict)
# ...
# ...

You see a dict and you might want to do this:

for key, value in info_dict.items():
    print(key, value)

Instead, you'll get an exception, because you were misled, and you will understand this only if you go to the definition of the variable and read the entire code from top to bottom, right down to the section from which you started the jump — this is the price of such variables.

Thus, when you indicate in the variable name the type of data stored in it, you are essentially a guarantee that this variable must contain the specified data type at any time during the execution of the program. Why should you take this responsibility if it is the direct responsibility of the interpreter or compiler? You shouldn't! Better to spend time thinking of a good variable name than trying to figure out why the variables do not behave as you expect.

In the example above, the choice of the name of a variable is rather bad, and you could give a name that more accurately expresses the context (no need to be afraid to use names related to the subject area), but even in this case, you could make this code better:

info_dict = {'name': 'Isaak', 'age': 25}
# ...
# ... 
info_keys = list(info_dict)
# ...
# ...

or even like this, which is more idiomatic:

info_dict = {'name': 'Isaak', 'age': 25}
# ...
# ... 
info_keys = info_dict.keys()
# ...
# ...

Another type of annoying variable is ones with a shortened name.

Let's go back to requests and consider this code:

s = requests.Session()
# ...
# ... 
s.close()

This is an example of an unnecessary shortening for a variable name. This is a terrible practice, and its horror becomes even more apparent when such code takes up more than 10-15 lines of code.

It is much better to write as is, namely:

session = requests.Session()
# ...
# ...
session.get('https://api.example.org/endpoint')
# ...
# ...
session.close()

with requests.Session() as session:
    session.get('https://api.example.org/endpoint')

You may argue that this is a more verbose option, but I will answer you that it pays off when you read the code and immediately understand that session is a Session.

Will you understand it by variable s without looking at its definition?

Methods

Smart naming of functions and methods is something that comes only with experience in designing an API, and therefore you can often find cases where methods do not behave as you expect.

Consider an example:

>>> person = Person()
>>> person.has_publications()
['Post 1', 'Post 2', 'Post 3']

We expressed a very clear-cut question in our code: “Does this person have publications?”, but what kind of answer did we get? Did we ask for a list of publications of a person?

The name of this method implies that the return value must be of Boolean type, namely True or False:

>>> person = Person()
>>> person.has_publications()
True

We can you can use a more appropriate method name for getting publications:

>>> person.get_publications()
['Post 1', 'Post 2', 'Post 3']

>>> person.publications()
['Post 1', 'Post 2', 'Post 3']

We often like to call programming a creative activity, and it really is. However, if you write not readable code, and then justify it with “creativity”, then I have bad news for you.

You can read more in my personal blog: https://isaak.dev

Twitter: https://twitter.com/likid_geimfari
GitHub: https://github.com/lk-geimfari
Telegram: @the_art_of_development

Generating mock data with Mimesis: Part II

Líkið Geimfari — Tue, 29 Aug 2017 15:55:24 +0000

We have already published how to generate mock data with the help of a Python library — Mimesis. The article you are reading now is the continuation of the previous one, therefore, we will not be going over the basics again. In case you missed out on the first article or you felt lazy at the time, you might want to go back to it now since this article assumes you are familiar with the library. Here we are going to speak about best practices and a number of most useful features of the library.

Note

First of all we would like to point out that Mimesis wasn’t developed to be used with a certain database or ORM. The main problem the library solves is generating valid data. Consequently, while there are no rigid rules of working with the library, here are a few recommendations that will help you keep your testing environment in order and will avert growth of entropy within your project. Recommendations are quite simple and are fully in tune with the Python spirit (if you disagree, feel free to let us know).

Despite the previous note that the library isn’t to be used with a certain database or ORM, the need for test data usually occurs in web-apps that perform certain operations (mostly CRUD) with a database. We have some advice on organizing test data generation for web-apps.
Functions responsible for data generation and importing it to the database should be kept close to the models, or even better as statistical methods of the model they are related to, as in the example of _bootstrap() method from the previous article. This is necessary to avoid running around files when the model structure changes and you need to add a new filed. Model Patient() from the previous article illustrates the idea:



class Patient(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    email = db.Column(db.String(120), unique=True)
    phone_number = db.Column(db.String(25))
    full_name = db.Column(db.String(100))
    weight = db.Column(db.String(64))
    height = db.Column(db.String(64))
    blood_type = db.Column(db.String(64))
    age = db.Column(db.Integer)

    def __init__(self, **kwargs):
        super(Patient, self).__init__(**kwargs)

    @staticmethod
    def _bootstrap(count=500, locale='en', gender):
        from mimesis import Personal
        person = Personal(locale)

        for _ in range(count):
            patient = Patient(
                email=person.email(),
                phone_number=person.telephone(),
                full_name=person.full_name(gender=gender),
                age=person.age(minimum=18, maximum=45),
                weight=person.weight(),
                height=person.height(),
                blood_type=person.blood_type()
            )

            db.session.add(patient)
            try:
                db.session.commit()
            except IntegrityError:
                db.session.rollback()

Keep in mind that the example above is a model of a Flask-app used by SQLAlchemy. Organizing mock data generators for such apps with the use of different frameworks is done in the same way.

Creating objects

If your app requires data in one particular language, it’s preferable to use class Generic(), giving access to all class providers through a single object, rather than through multiple separate class providers. Using Generic() will allow you to get rid of several extra lines of code.

Correct:



>>> from mimesis import Generic
>>> generic = Generic('ru')

>>> generic.personal.username()
'sherley3354'

>>> generic.datetime.date()
'14-05-2007'

Incorrect:



>>> from mimesis import Personal, Datetime, Text, Code

>>> personal = Personal('ru')
>>> datetime = Datetime('ru')
>>> text = Text('ru')
>>> code = Code('ru')

Still correct:



>>> from mimesis import Personal

>>> p_en = Personal('en')
>>> p_sv = Personal('sv')
>>> # …

It means that importing class providers separately makes sense only if you limit yourself to the data available through the class you imported, otherwise it’s better to use Generic().

Inserting data into database

If you need to generate data and import it into a database we strongly recommend generating data in chunks rather than 600k at once. Keep in mind the possible limitations of databases, ORM, etc. The smaller the generated data chunks are, the faster the process will go.

Good:



>>> Patient()._bootstrap(count=2000, locale='de')

Very bad:



>>> Patient()._bootstrap(count=600000, locale='de')

Importing images

Class Internet() boasts of several methods which generate image links (more details here). Links to images locate on remote servers would be enough, however, if you still want to have a number of random images locally, you can download images generated by the respective class Internet() methods with the help of function download_image() from model utils:



>>> from mimesis import Internet
>>> from mimesis.utils import download_image

>>> net = Internet()

>>> img_url = net.stock_image(category='food', width=1920, height=1080)
>>> download_image(url=img_url, save_path='/some/path/')

User providers

The library supports a vast amount of data and in most cases this would be enough. For those who want to create their own providers with more specific data. This can be done like this:



>>> class SomeProvider():
...     class Meta:
...         name = "some_provider"
...
...     @staticmethod
...     def one():
...         return 1

>>> class Another():
...     @staticmethod
...     def bye():
...         return "Bye!"

>>> generic.add_provider(SomeProvider)
>>> generic.add_provider(Another)

>>> generic.some_provider.one()
1

>>> generic.another.bye()
'Bye!'

You can also add multiple providers:



>>> generic.add_providers(SomeProvider, Another)
>>> generic.some_provider.one()
1
>>> generic.another.bye()
'Bye!'

Everything is pretty easy and self-explanatory here, therefore, we will only clarify one moment — attribute name, class Meta is the name of a class through which access to methods of user-class providers is carried out. By default class name is the name of the class in the lower register.

Built-in providers

Most countries, where only one language is official, have data typical only for these particular countries. For example, CPF for Brazil (pt-br), SSN for USA (en). This kind of data can cause discomfort and meddle with the order (or at least annoy) by being present in all the objects regardless of the chosen language standard. You can see that for yourselves by looking at the example (the code won’t run):



>>> from mimesis import Personal
>>> person = Personal('en')

>>> person.ssn()
>>> person.cpf()

We bet everyone would agree that this does not look too good. Perfectionists, as we are, have taken care of this in a way that some specific regional provider would not bother other providers for other regions. For this reason, class providers with locally-specific data are separated into a special sub-package (mimesis.builtins) for keeping a common class structure for all languages and their objects.

Here’s how it works:



>>> from mimesis import Generic
>>> from mimesis.builtins import BrazilSpecProvider

>>> generic = Generic('pt-br')
>>> generic.add_provider(BrazilProvider)
>>> generic.brazil_provider.cpf()
'696.441.186-00'

If you want to change default name of built-in provider, just change value of attribute name, class Meta of the builtin provider:



>>> BrazilSpecProvider.Meta.name = 'brasil'
>>> generic.add_provider(BrazilSpecProvider)
>>> generic.brasil.cpf()
'019.775.929-70'

Or just inherit the class and override the value of attribute name of class Meta of the provider (in our case this is BrazilSpecProvider()) :



>>> class Brasil(BrazilSpecProvider):
...
...     class Meta:
...         name = "brasil"
...
>>> generic.add_provider(Brasil)
>>> generic.brasil.cnpj()
'55.806.487/7994-45'

Generally, you don’t need to add built-it classes to the object Generic(). It was done in the example with the single purpose of demonstrating in which cases you should add a built-in class provider to the object Generic(). You can use it directly, as shown below:



>>> from mimesis.builtins import RussiaSpecProvider
>>> ru = RussiaSpecProvider()

>>> ru.patronymic(gender='female')
'Петровна'

>>> ru.patronymic(gender='male')
'Бенедиктович'

Generate data by schema

Mimesis support generating data by schema from version 0.0.5. This feature is still at an early stage of development. Actually is very easy feature, here how it works:



>>> from mimesis.schema import Schema
>>> schema = Schema('en')

>>> schema.load(schema={
...     "id": "cryptographic.uuid",
...     "name": "text.word",
...     "version": "development.version",
...     "owner": {
...         "email": "personal.email",
...         "token": "cryptographic.token",
...         "creator": "personal.full_name"
...     }
... }).create(iterations=2)

>>> # or you can load data from json file:
>>> schema.load(path='schema.json').create(iterations=2)

Result:



[
  {
    "id": "790cce21-5f75-2652-2ee2-f9d90a26c43d",
    "name": "container",
    "owner": {
      "email": "anjelica8481@outlook.com",
      "token": "0bf924125640c46aad2a860f40ec4b7f33a516c497957abd70375c548ed56978",
      "creator": "Ileen Ellis"
    },
    "version": "4.11.6"
  },
  ...
]

Which type of data do you usually need in your work? What is the library missing? We will be very happy to hear your suggestions and comments.

Useful links

Link to the first part of article.

Link to the repository of Mimesis

Post Scriptum

An article written by @likid_geimfari and translated to English by @wemake-services.

Generating mock data using Mimesis: Part I

Líkið Geimfari — Thu, 24 Aug 2017 13:56:22 +0000

The ability to generate mock but valid data comes in handy in app development, where you need to work with databases. Filling in the database by hand is a time-consuming and tedious process, which can be done in three stagesâ€Š–â€Šgathering necessary information, post-processing the data and coding the data generator itself. It gets really complicated when you need to generate not just 10–15 users, but 100–150 thousand users (or other types of data). In this article as well as the two following ones we will introduce you to a tool, which immensely simplifies generating mock data, initial database loading and testing in general.

Mimesis is a Python library, which helps generate mock data for various purposes. The library was written with the use of tools from the standard Python library, and therefore, it doesn’t have any side dependencies. Currently the library supports 32 languages and 21 class providers, supplying various data.

Installation

A normal way to install mimesis is via pip:

âžœ  ~ pip install mimesis

If for some reason you can not install the package with the help of pip, try installing it manually, as shown below:

(venv) âžœ  ~ git clone https://github.com/lk-geimfari/mimesis.git
(venv) âžœ  cd mimesis/
(venv) âžœ  python3 setup.py install
# or
(venv) âžœ make install

Generating data

Initially, we planned on showing data generation using the example of a small web-application Flask, but we decided against it because not everyone is familiar with Flask nor they are willing to change that. Therefore, we are going to showcase that solely on Python. In case you want to transfer everything to your project on Flask or Django, you simply need to define a static method that will run all the manipulations related to the current model and call it when you need the initial database loading, as demonstrated in the example below.
Model for Flask (Flask-SQLAlchemy) would look like this:

class Patient(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    email = db.Column(db.String(120), unique=True)
    phone_number = db.Column(db.String(25))
    full_name = db.Column(db.String(100))
    weight = db.Column(db.String(64))
    height = db.Column(db.String(64))
    blood_type = db.Column(db.String(64))
    age = db.Column(db.Integer)

    def __init__(self, **kwargs):
        super(Patient, self).__init__(**kwargs)

    @staticmethod
    def _bootstrap(count=500, locale='en', gender):
        from mimesis import Personal
        person = Personal(locale)

        for _ in range(count):
            patient = Patient(
                email=person.email(),
                phone_number=person.telephone(),
                full_name=person.full_name(gender=gender),
                age=person.age(minimum=18, maximum=45),
                weight=person.weight(),
                height=person.height(),
                blood_type=person.blood_type()
            )

            db.session.add(patient)
            try:
                db.session.commit()
            except IntegrityError:
                db.session.rollback()

Now let’s transition to shell-mode:

(venv) âžœ python3 manage.py shell

And generate data. Beforehand, we need to make sure that the database and the model in question are available.

>>> db
<SQLAlchemy engine='sqlite:///db.sqlite'>

>>> Patient

<class 'app.models.Patient'>

>>> Patient()._bootstrap(count=40000, locale='en') # generate 40Ðº entries in English.

Introduction

It is worth noting that we will be showing the basic capabilities of the library and we will be using a few most common class providers, since there are too many of them to cover each one in detail. If the article sparks your interest to the library you can visit the useful links listed in the end of the article and find out more.

The library is pretty simple. All you need to do to start working with the data is to create a class provider. The most common type of data in apps are personal users’ data, such as name, last name, credit card info, etc. There is a special class provider for this type of data â€Š– â€ŠPersonal(), which takes the code from the language standard in the form of a line as shown below:

>>> from mimesis import Personal
>>> person = Personal('is')
>>> for _ in range(0, 3):
...     person.full_name(gender='male')
...
'Karl BrynjÃºlfsson'
'RÃ¶gnvald EiÃ°sson'
'VÃ©steinn RÃkharÃ°sson'

Almost every web-application requires e-mail for registration. Naturally, the library supports the ability to generate e-mails with the help of email() method of the Personal() class, as below:

>>> person.email(gender='female')
'lvana6108@gmail.com'

>>> person.email(gender='male')
'john2454@yandex.com'

There is a little problem with the method above, which may cause the code to be slightly “dirty in case the app uses more than one type of class providers. In such situation you should use object Generic(), which grants access to all providers from one single object:

>>> from mimesis import Generic
>>> g = Generic('pl') # pl – code of Poland (ISO 639-1).
>>> g.personal.full_name()
'LonisÅ‚awa PodsiadÅ‚o'
>>> g.datetime.birthday(readable=True)
'Listopad 11, 1997'
>>> g.code.imei()
'011948003071013'
>>> g.food.fruit()
'Cytryna'
>>> g.internet.http_method()
'PUT'
>>> g.science.math_formula()
'A = (h * (a + b)) / 2'

Combining data gives you a vast field for experimentation. For example, you can create mock (female) Visa (Maestro, MasterCard) credit card holders:

>>> user = Personal('en')
>>>
>>> def get_card(sex='female'):
...     owner = {
...       'owner': user.full_name(sex),
...       'exp_date': user.credit_card_expiration_date(maximum=21),
...       'number': user.credit_card_number(card_type='visa')
...       }
...     return owner
>>>
>>> for _ in range(0, 3):
...     get_card()
...
{'exp_date': '02/20', 'owner': 'Laverna Morrison', 'card_number': '4920 3598 2121 3328'}
{'exp_date': '11/19', 'owner': 'Melany Martinez', 'card_number': '4980 9423 5464 1201'}
{'exp_date': '01/19', 'owner': 'Cleora Mcfarland', 'card_number': '4085 8037 5801 9703'}

As mentioned above, the library supports more than 21 class providers with data for all possible situations (if not, your PR with corrections of such an awful injustice are more than welcome). For example, if you are working on an app dedicated to transportation and logistics and you need to generate transportation models, you can easily do this by using Transport() class provider, which contains data related to transportation:

>>> from mimesis import Transport
>>> trans = Transport()
>>> for _ in range(0, 5):
...     trans.truck()
...
'Seddon-2537 IM'
'Karrier-7799 UN'
'Minerva-5567 YC'
'Hyundai-2808 XR'
'LIAZ-7174 RM'

Or you could indicate the transport mask model:

>>> for _ in range(0, 5):
...     # Here # (sharp) - placeholder for numbers, @ - for letters
...     trans.truck(model_mask="##@")
...
'Henschel-16G'
'Bean-44D'
'Unic-82S'
'Ford-05Q'
'Kalmar-58C'

Quite often when testing web-applications (blog would be an excellent example) you need to generate text data (text, sentences, tags, etc.). Manually inputting the text is long and boring, and Mimesis allows you to avoid this thanks to a class provider Text():

>>> from mimesis import Text
>>> text = Text('en')
>>> text.text(quantity=3)

'Language includes means for creating light parallel processes  and their interactions via exchanging asynchronous messages according to the actors’ model. Python supports several programming paradigms, including structural, object-oriented, functional, imperative and aspect-oriented. For instance, some functions that use comparison of examples to choose one calculating option or extracting data points looks similar to an equation.'

You can get a list of random words:

>>> text = Text('pt-br')
>>> text.words(quantity=5)
['poder', 'de', 'maior', 'sÃ³', 'cima']

Generate a street name:

>>> from mimesis import Address
>>> address = Address('en')
>>> address.address()
'77 Shephard Trace'

Get a name of a state/area/province, which is related to the chosen language.Â In this case it is an state of the USA:

>>> address.state()
'Texas'

The library also has means to Romanize Cyrillic languages (for the moment only Russian and Ukrainian are supported):

>>> from mimesis.decorators import romanized
>>> @romanized('ru')
... def name_ru():
...     return 'Ð’ÐµÑ€Ð¾Ð½Ð¸ÐºÐ° Ð”ÐµÐ½Ð¸ÑÐ¾Ð²Ð°'
...
>>> @romanized('uk')
>>> def name_uk():
...     return 'Ð•Ð¼Ñ–Ð»Ñ–Ñ ÐÐºÑƒÐ»ÐµÐ½ÐºÐ¾'
...
>>> name_ru()
'Veronika Denisova'
>>> name_uk()
'EmÑ–lÑ–ja Akulenko'

In reality there are a lot of possibilities and you can come up with a huge number of great use-cases, where the data would look more useful than in our examples. We are looking forward to getting them from our users. And we would be happy to read how you are successfully applying the library to your projects.

Useful links

The second part of the article.
Project available on GitHub.
Documentation available on ReadTheDocs.

DEV Community: Líkið Geimfari

Generating secrets in Clojure

Installation

Usage

secrets.core/randbelow(n)

secrets.core/choice(seq)

secrets.core/choices(seq)

secrets.core/token-hex(nbytes)

secrets.core/token-urlsafe(nbytes)

secrets.core/token-bytes(nbytes)

How many bytes should tokens use?

Links

Python libraries to make your code readable, reliable and maintainable

Testing Frameworks

Test Runners

E2E Testing (GUI / Frontend)

Fake Data

Mocking

Code coverage

Object Factories

Code Style

Typing

Thoughts about naming variables and methods

Variables

Methods

Further reading

Generating mock data with Mimesis: Part II

Note

Creating objects

Inserting data into database

Importing images

User providers

Built-in providers

Generate data by schema

Useful links

Post Scriptum

Generating mock data using Mimesis: Part I

Installation

Generating data

Introduction

Useful links