DEV Community: Aaron Harris

Django Database Migrations: A Comprehensive Overview

Aaron Harris — Wed, 23 Oct 2019 19:30:55 +0000

Django Database Migrations: A Comprehensive Overview
by Damian Hites

The Django web framework is designed to work with an SQL-based relational database backend, most commonly PostgreSQL or MySQL. If you’ve never worked directly with a relational database before, managing how your data is stored/accessed and keeping it consistent with your application code is an important skill to master.

You’ll need a contract between your database schema (how your data is laid out in your database) and your application code, so that when your application tries to access data, the data is where your application expects it to be. Django provides an abstraction for managing this contract in its ORM (Object-Relational Mapping).

Over your application’s lifetime, it’s very likely that your data needs will change. When this happens, your database schema will probably need to change as well. Effectively, your contract (in Django’s case, your Models) will need to change to reflect the new agreement, and before you can run the application, the database will need to be migrated to the new schema.

Django’s ORM comes with a system for managing these migrations to simplify the process of keeping your application code and your database schema in sync.

Django’s database migration solution

Django’s migration tool simplifies the manual nature of the migration process described above while taking care of tracking your migrations and the state of your database. Let’s take a look at the three-step migration process with Django’s migration tool.

1. Change the contract: Django’s ORM

In Django, the contract between your database schema and your application code is defined using the Django ORM. You define a data model using Django ORM’s models and your application code interfaces with that data model.

When you need to add data to the database or change the way the data is structured, you simply create a new model or modify an existing model in some way. Then you can make the required changes to your application code and update your unit tests, which should verify your new contract (if given enough testing coverage).

2. Plan for change: generate migrations

Django maintains the contract largely through its migration tool. Once you make changes to your models, Django has a simple command that will detect those changes and generate migration files for you.

3. Execute: apply migrations

Finally, Django has another simple command that will apply any unapplied migrations to the database. Run this command any time you are deploying your code to the production environment. Ideally, you’ll have deploy scripts that would run the migration command right before pushing your new code live.

Tracking changes with Django

Django takes care of tracking migrations for you. Each generated migration file has a unique name that serves as an identifier. When a migration is applied, Django maintains a database table for tracking applied migrations to make sure that only unapplied migrations are run.

The migration files that Django generates should be included in the same commit with their corresponding application code so that it’s never out-of-sync with your database schema.

Rolling back with Django

Django has the ability to rollback to a previous migration. The auto-generated operations feature built-in support for reversing an operation. In the case of a custom operation, it’s on you to make sure the operation can be reversed to ensure that this functionality is always available.

A simple Django database migrations example

Now that we have a basic understanding of how migrations are handled in Django, let’s look at a simple example of migrating an application from one state to the next. Let’s assume we have a Django project for our blog and we want to make some changes.

First, we want to allow for our posts to be edited before publishing to the blog. Second, we want to allow people to give feedback on each post, but we want to give them a curated list of options for that feedback. In anticipation of those options changing, we want to define them in our database rather than in the application code.

The initial Django application

For the purposes of demonstration, we’ll setup a very basic Django project called Foo:

django-admin startproject foo

Within that project, we’ll set up our blogging application. From inside the project’s base directory: ./manage.py startapp blog

INSTALLED_APPS = [
    ...
    'blog',
]

In blog/models.py we can define our initial data model:

class Post(models.Model):
    slug = models.SlugField(max_length=50, unique=True)
    title = models.CharField(max_length=50)
    body = models.TextField()

In our simple application, the only model we have represents a blog post. It has a slug for uniquely identifying the post, a title, and the body of the post.

Now that we have our initial data model defined, we can generate the migrations that will set up our database: ./manage.py makemigrations

Notice that the output of this command indicates that a new migration file was created at

blog/migrations/0001_initial.py containing a command to CreateModel name=‘Post’.

If we open the migration file, it will look something like this:

# Generated by Django 2.2 on 2019-04-21 18:04

from django.db import migrations, models

class Migration(migrations.Migration):
    initial = True

    dependencies = [
    ]

    operations = [
        migrations.CreateModel(
            name='Post',
            fields=[
                ('id', models.AutoField(
                    auto_created=True, 
                    primary_key=True, 
                    serialize=False, 
                    verbose_name='ID'
                )),
                ('slug', models.SlugField(unique=True)),
                ('title', models.CharField(max_length=50)),
                ('body', models.TextField()),
            ],
        ),
    ]

Most of the migration’s contents are pretty easy to make sense of. This initial migration was auto-generated, has no dependencies, and has a single operation: create the Post Model.

Now let’s set up an initial SQLite database with our data model:

./manage.py migrate

The default Django configuration uses SQLite3, so the above command generates a file called db.sqlite3 in your project’s root directory. Using the SQLite3 command line interface, you can inspect the contents of the database and of certain tables.

To enter the SQLite3 command line tool run:

sqlite3 db.sqlite3

Once in the tool, list all tables generated by your initial migration:

sqlite> .tables

Django comes with a number of initial models that will result in database tables, but the 2 that we care about right now are blog_post, the table corresponding to our Post Model, and django_migrations, the table Django uses to track migrations.

Still in the SQLite3 command line tool, you can print the contents of the django_migrations table:

sqlite> select * from django_migrations;

This will show all migrations that have run for your application. If you look through the list, you’ll find a record indicating that the 0001_initial migration was run for the blog application. This is how Django knows that your migration has been applied.

Changing the Django data model

Now that the initial application is setup, let’s make changes to the data model. First, we’ll add a field called published_on to our Post Model. This field will be nullable. When we want to publish something, we can simply indicate when it was published.

Our new Post Model will now be:

from django.db import models

class Post(models.Model):
    slug = models.SlugField(max_length=50, unique=True)
    title = models.CharField(max_length=50)
    body = models.TextField()
    published_on = models.DateTimeField(null=True, blank=True)

Next, we want to add support for accepting feedback on our posts. We want 2 models here: one for tracking the options we display to people, and one for tracking the actual responses

from django.conf import settings
from django.db import models

class FeedbackOption(models.Model):
    slug = models.SlugField(max_length=50, unique=True)
    option = models.CharField(max_length=50)

class PostFeedback(models.Model):
    user = models.ForeignKey(
        settings.AUTH_USER_MODEL, related_name='feedback',
        on_delete=models.CASCADE
    )
    post = models.ForeignKey(
        'Post', related_name='feedback', on_delete=models.CASCADE
    )
    option = models.ForeignKey(
        'FeedbackOption', related_name='feedback', on_delete=models.CASCADE
    )

Generate the Django database migration

With our model changes done, let’s generate our new migrations:

./manage.py makemigrations

Notice that this time, the output indicates a new migration file, blog/migrations/0002_auto_<YYYYMMDD>_<...>.py, with the following changes:

Create model FeedbackOption
Add field published_on to Post
Create model PostFeedback

These are the three changes that we introduced to our data model.

Now, if we go ahead and open the generated file, it will look something like this:

# Generated by Django 2.2 on 2019-04-21 19:31

from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion

class Migration(migrations.Migration):

    dependencies = [
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
        ('blog', '0001_initial'),
    ]

    operations = [
        migrations.CreateModel(
            name='FeedbackOption',
            fields=[
                ('id', models.AutoField(
                    auto_created=True,
                    primary_key=True,
                    serialize=False, verbose_name='ID'
                )),
                ('slug', models.SlugField(unique=True)),
                ('option', models.CharField(max_length=50)),
            ],
        ),
        migrations.AddField(
            model_name='post',
            name='published_on',
            field=models.DateTimeField(blank=True, null=True),
        ),
        migrations.CreateModel(
            name='PostFeedback',
            fields=[
                ('id', models.AutoField(
                    auto_created=True,
                    primary_key=True,
                    serialize=False,
                    verbose_name='ID'
                )),
                ('option', models.ForeignKey(
                    on_delete=django.db.models.deletion.CASCADE,
                    related_name='feedback',
                    to='blog.FeedbackOption'
                )),
                ('post', models.ForeignKey(
                    on_delete=django.db.models.deletion.CASCADE,
                    related_name='feedback',
                    to='blog.Post'
                )),
                ('user', models.ForeignKey(
                    on_delete=django.db.models.deletion.CASCADE,
                    related_name='feedback',
                    to=settings.AUTH_USER_MODEL
                )),
            ],
        ),
    ]

Similar to our first migration file, each operation maps to changes that we made to the data model. The main differences to note are the dependencies. Django has detected that our change relies on the first migration in the blog application and, since we depend on the auth user model, that is marked as a dependency as well.

Applying the Django database migration

Now that we have our migrations generated, we can apply the migrations:

./manage.py migrate

The output tells us that the latest generated migration is applied. If we inspect our modified SQLite database, we’ll see that our new migration file should be in the django_migrations table, the new tables should be present, and our new field on the Post Model should be reflected in the blog_post table.

Now, if we were to deploy our changes to production, the application code and database would be updated, and we would be running the new version of our application.

Bonus: data migrations

In this particular example, the blog_feedbackoption table (generated by our migration) will be empty when we push our code change. If our interface has been updated to surface these options, there is a chance that we forget to populate these when we push. Even if we don’t forget, we have the same problem as before: new objects are created in the database while the new application code is deploying, so there is very little time for the interface to show a blank list of options.

To help in scenarios where the required data is somewhat tied to the application code or to changes in the data model, Django provides utility for making data migrations. These are migration operations that simply change the data in the database rather than the table structure.

Let’s say we want to have the following feedback options: Interesting, Mildly Interesting, Not Interesting and Boring. We could put our data migration in the same migration file that we generated previously, but let’s create another migration file specifically for this data migration...

... check out the code on Kite's blog! Continue with "Bonus: Data Migrations"

Damian Hites is the CTO of Sylo, which is looking to improve Social Media Marketing by offering 3rd party trusted measurement. He has 10+ years of experience writing software and leading teams.

Django Templates: Best Practices

Aaron Harris — Tue, 15 Oct 2019 19:10:50 +0000

Introduction to Django templates

Django, as a web framework, uses templates as a way of producing static HTML from the output of a Django view. In practice, Django’s templates are simply HTML files, with some special syntax and a set of tools which lets Django render the HTML page on-the-fly for the visiting user. Templates are highly customizable, but are meant to be simple, with most of the “heavy” logic going into the view. Let’s dive deeper and learn some standard ways of dealing with common problems.

Simple start with Django templates

By default, Django comes with a ton of built-in template tags and filters that help us perform repeatable template tasks throughout our apps.

Tags: Tags provide arbitrary logic in the rendering process. Django leaves this definition fairly vague, but tags are able to output content, grab content from the database (more on this later), or perform control operations like if statements or for loops.

Examples of tags*:*

{% firstof user.is_active user.is_staff user.is_deleted %}

The firstof tag will output the first provided variable which evaluates to True. This is a good replacement for a large if/elif/elif/elif/elif block that’s just evaluating on truthiness within your Django templates.

<ul>
{% for product in product_list %}
    <li>{{ product.name }}: ${{ product.price }}</li>
{% endfor %}
</ul>

The for tag in Django will loop over each item in a list, making that item (product, in this case) available in the template context before the tag is closed with endfor. This is a widely used pattern when working with lists of Django model instances which have been returned from the view.

Filters: Filters transform the values of variables and arguments. Filters would be used in tasks like rendering a string in uppercase or formatting a date string into a user’s region.

Examples of filters*:*

{{ value|date:'D d M Y' }}

The date filter will format a date (value, in the example) given a string with some format characters. The example would output the string: Mon 01 Apr 2019.

{{ value|slugify }}

The slugify filter will convert the spaces of a string into hyphens and convert the string to lowercase, among other things. The output of this example would-look-something-like-this.

Project structure

Django, by default, will make some assumptions about the structure of our project when it’s looking for templates. Knowing this, we can set up our project with a template directory and application template directories.

Imagine a project, cloud*,* with the following structure:

cloud/
    accounts/
        urls.py
        models.py
        views.py
        templates/
            accounts/
                login.html
                register.html
    blog/
        urls.py
        views.py
        models.py
        templates/
            blog/
                create.html
                post.html
                list.html
    config/
        settings/
            base.py
            local.py
        urls.py
    manage.py
    templates/
        includes/
            messages.html
            modal.html
        base.html
        logged_in.html

How inheritance works for Django templates

An important aspect of Django’s templating system is template inheritance. Django applications are meant to be reusable, and we can apply the same methodology to our templates by inheriting common HTML from other templates.

A typical pattern is to have a common base template for common aspects of your application, logged-in pages, logged-out pages, or in places where significant changes are made to the underlying HTML. From our example above, base.htmlwould contain most of the core structure that would make up each page, with blocks defined for app or page-specific customizations.

For example, base.html may contain:

{% load static %}
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
  {% block page_meta %}
  {% endblock %}

  {# Vendor styles #}
  {% block vendor_css %}
    <link rel="stylesheet" type="text/css" media="all" href="{% static 'css/vendor.css' %}" />
  {% endblock %}

  {# Global styles #}
  {% block site_css %}
    <link rel="stylesheet" type="text/css" media="all" href="{% static 'css/application.css' %}" />
  {% endblock %}

  {# Page-specific styles #}
  {% autoescape off %}
    {% block page_css %}{% endblock %}
  {% endautoescape %}

  {% block extra_head %}
    {# Extra header stuff (scripts, styles, metadata, etc) #}
  {% endblock %}

  <title>{% block page_title %}{% endblock %}</title>
</head>
<body class="{% block body_class %}{% endblock %}">
{% block body %}
    {# Page content will go here #}
{% endblock %}

{# Modal HTML #}
{% block modals %}
{% endblock %}

{# Vendor javascript #}
{% block vendor_js %}
  <script src="{% static 'js/vendor.js' %}"></script>
{% endblock %}

{# Global javascript #}
{% block site_js %}
  <script src="{% static 'js/application.js' %}"></script>
{% endblock %}

{# Shared data for javascript #}
<script type="text/javascript">
  window._sharedData = {
    {% autoescape off %}
      {% block shared_data %}
        'DEBUG': {% if debug %}true{% else %}false{% endif %},
      {% endblock %}
    {% endautoescape %}
  }
</script>

{# Page javascript #}
{% autoescape off %}
  {% block page_js %}
  {% endblock %}
{% endautoescape %}
</body>
</html>

There are a few things done in this example specifically for the sake of inheritance. Most notably, this base template has blocks defined for nearly every customizable aspect of the underlying HTML. Blocks for including CSS, JavaScript, an HTML title, meta tags, and more are all defined.

We use Django’s autoescape template tag surrounding blocks where we don’t want Django to autoescape our HTML tags or JavaScript, but rather treat the contents of the block literally.

Our shared_data block allows us to populate a global JavaScript object with variables and data which we may want to share between Django and any running JavaScript on the page (populating React or Vue.js components, for example.)

For example, if we wanted to pass a Django URL to one of our JavaScript files, we could do something like this:

{% extends 'base.html' %}

{% block shared_data %}
  {{ block.super }}
  'USERS_AUTOCOMPLETE_ENDPOINT': '{% url 'api:users:autocomplete' %}',
{% endblock %}

Django loads the page and returns in a JavaScript object that you can then use within the JavaScript files on the page:

<script type="text/javascript">
    window._sharedData = {      
        'DEBUG': false,
        'USERS_AUTOCOMPLETE_ENDPOINT': '/api/users/autocomplete/',
    }
</script>

The inside of a JS console once the page has loaded:

>> window._sharedData.DEBUG
false
>> window._sharedData.USERS_AUTOCOMPLETE_ENDPOINT
'/api/users/autocomplete/'

Handling querysets

Properly handling querysets within your templates can be a performance bottleneck for Django depending on the complexities of your model definitions.

Django’s templating system is tightly coupled with Django’s object-relational mapping layer which returns us data from the database. Without proper consideration of this coupling you may, inadvertently, cause the number of queries run on each page load to jump to unmaintainable amounts. In some cases, this can cause the database to become too sluggish to operate certain pages on your site, or worse, crash and need to be restarted.

Thankfully, Django provides mechanisms and patterns which we can use to make sure our templates are running as fast as possible and we’re not killing the database server.

Consider this common Django pattern:

accounts/views.py

class UserListView(ListView):
    template_name = 'accounts/list.html'
    model = User
    paginate_by = 25
    context_object_name = 'users'
    queryset = User.objects.all()

accounts/templates/accounts/list.html

...
<table>
  <thead>
  <tr>
    <th>Username</th>
    <th>Email</th>
    <th>Profile photo URL</th>
    <th>Joined</th>
  </tr>
  </thead>
  <tbody>
  {% for user in users %}
    <tr>
      <td>{{ user.username }}</td>
      <td>{{ user.email_address }}</td>
      <td>{{ user.profile.avatar_url }}</td>
      <td>{{ user.created_at }}</td>
    </tr>
  {% endfor %}
  </tbody>
</table>
...

Can you spot the problem? It may not be obvious at first, but look at this line:

<td>{{ user.profile.avatar_url }}</td>

When Django is processing and rendering our template (line by line), it will need to do an additional query to grab information from the profile object as it’s a related field. In our example view, we’re paginating by 25 users, so this one line in the template could account for an additional 25 queries (on each page request as the profile object, as with all related objects and models in Django) which aren’t included in the original query for the 25 users. You can imagine how this could become a very slow page if we were including fields from other related objects in our table, or if we were paginating by 100 users instead of 25.

To resolve this, we’ll change one line in our view, accounts/views.py, to select related objects when we’re running our original query for users:

class UserListView(ListView):
    template_name = 'accounts/list.html'
    model = User
    paginate_by = 25
    context_object_name = 'users'
    queryset = User.objects.select_related('profile')

By replacing our User.objects.all() with User.objects.select_related(‘profile’), we’re telling Django to include related profile instances when it’s performing its query for our users. This will include the Profile model on each User instance, preventing Django from needing to run an extra query each time we ask for information from the profile within the template.

Django’s select_related functionality does not work with many-to-many model relationships, or with many-to-one relationships. For this, we’d want to use Django’s prefetch_related method.

Unlike select_related, prefetch_related does its magic in Python, as opposed to SQL select statements, by joining related objects into instances which can be accessed in templates as we’ve done above. It doesn’t perform things in a single query like select_related is able to, but it’s much more efficient than running a query each time you request a related attribute.

A prefetch for related projects and organizations and one-to-many relationships off of the User model would look like this:

... check out the full article and source code from kite.com!

Zac Clancy is Vice President at Global DIRT (Disaster Immediate Response Team)

Using Custom Authentication Backends in Django

Aaron Harris — Tue, 08 Oct 2019 18:02:24 +0000

Many organizations use widely-adopted authentication systems provided by services like Google, Facebook, or GitHub. A few Python packages provide authentication integration with these services, but most of them expect you to be handling the final user accounts on with Django. What happens when you need to work with user accounts that live in another system altogether?

In this article, you’ll see the interface that Django exposes for authenticating to an external system. By the end, you should understand the pieces involved in mapping an external system’s information to Django’s native User objects in order to work with them on your own site.

Django’s default authentication

In the Django User Authentication System, we covered the basics of how default authentication works in Django. Ultimately, you can interact with User objects and understand if a user is_authenticated or not. Using the default authentication system, you can make use of many of Django’s built-in features like its login and logout views and password reset workflow.

When working with an external authentication system, you have to manage these pieces yourself. Some of them may not make sense to you depending on how your authentication system works.

Authentication backends

As with many of Django’s systems, authentication is modeled as a plugin system. Django will try to authenticate users through a series of authentication backends. The default backend checks a user’s username and password against all the existing User objects in the database to authenticate them. The AUTHENTICATION_BACKENDS setting is your entrypoint to intercept this workflow and point Django to your external system.

An authentication backend is a class that, minimally, implements two methods:

get_user(user_id) — a user_id can be whatever unique identifier your external system uses to distinguish users, and get_user returns either a user object matching the given user_id or None.
authenticate(request, **credentials) — the request is the current HTTP request, and the credentials keyword arguments are whatever credentials your external system needs to check if a user should be authenticated or not. This is often a username and password, but it could be an API token or some other scheme. authenticate returns an authenticated User object or None.

Inside your authentication backend’s authenticate method, you can pass along the credentials to your external system via a REST API or another common authentication scheme like LDAP or SAML.

Using the wonderful Yes or No? API, you could build an authentication backend that authenticates a user occasionally if the API permits:

import requests

class FickleAuthBackend:
    def authenticate(self, request, username):
        response = requests.get(
            'https://yesno.wtf/api/'
        ).json()
        return User(username=username, password='') if response['answer'] == 'yes' else None

While authenticate can return a user object or None, it may also return an AnonymousUser object, or raise PermissionDenied to explicitly halt any further authentication checks. This allows for a variety of ways to proceed, and anonymous users may still have certain permissions. You’ll want to account for that in your middleware and views.

If the external user service provides additional information about the user, get_user might be a good place to grab some of that data. You can add attributes to the user object in authenticate before you return it if you’d like, but be careful of how many attributes you add dynamically.

Permissions

I also covered Django’s permission scheme in The Django User Authentication System: when given a user, you can inquire about their permissions generally or against specific objects using the has_perm method. Custom authentication backends can override permission checking methods and Django will check against those first before falling back to its default checks. This allows you to make queries to your external system about permissions in addition to authentication:

... continue with Permissions see the code, and more Django tutorials by Dane!

Dane Hillard has an upcoming book "Practices of the Python Pro" coming this month (October 2019)

Advanced Django Models: Improve Your Python Development

Aaron Harris — Fri, 04 Oct 2019 17:25:01 +0000

Models are a core concept of the Django framework. According to Django’s design philosophies for models, we should be as explicit as possible with the naming and functionality of our fields, and ensure that we’re including all relevant functionality related to our model in the model itself, rather than in the views or somewhere else. If you’ve worked with Ruby on Rails before, these design philosophies won’t seem new as both Rails and Django implement the Active Record pattern for their object-relational mapping (ORM) systems to handle stored data.

In this post we’ll look at some ways to leverage these philosophies, core Django features, and even some libraries to help make our models better.

`getter/setter/deleter` properties

As a feature of Python since version 2.2, a property’s usage looks like an attribute but is actually a method. While using a property on a model isn’t that advanced, we can use some underutilized features of the Python property to make our models more powerful.

If you’re using Django’s built-in authentication or have customized your authentication using AbstractBaseUser, you’re probably familiar with the last_login field defined on the User model, which is a saved timestamp of the user’s last login to your application. If we want to use last_login, but also have a field named last_seen saved to a cache more frequently, we could do so pretty easily.

First, we’ll make a Python property that finds a value in the cache, and if it can’t, it returns the value from the database.

accounts/models.py
from django.contrib.auth.base_user import AbstractBaseUser
from django.core.cache import cache


class User(AbstractBaseUser):
    ...

    @property
    def last_seen(self):
        """
        Returns the 'last_seen' value from the cache for a User.
        """
        last_seen = cache.get('last_seen_{0}'.format(self.pk))

        # Check cache result, otherwise return the database value
        if last_seen:
            return last_seen

        return self.last_login

Note: I’ve slimmed the model down a bit as there’s a separate tutorial on this blog about specifically customizing the built-in Django user model.

The property above checks our cache for the user’s last_seen value, and if it doesn’t find anything, it will return the user’s stored last_login value from the model. Referencing <instance>.last_seen now provides a much more customizable attribute on our model behind a very simple interface.

We can expand this to include custom behavior when a value is assigned to our property (some_user.last_seen = some_date_time), or when a value is deleted from the property (del some_user.last_seen).

...

@last_seen.setter
def last_seen(self, value):
    """
    Sets the 'last_seen_[uuid]' value in the cache for a User.
    """
    now = value

    # Save in the cache
    cache.set('last_seen_{0}'.format(self.pk), now)

@last_seen.deleter
def last_seen(self):
    """
    Removes the 'last_seen' value from the cache.
    """
    # Delete the cache key
    cache.delete('last_seen_{0}'.format(self.pk))

...

Now, whenever a value is assigned to our last_seen property, we save it to the cache, and when a value is removed with del, we remove it from the cache. Using setter and deleter is described in the Python documentation but is rarely seen in the wild when looking at Django models.

You may have a use case like this one, where you want to store something that doesn’t necessarily need to be persisted to a traditional database, or for performance reasons, shouldn’t be. Using a custom property like the above example is a great solution.

In a similar use case, the python-social-auth library, a tool for managing user authentication using third-party platforms like GitHub and Twitter, will create and manage updating information in your database based on information from the platform the user logged-in with. In some cases, the information returned won’t match the fields in our database. For example, the python-social-auth library will pass a fullname keyword argument when creating the user. If, perhaps in our database, we used full_name as our attribute name then we might be in a pinch.

A simple way around this is by using the getter/setter pattern from above:

@property
def fullname(self) -> str:
    return self.full_name

@fullname.setter
def fullname(self, value: str):
    self.full_name = value

Now, when python-social-auth saves a user’s fullname to our model (new_user.fullname = 'Some User'), we’ll intercept it and save it to our database field, full_name, instead.

`through` model relationships

Django’s many-to-many relationships are a great way of handling complex object relationships simply, but they don’t afford us the ability to add custom attributes to the intermediate models they create. By default, this simply includes an identifier and two foreign key references to join the objects together.

Using the Django ManyToManyField through parameter, we can create this intermediate model ourselves and add any additional fields we deem necessary.

If our application, for example, not only needed users to have memberships within groups, but wanted to track when that membership started, we could use a custom intermediate model to do so.

accounts/models.py
import uuid

from django.contrib.auth.base_user import AbstractBaseUser
from django.db import models
from django.utils.timezone import now


class User(AbstractBaseUser):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    …

class Group(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    members = models.ManyToManyField(User, through='Membership')

class Membership(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    group = models.ForeignKey(Group, on_delete=models.CASCADE)
    joined = models.DateTimeField(editable=False, default=now)

In the example above, we’re still using a ManyToManyField to handle the relationship between a user and a group, but by passing the Membership model using the through keyword argument, we can now add our joined custom attribute to the model to track when the group membership was started. This through model is a standard Django model, it just requires a primary key (we use UUIDs here), and two foreign keys to join the objects together.

Using the same three model pattern, we could create a simple subscription database for our site:

import uuid

from django.contrib.auth.base_user import AbstractBaseUser
from django.db import models
from django.utils.timezone import now


class User(AbstractBaseUser):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    ...

class Plan(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    name = models.CharField(max_length=50, unique=True, default='free')
    subscribers = models.ManyToManyField(User, through='Subscription', related_name='subscriptions', related_query_name='subscriptions')

class Subscription(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    plan = models.ForeignKey(Plan, on_delete=models.CASCADE)
    created = models.DateTimeField(editable=False, default=now)
    updated = models.DateTimeField(auto_now=True)
    cancelled = models.DateTimeField(blank=True, null=True)

Here we’re able to track when a user first subscribed, when they updated their subscription, and if we added the code paths for it, when a user canceled their subscription to our application.

Using through models with the ManyToManyField is a great way to add more data to our intermediate models and provide a more thorough experience for our users without much added work.

Proxy models

Normally in Django, when you subclass a model (this doesn’t include abstract models) into a new class, the framework will create new database tables for that class and link them (via OneToOneField) to the parent database tables. Django calls this “multi-table inheritance” and it’s a great way to re-use existing model fields and structures and add your own data to them. “Don’t repeat yourself,” as the Django design philosophies state.

Multi-table inheritance example:

from django.db import models

class Vehicle(models.Model):
    model = models.CharField(max_length=50)
    manufacturer = models.CharField(max_length=80)
    year = models.IntegerField(max_length=4)

class Airplane(Vehicle):
    is_cargo = models.BooleanField(default=False)
    is_passenger = models.BooleanField(default=True)

This example would create ...

...continue with Proxy Models

Zac Clancy is Vice President at Global DIRT (Disaster Immediate Response Team)

Handling Imbalanced Datasets with SMOTE in Python

Aaron Harris — Tue, 01 Oct 2019 19:04:28 +0000

Close your eyes.

Now imagine a perfect data world. What do you see? What do you wish to see? Exactly, me too. A flawlessly balanced dataset. A collection of data whose labels form a magnificent 1:1 ratio: 50% of this, 50% of that; not a bit to the left, nor a bit to the right. Just perfectly balanced, as all things should be. Now open your eyes, and come back to the real world.

The opposite of a pure balanced dataset is a highly imbalanced dataset, and unfortunately for us, these are quite common. An imbalanced dataset is a dataset where the number of data points per class differs drastically, resulting in a heavily biased machine learning model that won’t be able to learn the minority class. When this imbalanced ratio is not so heavily skewed toward one class, such dataset is not that horrible, since many machine learning models can handle them.

Nevertheless, there are some extreme cases in which the class ratio is just wrong, for example, a dataset where 95% of the labels belong to class A, while the remaining 5% fall under class B– a ratio not so rare in use cases such as fraud detection. In these extreme cases, the ideal course of action would be to collect more data.

However, this is typically not feasible; in fact, it’s costly, time-consuming and in most cases, impossible. Luckily for us, there’s an alternative known as oversampling. Oversampling involves using the data we currently have to create more of it.

What is data oversampling?

Data oversampling is a technique applied to generate data in such a way that it resembles the underlying distribution of the real data. In this article, I explain how we can use an oversampling technique called Synthetic Minority Over-Sampling Technique or SMOTE to balance out our dataset.

What is SMOTE?

SMOTE is an oversampling algorithm that relies on the concept of nearest neighbors to create its synthetic data. Proposed back in 2002 by Chawla et. al., SMOTE has become one of the most popular algorithms for oversampling.

The simplest case of oversampling is simply called oversampling or upsampling, meaning a method used to duplicate randomly selected data observations from the outnumbered class.

Oversampling’s purpose is for us to feel confident the data we generate are real examples of already existing data. This inherently comes with the issue of creating more of the same data we currently have, without adding any diversity to our dataset, and producing effects such as overfitting.

Hence, if overfitting affects our training due to randomly generated, upsampled data– or if plain oversampling is not suitable for the task at hand– we could resort to another, smarter oversampling technique known as synthetic data generation.

Synthetic data is intelligently generated artificial data that resembles the shape or values of the data it is intended to enhance. Instead of merely making new examples by copying the data we already have (as explained in the last paragraph), a synthetic data generator creates data that is similar to the existing one. Creating synthetic data is where SMOTE shines.

How does SMOTE work?

To show how SMOTE works, suppose we have an imbalanced two-dimensional dataset, such as the one in the next image, and we want to use SMOTE to create new data points.

Example of an imbalanced dataset

For each observation that belongs to the under-represented class, the algorithm gets its K-nearest-neighbors and synthesizes a new instance of the minority label at a random location in the line between the current observation and its nearest neighbor.

In our example (shown in the next image), the blue encircled dot is the current observation, the blue non-encircled dot is its nearest neighbor, and the green dot is the synthetic one.

SMOTE’s new synthetic data point

Now let’s do it in Python.

SMOTE tutorial using imbalanced-learn

In this tutorial, I explain how to balance an imbalanced dataset using the package imbalanced-learn.

First, I create a perfectly balanced dataset and train a machine learning model with it which I’ll call our “base model”. Then, I’ll unbalance the dataset and train a second system which I’ll call an “imbalanced model.”

Finally, I’ll use SMOTE to balance out the dataset, followed by fitting a third model with it which I’ll name the “SMOTE’d” model. By training a new model at each step, We’ll be able to better understand how an imbalanced dataset can affect a machine learning system.

Base model

Example code for this article may be found at the Kite Blog repository.

For the initial task, I’ll fit a support-vector machine (SVM) model using a created, perfectly balanced dataset. I chose this kind of model because of how easy it is to visualize and understand its decision boundary, namely, the hyperplane that separates one class from the other.

To generate a balanced dataset, I’ll use scikit-learn’s make_classification function which creates n clusters of normally distributed points suitable for a classification problem.

My fake dataset consists of 700 sample points, two features, and two classes. To make sure each class is one blob of data, I’ll set the parameter n_clusters_per_class to 1.

To simplify it, I’ll remove the redundant features and set the number of informative features to 2. Lastly, I’ll useflip_y=0.06 to reduce the amount of noise.

The following piece of code shows how we can create our fake dataset and plot it using Python’s Matplotlib.

import matplotlib.pyplot as plt
import pandas as pd

from sklearn.datasets import make_classification
from imblearn.datasets import make_imbalance

# for reproducibility purposes
seed = 100

# create balanced dataset
X1, Y1 = make_classification(n_samples=700, n_features=2, n_redundant=0,
                            n_informative=2, n_clusters_per_class=1,
                            class_sep=1.0, flip_y=0.06, random_state=seed)

plt.title('Balanced dataset')
plt.xlabel('x')
plt.ylabel('y')
plt.scatter(X1[:, 0], X1[:, 1], marker='o', c=Y1,
           s=25, edgecolor='k', cmap=plt.cm.coolwarm)
plt.show()
# concatenate the features and labels into one dataframe
df = pd.concat([pd.DataFrame(X1), pd.DataFrame(Y1)], axis=1)
df.columns = ['feature_1', 'feature_2', 'label']
# save the dataset because we'll use it later
df.to_csv('df_base.csv', index=False, encoding='utf-8')

A balanced dataset

As you can see in the previous image, our balanced dataset looks tidy and well defined. So, if we fit an SVM model with this data (code below), how will the decision boundary look?

Since we’ll be training several models and visualizing their hyperplanes, I wrote two functions that will be reused several times throughout the tutorial. The first one, train_SVM, is for fitting the SVM model, and it takes the dataset as a parameter.

The second function, plot_svm_boundary, plots the decision boundary of the SVM model. Its parameters also include the dataset and the caption of the plot.

These are the functions:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.svm import SVC


def train_SVM(df):
   # select the feature columns
   X = df.loc[:, df.columns != 'label']
   # select the label column
   y = df.label

   # train an SVM with linear kernel
   clf = SVC(kernel='linear')
   clf.fit(X, y)

   return clf


def plot_svm_boundary(clf, df, title):
   fig, ax = plt.subplots()
   X0, X1 = df.iloc[:, 0], df.iloc[:, 1]

   x_min, x_max = X0.min() - 1, X0.max() + 1
   y_min, y_max = X1.min() - 1, X1.max() + 1
   xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02), np.arange(y_min, y_max, 0.02))

   Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
   Z = Z.reshape(xx.shape)
   out = ax.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.8)

   ax.scatter(X0, X1, c=df.label, cmap=plt.cm.coolwarm, s=20, edgecolors='k')
   ax.set_ylabel('y')
   ax.set_xlabel('x')
   ax.set_title(title)
   plt.show()

To fit and plot the model, do the following:

df = pd.read_csv('df_base.csv', encoding='utf-8', engine='python')
clf = train_SVM(df)
plot_svm_boundary(clf, df, 'Decision Boundary of SVM trained with a balanced dataset')

Blue dots on the blue side and red dots on the red side means that the model was able to find a function that separates the classes

The image above presents the hyperplane of the base model. On it, we can observe how clear the separation between our classes is. However, what would happen if we imbalance our dataset? How would the decision boundary look? Before doing so, let’s imbalance the dataset by calling the function make_imbalance from the package, imbalanced-learn.

... continue with imbalanced-learn on the Kite blog!

Juan De Dios Santos is a travelling Data Storyteller and Machine Learning professional working on the Wander Data project.

Time Series Analysis with Pandas

Aaron Harris — Thu, 26 Sep 2019 18:16:58 +0000

We’ll be analyzing stock data with Python 3, pandas and Matplotlib. To fully benefit from this article, you should be familiar with the basics of pandas as well as the plotting library called Matplotlib.

Time series data

Time series data is a sequence of data points in chronological order that is used by businesses to analyze past data and make future predictions. These data points are a set of observations at specified times and equal intervals, typically with a datetime index and corresponding value. Common examples of time series data in our day-to-day lives include:

Measuring weather temperatures
Measuring the number of taxi rides per month
Predicting a company’s stock prices for the next day

Variations of time series data

Trend Variation: moves up or down in a reasonably predictable pattern over a long period of time.
Seasonality Variation: regular and periodic; repeats itself over a specific period, such as a day, week, month, season, etc.
Cyclical Variation: corresponds with business or economic ‘boom-bust’ cycles, or is cyclical in some other form
Random Variation: erratic or residual; doesn’t fall under any of the above three classifications.

Here are the four variations of time series data visualized:

Importing stock data and necessary Python libraries

To demonstrate the use of pandas for stock analysis, we will be using Amazon stock prices from 2013 to 2018. We’re pulling the data from Quandl, a company offering a Python API for sourcing a la carte market data. A CSV file of the data in this article can be downloaded from the article’s repository.

Fire up the editor of your choice and type in the following code to import the libraries and data that correspond to this article.

Example code for this article may be found at the Kite Blog repository on Github.

# Importing required modules
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Settings for pretty nice plots
plt.style.use('fivethirtyeight')
plt.show()

# Reading in the data
data = pd.read_csv('amazon_stock.csv')

A first look at Amazon’s stock Prices

Let’s look at the first few columns of the dataset:

# Inspecting the data
data.head()

Let’s get rid of the first two columns as they don’t add any value to the dataset.

data.drop(columns=['None', 'ticker'], inplace=True)
data.head()

Let us now look at the datatypes of the various components.

data.info()

It appears that the Date column is being treated as a string rather than as dates. To fix this, we’ll use the pandas to_datetime() feature which converts the arguments to dates.

# Convert string to datetime64
data['Date'] = data['Date'].apply(pd.to_datetime)
data.info()

Lastly, we want to make sure that the Date column is the index column.

data.set_index('Date', inplace=True)
data.head()

Now that our data has been converted into the desired format, let’s take a look at its columns for further analysis.

The Open and Close columns indicate the opening and closing price of the stocks on a particular day.
The High and Low columns provide the highest and the lowest price for the stock on a particular day, respectively.
The Volume column tells us the total volume of stocks traded on a particular day.

The Adj_Close column represents the adjusted closing price, or the stock’s closing price on any given day of trading, amended to include any distributions and/or corporate actions occurring any time before the next day’s open. The adjusted closing price is often used when examining or performing a detailed analysis of historical returns.

data['Adj_Close'].plot(figsize=(16,8),title='Adjusted Closing Price')

Interestingly, it appears that Amazon had a more or less steady increase in its stock price over the 2013-2018 window. We’ll now use pandas to analyze and manipulate this data to gain insights.

Pandas for time series analysis

As pandas was developed in the context of financial modeling, it contains a comprehensive set of tools for working with dates, times, and time-indexed data. Let’s look at the main pandas data structures for working with time series data.

Manipulating `datetime`

Python’s basic tools for working with dates and times reside in the built-in datetime module. In pandas, a single point in time is represented as a pandas.Timestamp and we can use the datetime()function to create datetime objects from strings in a wide variety of date/time formats. datetimes are interchangeable with pandas.Timestamp.

from datetime import datetime
my_year = 2019
my_month = 4
my_day = 21
my_hour = 10
my_minute = 5
my_second = 30

We can now create a datetime object, and use it freely with pandas given the above attributes.

test_date = datetime(my_year, my_month, my_day)
test_date


# datetime.datetime(2019, 4, 21, 0, 0)

For the purposes of analyzing our particular data, we have selected only the day, month and year, but we could also include more details like hour, minute and second if necessary.

test_date = datetime(my_year, my_month, my_day, my_hour, my_minute, my_second)
print('The day is : ', test_date.day)
print('The hour is : ', test_date.hour)
print('The month is : ', test_date.month)

# Output

The day is :  21
The hour is :  10
The month is :  4

For our stock price dataset, the type of the index column is DatetimeIndex. We can use pandas to obtain the minimum and maximum dates in the data.

print(data.index.max())
print(data.index.min())

# Output

2018-03-27 00:00:00
2013-01-02 00:00:00

We can also calculate the latest date location and the earliest date index location as follows:

# Earliest date index location
data.index.argmin()
#Output
1315
# Latest date location
data.index.argmax()
#Output
0

Time resampling

Examining stock price data for every single day isn’t of much use to financial institutions, who are more interested in spotting market trends. To make it easier, we use a process called time resampling to aggregate data into a defined time period, such as by month or by quarter. Institutions can then see an overview of stock prices and make decisions according to these trends.

The pandas library has a resample() function which resamples such time series data. The resample method in pandas is similar to its groupby method as it is essentially grouping according to a certain time span. The resample() function looks like this:

data.resample(rule = 'A').mean()

To summarize:

data.resample() is used to resample the stock data.
The ‘A’ stands for year-end frequency, and denotes the offset values by which we want to resample the data.
mean() indicates that we want the average stock price during this period.

The output looks like this, with average stock data displayed for December 31st of each year

Below is a complete list of the offset values. The list can also be found in the pandas documentation.

Offset aliases for time resampling

We can also use time sampling to plot charts for specific columns.

data['Adj_Close'].resample('A').mean().plot(kind='bar',figsize = (10,4))
plt.title('Yearly Mean Adj Close Price for Amazon')

The above bar plot corresponds to Amazon’s average adjusted closing price at year-end for each year in our data set.

Similarly, monthly maximum opening price for each year can be found below.

Monthly maximum opening price for Amazon

Time shifting

Sometimes, we may need to shift or move the data forward or backwards in time. This shifting is done along a time index by the desired number of time-frequency increments.

...continue with Time Shifting and see the code in the Kite Github repo.

Parul Pandey is a Data Science Evangelist at H2O.ai and author for the Kite Blog.

Guide: Type Hinting in Python

Aaron Harris — Mon, 23 Sep 2019 18:08:23 +0000

Guide: Type Hinting in Python

Since version 3.5, Python supports type hints: code annotations that, through additional tooling, can check if you’re using your code correctly.

Introduction

With the release of version 3.5, Python has introduced type hints: code annotations that, through additional tooling, can check if you’re using your code correctly.

Long-time Python users might cringe at the thought of new code needing type hinting to work properly, but we need not worry: Guido himself wrote in PEP 484, “no type checking happens at runtime.”

The feature has been proposed mainly to open up Python code for easier static analysis and refactoring.

For data science–and for the data scientist– type hinting is invaluable for a couple of reasons:

It makes it much easier to understand the code, just by looking at the signature, i.e. the first line(s) of the function definition;
It creates a documentation layer that can be checked with a type checker, i.e. if you change the implementation, but forget to change the types, the type checker will (hopefully) yell at you.

Of course, as is always the case with documentation and testing, it’s an investment: it costs you more time at the beginning, but saves you (and your co-worker) a lot in the long run.

Note: Type hinting has also been ported to Python 2.7 (a.k.a Legacy Python). The functionality, however, requires comments to work. Furthermore, no one should be using Legacy Python in 2019: it’s less beautiful and only has a couple more months of updates before it stops receiving support of any kind.

Getting started with types

The code for this article may be found at Kite’s Github repository.

The hello world of type hinting is:

# hello_world.py
def hello_world(name: str = 'Joe') -> str:
    print(f'Hello {name}')

We have added two type hint elements here. The first one is : str after name and the second one is -> str towards the end of the signature.

The syntax works as you would expect: we’re marking name to be of type str and we’re specifying that the hello_world function should output a str. If we use our function, it does what it says:

> hello_world(name='Mark')
'Hello Mark'

Since Python remains a dynamically unchecked language, we can still shoot ourselves in the foot:

> hello_world(name=2)
'Hello 2'

What’s happening? Well, as I wrote in the introduction, no type checking happens at runtime.

So as long as the code doesn’t raise an exception, things will continue to work fine.

What should you do with these type definitions then? Well, you need a type checker, or an IDE that reads and checks the types in your code (PyCharm, for example).

Type checking your program
There are at least four major type checker implementations: Mypy, Pyright, pyre, and pytype:

Mypy is actively developed by, among others, Guido van Rossum, Python’s creator.
Pyright has been developed by Microsoft and integrates very well with their excellent Visual Studio Code;
Pyre has been developed by Facebook with the goal to be fast (even though mypy recently got much faster);
Pytype has been developed by Google and, besides checking the types as the others do, it can run type checks (and add annotations) on unannotated code. Since we want to focus on how to use typing from a Python perspective, we’ll use Mypy in this tutorial. We can install it using pip (or your package manager of choice):

$ pip install mypy
$ mypy hello_world.py

More advanced types
In principle, all Python classes are valid types, meaning you can use str, int, float, etc. Using dictionary, tuples, and similar is also possible, but you need to import them from the typing module.

# tree.py
from typing import Tuple, Iterable, Dict, List, DefaultDict
from collections import defaultdict

def create_tree(tuples: Iterable[Tuple[int, int]]) -> DefaultDict[int, List[int]]:
    """
    Return a tree given tuples of (child, father)

    The tree structure is as follows:

        tree = {node_1: [node_2, node_3], 
                node_2: [node_4, node_5, node_6],
                node_6: [node_7, node_8]}
    """
    tree = defaultdict(list) 
    for child, father in tuples:
        if father:
            tree[father].append(child)
    return tree

print(create_tree([(2.0,1.0), (3.0,1.0), (4.0,3.0), (1.0,6.0)]))
# will print
# defaultdict( 'list'="">, {1.0: [2.0, 3.0], 3.0: [4.0], 6.0: [1.0]}

While the code is simple, it introduces a couple of extra elements:

First of all, the Iterable type for the tuples variable. This type indicates that the object should conform to the collections.abc.Iterable specification (i.e. implement __iter__). This is needed because we iterate over tuples in the for loop;
We specify the types inside our container objects: the Iterable contains Tuple, the Tuples are composed of pairs of int, and so on.

Ok, let’s try to type check it!

$ mypy tree.py
tree.py:14: error: Need type annotation for 'tree'

Uh-oh, what’s happening? Basically Mypy is complaining about this line:

tree = defaultdict(list)

While we know that the return type should be DefaultDict[int, List[int]], Mypy cannot infer that tree is indeed of that type. We need to help it out by specifying tree’s type. Doing so can be done similarly to how we do it in the signature:

tree: DefaultDict[int, List[int]] = defaultdict(list)

If we now re-run Mypy again, all is well:

$ mypy tree.py
$

Type aliases

Sometimes our code reuses the same composite types over and over again. In the above example, Tuple[int, int] might be such a case. To make our intent clearer (and shorten our code), we can use type aliases. Type aliases are very easy to use: we just assign a type to a variable, and use that variable as the new type...

Relation = Tuple[int, int]

def create_tree(tuples: Iterable[Relation]) -> DefaultDict[int, List[int]]:
    """
    Return a tree given tuples of (child, father)

    The tree structure is as follow:

        tree = {node_1: [node_2, node_3], 
                node_2: [node_4, node_5, node_6],
                node_6: [node_7, node_8]}
    """
    # convert to dict
    tree: DefaultDict[int, List[int]] = defaultdict(list) 
    for child, father in tuples:
        if father:
            tree[father].append(child)

    return tree

Generics

Experienced programmers of statically typed languages might have noticed that defining a Relation as a tuple of integers is a bit restricting. Can’t create_tree work with a float, or a string, or the ad-hoc class that we just created?

In principle, there’s nothing that prevents us from using it like that:

# tree.py
from typing import Tuple, Iterable, Dict, List, DefaultDict
from collections import defaultdict

Relation = Tuple[int, int]

def create_tree(tuples: Iterable[Relation]) -> DefaultDict[int, List[int]]:
    ...

print(create_tree([(2.0,1.0), (3.0,1.0), (4.0,3.0), (1.0,6.0)]))
# will print
# defaultdict( 'list'="">, {1.0: [2.0, 3.0], 3.0: [4.0], 6.0: [1.0]})

However if we ask Mypy’s opinion of the code, we’ll get an error:

$ mypy tree.py
tree.py:24: error: List item 0 has incompatible type 'Tuple[float, float]'; expected 'Tuple[int, int]'
...

There is a way in Python to fix this. It’s called TypeVar, and it works by creating a generic type that doesn’t require assumptions: it just fixes it throughout our module. Usage is pretty simple...

Check out the code -- Guide: Type Hinting in Python 3.5
on Kite's blog.
Giovanni Lanzani is the Director of Learning and Development at GoDataDriven.

Statistical Modeling with Python: How-to & Top Libraries

Aaron Harris — Thu, 12 Sep 2019 03:51:43 +0000

Statistical Modeling with Python: How-to & Top Libraries

This post covers some of the essential statistical modeling frameworks and methods for Python, which can help us do statistical modeling and probabilistic computation.

Introduction: Why Python for data science
Why these frameworks are necessary
Start with NumPy
Matplotlib and Seaborn for visualization
Using Seaborn and Matplotlib
SciPy for inferential statistics
Statsmodels for advanced modeling
Scikit-learn for statistical learning
Conclusion

Why these frameworks are necessary

While Python is most popular for data wrangling, visualization, general machine learning, deep learning and associated linear algebra (tensor and matrix operations), and web integration, its statistical modeling abilities are far less advertised. A large percentage of data scientists still use other special statistical languages such as R, MATLAB, or SAS over Python for their modeling and analysis.

While each of these alternatives offer their own unique blend of features and power for statistical analyses, it’s useful for an up-and-coming data scientist to know more about various Python frameworks and methods that can be used for routine operations of descriptive and inferential statistics.

The biggest motivation for learning about these frameworks is that statistical inference and probabilistic modeling represent the bread and butter of a data scientists’ daily work. However, only by using such Python-based tools can a powerful end-to-end data science pipeline (a complete flow extending from data acquisition to final business decision generation) be built using a single programming language.

If using different statistical languages for various tasks, you may face some problems. For example:

Conducting any web scraping and database access using SQL commands and Python libraries such as BeautifulSoup and SQLalchemy
Cleaning up and preparing your data tables using Pandas, but then switching to R or SPSS for performing statistical tests and computing confidence intervals
Using ggplot2 for creating visualization, and then using a standalone LaTeX editor to type up the final analytics report
Switching between multiple programmatic frameworks makes the process cumbersome and error-prone.

What if you could do statistical modeling, analysis, and visualization all inside a core Python platform? Let’s see what frameworks and methods exist for accomplishing such tasks.

Start with NumPy

NumPy is the de-facto standard for numerical computation in Python, used as the base for building more advanced libraries for data science and machine learning applications such as TensorFlow or Scikit-learn. For numeric processing, NumPy is much faster than native Python code due to the vectorized implementation of its methods and the fact that many of its core routines are written in C (based on the CPython framework).

Although the majority of NumPy related discussions are focused on its linear algebra routines, it offers a decent set of statistical modeling functions for performing basic descriptive statistics and generating random variables based on various discrete and continuous distributions.

For example, let’s create a NumPy array from a simple Python list and compute basic descriptive statistics like mean, median, standard deviation, quantiles, etc.

The code for this article may be found at Kite’s Github repository.

import numpy as np

# Define a python list
a_list = [2, 4, -1, 5.5, 3.5, -2, 5, 4, 6.5, 7.5]

# Convert the list into numpy array
an_array = np.array(a_list)

# Compute and print various statistics
print('Mean:', an_array.mean())
print('Median:', np.median(an_array))
print('Range (Max - min):', np.ptp(an_array))
print('Standard deviation:', an_array.std())
print('80th percentile:', np.percentile(an_array, 80))
print('0.2-quantile:', np.quantile(an_array, 0.2))

The results are as follows:

Mean: 3.5
Median: 4.0
Range (Max - min): 9.5
Standard deviation: 2.9068883707497264
80th percentile: 5.699999999999999
0.2-quantile: 1.4000000000000001

To read more about Numpy, Matplotlib, Seaborn, and Statsmodels, check out the full article by Tirtha Sarkar.

Tirtha Sarkar is a semiconductor technologist, data science author, and author of pydbgen, MLR, and doepy packages. He holds a Ph.D. in Electrical Engineering and M.S. in Data Analytics.

Data Science, the Good, the Bad, and the... Future

Aaron Harris — Thu, 01 Aug 2019 19:08:51 +0000

Positives: astrophysics, biology, and sports

Part of a new article by Kirit Thadaka on the Kite Blog

Data science made a huge positive impact on the way technology influences our lives. Some of these impacts have been nice and some have been otherwise. looks at Facebook But, technology can’t inherently be good or bad, technology is… technology. It’s the way we use it that has good or bad outcomes.

We recently had a breakthrough in astrophysics with the first ever picture of a black hole. This helps physicists confirm more than a century of purely theoretical work around black holes and the theory of relativity.

To capture this image, scientists used a telescope as big as the earth (Event Horizon Telescope or EHT) by combining data from an array of eight ground-based radio telescopes and making sense of it all to construct an image. Analyzing data and then visualizing that data – sounds like some data science right here.

A cool side note on this point: a standard Python library of functions for EHT Imaging was developed by Andrew Chael from Harvard to simulate and manipulate VLBI (Very-long-baseline interferometry) data helping the process of creating the black hole image.

Olivier Elemento at Cornell uses Big Data Analytics to help identity mutations in genomes that result in tumor cells spreading so that they can be killed earlier – this is a huge positive impact data science has on human life. You can read more about his incredible research here.

Python is used by researchers in his lab while testing statistical and machine learning models. Keras, NumPy, Scipy, and Scikit-learn are some top notch Python libraries for this.

If you’re a fan of the English Premier League, you’ll appreciate the example of Leicester City winning the title in the 2015-2016 season.

At the start of the season, bookmakers had the likelihood Leicester City winning the EPL at 10 times less than the odds of finding the Loch Ness monster. For a more detailed attempt at describing the significance of this story, read this.

Everyone wanted to know how Leicester was able to do this, and it turns out that data science played a big part! Thanks to their investment into analytics and technology, the club was able to measure players’ fitness levels and body condition while they were training to help prevent injuries, all while assessing best tactics to use in a game based on the players’ energy levels.

All training sessions had plans backed by real data about the players, and as a result Leicester City suffered the least amount of player injuries of all clubs that season.

Many top teams use data analytics to help with player performance, scouting talent, and understanding how to plan for certain opponents.

Here’s an example of Python being used to help with some football analysis. I certainly wish Chelsea F.C. would use some of these techniques to improve their woeful form and make my life as a fan better. You don’t need analytics to see that Kante is in the wrong position, and Jorginho shouldn’t be in that team and… Okay I’m digressing – back to the topic now!

Now that we’ve covered some of the amazing things data science has uncovered, I’m going to touch on some of the negatives as well – it’s important to critically think about technology and how it impacts us.

The amount that technology impacts our lives will undeniably increase with time, and we shouldn’t limit our understanding without being aware of the positive and negative implications it can have.

Some of the concerns I have around this ecosystem are data privacy (I’m sure we all have many examples that come to mind), biases in predictions and classifications, and the impact of personalization and advertising on society.

Negatives: gender bias and more

This paper published in NIPS talks about how to counter gender biases in word embeddings used frequently in data science.

For those who aren’t familiar with the term, word embeddings are a clever way of representing words so that neural networks and other computer algorithms can process them.

The data used to create Word2Vec (a model for word embeddings created by Google) has resulted in gender biases that show close relations between “men” and words like “computer scientist”, “architect”, “captain”, etc. while showing “women” to be closely related to “homemaker”, “nanny”, “nurse”, etc.

Here’s the Python code used by the researchers who published this paper. Python’s ease of use makes it a good choice for quickly going from idea to implementation...

Check out the code and examples, and rest of the negatives on our blog!

First, use the right tool for the job. Check out the options in this pocket-sized guide.

Aaron Harris — Fri, 19 Apr 2019 23:13:26 +0000

So you want to set up a superior Python environment, but you don’t want it to be a major hassle. Well, this was written for you!

In this post, we explore the top IDEs and general-purpose editors for all your Python programming needs.

What Are IDEs?
An integrated development environment (IDE) provides Python programmers with a suite of tools that streamline the coding, testing, and debugging process for specific use cases.

The best Python IDE for you is the one that will help you ship code faster by automating repetitive tasks, organizing information, and helping reduce errors.

What Is a Code Editor?
Code editors are tools that make writing code easier, offering syntax highlighting and code formatting, among other things. They differ from IDEs in that they have less features outside of their primary use case which is writing code.

For this reason, they are typically quicker and lighter weight, which leads some developers to preference using a code editor over an IDE.

Some code editors also deliver additional functionality, like debugging and code execution, however.

Want to know more? Check out:
https://owlskip.com/s/top-ides

DEV Community: Aaron Harris

Django Database Migrations: A Comprehensive Overview

Django’s database migration solution

1. Change the contract: Django’s ORM

2. Plan for change: generate migrations

3. Execute: apply migrations

Tracking changes with Django

Rolling back with Django

A simple Django database migrations example

The initial Django application

Changing the Django data model

Generate the Django database migration

Applying the Django database migration

Bonus: data migrations

Django Templates: Best Practices

Introduction to Django templates

Simple start with Django templates

Project structure

How inheritance works for Django templates

Handling querysets

Using Custom Authentication Backends in Django

Django’s default authentication

Authentication backends

Permissions

Advanced Django Models: Improve Your Python Development

getter/setter/deleter properties

through model relationships

Proxy models

Handling Imbalanced Datasets with SMOTE in Python

What is data oversampling?

What is SMOTE?

How does SMOTE work?

SMOTE tutorial using imbalanced-learn

Base model

Time Series Analysis with Pandas

Time series data

Variations of time series data

Importing stock data and necessary Python libraries

A first look at Amazon’s stock Prices

Pandas for time series analysis

Manipulating datetime

Time resampling

Time shifting

Guide: Type Hinting in Python

Guide: Type Hinting in Python

Introduction

Getting started with types

Type aliases

Generics

Statistical Modeling with Python: How-to & Top Libraries

Statistical Modeling with Python: How-to & Top Libraries

Why these frameworks are necessary

Start with NumPy

Data Science, the Good, the Bad, and the... Future

Positives: astrophysics, biology, and sports

Negatives: gender bias and more

First, use the right tool for the job. Check out the options in this pocket-sized guide.

`getter/setter/deleter` properties

`through` model relationships

Manipulating `datetime`