DEV Community: Farid MUSA

How to Create a Fancy Range-Slider Filter in Django

Farid MUSA — Wed, 07 Apr 2021 00:00:00 +0000

Introduction

In this tutorial, we are going to create a nice crispy range slider using django-crispy-forms for an integer filter provided by django-filters. Tutorial can be split into four sections. In the first or prerequisite section, we will install the required packages and initiate the Django project. In the next section, we will create the simple app with a model, view, and template. In the third section, we will create the simple range filter using django-filters package without a slider. In the fourth and last section, we will describe how to create a range-slider and integrate it into our Django app created in the third section.

Prerequisite

First things first, let’s create a directory with the working environment:

$ mkdir myproject
$ cd myproject
$ pipenv shell

Then, install packages that are required in this tutorial using pip:

$ pip install Django
$ pip install django-crispy-forms
$ pip install django-filter

Next, create a new Django project called myproject :

$ django-admin startproject myproject
$ mv myproject src

Similarly, create a new Django app called myapp :

$ python manage.py startapp myapp

In the following sections, you are going to need to generate sample data for our model. Hence, let’s create a new Django admin super-user using the following command:

$ python manage.py createsuperuser

To enable packages in Django project, add following lines to the INSTALLED_APPS in /src/myproject/settings.py:

INSTALLED_APPS = [
    ...
    'django.forms', # Required in the last section.
    'django_filters', # Django-filter
    'crispy_forms',
    'myapp'
]

Then, add the following line to TEMPLATES in /src/myproject/settings.py:

TEMPLATES = [
    {
        ...
        'DIRS': [BASE_DIR / 'templates'],
        ...
    },
]

Next, add path /src/myproject/static to STATICFILES_DIRS in /src/myproject/settings.py to enable the CSS and JS files, which will be required in upcoming sections:

...
STATICFILES_DIRS = [BASE_DIR / 'static']
...

Finally, add the following line of code to /src/myproject/settings.py to enable widget customization in our Django project.

...
FORM_RENDERER = 'django.forms.renderers.TemplatesSetting'
...

Getting Ready

In this section, we will create a model called People, a view for this model, and a template for that view.

The Model

Create a model called People using three fields name , surname and age. The target filter-field is IntegerField named age :

class People(models.Model):
    name = models.CharField(null=True,blank=True,max_length=50)
    surname = models.CharField(null=True,blank=True,max_length=50)
    age = models.IntegerField()

Run makemigrations and then migrate to apply change to the default SQLite database:

$ python manage.py makemigrations
$ python manage.py migrate

Then, register the model in Django admin by adding the following code to file /src/myapp/admin.py:

from django.contrib import admin
from .models import People

class PeopleAdmin(admin.ModelAdmin):
    pass

admin.site.register(People, PeopleAdmin)

NOTE: Add some items to the database from the Django admin page at http://127.0.0.1:8000/admin/.

The View

Now let’s create a simple view that will print all instances of People model in /src/myapp/views.py.

from django.shortcuts import render
from .models import People

def index(request):
    all_people = People.objects.all()
    return render(request, 'index.html', {'all_people':all_people})

URLs

Create a URL path by adding following line to the file /src/myproject/urls.py:

...
from myapp.views import index

urlpatterns = [
    ...
    path('', index),
]

The Template

In order to create the simplest template to render all People instances, create a file /src/templates/index.html with the following content:

<table border='1' style="width:100%; text-align:center">
  <thead>
    <tr>
      <th> Name </th>
      <th> Surname </th>
      <th> Age </th>
    </tr>
  </thead>

  <tbody>
    {% for person in all_people %}
    <tr>
      <td> {{ person.name }} </td>
      <td> {{ person.surname }} </td>
      <td> {{ person.age }} </td>
    </tr>
    {% endfor %}
  </tbody>
</table>

Recap

In this section, we created a simple view and template to print database records for People model. Executing $ python manage.py runserver should make available the following screen:

Naive Range Filter

In order to ensure coherence let’s first create a simple(or naive) RangeFilter provide by django-filters package.

The Filter

Create in a new file /src/myapp/filters.py and insert the following code:

import django_filters
from .models import People

class PeopleFilter(django_filters.FilterSet):
  age = django_filters.AllValuesFilter()

  class Meta:
      model = People
      fields = ['age']

This will create a simple range-filter with minimum and maximum value requirements for age field of People model.

The View

Now that filtering logic is ready, let’s add the filtering feature to the main view in /src/myapp/views.py.

from django.shortcuts import render
from .filters import PeopleFilter

def index(request):
    people_filter = RangeFilter(request.GET)
    return render(request, 'index.html', {'people_filter':people_filter})

In the above code, RangeFilter instantiation takes request.GET as a single parameter since our form is set to GET mode.

The Template

With our filter ready, we can add filter controls in the front. Once again change the primary template file /src/template/index.html to look like this:

<form method="get">
    {{ people_filter.form.as_p }}
    <input type="submit" />
</form>

<table border='1' style="width:100%; text-align:center">
  <thead>
    <tr>
      <th> Name </th>
      <th> Surname </th>
      <th> Age </th>
    </tr>
  </thead>

  <tbody>
    {% for person in people_filter.qs %}
    <tr>
      <td> {{ person.name }} </td>
      <td> {{ person.surname }} </td>
      <td> {{ person.age }} </td>
    </tr>
    {% endfor %}
  </tbody>
</table>

Notice that we now have an additional filtering form provided by django-filters. In addition, observe that for statement loops over people_filter.qs instead of all_people. The .qs stands for query set, which is self-explanatory.

Recap

In this section, we created the simplest or naive filter. The final result should look like this:

Crispy Range-Slider

To create a working crispy range-slider we need the following:

Front-end Backbone: Actual range-slider with HTML, CSS and JavaScript.
Django Widget: Custom Django widget for the actual range-slider backbone.
Crispy Form: Crispy form with a layout template for the custom widget.
Range Filter: Custom filter from django-filters package that utilizes the range-slider widget with the Crispy form.

Each point will be described in details so let’s move step-by-step:

The Front-End

The first and obvious step is to create an actual slider. Since range-slider is fancy filtering “thing” and not a real HTML form element, let’s use a popular trick to make such a fancy “thing” act like an HTML form element. Particularly, we use jQuery’s range slider feature to make our range-slider work. Here is a sample HTML blueprint for our slider:

<div id="my-numeric-slider"
     class="form-group numeric-slider"
     data-range_min="[Min. Possible Value]"
     data-range_max="[Max. Possible Value]"
     data-cur_min="[Current Min. Value]"
     data-cur_max="[Current Max. Value]">
<div class="numeric-slider-range ui-slider ui-slider-horizontal ui-slider-range"></div>
<span class="numeric-slider-range_text" id='my-numeric-slider_text'>[Lower Value] - [Upper Value]</span>
<input type='hidden' id='my-numeric-slider_min' name='slider-min'/>
<input type='hidden' id='my-numeric-slider_max' name='slider-max'/>
</div>

The above HTML markup is comprised of outer Div element where the first two data- attributes represent possible minimum and maximum values of the range filter, last two data- attributes represent current lower and upper values of the range filter that were established when the page is loaded. Likewise, the first inner elements Div with the class numeric-slider-range is the main element transformed to the range slider by jQuery when the page is loaded. The last two hidden Input form-elements represent the primary means by which the data is passed from the client to the server-side when the form is submitted. Additionally, the above HTML markup requires a JS script to make the slider work and a CSS markup to render the elements properly. Both can be found in GitHub repo. Finally, apply the last template code bellow to the file /src/templates/index.html:

{% load static %}
{% load crispy_forms_tags %}

<head>
  <link rel="stylesheet" href="{% static 'custom_slider.css' %}"> # CSS of our range-slider.
  <link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">

  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
  <script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>
</head>

<body>
  {% crispy people_filter.form %}

  <table border='1' style="width:100%; text-align:center">
    <thead>
      <tr>
        <th> Name </th>
        <th> Surname </th>
        <th> Age </th>
      </tr>
    </thead>

    <tbody>
      {% for person in people_filter.qs %}
      <tr>
        <td> {{ person.name }} </td>
        <td> {{ person.surname }} </td>
        <td> {{ person.age }} </td>
      </tr>
      {% endfor %}
    </tbody>
  </table>

  <script src="{% static 'custom_slider.js' %}"></script> # JS of our range-slider.
</body>

Notice that in the HTML above we load the crispy_forms_tags and use {% crispy people_filter.form %} instead of .as_p. Doing so we let django-crispy-forms package handle our form rendering.

NOTE: The django-crispy-forms package provides two ways to render crispy forms. Common way is to use |crispy filter to render the form but it expects to be wrapped in <form>...</form> HTML tag. In our tutorial, we use {% crispy %} tag because we will generate our form using the FormHelper.

The Django Widget

Django uses widgets to render form-fields and present them as final HTML markup. Therefore, to let Django handle the rendering of our front-end backbone, we need a working widget. In Django, a widget consists of two parts:

a widget-template that represents the final HTML
a class that inherits Django’s Widget class with render() method.

Widget Class

The django-filters package provides a working RangeFilter with a predefined RangeWidget that we seamlessly/automagically used in the previous section. This widget uses two text-fields associated with two text-input HTML form-elements. Notice that it is similar to hidden input-elements required in our case. To make our widget work we will simply rewrite the default RangeWidget provided by django-filter to be compatible with our range-slider. For simplicity let’s call it the CustomRangeWidget:

from django.forms.widgets import HiddenInput
from django_filters.widgets import RangeWidget

class CustomRangeWidget(RangeWidget):
    template_name = 'forms/widgets/range-slider.html'

    def __init__ (self, attrs=None):
        widgets = (HiddenInput(), HiddenInput())
        super(RangeWidget, self). __init__ (widgets, attrs)

    def get_context(self, name, value, attrs):
        ctx = super().get_context(name, value, attrs)
        cur_min, cur_max = value
        if cur_min is None:
            cur_min = ctx['widget']['attrs']['data-range_min']
        if cur_max is None:
            cur_max = ctx['widget']['attrs']['data-range_max']
        ctx['widget']['attrs'].update({'data-cur_min':cur_min,
                                        'data-cur_max':cur_max})
        base_id = ctx['widget']['attrs']['id']
        for swx, subwidget in enumerate(ctx['widget']['subwidgets']):
            subwidget['attrs']['id'] = base_id + "_" + self.suffixes[swx]
        ctx['widget']['value_text'] = "{} - {}".format(cur_min,cur_max)
        return ctx

Widget Template

The widget also requires an associated template. Let’s create a file in /src/templates/forms/widgets/range-slider.html and insert the following content.

<div class="form-group numeric-slider" {% include "django/forms/widgets/attrs.html" %}>
  <div class="numeric-slider-range ui-slider ui-slider-horizontal ui-slider-range"></div>
  <span class="numeric-slider-range_text" id='{{ widget.attrs.id }}_text'>
    {{ widget.value_text }}
  </span>
  {% for widget in widget.subwidgets %}
    {% include widget.template_name %}
  {% endfor %}
</div>

In the above widget-template, we use {% include "django/forms/widgets/attrs.html" %} to let Django handle the widget attributes. It does so by parsing dictionary ctx['widget']['attrs'] from previous part. Likewise, the for loop add widget’s HiddenInput elements.

The Crispy Form

At last, we have our actual widget ready and now we can create a crispy-form with a special template for our slider. This crispy layout template basically helps our widget to fit the Bootstrap markup logic. In other words, it makes it crispy.

Crispy Template

Create a new file /src/templates/forms/fields/range-slider.html. Then add the following template code:

{% load crispy_forms_field %}

<div class="form-group{% if 'form-horizontal' in form_class %} row{% endif %}">
  <label for="{{ field.id_for_label }}" class="{% if 'form-horizontal' in form_class %}col-form-label {% endif %}{{ label_class }}{% if field.field.required %} requiredField{% endif %}">
    {{ field.label|safe }}
    {% if field.field.required %}
      <span class="asteriskField">*</span>
    {% endif %}
  </label>
  {% crispy_field field %}
</div>

NOTE: the above code is based on django-crispy-forms’s bootstrap4 templates and was not tested in bootstrap3 or other crispy template-engine.

Crispy Form Helper

Once the crispy template is ready we need a form where the template will be utilized. Create a file /src/myapp/forms.py and add the following code:

from crispy_forms.helper import FormHelper
from crispy_forms.bootstrap import StrictButton
from crispy_forms.layout import Field, Layout
from django import forms
from django_filters.fields import RangeField

class PeopleFilterFormHelper(forms.Form):
    def __init__ (self, *args, **kwargs):
        super(). __init__ (*args, **kwargs)
        self.helper = FormHelper(self)
        self.helper.form_method = 'get'
        layout_fields = []
        for field_name, field in self.fields.items():
            if isinstance(field, RangeField):
                layout_field = Field(field_name, template="forms/fields/range-slider.html")
            else:
                layout_field = Field(field_name)
            layout_fields.append(layout_field)
        layout_fields.append(StrictButton("Submit", name='submit', type='submit', css_class='btn btn-fill-out btn-block mt-1'))
        self.helper.layout = Layout(*layout_fields)

In the code above the class PeopleFilterFormHelper is nothing different than a simple Django form with a fancy name. However, instead of a common way of constructing Django form we use the Crispy approach with its FormHelper.

NOTE: the FormHelper simply helps you to create a fancy form, which would most certainly be possible to create with the same Django means but with more effort. Our form is rather basic for the sake of clarity of our tutorial, so it is not obvious.

Range Filter

At last, we have everything ready except the actual filtering logic.

Custom Range Filter

Insert following final filter code to the file /src/myapp/filters.py.

from django_filters import FilterSet
from django_filters.filters import RangeFilter
from .models import People
from .forms import PeopleFilterFormHelper
from .widgets import CustomRangeWidget

class AllRangeFilter(RangeFilter):
    def __init__ (self, *args, **kwargs):
        super(). __init__ (*args, **kwargs)
        values = [p.age for p in People.objects.all()]
        min_value = min(values)
        max_value = max(values)
        self.extra['widget'] = CustomRangeWidget(attrs={'data-range_min':min_value,'data-range_max':max_value})

class PeopleFilter(FilterSet):
  age = AllRangeFilter()

  class Meta:
      model = People
      fields = ['age']
      form = PeopleFilterFormHelper

In the code above, PeopleFilter represents the set of filters for model People. Notice the line form = PeopleFilterFormHelper, which overrides default form builder of django-filters to our custom PeopleFilterFormHelper. The actual filter is the AllRangeFilter class, which is a customized version of the original django-filters package’s RangeFilter. We override its __init__ method and initiate our custom widget CustomRangeWidget with initial minimum and maximum values of all possible age values from People model’s age field.

NOTE: I totally agree that list comprehension is far not the best way to get the min. and max. values but this is a tutorial and its for only for educational purpose.

Recap

At last, we created our range filter with a fancy slider that should looks like this:

Summary

In this tutorial, you learned how to create a fancy jQuery range slider with the custom widget for custom range filter provided by django-filters package. Moreover, you learned how to use render the custom widget using django-crispy-forms package. The source code for this tutorial can be found on my GitHub repo. I hope this tutorial was helpful to the reader and eased his suffering while learning these amazing Django packages.

Guide to improve Python performance.

Farid MUSA — Wed, 18 Mar 2020 00:00:00 +0000

Python is an amazing programming language, but it has two huge handicaps compared to compiled languages.

The first one is GIL or Global Interpreter Lock. GIL is the actual process lock that forces the Python interpreters to work on a single process and only uses one of the CPU cores. Due to this lock, Python is both simple and stable but also is very slow compared to other programming languages like C or Java.
The second handicap is rather universal for non-statically typed aka dynamically typed programming languages. Simply put, when you do not specify the data type of the variable that you are going to use and rely on dynamic type assignment, you end up with much slower execution performance.

Luckily there are several ways to speed up your Python code.

Bypass GIL (Concurrency/GIL persist)

In this approach, the execution of the target code is performed in such a way that data is processed in parallel or concurrently. Essentially, it simply means breaking a single task into multiple separate sub-tasks and processing each in a different thread or process. This is also known as multithreading or multiprocessing. This approach is very effective, only if your task can be separated into different tasks.

Before moving on it is important to distinguish between thread and process. A thread and process are two different things, which simply can be used in a similar fashion. However, for different purposes. A single process can create multiple threads, which are limited by OS. All of the threads of any single process share the same memory heap. However, in Python new thread does not mean a new CPU core; hence, the thread does not give you an actual performance boost and rather helps you focus on several tasks simultaneously. Therefore, threads are cheap to create and mostly applied for I/O operations. In contrast, when you create a new process the original memory heap is copied and a new one is created. Two processes cannot see each other’s memory heap, but do work on separate CPU cores. Creating a new process is expensive and ends up in larger overall memory usage. However, the new process provides you with additional CPU cores (up to maximum) and can be created locally or in different computers within a cluster. [3,4]

Here are some examples:

Say that you have a very large file on a disk or maybe a list/array on your memory that you would like to process. Instead of processing all of the data in one single loop and using a single core on your CPU, you can break your objective/data into smaller tasks/chunks. Finally, you can then process each chunk separately in different threads or a process.
Similarly, you can run some piece of Python code inside a single thread that would listen to a network socket for the packages that you might receive from your TCP connection.
Another popular application of concurrency can be seen in data analysis problems. Say that you want to perform computationally expensive operations on large amounts of data but in a short time inside a cluster of computers/servers. In such a case, each process can be run inside a different computer and data can be shared via a network interface.

Python provides several built-in packages for concurrent processing.

concurrent.futures - This module is very easy to use but lacks advanced control.
threading - Useful if you want to do more complicated things with threads like locking threads or running tasks in a queue.
multiprocessing - Useful if you want to do more complicated things with processes like sharing a memory or running tasks in a queue.

Concurrency vs True Parallelism: Parallelism based on concurrency is not true parallelism. A piece of code is executed in a truly parallel way when all of the CPU cores literally work on the same task simultaneously. True parallelism is based on the actual hardware and in general a more complicated topic than concurrency. Python’s GIL prevents true parallel execution of the code. However, fortunately, there are ways to release the GIL and achieve truly parallel processing. [1]

No GIL (True parallelism) + Optional Static Typing

To release GIL and boost your Python code, you need to get your hands a bit dirty. Python is a great high-level programming language, which is also written in other low-level programming languages. The original and most popular Python is in fact is implemented in C and this reference implementation is called CPython. However, there are other implementations of Python with their own advantages and issues. Each is developed by a different community; hence, every Python implementation is as strong as its community. [5]

What about Python packages? At this point, you may ask yourself a question. Are Python packages that I currently use are also available in other Python implementations? Unfortunately, the answer is ambiguous, since it really depends on the implementation. It is definite that there are going to be some incompatibilities. In order to make sure, always check the documentation material.

Here are some popular alternative implementations of Python.

IronPython - is .NET Framework-based implementation of Python written in C# language. It is great if you love writing Python code and need to use features of the .NET Framework.
Jython - is a JVM-based implementation of Python written in Java. It is great if you love writing Python code and need to tightly integrate it with your pure Java backend.
PyPy - is an alternative implementation of Python with a just-in-time (JIT) compiler. Compared to the reference implementation of Python (CPython), PyPy makes use of JIT that provides better performance than the original bytecode compiler implemented in CPython. Therefore, PyPy/Python has better performance than CPython/Python. Also according to PyPy documentation, PyPy supports almost all Python packages so it is a huge plus for PyPy. However, PyPy supports only 32-bit architecture and is not updated as frequently as Python.

Following are not Python implementations

NumPy/Pandas/Dask - “Correct” usage of the many methods implemented in these data analysis packages will almost always give you a significant performance improvement.
Cython - is not an implementation of Python. It is rather a superset of Python that compiles to C. Its syntax is almost the same as Python’s but more C-like because it can understand static typing. Almost any Python code can be compiled using Cython to generate modules with the same functionality but with faster execution time. Similarly, most of the time it is easy to rewrite your C code to Cython code. In addition, many of the built-in modules and methods are optimized for Cython to produce minimal C code before compilation. Furthermore, Cython also supports NumPy and runs very efficiently when using numpy.ndarray. [2]
Numba - is a NumPy-aware JIT compiler for Python. Numba is very user-friendly and can be easily applied to specific pieces of Python code that you would like to speed up.

Important Note: Every approach described in this section has comparable advantages and disadvantages, best usage applications, best practices, learning curves. However, you will need to get your hands dirty if you want to get the best performance out of any approach.

No GIL (True parallelism) + Mandatory Static Typing

The following methods are different from all the methods above because they require sufficient knowledge of a second low-level programming language like C/C++. This section rather describes tools/methods that act as a “bridge” between Python and extension written in a low-level programming language. Therefore, performance is only limited by the low-level programming language and the bridge. At this point, two important questions must be stated before we continue. [5]

If the compiled languages like C, require compilation and linkage of the source code, is it possible to automate compilation during package distribution?
Since compiling C code can be different based on the OS, does that mean that your extension code will not be cross-OS anymore?
How to convert Python data types to C types and vice versa?

There is no single answer to these questions because it depends on what you want to do. Both compiler configuration and automation can be managed using a built-in Python module called distutils, which is very easy to use. However, it really depends on how you design your extension if you want to use it cross-OS. Nevertheless, when it comes to data types there are three logical approaches.

Let C side handle data types:

Python API (CPython) - CPython is a reference implementation of the Python programming language. In other words, CPython is essentially the primary C code, which when compiled generates a Python interpreter and its bytecode compiler. Luckily, CPython/Python developers provide access to the core Python API for C/C++ programmers via Python.h include file. Extension written using this approach can be easily distributed using distutils. It is also possible to embed Python functionality using CPython into your C program.[5]

Let Python side handle data types

cffi (C Foreign Function Interface for Python) - A very powerful tool with a world of its own. CFFI provides several modes for both extending Python with C extension and embedding Python inside the C program. Furthermore, in addition to API mode CFFI also provides ABI mode that can be used to access functions available in any compiled libraries like .dll for Windows or .so for Linux/macOS.
ctypes - This is a built-in Python package that provides a foreign function interface for Python-C. However, this package only provides ABI access to libraries similar to CFFI.

Let the bridge handle data types.

SWIG (Simplified Wrapper and Interface Generator) - This is an old but popular tool that lets you interconnect different programming languages including Python and C. SWIG is one of the oldest tools that made possible interfacing different programming languages but compared to other newer options to bridge Python with C it seems to lose its popularity.
Cython - Cython deserves to re-appear here because of its flexibility. Cython extensions also allow using Python API via Python.h and even can act as an interface for pure C extensions. Cython is very good at understanding both C and Python. [2]

Conclusion

To summarize, let’s make very rough generalizations. Bypassing GIL is very easy to implement and can significantly improve overall performance. In addition, multithreading allows simultaneous I/O without blocking your main code. However, concurrency does not mean true parallelism and can be applied only for specific tasks. Nevertheless, in the second section, we learned about different ways on how to release GIL and still keep it Python. Alternative Python implementations can achieve true parallelism by releasing GIL without losing the flexibility of Python. On the other hand, Cython may require some experience to get the most out of it. Similarly, data analysis packages like Pandas can produce lightning performance if used correctly, but its application depends on the nature of the task. Finally, for those who are not afraid to get their hands dirty, GIL can be released via C extensions. C extensions can provide total control over your code and the best execution speed. However, C knowledge is required because too much control is too much responsibility.

References

“Concurrency vs. Parallelism,” HowToDoInJava. [Online]. Available: https://howtodoinjava.com/java/multi-threading/concurrency-vs-parallelism/. [Accessed: 18-Mar-2020].
K. W. Smith, Cython: a guide for python programmers, First Edition. Beijing Cambridge Farnham Köln Sebastopol Tokyo: O’Reilly, 2015.
A. J. J. Davis, “Grok the GIL: Write Fast and Thread-Safe Python.” [Online]. Available: https://emptysqua.re/blog/grok-the-gil-fast-thread-safe-python/. [Accessed: 18-Mar-2020].
“Java Multi-threading Tutorials,” HowToDoInJava. [Online]. Available: https://howtodoinjava.com/java/multi-threading/. [Accessed: 18-Mar-2020].
M. Summerfield, Python in practice: create better programs using concurrency, libraries, and patterns. Addison-Wesley, 2013.

Educational Resources for for Amplicon Metagenomics

Farid MUSA — Tue, 03 Sep 2019 00:00:00 +0000

Articles

mothur and QIIME
- Description: Very nice article that describes how mothur and QIIME are different and/or similar.
- Link: http://blog.mothur.org/2016/01/12/mothur-and-qiime

Workshops

CoDaSeq workshop
- Description: The intention of this workshop is to give participants a background in Compositional Data Analysis (CoDa) for transcription studies. At the end of the workshop you will have been introduced to CoDa, why you’d use CoDa, and gain hands-on experience using CoDa analysis in R.
- Link: https://github.com/ggloor/CoDa_microbiome_tutorial/wiki
Microbiota Analysis in R [workshop/tutorial]
- Description: The goal of this tutorial is to demonstrate basic analyses of microbiota data to determine if and how communities differ by variables of interest. In general, this pipeline can be used for any microbiota data set that has been clustered into operational taxonomic units (OTUs).
- Link: https://rpubs.com/dillmcfarlan/R_microbiotaSOP

Tutorials

Microbiota Procesing in mothur: standard operating procedure (SOP)
- Description: The goal of this tutorial is to demonstrate standard operating procedures (SOP) for processing amplicon data that are generated using Illumina’s MiSeq platform with paired end reads.
- Link: https://rpubs.com/dillmcfarlan/mothurSOP
16S Metagenomic Analysis Tutorial
- Description: In this tutorial we will explore the primary characterization of a specific human gut microbiome dataset, namely the enterotype dataset from…
- Link: http://web.evolbio.mpg.de/~wang/Site/R_tutorial_files/16S Metagenomic Analysis Tutorial.pdf
An Introduction to Applied Bioinformatics[OnlineBook]
- Description: An Introduction to Applied Bioinformatics (or IAB) is a free, open source interactive text that introduces readers to core concepts of bioinformatics in the context of their implementation and application.
- Link: http://readiab.org Focus mainly on following chapters
- Phylogenetic reconstruction
- Description: Explore the goals, approaches, and challenges for creating phylogenetic trees, or phylogenies.
- Link: http://readiab.org/book/0.1.3/2/4 2. Studying Microbial Diversity
- Description: Explore terminology, methods and approaches for performing microbial diversity analysis.
- Link: http://readiab.org/book/0.1.3/3/1
Microbiome data analysis
- Description: The purpose of this tutorial is to describe the steps required to perform microbiome data analysis, explain how to build your own data flow, and finally, discuss the results obtained in such analysis.
- Link: https://genestack-user-tutorials.readthedocs.io/tutorials/Microbiome_analysis

QIIME2 Tutorials
1. QIIME2 Glossary
2. Description: It is important to first know QIIME2 terminology before starting to learn it.
3. Link: https://docs.qiime2.org/2019.1/glossary
4. Overview of QIIME2 Plugin Workflows
5. Description: After terminology this tutorial is your first step to learn QIIME2.
6. Link: https://docs.qiime2.org/2019.1/tutorials/overview/
7. Microbiome Analysis with QIIME2: A Hands-On Tutorial[Presentation]
8. Description: Nice tutorial for introduction to QIIME2
9. Link: http://compbio.ucsd.edu/wp-content/uploads/2018/07/20180621_oslo_university_microbiome_analysis_with_qiime2_tutorial.pdf
10. “Moving Pictures” tutorial [VeryNiceTutorial!]
11. Description: In this tutorial you’ll use QIIME 2 to perform an analysis of human microbiome samples from two individuals at four body sites at five timepoints, the first of which immediately followed antibiotic usage.
12. Link: https://docs.qiime2.org/2019.1/tutorials/moving-pictures
13. QIIME2 workflow
14. Description: Using the Quantitative Insights Into Microbial Ecology (QIIME2) software pipeline for analysis of marker gene-based microbiome sequencing data
15. Link: https://chmi-sops.github.io/mydoc_qiime2.html
Amplicon Metagenomics
- Description: This tutorial describes a strategy for assembling, filtering and analysing an NGS metagenomic data set in Geneious Prime.
- Link: https://www.geneious.com/tutorials/metagenomic-analysis
Taxonomy from species name in R
- Description: This tutorial describes one of many approaches to generate phylogenetic tree from specie names.
- Link: https://taylorreiter.github.io/2017-07-28-Taxonomy-from-Species-Name-in-R

Software and Tools for Amplicon Metagenomics

Farid MUSA — Tue, 03 Sep 2019 00:00:00 +0000

WebApps/WebTools

Phylo.io
- Description: Nice and clean web application for visualizing phylogenetic trees.
- Link: http://phylo.io/
MicrobiomeAnalyst
- Description: MicrobiomeAnalyst is a comprehensive statistical, visual and meta-analysis of microbiome data.
- Link: https://www.microbiomeanalyst.ca

Software and Tools

QIIME2
- Description: QIIME2 is a next-generation microbiome bioinformatics platform that is extensible, free, open source, and community developed.
- Link: https://docs.qiime2.org/2019.7/install
mothur
- Description: mothur aims to be a comprehensive software package that allows users to use a single piece of software to analyze community sequence data.
- Link: https://github.com/mothur/mothur/releases/tag/v.1.43.0
KRAKEN
- Description: Kraken is an ultrafast and highly accurate program for assigning taxonomic labels to metagenomic DNA sequences.
- Link: http://ccb.jhu.edu/software/kraken
USEARCH
- Description: USEARCH is a unique sequence analysis tool with thousands of users world-wide. USEARCH offers search and clustering algorithms that are often orders of magnitude faster than BLAST.
- Link: https://www.drive5.com/usearch
VSEARCH
- Description: VSEARCH is an open source and free of charge multithreaded 64-bit tool for processing and preparing metagenomics, genomics and population genomics nucleotide sequence data. It is designed as an alternative to the widely used USEARCH tool.
- Link: https://github.com/torognes/vsearch
MG-RAST
- Description: MG-RAST is an open source, open submission web application server that suggests automatic phylogenetic and functional analysis of metagenomes.
- Link: https://www.mg-rast.org
PICRUSt
- Description: PICRUSt (pronounced “pie crust”) is a bioinformatics software package designed to predict metagenome functional content from marker gene (e.g., 16S rRNA) surveys and full genomes.
- Link: http://picrust.github.io/picrust
pplacer
- Description: Pplacer places query sequences on a fixed reference phylogenetic tree to maximize phylogenetic likelihood or posterior probability according to a reference alignment.
- Link: https://matsen.fhcrc.org/pplacer
The Exelixis Lab [Curated Collection of Software/Tools]
- Description: Include software/tools for:
- Phylogenetic Inference
- Phylogenetic Post-Analysis
- Next Generation Sequencing & Sequence Analysis
- Population genetics
- Web services
- etc.
- Link: https://cme.h-its.org/exelixis/software.html

R packages

microbiome
- Description: Tools for the exploration and analysis of microbiome profiling data sets, in particular large-scale population studies and 16S taxonomic profiling.
- Tutorial: https://microbiome.github.io/tutorials
- Link: https://github.com/microbiome/microbiome
microbiomeSeq
- Description: An R package for microbial community analysis in an environmental context
- Tutorial: http://userweb.eng.gla.ac.uk/umer.ijaz/projects/microbiomeSeq_Tutorial.html
- Link: https://github.com/umerijaz/microbiomeSeq
matR - Metagenomics Analysis Tools
- Description: An analysis platform for metagenomics combining specialized tools and workflows, easy handling of the BIOM format, and transparent access to MG-RAST resources. Integrates easily with other R packages and non-R software.
- Tutorial: https://www.mcs.anl.gov/~braithwaite/matR/docs-and-pubs/matR-user-manual.pdf
- Link: https://rdrr.io/cran/matR
MonoPhy
- Description: The R package MonoPhy allows assessment and exploration of monophyly of taxa in a phylogeny.
- Tutorial: https://cran.r-project.org/web/packages/MonoPhy/vignettes/MonoPhyVignette.html
- Link: https://cran.r-project.org/web/packages/MonoPhy/index.html