DEV Community: Amartya Gaur

I Built Format-on-Save for Everything That Isn't an Editor

Amartya Gaur — Sun, 23 Aug 2026 11:15:57 +0000

Editors have had format-on-save for years. The rest of the development environment has not.

When a generator, shell command or coding agent writes a file, the bytes usually land exactly as produced. The repository may already have Prettier, Ruff or gofmt, but nothing asks it to run. The mistake is found later by a pre-commit hook, CI, or the agent itself after it has moved on.

I built onwrite to run the project's own tools at the point of the write.

The command I use most is:

$ onwrite run -- ./scripts/codegen.sh
onwrite: formatted src/client.ts (eslint, prettier)
onwrite: formatted api/models.py (ruff-check, ruff-format)

onwrite run waits for the wrapped command to finish, works out which files it changed, and formats only those files. The command's input, output and exit code pass through unchanged.

There is also a watcher for writes that cannot be wrapped:

$ onwrite watch

And a diagnostic command, because automatic detection is only useful when it can explain itself:

$ onwrite doctor

onwrite does not bring its own formatting rules. It looks for the choices already present in the repository: a Prettier config, Ruff settings in pyproject.toml, .clang-format, Cargo.toml, the Go toolchain, and so on. Project-local binaries are resolved before global ones, so the pinned version wins.

The implementation is deliberately cautious. A formatter never receives the real source file. The contents go in through stdin, the result comes back through stdout, and the file is replaced atomically only after the formatter chain has succeeded.

That choice handles the failure case I cared about most. If a watcher catches a heredoc halfway through, the partial file will normally fail to parse. The formatter exits with an error and onwrite leaves the file alone. A formatter that hangs is timed out. One that exits successfully but returns nothing is not allowed to empty the file.

The watcher also avoids the usual “formatter wrote the file, so run the formatter again” loop without using a blind delay. It remembers the exact content it wrote. Its own write is ignored, while a real edit made immediately afterwards still has different content and is processed.

For Claude Code, a hook reports when formatting changed a file the agent just wrote. That gives the agent a chance to re-read it instead of continuing with a stale copy. The core tool is not tied to Claude; anything that can run in a shell can be placed after onwrite run --.

The first release has 87 tests, including partial writes, hanging tools, ignored files and write-loop cases. CI runs the race detector and integration tests against real formatters on Linux and macOS. Release binaries are available for macOS, Linux and Windows.

It is still v0.1.0. I expect real repositories to uncover combinations I have not seen yet.

Install it on macOS or Linux:

curl -fsSL https://raw.githubusercontent.com/hunr-ai/onwrite/main/scripts/install.sh | sh

Then run:

onwrite doctor
onwrite run -- your-command-here

The project is MIT licensed: github.com/hunr-ai/onwrite.

I wrote a more detailed account of the safety model and the decisions behind run, tool detection and agent synchronization on my site: Format-on-save, after the editor.

If you try it, tell me where its detection was wrong. That is the feedback most likely to improve the next release.

A container per job, without a daemon

Amartya Gaur — Thu, 20 Aug 2026 11:46:36 +0000

Some workloads want an image each. A build that needs a specific toolchain, a test that needs a specific database client, a task whose dependencies are described by something other than a human.

The instinct is a Dockerfile and a builder. That means a daemon or a rootless equivalent, a build job in the path of every request, a registry filling with near-identical layers, and, the part that turned out to matter, an arbitrary program running at build time on input you did not write.

There is another way to get an image, and it starts by giving up the thing everyone assumes you need.

An image is just files and a manifest

A container image is a set of tarballs plus a JSON document describing them. Nothing requires that the tarballs were produced by executing commands. If you can state which packages you want, a tool can resolve them, lay them out, and write the manifest. No shell, no intermediate containers, nothing executed at all.

That is what apko does. You hand it a package list:

contents:
  repositories: [https://packages.wolfi.dev/os]
  packages:
    - python-3.12
    - py3.12-pytest
    - busybox
accounts:
  users: [{ username: runner, uid: 65532 }]
  run-as: 65532
entrypoint:
  command: /bin/sh -l

and it produces an OCI image. Reproducibly, same list and same image digest, because there is no build-time entropy to be non-deterministic about.

The absence of RUN is the whole point

Look at what is missing from that file. There is no RUN. There is nowhere to put one.

If a person, or a model, or a pipeline describes an environment, and that description is a package list, then the worst thing a malicious description can do is ask for a package. It cannot curl a script. It cannot read your build secrets. It cannot write to a layer and then delete the evidence. The input is data, and it is treated as data.

With a Dockerfile the input is a program, and the build is an execution of it, on your builder, with whatever that builder can reach. Every hardening measure after that point, rootless and no network and a scrubbed environment, is you re-imposing a constraint that the format gave away by design.

Removing the build step does not weaken the sandbox. It removes a sandbox you would otherwise have had to build.

Assembly is a manifest rewrite

The other benefit shows up when you want many similar images: one base, small variations.

Because an image is layers plus a manifest, swapping a base underneath a set of layers is a metadata operation. crane rebase does it against the registry:

crane rebase myimg:variant \
  --old_base base:v1 \
  --new_base base:v2 \
  --tag       myimg:variant-v2

No pull, no push of anything large, no rebuild. In my case the base is cached by the fingerprint of its package list. Building it fresh takes about fourteen seconds, and thereafter assembling a variant is a manifest write measured at 0.227 seconds with zero layer uploads.

Step	Time
Base, built fresh	14 s
Variant, rebased	0.227 s

Time to a runnable image, once the base is cached. The second row is a manifest write. Nothing is pulled and nothing large is pushed, because swapping a base underneath a set of layers is a metadata operation rather than a build.

The shape that falls out:

Fingerprint the package list. Base already built? Reuse it.
Lay the job's own files in as one thin layer.
Rebase and tag. Done in under a second.
The runner pulls by digest.

Pin by digest rather than tag, and add a CI check that fails if a manifest's digest drifts from what the repository says it should be. Otherwise "reproducible" quietly becomes "reproducible until somebody moves a tag."

What you give up

This is not a general replacement for Docker builds, and pretending otherwise would waste your time.

You are limited to what is packaged. Wolfi's repository is good and fast-moving, but if your dependency is not there you are building an apk, which is a real cost. Anything genuinely requiring a build step, compiling from source or running a bundler, still needs a builder; the honest pattern is to build those artefacts somewhere with a real toolchain and have apko compose the runtime image around them.

And you need to actually want the property. If the environment descriptions all come from your own team and go through review, a Dockerfile is fine and you know how it works. This trade only pays when something other than a person is writing the description, and in that case it pays immediately, because the question "what can this input do to my builder" gets the best possible answer, which is nothing.

Taking the network away from a Cloud Run job

Amartya Gaur — Thu, 20 Aug 2026 11:46:20 +0000

If you execute other people's code, at some point you have to decide what it is allowed to reach. Mine should reach nothing: no package registry, no pastebin, no callback to a server that tells it the answer. It should see its own filesystem, its own loopback interface, and the end of the world in every other direction.

The usual answers are all heavy. Run your own Kubernetes with a deny-all NetworkPolicy and now you operate a cluster. Use Firecracker directly and now you operate a fleet. Both are correct and both cost you an ops function you may not want.

What I wanted was the managed thing, where you hand a container to a platform and get told when it exited, with the network removed. Here is how that works on Cloud Run, and the two details that are not in the docs.

The prefix

A Cloud Run Job runs whatever command you give it. So give it a command that drops the network first:

unshare -rn -- ./run-the-untrusted-thing

unshare -r maps your current user to root inside a new user namespace, which is what earns you the right to create the next one without privileges. -n is the one that matters: a new network namespace. A fresh network namespace contains no interfaces except a loopback device, and no route to anything. Nothing you can do inside it reaches the outside, because from in there, there is no outside.

cloud run job · gen2 microvm
┌──────────────────────────────────────────────────────────────┐
│  ┌─ unshare -rn ───────────────────┐                         │
│  │  ┌──────────────────┐           │                         │
│  │  │ candidate code   │           │    registries,          │
│  │  └──────────────────┘           │    pastebins,           │
│  │  ┌──────────────────┐           │    a server with        │
│  │  │ lo · 127.0.0.1   │──── ✗ ────┼──  the answer           │
│  │  └──────────────────┘  no route │                         │
│  └─────────────────────────────────┘                         │
└──────────────────────────────────────────────────────────────┘

Pull happens before this. The image is already on the machine when your command starts, so removing egress at the top of the run costs you nothing you needed.

Why the execution environment matters

That prefix only works on Cloud Run's second-generation execution environment, and the difference is the whole reason to care which one you are on.

Gen1 is gVisor: a user-space kernel that intercepts syscalls and implements a subset of them. Gen2 is a microVM with a real Linux kernel. Creating user and network namespaces is exactly the sort of operation a syscall-filtering layer declines to support, so on gen1 the unshare fails and you are left wondering why a command that works on your laptop does not work in the job.

executionEnvironment: EXECUTION_ENVIRONMENT_GEN2

One field. Worth writing a comment above it, because it looks like a performance knob and it is actually a capability boundary.

The detail that will waste your afternoon

Drop the network and every test that talks to a local server fails. Not the interesting ones. All of them. A web framework's test client, a database on 127.0.0.1, anything that binds a port and connects to itself.

This is not the sandbox working. This is a fresh network namespace starting with its loopback interface down. lo exists, it is just not up, and nothing has told it to be.

So the run phase needs a shim that raises loopback and then gets out of the way:

import fcntl, socket, struct, os, sys

SIOCGIFFLAGS, SIOCSIFFLAGS, IFF_UP = 0x8913, 0x8914, 0x1

def raise_loopback():
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    ifr = struct.pack("16sh", b"lo", 0)
    flags = struct.unpack("16sh", fcntl.ioctl(sock, SIOCGIFFLAGS, ifr))[1]
    fcntl.ioctl(sock, SIOCSIFFLAGS, struct.pack("16sh", b"lo", flags | IFF_UP))

raise_loopback()
os.execvp(sys.argv[1], sys.argv[1:])

Ten lines, and it does not weaken anything. Loopback goes nowhere by definition; the namespace still has no route off the box. You get the isolation you asked for and tests that pass for the right reasons.

The execvp at the end is deliberate: the shim replaces itself with the real command rather than supervising it, so there is no extra process in the tree, no signal forwarding to get wrong, and the exit code is the exit code.

Turn off retries

The last one is not about networking, and it is the one I would most want somebody to tell me.

Managed job runners retry failed executions by default. That is right for a data pipeline and quietly catastrophic for anything that grades. A flaky test that fails, retries, and passes has just been recorded as a pass. Nobody in the system is lying; the platform did what it was configured to do, and the result is wrong.

maxRetries: 0

If a run fails, it failed. Deciding what that means is your job, not the platform's.

What this does not give you

A namespace is an isolation boundary, not a security guarantee against a determined attacker with a kernel exploit, and you are on a shared managed platform. Set your CPU and memory limits, set a deadline, treat everything the run produces as hostile input, and do not confuse "cannot reach the network" with "cannot do anything".

It does, however, get you a very strong property for one config field, one prefix, and ten lines of ioctl: code you did not write, running on infrastructure you do not operate, with nowhere to phone.

Building a Dynamic MCP Proxy Server in Python

Amartya Gaur — Mon, 09 Feb 2026 07:52:13 +0000

The Model Context Protocol (MCP) is rapidly becoming the standard for connecting AI models to external tools and data. As you start building or using more MCP servers—one for your database, one for web search, another for local files—you might find yourself juggling multiple connections.

What if you could have a single entry point? A "Proxy" server that sits in the middle and dynamically routes your AI's requests to the right place?

In this post, I'll show you how to build a simple, dynamic MCP Proxy Server using Python and the fastmcp library.

What We're Building

We are going to build a server that:

Acts as an MCP Server itself: Your AI client (like Claude Desktop or an IDE) connects to this one server.
Manages other Servers: It has tools to add_proxy and list_proxies.
Persists Configuration: It saves your proxy list to a JSON file so they survive restarts.
Mounts Dynamically: It automatically connects to your registered proxies on startup.

Prerequisities

You'll need Python installed. I highly recommend using uv for modern Python package management, but pip works too.

uv init mcp-proxy
cd mcp-proxy
uv add "fastmcp>=2.14.5" "uvicorn>=0.40.0"

Step 1: The Configuration

First, we need a way to save our proxy configurations. We'll use a simple proxies.json file.

import json
import os

CONFIG_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "proxies.json")

def load_proxies():
    if not os.path.exists(CONFIG_FILE):
        return []
    try:
        with open(CONFIG_FILE, "r") as f:
            data = json.load(f)
            return data.get("proxies", [])
    except Exception as e:
        print(f"Error loading config: {e}")
        return []

def save_proxy(name, url):
    proxies = load_proxies()
    # Update if exists, otherwise append
    for p in proxies:
        if p["name"] == name:
            p["url"] = url
            break
    else:
        proxies.append({"name": name, "url": url})

    with open(CONFIG_FILE, "w") as f:
        json.dump({"proxies": proxies}, f, indent=4)

Step 2: The Proxy Server

Now for the magic. We use FastMCP to create our server and FastMCPProxy to wrap remote servers.

from fastmcp import FastMCP
from fastmcp.server.proxy import ProxyClient as _ProxyClient
from fastmcp.server.proxy import FastMCPProxy

# Initialize our main server
mcp = FastMCP("MCP Proxy")

@mcp.tool()
async def add_proxy(name: str, url: str) -> str:
    """Register a new MCP server to proxy."""
    try:
        save_proxy(name, url)

        # Create a factory that returns a client for the URL
        def client_factory():
            return _ProxyClient(url)

        # Create a proxy wrapper
        proxy_server = FastMCPProxy(client_factory=client_factory, name=name)

        # Mount it to our main server
        mcp.mount(proxy_server)

        return f"Added and mounted {name}"
    except Exception as e:
        return f"Error adding proxy: {str(e)}"

@mcp.tool()
def list_proxies() -> str:
    """List all registered proxy servers."""
    proxies = load_proxies()
    if not proxies:
        return "No proxies registered."
    return "\n".join([f"{p['name']}: {p['url']}" for p in proxies])

Step 3: Startup Logic

We want our saved proxies to be available immediately when we restart the server.

def mount_existing_proxies():
    """Mount proxies defined in the config file on startup."""
    proxies = load_proxies()
    for p in proxies:
        try:
            def make_factory(u):
                return lambda: _ProxyClient(u)

            ps = FastMCPProxy(client_factory=make_factory(p["url"]), name=p["name"])
            mcp.mount(ps)
            print(f"Mounted proxy: {p['name']}")
        except Exception as e:
            print(f"Startup mount error for {p['name']}: {e}")

# Run this before starting the server
mount_existing_proxies()

if __name__ == "__main__":
    mcp.run()

Implementation Details

The key component here is mcp.mount(). FastMCP allows you to mount one server onto another.

When we use FastMCPProxy, we are essentially saying: "Create a virtual MCP server that just forwards everything to this URL." By mounting that virtual server onto our main mcp instance, our main server instantly gains all the tools and resources of the remote server.

Running It

Run your server using uv:

uv run server.py

Now, connect your MCP client (like Claude Desktop) to this server. You can ask Claude:

"Add a proxy named 'weather' pointing to http://localhost:8100/sse"

And just like that, Claude now has access to the weather tools through your proxy.

Conclusion

This pattern is incredibly powerful. It allows you to build a "Router" for your AI agents, decoupling the implementation of specific tools from the main connection. You can add, remove, or update backend services without ever disconnecting your AI client.

Check out the full code on GitHub!

A cookiecutter for Django Rest Framework projects.

Amartya Gaur — Mon, 12 Dec 2022 06:10:47 +0000

TLDR;

I made a cookiecutter for DRF projects, you can check it out here:
https://github.com/Ohuru-Tech/drf-cookiecutter

Now, if you do want to read the rest of it:

DRF (Django Rest Framework) is just awesome !! I have been working a lot in DRF and I find myself writing same code multiple times for different projects, it is a tedious process and goes strongly against the DRY (don't repeat yourselves) principle... (I know that applies to a single application, bear with me..). After writing the exact same configuration code for setting up a new project thousands of times, I finally came across this gem:

https://cookiecutter.readthedocs.io/en/stable/

This helped me come up with a template (sort of) which will serve as the starting point for a new Django Rest Framework project. I am writing this post to talk about it's features and urge anyone interested to try and help make it better :)

So, let's get to it:

Project outline

What's included:

Settings via environment variables with django-environ.
Celery configuration.
Django REST Framework configuration + Swagger with drf-yasg.
Typical user CRUD APIs.
Code formatting with Black.
Imports sorting with isort.
Pre-commit hook for running test & linters on each commit.
Custom user model
Custom documentation with drf-yasg.
Authentication setup using dj-rest-auth

If you want to just see it in action, you can skip the details below and just try it out:

Create python virtual environment.

python3 -m venv venv
source venv/bin/activate

Install cookiecutter and packaging:

pip install cookiecutter packaging

Step into directory you want to create project and generate project with cookiecutter:

cd /path/to/directory
cookiecutter https://github.com/Ohuru-Tech/drf-cookiecutter

Answer the questions in wizard.

Steps after project setup

Install the dependencies:

pip install -r requirements_Dev.txt

Copy .env file from example file and set your settings:

cp .env.template .env

Run migrations:

python manage.py migrate

Start building your awesome project!

This section will mostly talk about specific project details:

App Structure

All the apps are located in the {{ cookiecutter.project_slug }}/apps folder. The project uses a custom startapp command, refer {{ cookiecutter.project_slug }}/management/commands/startapp.py for more information.

The command initializes a new app with the following:

A models directory as opposed to a generic models.py file which is created by Django's default startapp command, this allows you to have models separated by their use-case and helps to keep the project clean.
A views directory which corresponds to the above reasoning for having a models directory.
A seriazliers directory which corresponds to the above reasoning for having a models directory.

The other files and folders are similar to how Django does (admin.py, apps.py).

This command ensures that the apps are added to the apps folder and are initialized in accordance with the project-setup.
To add a new app run:

$ python manage.py startapp <app_name>

Then include the app to the INSTALLED_APPS list in config/base.py like so:

INSTALLED_APPS = [
    ...
    '{{ cookiecutter.project_slug }}.apps.<app_name>.apps.<app_name_capitalized>Config',
    ...
]

eg: if your app_name is foo then this would be:

INSTALLED_APPS = [
    ...
    '{{ cookiecutter.project_slug }}.apps.foo.apps.FooConfig',
    ...
]

It is always nice to include the complete config while adding the app in the INSTALLED_APPS list. This maintains consistency when using signals.

Settings

The project uses django-configurations for class based settings, there are two configurations available, Dev and Prod, the common settings exist in config/base.py, Dev in config/dev.py and Prod in config/prod.py.

You can use the DJANGO_CONFIGURATION environment variable to switch between the configurations, the Dev settings are used by default.

Environment Variables

Most of the configurations in the project are based on environment variables, you have the following options to use them:

Set them up in a .env file in the root of the project, i.e. the place where you have this README.md file, you can refer to the .env.template for the available options and easily create your configuration file using that.
Set them up via the environment variables themselves.
Set them up in the docker / kubernetes runtime via the .env file, most of the things still remain the same except that you will need to get rid of the comments in .env.template while creating the .env file.

You can still run the project without creating any of those files because there are defaults available to almost all of the required settings but, you will need to set up the db instances, rabbit mq (in case you want to use celery) and make sure the settings match the ones used by default in the configurations. It is recommended to always have a .env file managing these settings.

NOTE:- The values for {{ cookiecutter.project_slug | upper() }}_PASSWORD_RESET_URL and {{ cookiecutter.project_slug | upper() }}_EMAIL_VERIFICATION_URL needs to be set in order for these flows to work properly, they make sure that the redirection happens to the frontend URLs for password reset and email verification.

The email verification could either be a page that gets and processes the link and sends the corresponding request to BE for email verification or have a page that waits for an explicit confirmation with the user.

Documentation

The project uses drf-yasg to auto-create documentation for the API endpoints, but the generated documentation is not very good, hence, this project uses a custom schema to manage the documentation.

The details for each endpoint are managed using a class, for example, the accounts app supports profile update, so the ProfileUpdate class is used to manage the documentation for that endpoint, it looks something like this:

class ProfileUpdate:
    desc = f"""
Profile Update is an API endpoint to update the profile associated
with a particular user.

By using this endpoint, you can make a patch call to update the profile
associated with any user. The {fields_to_md(ProfileCreateUpdateSerializer.fields_names)}
are the fields that can currently be updated
    """  # noqa

    responses = {
        "200": openapi.Response(
            description="OK",
            examples=profile_update_200_example,
            schema=UserInfoSerializer,
        ),
        "401": openapi.Response(
            description="Unauthorized", examples=profile_update_401_example
        ),
        "403": openapi.Response(
            description="Permission Denied",
            examples=profile_update_403_example,
        ),
    }

    code_examples = [{"lang": "Bash", "source": profile_update_curl}]

    swagger_setup = {
        "operation_id": "Profile Update",
        "operation_description": desc,
        "request_body": ProfileCreateUpdateSerializer,
        "responses": responses,
        "code_examples": code_examples,
    }

The examples come from config/examples and look something like this:

profile_update_200_example = {
    "application/json": {
        "id": 1,
        "name": "Test developer",
        "profile_pic": "/some_pic.jpg"
    },
}

profile_update_401_example = {
    "application/json": {
        "error": "NotAuthenticated",
        "detail": "Incorrect authentication credentials.",  # noqa
    }
}

profile_update_403_example = {
    "application/json": {
        "error": "PermissionDenied",
        "detail": "You do not have permission to perform this action."
    }
}

After setting up the class, you can use the swagger_auto_schema decorator to plug the specification into the view.

@method_decorator(
    swagger_auto_schema(**ProfileUpdate.swagger_setup), "partial_update"
)
class ProfileViewSet(UpdateModelMixin, PsqMixin, GenericViewSet):
    ...

In the above example, the ProfileUpdate class we declared earlier is used to plug those settings onto the partial_update action.

The views

The project is set up to use GenericViewSet, it further uses different mixins to provide the create, retrieve, update and destroy actions.
You can set up custom actions for anything else that you might need to do.

The views also use the PsqMixin to configure different serizlizers and permissions for the actions. The serializers come from the serializers directory and the permissions are from the permissions directory (or file, as per your convinience).

Routers

The GenericViewSet automatically sets up the URLs based on the mixins you use with it, it also adds the custom actions as urls. Thus, we have urls/versioned_urls.py which uses a SimpleRouter to add these viewsets and plug them at different endpoints.

Versioning

The version comes from the environment variables, you can set it up in the .env file. To add and support different versions, you can have differnt urls list in the urls/versioned_urls.py file, then update the environment to default to the latest version. You can manually include the other versions in urls/__init__.py as and when needed.

Database

The project supports PostgreSQL, PostGIS, MySQL, MySQL (GIS), Oracle, Oracle (GIS), Redshift, CockroachDB, and SQLite, you can refer this table in order to find out the details on forming the url corresponding to your database.

Deployment

I have included a Dockerfile with a custom bash script entrypoint that can help you get both the django backend and celery worker + beat scheduler up and running in a jiffy...

I plan to leverage kubernetes and make a helm chart in future to easily set up a production grade server with nginx but I can use some help around this :P...

Anyway, that's almost as much as I wanted to say about this, please check the project out, do leave a star on the repository if you like it and feel free to make PRs to improve it.

Its Prime Media, a website built completely with Django

Amartya Gaur — Tue, 27 Oct 2020 04:30:11 +0000

While people are quickly switching over to MEAN stack, MERN stack, MEVN stack, and what not? Django still is a very powerful framework for web development and let's not forget the plain and simple HTML, CSS, Bootstrap, and Vanilla JS.

The combination of Django and Bootstrap templates are a very good way to make a good looking fully mobile-friendly website, Its Prime Media is a great example of that.

About the website

Its Prime Media is a global media brand that enables people to stay updated and informed with all the latest happenings around the world. They curate, aggregate, and convene the latest articles about News, Movies, Lifestyle, Millionaires, Fashion, and Technology.

They as a media platform recognize their purpose-driven responsibility to amplify stories and voices of the general public that compel change and ensure that each person is heard via citizen journalism.

Its Prime Media is also home to creative people for showcasing their work to millions of users and also aims to provide employment opportunities to them.

Latest articles

Get the latest updates that matter, read data aggregated from your favorite and trusted sources, and stay updated with all the happenings around the globe.

The voice of people

Be heard. All voices matter at Its Prime Media, you can publish any matters of interest or concern.

Get Paid for your Creativity

Display your works on Its Prime Media, possibly get hired but most importantly get applauded for your work in form of small donations. They are open to fashion designers, photographers, graphic designers, poets, and artists.

How it works?

Since this is a technical blog, this is the section you probably have been waiting for :)

If you go and look at the website, you will be able to notice that the server automatically picks your country and displays the articles filtered on your country (currently only the USA, UK, and India are supported). If you want them to extend support to your country, you can always contact them. This is handled by geoIP2.

def get_client_ip(req):
    x_forwarded_for = req.META.get("HTTP_X_FORWARDED_FOR")
    if x_forwarded_for:
        ip = x_forwarded_for.split(",")[0]
    else:
        ip = req.META.get("REMOTE_ADDR")
    return ip

def get_current_country(request):
    current_ip = get_client_ip(request)
    logger.info(f"Recognized IP: {current_ip}")

    try:
        geoIpObject = GeoIP2()
        current_country = geoIpObject.country(current_ip)["country_name"]
        logger.info(f"Recognized Country : {current_country}")

    except AddressNotFoundError:
        current_country = "United States"

    return current_country

The website gets updated every three minutes, gathering data from major sources from those three countries, that is handled using Celery and background asynchronous tasks.

@shared_task
def periodic_update(feed_id):
    feed = get_object_or_404(Feed, id=feed_id)
    urls_in_feed = sc.get_urls_from_feed(feed.url)
    for url in urls_in_feed:
        try:
            UrlsForFeed.objects.create(
                url=url,
                feed=feed
            )
            logger.info("Getting latest content now")
            NewsArticle.objects.create(...data)                
            logger.info(f"Done updating the article titled : {data['title']}")
        except IntegrityError as error:
            logger.info("Skipping this article")
            logger.debug(f"Encountered an Integrity error {error}")
            pass

This task gets added as soon as a new source is added.

User management is pretty much automatic except for an added profile and email verification.

Since the website provides its users with an opportunity to write their own content and get it published, it calls for a rich text uploading feature, which can be very easily added using ckeditor.

Conclusion

What I wanted to show with this blog is that making powerful websites completely with Django is fast, easy, and completely doable, while MEAN, MERN, MEVN have their advantages, Django can also handle async with extensions (this might change to inbuilt support soon). Another option is to use the DVN or DRN or DAN stacks (yeah made them up just now, Django Vue Node or Django React Node or Django Angular Node). The decision is up to you.

PS:-
All Django lovers:
DO CHECKOUT THE WEBSITE

A voting app (CRUD) using Django Rest Framework and Vue.JS

Amartya Gaur — Sun, 09 Aug 2020 12:05:21 +0000

I am going to keep it short and simple, I wanted to experiment with Vue.JS so I just read up a bit and gathered resources from here and there to build a basic app using DRF and Vue.JS, this will not be perfect as this blog is made after spending less than 3 hours on Vue, let me know in the comments if there is something that could have been handled in a better way...

What are we building?

In this post we will be making a very simple voting app with the following functionalities:

The admin is able to register new candidates
Normal users are able to view and vote for candidates (only once, tracked using the IP address of the request)

If you just want the code then: https://github.com/amartya-dev/vote_drf_vue (Do star the repo in case you find it useful :P)

The Backend with Django Rest Framework

The application architecture

The requirements are simple, we need a candidate table (model/entity or whatever you want to call it) which will contain the details about the candidates and to track the votes and the IP addresses we would need another Vote table which contains the IP address and the candidate voted for.

We want to be able to get the votes directly with the candidate information for easy access thus, it might be a good idea to include the total number of votes there.

We would need to set up your Django project at this point, so let us quickly create a project and the main app inside it via:

django-admin startproject coda
cd coda/
python manage.py startapp main

As it is pretty clear at this point, our project is called coda, and the app is called main.

Let us code the models for our application according to the above constraints (the following code goes in coda/main/models.py):

class Candidate(models.Model):
    name = models.CharField(max_length=250)
    no_challenges_solved = models.IntegerField()
    votes = models.IntegerField(default=0)
    python_rating = models.IntegerField(default=1)
    dsa_rating = models.IntegerField(default=1)
    cplus_rating = models.IntegerField(default=1)
    java_rating = models.IntegerField(default=1)

    def __str__(self):
        return self.name


class Vote(models.Model):
    ip_address = models.CharField(
        max_length=50,
        default="None",
        unique=True
    )
    candidate = models.ForeignKey(
        to=Candidate,
        on_delete=models.CASCADE,
        related_name='vote'
    )

    def save(self, commit=True, *args, **kwargs):

        if commit:
            try:
                self.candidate.votes += 1
                self.candidate.save()
                super(Vote, self).save(*args, **kwargs)

            except IntegrityError:
                self.candidate.votes -= 1
                self.candidate.save()
                raise IntegrityError

        else:
            raise IntegrityError

    def __str__(self):
        return self.candidate.name

I have overridden the save() method of the Vote model to achieve the following:

As I am maintaining the number of votes for each candidate, as soon as there is a request for a vote I append the number of votes of the associated candidate. The catch here is that in case there is a repeated request the incremented votes' value needs to be decremented again. Thus, I use the except block to do precisely that.
I wrote the conditional to check the commit flag so that I can save an instance of the model without committing the transaction to the database.

The serializers

To be able to write the API and corresponding views we would need the serializers to parse the data to JSON and vice versa.
Create a file called serializers.py inside coda/main/, I will be creating two serializers here:

The candidate serializer which I am going to use for the CRUD operations for candidates and
The Vote serializer which I am going to use to just allow the cast of a vote. Thus, I have overridden the create() method where I am just returning an object of the Vote class without committing the entry into our DB, reason: I would be adding the IP address in views for which I just need the object as sort of a baseline. Also, I am using candidate_name to make easily send that data from the frontend and get the corresponding candidate instance. You might want to change that to id in case the uniqueness of candidate names is not guaranteed.

from rest_framework import serializers
from main.models import Candidate, Vote
from django.shortcuts import get_object_or_404
from django.db import IntegrityError


class CandidateSerializer(serializers.ModelSerializer):
    votes = serializers.ReadOnlyField()

    class Meta:
        model = Candidate
        fields = "__all__"


class VoteSerializer(serializers.ModelSerializer):
    candidate_name = serializers.CharField()

    def create(self, validated_data):
        candidate = get_object_or_404(Candidate, name=validated_data["candidate_name"])
        vote = Vote()
        vote.candidate = candidate
        try:
            vote.save(commit=False)
        except IntegrityError:
            return vote
        return vote

    class Meta:
        model = Vote
        exclude = ("id", "ip_address", "candidate")

Views

The time to finally write our logic for all the operations we need from this application, I am using generic viewsets and views provided by Django Rest Framework, I use a ModelViewSet for candidates CRUD operations and very generic APIView for casting the vote like so:

from rest_framework.viewsets import ModelViewSet
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
from rest_framework.permissions import IsAdminUser
from main.serializers import VoteSerializer, CandidateSerializer
from main.models import Candidate
from django.db import IntegrityError


def get_client_ip(request):
    x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
    if x_forwarded_for:
        ip = x_forwarded_for.split(',')[0]
    else:
        ip = request.META.get('REMOTE_ADDR')
    return ip


class CandidateViewSet(ModelViewSet):
    queryset = Candidate.objects.all().order_by('-votes')
    serializer_class = CandidateSerializer
    permission_classes = [IsAdminUser, ]


class CastVoteView(APIView):

    def post(self, request):
        serializer = VoteSerializer(data=request.data)
        if serializer.is_valid(raise_exception=ValueError):
            created_instance = serializer.create(validated_data=request.data)
            created_instance.ip_address = get_client_ip(request)

            try:
                created_instance.save()

            except IntegrityError:
                return Response(
                    {
                        "message": "Already voted"
                    },
                    status=status.HTTP_400_BAD_REQUEST
                )

            return Response(
                {
                    "message": "Vote cast successful"
                },
                status=status.HTTP_200_OK
            )

I use the uncommitted object I get from serializer's create() function and add the IP address from request to it before finally committing the entry to the database.

The URLS

Let us wrap this up by binding our views to URLs, create a file called coda/main/urls.py and add:

from django.urls import include, path
from rest_framework import routers
from main import views as main_views

router = routers.DefaultRouter()
router.register(r'candidate', main_views.CandidateViewSet)

app_name = 'api'
urlpatterns = [
    path('', include(router.urls)),
    path('vote/', main_views.CastVoteView.as_view(), name='vote')
]

Then add these to the main URLs i.e. coda/urls.py :

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path('api/', include('main.urls', namespace='api')),
    path('admin/', admin.site.urls),
]

Finally, we would need to allow cross origins requests and add this app to the settings:
So first install django-cors-headers by:

pip install django-cors-headers

Then modify coda/settings.py:

...
INSTALLED_APPS = [
    'main.apps.MainConfig',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    'corsheaders'
]
...

CORS_ORIGIN_ALLOW_ALL = True

Time to make, run migrations, and run our server:

python manage.py makemigrations
python manage.py migrate
python manage.py runserver

The vue frontend:

Let us quickly write the frontend for our app, I used vue-cli and yarn in order to set up the project, thus:

vue create vote-app

Use the default settings then add the following packages:

yarn add axios router vuetify @mdi/font

Yes you saw it right, I am going to use vuetify in order to make my application look a bit cool xD

The first thing to do is to set up our application to use Vuetify thus, create a folder called plugins in the src directory and create a file called vuetify.js inside it:

import Vue from 'vue'
import Vuetify from 'vuetify'
import 'vuetify/dist/vuetify.min.css'
import '@mdi/font/css/materialdesignicons.css'

Vue.use(Vuetify, {
  iconfont: 'md',
})

export default new Vuetify({})

Now we need to modify our main.js file in order to use Vuetify and Router with our application like so:

import Vue from 'vue'
import App from './App.vue'
import router from "./router";

import BootstrapVue from "bootstrap-vue";
// import VeeValidate from "vee-validate";
import vuetify from '@/plugins/vuetify' // path to vuetify export

Vue.config.productionTip = false

new Vue({
  router,
  vuetify,
  render: h => h(App),
}).$mount('#app')

Let us define the routes in our router, create a file called router.js in your src folder and add the following routes to it:

import Vue from "vue";
import Router from "vue-router";

Vue.use(Router);

export default new Router({
  routes: [
    {
      path: "/",
      redirect: '/index'
    },
    {
      path: "/register",
      name: "register",
      component: () => import("./components/Register.vue")
    },
    {
      path: "/index",
      name: "index",
      component: () => import("./components/Index.vue")
    },
  ]
});

Now that we are all set up it is time to create our components, let us start with index, create a file called Index.vue inside the components folder and add the following code:

<template>
  <v-card class="mx-auto">
    <v-row>
      <v-col v-for="(item, i) in candidates" :key="i" cols="10" style="margin: 2%">
        <v-card :color="white" light>
          <div class="d-flex flex-no-wrap justify-space-between">
            <div>
              <v-card-title class="headline" v-text="item.name"></v-card-title>
              <v-card-subtitle style="color:black">Votes: {{ item.votes }}</v-card-subtitle>
              <v-card-subtitle>
                <v-expansion-panels v-model="panel" :disabled="disabled">
                  <v-expansion-panel>
                    <v-expansion-panel-header>Details</v-expansion-panel-header>
                    <v-expansion-panel-content>
                      <b>Number of Challenges Solved:</b> {{ item.no_challenges_solved }}
                      <br />
                      <b>Python Rating:</b> {{ item.python_rating }}
                      <br />
                      <b>DSA Rating:</b> {{ item.dsa_rating }}
                      <br />
                      <b>Java Rating:</b> {{ item.java_rating }}
                      <br />
                      <b>C++ Rating:</b> {{ item.cplus_rating }}
                      <br />
                    </v-expansion-panel-content>
                  </v-expansion-panel>
                </v-expansion-panels>
              </v-card-subtitle>
              <v-card-actions>
                <v-btn class="btn-success" style="color:white" text v-on:click="vote(item)">Vote</v-btn>
              </v-card-actions>
            </div>
          </div>
        </v-card>
      </v-col>
    </v-row>
  </v-card>
</template>

<script>
import axios from "axios";
export default {
  data() {
    return {
      candidates: [],
    };
  },
  created() {
    console.log("Here");
    this.all();
  },
  methods: {
    vote: function (candidate) {
      if (confirm("Vote " + candidate.name)) {
        axios
          .post(`http://localhost:8000/api/vote/`, {
            candidate_name: candidate.name,
          })
          .then((response) => {
            console.log(response);
            alert("Voted for" + candidate.name)
            this.all()
          })
          .catch(function (error) {
            if (error.response) {
              console.log(error);
              alert("You are only allowed to vote once");
            }
          });
      }
    },
    all: function () {
      console.log("Getting data");
      axios.get("http://localhost:8000/api/candidate/", {
        auth: {
          username: "admin",
          password: "hello@1234"
        }
      }).then((response) => {
        this.candidates = response.data;
        console.log(response);
      });
    },
  },
};
</script>

We used axios to make a request for the available candidates since we have set up the django application to use basic authentication for allowing CRUD on the Candidates, you would need to hard code the admin id and password here.
Also, we used a function vote to make a request to vote for a particular candidate after confirmation via an alert window, and if the response is successful create a corresponding alert and vice-versa.

Let us now create the other component called Register.Vue in order to allow for registering new candidates:

<template>
  <v-container>
    <v-form @submit="create" ref="form" lazy-validation>
      <v-text-field v-model="admin_id" :counter="250" label="Admin Id" required></v-text-field>
      <v-text-field v-model="admin_password" label="Admin Password" type="password" required></v-text-field>
      <v-text-field v-model="candidate.name" :counter="250" label="Name" required></v-text-field>
      <v-text-field
        v-model="candidate.no_challenges_solved"
        label="Number of challenges solved"
        type="number"
      ></v-text-field>
      <v-select
        v-model="candidate.python_rating"
        :items="ratings"
        :rules="[v => !!v || 'Python Rating is required']"
        label="Python Rating"
        required
      ></v-select>
      <v-select
        v-model="candidate.java_rating"
        :items="ratings"
        :rules="[v => !!v || 'Java Rating is required']"
        label="Java Rating"
        required
      ></v-select>
      <v-select
        v-model="candidate.dsa_rating"
        :items="ratings"
        :rules="[v => !!v || 'DSA Rating is required']"
        label="DSA Rating"
        required
      ></v-select>
      <v-select
        v-model="candidate.cplus_rating"
        :items="ratings"
        :rules="[v => !!v || 'C++ Rating is required']"
        label="C++ Rating"
        required
      ></v-select>
      <v-btn color="primary" type="submit">Submit</v-btn>
    </v-form>
  </v-container>
</template>

<script>
import axios from "axios";
export default {
  data() {
    return {
      ratings: [1, 2, 3, 4, 5],
      num: 1,
      candidate: {
        name: "",
        no_challenges_solved: 0,
        java_rating: 1,
        cplus_rating: 1,
        dsa_rating: 1,
        python_rating: 1,
      },
      admin_id: "",
      admin_password: "",
      submitted: false,
    };
  },
  methods: {
    create: function () {
      axios
        .post("http://127.0.0.1:8000/api/candidate/", this.candidate, {
          auth: {
            username: this.admin_id,
            password: this.admin_password,
          },
        })
        .then((response) => {
          console.log(response);
          alert("Registered Succesfuly");
          this.$router.push("/");
        })
        .catch((error) => {
          console.log(error);
        });
    },
  },
};
</script>

Last but not the least we would need to create the navigation drawer in the App.Vue file in order to create the navigation and link it with our router, thus, the router will exist with the navigation drawer of Vuetify:

<template>
  <v-app id="inspire">
    <v-navigation-drawer v-model="drawer" app>
      <v-list dense>
        <v-list-item link>
          <v-list-item-action>
            <v-icon>mdi-home</v-icon>
          </v-list-item-action>
          <v-list-item-content>
            <v-list-item-title>
              <router-link to="/index">Candidates</router-link>
            </v-list-item-title>
          </v-list-item-content>
        </v-list-item>
        <v-list-item link>
          <v-list-item-action>
            <v-icon>mdi-account-plus</v-icon>
          </v-list-item-action>
          <v-list-item-content>
            <v-list-item-title>
              <router-link to="/register">Register New Candidate<br> (Only Admins)</router-link>
            </v-list-item-title>
          </v-list-item-content>
        </v-list-item>
      </v-list>
    </v-navigation-drawer>

    <v-app-bar app color="indigo" dark>
      <v-app-bar-nav-icon @click.stop="drawer = !drawer"></v-app-bar-nav-icon>
      <v-toolbar-title>Application</v-toolbar-title>
    </v-app-bar>
    <v-main>
      <router-view />
    </v-main>
    <v-footer color="indigo" app>
      <span class="white--text">&copy; {{ new Date().getFullYear() }}</span>
    </v-footer>
  </v-app>
</template>
<script>
  export default {
    props: {
      source: String,
    },
    data: () => ({
      drawer: null,
    }),
  }
</script>

AND DONE...

You should be able to run the app via:

yarn serve

Enough talk just show me how it looks :P, sure here is how it looks:

Screenshots

Index

Detail View

Registering candidates

Voting

Vote twice error (based on IP)

Leave a star at the repo if you like the work :P. Let me know in the comments about what you would like me to write more about. Stay tuned because I am soon going to start my youtube channel.

Planning to start a YouTube channel, need suggestions

Amartya Gaur — Wed, 15 Jul 2020 17:37:54 +0000

I am planning to start a YouTube channel with tutorials on any of the following:

Django + Flutter (Can be a series of a full stack application).
Machine Learning + Django Application (From developing a model to applying it in a real scenario)

Fields I am comfortable with:

Web Development (Django or Spring Boot)
Mobile Application Development (Flutter)
ML (Python with all them libraries)
Basic Image processing (With python)
A bit of devops (Making a kubernetes cluster, deploying a web app etc.)
Scheduling with crons and recurring tasks (airflow or celery)
BigChain DB with Django (experimental not sure how perfect because here security is involved).

Please let me know what of these will be helpful to you. Waiting for your comments .......

Some Feature Engineering methods

Amartya Gaur — Tue, 23 Jun 2020 13:16:44 +0000

Introduction

Machine learning is explained in a single line is the set of rules that the system automatically generates to solve a problem based on certain patterns in the data. The machine learning problems can be classified into three main categories:

Supervised Learning methods - both data and labels available
Unsupervised Learning methods - only data and no labels
Reinforcement Learning methods - Rewards and punishment-based systems mostly used in robotics

Machine learning models are greatly influenced by the features of our data.

Feature Engineering

All machine learning models take certain input and generate certain output which depends upon the type of problem at hand, it might be a float quantity if it is a regression problem or an integer representing the predicted class in case of classification or it can be a decision if it is a reinforcement learning problem.

The data that we feed into the machine learning models are nothing but features that ultimately decide the output. The better the features, the better the model performance.

Feature engineering refers to the selection of appropriate and important features for your machine learning models.

Possible problems with raw data:

Missing Data

Most ML algorithms can not work with missing data, it becomes necessary to deal with missing data before proceeding further, this calls for Data Cleaning which can be accomplished by either of the following strategies:

Get rid of the missing features

dataFrame.dropna(inplace=True)

Set the values to mean / median / mode The SimpleImputer can be very beneficial in this case

from sklearn.impute import SimpleImputer
computer = SimpleImputer(strategy="mean") # can be median or mode
imputer.fit(dataFrame)

Categorical data

The machine learning models deal with numerical values so it is often required to change your categorical data and encode it to numerical data. sklearn provides you with inbuilt methods to achieve this:

Ordinal Encoding

Ordinal Encoding provides the categories an encoded integer while preserving the ordinal nature of the variables:

from sklearn.preprocessing import OrdinalEncoder
ordinal_encoder = OrdinalEncoder()
encoded_dframe = ordinal_encoder.fit_transform(dataFrame)

One Hot Encoding

One Hot Encoding creates one binary attribute per category: one attribute equal to 1 when the category is "cold" for example and 0 otherwise and similarly 1 for "hot" and 0 otherwise and so on. This can be implemented by:

from sklearn.preprocessing import OneHotEncoder
cat_encoder = OneHotEncoder()
dframe_1hot = cat_encoder.fit_transform(dataFrame)

This produces a SciPy matrix instead of a NumPy array. This is very useful when we have categorical attributes with thousands of categories.

Unscaled Data

The data at hand is mostly on different scales for example in a dataset for housing price prediction, the number of rooms can range from 2-10 whereas the area can vary from 1000-5000. It is often found that scaled data performs better in machine learning.

Min Max Scaler

The MinMaxScaler is a transformer available in sklearn in which the numerical values are shifted and recalculated so that they range from 0 to 1. This is done simply by subtracting the min value and then dividing by the max minus min value. This is sometimes also called normalization.

Standard Scaler

Standardization first subtracts the mean value (so that the standardized values always have 0 mean) and then divides by the standard deviation (so that the resulting values have a unit standard deviation). Unlike min max scaler standard scaler does not bound the data to be on a specific scale. It is however less affected by the outliers.

from sklearn.preprocessing import StandardScaler
sc=StandardScaler()
dframe_std=sc.fit_transform(dataFrame)

Imbalanced Data

When the examples of a particular class are way too less as compared to the other class, this is a situation we call imbalanced data. It is important to have balanced data in order for the ML model to properly perform.

Imbalanced Data can be handled via one of the following techniques:

SMOTE (Synthetic Minority Oversampling Technique) – Oversampling

SMOTE (synthetic minority oversampling technique) is one of the most commonly used oversampling methods to solve the imbalance problem.
It aims to balance class distribution by randomly increasing minority class examples by replicating them.
SMOTE synthesizes new minority instances between existing minority instances. It generates virtual training records by linear interpolation for the minority class. These synthetic training records are generated by randomly selecting one or more of the k-nearest neighbors for each example in the minority class. After the oversampling process, the data is reconstructed and several classification models can be applied for the processed data.

from imblearn.over_sampling import SMOTE
oversample = SMOTE()
X, y = oversample.fit_resample(X, y)

SMOTEENN (A combination of over and under sampling)

SMOTE can generate noisy samples by interpolating new points between marginal outliers and inliers. This issue can be solved by cleaning the space resulting from over-sampling.

In this regard, Tomek’s link and edited nearest-neighbors are the two cleaning methods that have been added to the pipeline after applying SMOTE over-sampling to obtain a cleaner space. The two ready-to-use classes imbalanced-learn implements for combining over- and undersampling methods are (i) SMOTETomek and (ii) SMOTEENN.

from collections import Counter
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=5000, n_features=2, n_informative=2,
...                            n_redundant=0, n_repeated=0, n_classes=3,
...                            n_clusters_per_class=1,
...                            weights=[0.01, 0.05, 0.94],
...                            class_sep=0.8, random_state=0)
print(sorted(Counter(y).items()))
[(0, 64), (1, 262), (2, 4674)]
from imblearn.combine import SMOTEENN
smote_enn = SMOTEENN(random_state=0)
X_resampled, y_resampled = smote_enn.fit_resample(X, y)
print(sorted(Counter(y_resampled).items()))
[(0, 4060), (1, 4381), (2, 3502)]
from imblearn.combine import SMOTETomek
smote_tomek = SMOTETomek(random_state=0)
X_resampled, y_resampled = smote_tomek.fit_resample(X, y)
print(sorted(Counter(y_resampled).items()))
[(0, 4499), (1, 4566), (2, 4413)]

SMOTEENN tends to clean more noisy samples than SMOTETomek.

Correlation in Data and Output

In regression problems, the correlation between certain features and the target variable can be beneficial. Sometimes a combination of one or more variables can better correlate with the target variable than individual variables. It is thus necessary to look for such patterns in the dataset.

Some feature selection that can be beneficial:

Feature selection can be very important and the selection of best features can reduce the computation power required in processing a lot of them, which can also help in less time complexity in certain algorithms.

Brute Force Correlation Processing

We manually calculate the correlation between various features and drop the ones that are more correlated than the predetermined threshold

Sample code on the Paribas-Cardif-Claim-Data can be (for code and dataset refer https://github.com/amartya-dev/feature_selection/blob/master/feature_selection.ipynb :

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import chi2
from sklearn.feature_selection import SelectKBest, SelectPercentile
from sklearn.feature_selection import mutual_info_classif, mutual_info_regression
from sklearn.feature_selection import SelectKBest, SelectPercentile

df = pd.read_csv('./dataset/Paribas-Cardif-Claim-Data/train.csv', nrows=50000)
print(df.shape)
print(df.head())

# Get Numerical features from dataset
numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
numerical_features = list(df.select_dtypes(include=numerics).columns)

data = df[numerical_features]
print("Dropping the non numeric data : ", data.shape, " is the shape")
print(data.head())

X = data.drop(['target', 'ID'], axis=1)
print("Dropping off target and ID : ", X.shape)

# Visualize Correlated Features
print("Visualizing the correlation features : ")
corr = X.corr()
fig, ax = plt.subplots()
fig.set_size_inches(11, 11)
sns.heatmap(corr)


# Brute Force Method to find Correlation between features
def correlation(data, threshold=None):
    # Set of all names of correlated columns
    col_corr = set()
    corr_mat = data.corr()
    for i in range(len(corr_mat.columns)):
        for j in range(i):
            if (abs(corr_mat.iloc[i, j]) > threshold):
                colname = corr_mat.columns[i]
                col_corr.add(colname)
    return col_corr


correlated_features = correlation(data=X, threshold=0.8)
print("Features correlated with each other in dataaset : ", len(set(correlated_features)))

print("Drpping these features ...")

X.drop(labels=correlated_features, axis=1, inplace=True)

X.to_csv(r'./dataset/brute_force_corr_processed.csv')

The Output:

Highly correlated Feature Groups

Another method which is slightly better than the brute force correlation is creating feature groups from the data available and then further identify the best features in that group using a classifier such as the Random Forest Classifier:


df = pd.read_csv('./dataset/Paribas-Cardif-Claim-Data/train.csv', nrows=50000)
print(df.shape)
print(df.head())

# Get Numerical features from dataset
numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
numerical_features = list(df.select_dtypes(include=numerics).columns)
pd.options.mode.chained_assignment = None
data = df[numerical_features]
print("Dropping the non numeric data : ", data.shape, " is the shape")
print(data.head())

X = data.drop(['target', 'ID'], axis=1)
print("Dropping off target and ID : ", X.shape)

# Build a Dataframe with Correlation between Features
corr_matrix = X.corr()
# Take absolute values of correlated coefficients
corr_matrix = corr_matrix.abs().unstack()
corr_matrix = corr_matrix.sort_values(ascending=False)
# Take only features with correlation above threshold of 0.8
corr_matrix = corr_matrix[corr_matrix >= 0.8]
corr_matrix = corr_matrix[corr_matrix < 1]
corr_matrix = pd.DataFrame(corr_matrix).reset_index()
corr_matrix.columns = ['feature1', 'feature2', 'Correlation']
print("Correlation matrix \n : ", corr_matrix.head())

# Get groups of features that are correlated amongs themselves
grouped_features = []
correlated_groups = []

for feature in corr_matrix.feature1.unique():
    if feature not in grouped_features:
        # Find all features correlated to a single feature
        correlated_block = corr_matrix[corr_matrix.feature1 == feature]
        grouped_features = grouped_features + list(correlated_block.feature2.unique()) + [feature]

        # Append block of features to the list
        correlated_groups.append(correlated_block)

print('Found {} correlated feature groups'.format(len(correlated_groups)))
print('out of {} total features.'.format(X.shape[1]))

for group in correlated_groups:
    print(group)
    print('\n')

# Investigating features further within one group
group = correlated_groups[3]
print(group)

# Select features with less missing data
for features in list(group.feature2.unique()) + ['v17']:
    print(X[feature].isnull().sum())

print("Using Random forest classifier to find best features : ")
y = data['target']
y.shape
# Train Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)
print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

from sklearn.ensemble import RandomForestClassifier

features = list(group.feature2.unique()) + ['v17']
rfc = RandomForestClassifier(n_estimators=20, random_state=101, max_depth=4)
rfc.fit(X_train[features].fillna(0), y_train)

# Get Feature Importance using RFC
importance = pd.concat([pd.Series(features), pd.Series(rfc.feature_importances_)], axis=1)
importance.columns = ['feature', 'importance']
importance.sort_values(by='importance', ascending=False)

to_drop = set()
for i in ('v8', 'v105', 'v54', 'v63', 'v25', 'v89'):
    to_drop.add(i)

X_train.drop(labels=to_drop, axis=1, inplace=True)
X_test.drop(labels=to_drop, axis=1, inplace=True)

X = pd.merge(X_train, X_test)
X.to_csv(r'./dataset/Highly_corr_group_processed.csv')

Fisher Score Chi-Squared

This is the measure of independent variables, we can use the fisher score in order to determine the best variable for the problem at hand. Sample code:

df = pd.read_csv('./dataset/Paribas-Cardif-Claim-Data/train.csv', nrows=50000)

df.boxplot('target', 'v75', rot=30, figsize=(5, 6))
df.boxplot('target', 'v52', rot=30, figsize=(5, 6))
df.boxplot('target', 'v125', rot=30, figsize=(5, 6))
df.boxplot('target', 'v91', rot=30, figsize=(5, 6))
df.boxplot('target', 'v107', rot=30, figsize=(5, 6))

print("\nGetting to know categorical data")

cat_df = df.select_dtypes(include=['object']).copy()

print(cat_df.head())

# Encode categorical variables into numbers
label1 = {k: i for i, k in enumerate(df['v3'].unique(), 0)}
label2 = {k: i for i, k in enumerate(df['v22'].unique(), 0)}
label3 = {k: i for i, k in enumerate(df['v24'].unique(), 0)}
label4 = {k: i for i, k in enumerate(df['v30'].unique(), 0)}
label5 = {k: i for i, k in enumerate(df['v31'].unique(), 0)}
label6 = {k: i for i, k in enumerate(df['v47'].unique(), 0)}
label7 = {k: i for i, k in enumerate(df['v52'].unique(), 0)}
label8 = {k: i for i, k in enumerate(df['v56'].unique(), 0)}
label9 = {k: i for i, k in enumerate(df['v66'].unique(), 0)}
label10 = {k: i for i, k in enumerate(df['v71'].unique(), 0)}
label11 = {k: i for i, k in enumerate(df['v74'].unique(), 0)}
label12 = {k: i for i, k in enumerate(df['v75'].unique(), 0)}
label13 = {k: i for i, k in enumerate(df['v79'].unique(), 0)}
label14 = {k: i for i, k in enumerate(df['v91'].unique(), 0)}
label15 = {k: i for i, k in enumerate(df['v107'].unique(), 0)}
label16 = {k: i for i, k in enumerate(df['v110'].unique(), 0)}
label17 = {k: i for i, k in enumerate(df['v112'].unique(), 0)}
label18 = {k: i for i, k in enumerate(df['v113'].unique(), 0)}
label19 = {k: i for i, k in enumerate(df['v125'].unique(), 0)}

df['v3'] = df['v3'].map(label1)
df['v22'] = df['v22'].map(label2)
df['v24'] = df['v24'].map(label3)
df['v30'] = df['v30'].map(label4)
df['v31'] = df['v31'].map(label5)
df['v47'] = df['v47'].map(label6)
df['v52'] = df['v52'].map(label7)
df['v56'] = df['v56'].map(label8)
df['v66'] = df['v66'].map(label9)
df['v71'] = df['v71'].map(label10)
df['v74'] = df['v74'].map(label11)
df['v75'] = df['v75'].map(label12)
df['v79'] = df['v79'].map(label13)
df['v91'] = df['v91'].map(label14)
df['v107'] = df['v107'].map(label15)
df['v110'] = df['v110'].map(label16)
df['v112'] = df['v112'].map(label17)
df['v113'] = df['v113'].map(label18)
df['v125'] = df['v125'].map(label19)

X = df[['v3', 'v22', 'v24', 'v30', 'v31', 'v47', 'v52', 'v56', 'v66', 'v71', 'v74', 'v75', 'v79', 'v91', 'v107', 'v110',
        'v112', 'v113', 'v125']]
print(X.head())

# Train Test Split
y = df['target']
y.shape
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)
X_train.shape, y_train.shape, X_test.shape, y_test.shape

# Calcualte the Fisher Score (chi2) between each feature and target
fisher_score = chi2(X_train.fillna(0), y_train)
fisher_score

p_values = pd.Series(fisher_score[1])
p_values.index = X_train.columns
p2 = p_values.sort_values(ascending=True)

print(p2.index[0], " is the most important feature here")

return (p2.index[0])

Output:

Please feel free to drop a message at Ohuru in order to avail various development services offered by us.

Flutter signup/login application with Django backend #3

Amartya Gaur — Mon, 20 Apr 2020 11:29:26 +0000

What this post is about

In the previous two posts, I made an API in Django with the help of DRF (please read them: post #1, post #2 before this post in case you want to develop the API as well). This post will be about the flutter application that will function with that API. We are going to create an application to allow a user to log in and log out.

You can also view the source code at this repo.

The application demo:

Prerequisites

To follow along with the post, you need to have flutter installed (refer: https://flutter.dev/docs/get-started/install) along with visual studio code and the extension for flutter (refer: https://flutter.dev/docs/development/tools/vs-code) and bloc (refer: https://marketplace.visualstudio.com/items?itemName=FelixAngelov.bloc).

Let's Start

Begin with creating a project (it would be better to use the vscode extension for the same) called bloc_login (Ctrl + shift + p, type flutter and you will get an option to create a new project).
Next, we will create certain folders in order to separate different components of our code. Create the following directories in bloc_login/
- api_connection (will contain the code to communicate with our API)
- common (Will contain utility widgets like loading indicator)
- dao (Will contain a helper file in order to communicate with our sqlite database)
- model (Will contain schema of our database)
- repository (Will act as a bridge between our API, blocs, and database)
- splash (Will contain splash screen code)
- login (Will contain login form and other details pertaining to it)
- database (Will contain files in order to initialize db and create the table to start inserting data)
- home (Will contain our main UI)
You can refer to this page in order to get a basic understanding of how flutter blocs work. To understand what blocs actually are you can give this page a read. In short, they are a way to map our incoming events to states which flutter will consume in order to rebuild UI in accordance.
Add dependencies. We are going to use Equatable, flutter_bloc, sqflite and http packages, thus modify your pubspec.yaml file and include the following dependencies:

dependencies:
  flutter:
    sdk: flutter
  flutter_bloc: ^3.2.0
  meta: ^1.1.6
  equatable: ^1.1.0
  http: ^0.12.0+4
  sqflite: ^1.3.0
  path_provider: ^1.6.5
  # The following adds the Cupertino Icons font to your application.
  # Use with the CupertinoIcons class for iOS style icons.
  cupertino_icons: ^0.1.2

Note: I have just shown the dependencies part where you need to add the files for dependencies.
Just saving the file in VScode will automatically trigger the command to get these packages but, for reference, run the following command from your bloc_login/ directory:

flutter pub get

Let us start by making our database helper thus, create a file user_database.dart inside your bloc_directory/database directory and add the following code to it:

import 'dart:async';
import 'dart:io';

import 'package:path/path.dart';
import 'package:path_provider/path_provider.dart';
import 'package:sqflite/sqflite.dart';

final userTable = 'userTable';

class DatabaseProvider {
  static final DatabaseProvider dbProvider = DatabaseProvider();

  Database _database;

  Future <Database> get database async {
    if (_database != null){
      return _database;
    }
    _database = await createDatabase();
    return _database;
  }

  createDatabase() async {
    Directory documentsDirectory = await getApplicationDocumentsDirectory();
    String path = join(documentsDirectory.path, "User.db");

    var database = await openDatabase(
      path,
      version: 1,
      onCreate: initDB,
      onUpgrade: onUpgrade,
    );
    return database;
  }

  void onUpgrade(
    Database database,
    int oldVersion,
    int newVersion,
  ){
    if (newVersion > oldVersion){}
  }

  void initDB(Database database, int version) async {
    await database.execute(
      "CREATE TABLE $userTable ("
      "id INTEGER PRIMARY KEY, "
      "username TEXT, "
      "token TEXT "
      ")"
    );
  }
}

Here we return the opened database as the database variable in our DatabaseProvider class. Thus, dbProvider will initialize the DatabaseProvider class and will have a get database function that will return the opened database.

We can use the onUpgrade function in case we need to migrate some changes between different versions. Also our final userTable variable is used to store the table name for our database.

Let us next create our model for the user that corresponds to the table we just created. Thus, create a file bloc_login/model/user_model.dart and add the following code to it:

class User {
  int id;
  String username;
  String token;

  User(
      {this.id,
      this.username,
      this.token});

  factory User.fromDatabaseJson(Map<String, dynamic> data) => User(
      id: data['id'],
      username: data['username'],
      token: data['token'],
  );

  Map<String, dynamic> toDatabaseJson() => {
        "id": this.id,
        "username": this.username,
        "token": this.token
      };
}

Here we also defined the factory function fromDatabaseJson which will return the user as a JSON object and the toDatabaseJson which will be responsible for converting the incoming JSON object to a database object which can be stored to our database.

Let us add API functionality thus create the file bloc_login/api_connection/api_connection.dart and add the following code to it:

import 'dart:async';
import 'dart:convert';
import 'package:http/http.dart' as http;
import 'package:bloc_login/model/api_model.dart';

final _base = "https://home-hub-app.herokuapp.com";
final _tokenEndpoint = "/api-token-auth/";
final _tokenURL = _base + _tokenEndpoint;

Future<Token> getToken(UserLogin userLogin) async {
  print(_tokenURL);
  final http.Response response = await http.post(
    _tokenURL,
    headers: <String, String>{
      'Content-Type': 'application/json; charset=UTF-8',
    },
    body: jsonEncode(userLogin.toDatabaseJson()),
  );
  if (response.statusCode == 200) {
    return Token.fromJson(json.decode(response.body));
  } else {
    print(json.decode(response.body).toString());
    throw Exception(json.decode(response.body));
  }
}

I have used the above URL because I named my app home_hub for heroku, you need to replace that with the URL you got from the previous post. The rest can remain the same.

We just have a helper function here in order to get token for any particular user and throw an error in case the details are not right.

Let us add a model in order to deal with requests to API and subsequent responses. Create a file bloc_login/model/api_model.dart and add the following code to it:

class UserLogin {
  String username;
  String password;

  UserLogin({this.username, this.password});

  Map <String, dynamic> toDatabaseJson() => {
    "username": this.username,
    "password": this.password
  };
}

class Token{
  String token;

  Token({this.token});

  factory Token.fromJson(Map<String, dynamic> json) {
    return Token(
      token: json['token']
    );
  }
}

We are going to use this model in order to just get the username and password from the user and then send them to the server. The Token class gets the result and gives us the string from the JSON object we receive from the server.

Let us create a dao object to provide some basic operations for User model and the database, create a file bloc_login/dao/user_dao.dart and add the following code to it:

import 'package:bloc_login/database/user_database.dart';
import 'package:bloc_login/model/user_model.dart';

class UserDao {
  final dbProvider = DatabaseProvider.dbProvider;

  Future<int> createUser(User user) async {
    final db = await dbProvider.database;

    var result = db.insert(userTable, user.toDatabaseJson());
    return result;
  }

  Future<int> deleteUser(int id) async {
    final db = await dbProvider.database;
    var result = await db
        .delete(userTable, where: "id = ?", whereArgs: [id]);
    return result;
  }

  Future<bool> checkUser(int id) async {
    final db = await dbProvider.database;
    try {
      List<Map> users = await db
          .query(userTable, where: 'id = ?', whereArgs: [id]);
      if (users.length > 0) {
        return true;
      } else {
        return false;
      }
    } catch (error) {
      return false;
    }
  }
}

This basically provides us with methods to create a user / delete a user and search if a user exists or not. Since our app will provide login facilities to only one user at max, we are forcing the user to be created with id 0. Thus, we can always check for the existence of a user by checking the first entry in our database. The best part is that we are just storing the username and token and not the password entered. The password is just used to send a request to the server and we never store than anywhere else. There is an alternative here, you can always use flutter_secure_storage in order to store the token and use it but I wanted to use SQL as I want to further extend the application to include other functionalities.

Let us come to the logical implementation of our apps i.e. the blocs. I recommend using the vscode extension to create a new bloc (refer: here) although, you can always manually create the files. Create new bloc in bloc_login. directory and name it authentication_bloc. The extension will automatically create the bloc_login/bloc folder and create three files in it corresponding to the states, events and the bloc for authentication_bloc. Modify your bloc_login/bloc/authentication_state.dart and add the following code:

part of 'authentication_bloc.dart';

abstract class AuthenticationState extends Equatable {
  @override
  List<Object> get props => [];
}

class AuthenticationUnintialized extends AuthenticationState {}

class AuthenticationAuthenticated extends AuthenticationState {}

class AuthenticationUnauthenticated extends AuthenticationState {}

class AuthenticationLoading extends AuthenticationState {}

Here we define four authentication states which are Uninitialized which can correspond to the state when the app is waiting to check if a user exists in our database or not, the loading state which can be a state when we are waiting for the app to store a token or delete a token, authenticated state corresponding to the successful login of a user and unauthenticated which corresponds to the user not being authenticated yet / logged out.

Let us define the events for our authentication bloc modify your bloc_login/bloc/authentication_event.dart and add the following code:

part of 'authentication_bloc.dart';

abstract class AuthenticationEvent extends Equatable {
  const AuthenticationEvent();

  @override
  List<Object> get props => [];
}

class AppStarted extends AuthenticationEvent {}

class LoggedIn extends AuthenticationEvent {
  final User user;

  const LoggedIn({@required this.user});

  @override
  List<Object> get props => [user];

  @override
  String toString() => 'LoggedIn { user: $user.username.toString() }';
}

class LoggedOut extends AuthenticationEvent {}

This corresponds to three events that might occur, AppStarted which will notify the block to check if a user exists, LoggedIn which will be the event suggesting that the user has logged in successfully and LoggedOut which will tell that the user has logged out.

Now we need to implement our bloc that is map our events to states, modify your bloc_login/bloc/authentication_bloc.dart and add the following:

import 'dart:async';

import 'package:bloc/bloc.dart';
import 'package:meta/meta.dart';
import 'package:equatable/equatable.dart';

import 'package:bloc_login/repository/user_repository.dart';
import 'package:bloc_login/model/user_model.dart';

part 'authentication_event.dart';
part 'authentication_state.dart';

class AuthenticationBloc
    extends Bloc<AuthenticationEvent, AuthenticationState> {
  final UserRepository userRepository;

  AuthenticationBloc({@required this.userRepository})
      : assert(UserRepository != null);

  @override
  AuthenticationState get initialState => AuthenticationUnintialized();

  @override
  Stream<AuthenticationState> mapEventToState(
    AuthenticationEvent event,
  ) async* {
    if (event is AppStarted) {

      final bool hasToken = await userRepository.hasToken();

      if (hasToken) {
        yield AuthenticationAuthenticated();
      } else {
        yield AuthenticationUnauthenticated();
      }
    }

    if (event is LoggedIn) {
      yield AuthenticationLoading();

      await userRepository.persistToken(
        user: event.user
      );
      yield AuthenticationAuthenticated();
    }

    if (event is LoggedOut) {
      yield AuthenticationLoading();

      await userRepository.delteToken(id: 0);

      yield AuthenticationUnauthenticated();
    }
  }
}

Here we did the following:
- Initialized state to be AuthenticationUninitialized
- Yielded the AuthenticationAuthenticated state which corresponds to the existence of users in our database. This will help in the persistence of the state because the user will exist even after we clear our app from the ram.
- If the event was LoggedIn, we saved the user to the database by calling the persistToken function we defined.
- If the event was LoggedOut, we deleted the user from the database.

Enters UI, let us create our splash screen, create the file bloc_login/splash_page.dart and the following code to it:

import 'package:flutter/material.dart';

class SplashPage extends StatelessWidget {
  @override
  Widget build (BuildContext context) {
    return Scaffold(
      body: Center(
        child: Text('Splash Screen'),
      ),
    );
  }
}

We will also export this code in the file bloc_login/splash/splash.dart for easy access:

export 'splash_page.dart';

Let us also create our home page so that we can navigate users when they are logged in. Thus, create the file bloc_login/home/home_page.dart and add the following code:

import 'package:flutter/material.dart';
import 'package:flutter_bloc/flutter_bloc.dart';
import 'package:bloc_login/bloc/authentication_bloc.dart';

class HomePage extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Home | Home Hub'),
      ),
      body: Container(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          crossAxisAlignment: CrossAxisAlignment.center,
          children: <Widget>[
            Padding(padding: EdgeInsets.only(left: 30.0),
            child: Text(
              'Welcome',
              style: TextStyle(
                fontSize: 24.0,
              ),
            ),),
            Padding(
              padding: EdgeInsets.fromLTRB(34.0, 20.0, 0.0, 0.0),
              child: Container(
                width: MediaQuery.of(context).size.width * 0.85,
                height: MediaQuery.of(context).size.width * 0.16,
                child: RaisedButton(
                  child: Text(
                    'Logout',
                    style: TextStyle(
                      fontSize: 24,
                    ),
                  ),
                  onPressed: () {
                    BlocProvider.of<AuthenticationBloc>(context)
                        .add(LoggedOut());
                  },
                  shape: StadiumBorder(
                    side: BorderSide(
                      color: Colors.black,
                      width: 2,
                    ),
                  ),
                ),
              ),
            ),
          ],
        ),
      ),
    );
  }
}

Here we are displaying Welcome and a button to Log Out a user, notice that we got our bloc provider from the build context (we will add the code for it in main.dart) and notified it that the LoggedOut event has occurred by adding that event on button press.

We also need to create a login bloc to handle our log in states and events. Thus, make another bloc called login using the extension in the bloc_login/login directory we created and add the following code in the bloc_login/login/bloc/login_states.dart

part of 'login_bloc.dart';

abstract class LoginState extends Equatable {
  const LoginState();

  @override
  List<Object> get props => [];
}

class LoginInitial extends LoginState {}

class LoginLoading extends LoginState {}

class LoginFaliure extends LoginState {
  final String error;

  const LoginFaliure({@required this.error});

  @override
  List<Object> get props => [error];

  @override
  String toString() => ' LoginFaliure { error: $error }';
}

Here the LoginInitial state is the initial state for our login form, LoginLoading is the state when our form is validating the credentials and LoginFaliure indicates that the login attempt was unsuccessful.

Let us create our login events, modify your bloc_login/login/login_event.dart file and add the following code:


part of 'login_bloc.dart';

abstract class LoginEvent extends Equatable {
  const LoginEvent();
}

class LoginButtonPressed extends LoginEvent {
  final String username;
  final String password;

  const LoginButtonPressed({
    @required this.username,
    @required this.password
  });

  @override
  List<Object> get props => [username, password];

  @override
  String toString() => 'LoginButtonPressed { username: $username, password: $password }';
}



Let us also add the login to map the events to state in bloc_login/login/login_bloc.dart. Add the following code in this file:


import 'dart:async';

import 'package:bloc/bloc.dart';
import 'package:bloc_login/bloc/authentication_bloc.dart';
import 'package:bloc_login/repository/user_repository.dart';
import 'package:meta/meta.dart';
import 'package:equatable/equatable.dart';

part 'login_event.dart';
part 'login_state.dart';

class LoginBloc extends Bloc<LoginEvent, LoginState> {
  final UserRepository userRepository;
  final AuthenticationBloc authenticationBloc;

  LoginBloc({
    @required this.userRepository,
    @required this.authenticationBloc,
  })  : assert(userRepository != null),
        assert(authenticationBloc != null);

  @override
  LoginState get initialState => LoginInitial();

  @override
  Stream<LoginState> mapEventToState(
    LoginEvent event,
  ) async* {
    if (event is LoginButtonPressed) {
      yield LoginInitial();

      try {
        final user = await userRepository.authenticate(
          username: event.username,
          password: event.password,
        );

        authenticationBloc.add(LoggedIn(user: user));
        yield LoginInitial();
      } catch (error) {
        yield LoginFaliure(error: error.toString());
      }
    }
  }
}




Let us create our login form. Create a file bloc_login/login/login_form.dart and add the following code:



import 'package:flutter/material.dart';
import 'package:flutter_bloc/flutter_bloc.dart';
import 'package:bloc_login/login/bloc/login_bloc.dart';

class LoginForm extends StatefulWidget {
  @override
  State<LoginForm> createState() => _LoginFormState();
}

class _LoginFormState extends State<LoginForm> {
  final _usernameController = TextEditingController();
  final _passwordController = TextEditingController();

  @override
  Widget build(BuildContext context) {
    _onLoginButtonPressed() {
      BlocProvider.of<LoginBloc>(context).add(LoginButtonPressed(
        username: _usernameController.text,
        password: _passwordController.text,
      ));
    }

    return BlocListener<LoginBloc, LoginState>(
      listener: (context, state) {
        if (state is LoginFaliure) {
          Scaffold.of(context).showSnackBar(SnackBar(
            content: Text('${state.error}'),
            backgroundColor: Colors.red,
          ));
        }
      },
      child: BlocBuilder<LoginBloc, LoginState>(
        builder: (context, state) {
          return Container(
            child: Form(
              child: Padding(
                padding: EdgeInsets.all(40.0),
                child: Column(
                  mainAxisAlignment: MainAxisAlignment.center,
                  crossAxisAlignment: CrossAxisAlignment.center,
                  children: <Widget>[
                    TextFormField(
                      decoration: InputDecoration(
                          labelText: 'username', icon: Icon(Icons.person)),
                      controller: _usernameController,
                    ),
                    TextFormField(
                      decoration: InputDecoration(
                          labelText: 'password', icon: Icon(Icons.security)),
                      controller: _passwordController,
                      obscureText: true,
                    ),
                    Container(
                      width: MediaQuery.of(context).size.width * 0.85,
                      height: MediaQuery.of(context).size.width * 0.22,
                      child: Padding(
                        padding: EdgeInsets.only(top: 30.0),
                        child: RaisedButton(
                          onPressed: state is! LoginLoading
                              ? _onLoginButtonPressed
                              : null,
                          child: Text(
                            'Login',
                            style: TextStyle(
                              fontSize: 24.0,
                            ),
                          ),
                          shape: StadiumBorder(
                            side: BorderSide(
                              color: Colors.black,
                              width: 2,
                            ),
                          ),
                        ),
                      ),
                    ),
                    Container(
                      child: state is LoginLoading
                          ? CircularProgressIndicator()
                          : null,
                    ),
                  ],
                ),
              ),
            ),
          );
        },
      ),
    );
  }
}





Also, we will create a file bloc_login/login/login_page.dart and add the following code:



import 'package:flutter/material.dart';

import 'package:flutter_bloc/flutter_bloc.dart';
import 'package:bloc_login/repository/user_repository.dart';

import 'package:bloc_login/bloc/authentication_bloc.dart';
import 'package:bloc_login/login/bloc/login_bloc.dart';
import 'package:bloc_login/login/login_form.dart';

class LoginPage extends StatelessWidget {
  final UserRepository userRepository;

  LoginPage({Key key, @required this.userRepository})
      : assert(userRepository != null),
        super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Login | Home Hub'),
      ),
      body: BlocProvider(
        create: (context) {
          return LoginBloc(
            authenticationBloc: BlocProvider.of<AuthenticationBloc>(context),
            userRepository: userRepository,
          );
        },
        child: LoginForm(),
      ),
    );
  }
}




We are all set, now all that is needed to be done is to create our bloc_login/main.dart file and also create a loading indicator. Let us first create the loading indicator, create bloc_login/common/loading_indicator.dart and add the following code:



import 'package:flutter/material.dart';

class LoadingIndicator extends StatelessWidget {
  @override
  Widget build(BuildContext context) => Center(
        child: CircularProgressIndicator(),
      );
}




We will also export it from bloc_login/common/common.dart for easy access:



export './loading_indicator.dart';





Finally, add the following code in bloc_login/main.dart



import 'package:flutter/material.dart';

import 'package:bloc/bloc.dart';
import 'package:flutter_bloc/flutter_bloc.dart';
import 'package:bloc_login/repository/user_repository.dart';

import 'package:bloc_login/bloc/authentication_bloc.dart';
import 'package:bloc_login/splash/splash.dart';
import 'package:bloc_login/login/login_page.dart';
import 'package:bloc_login/home/home.dart';
import 'package:bloc_login/common/common.dart';

class SimpleBlocDelegate extends BlocDelegate {
  @override
  void onEvent(Bloc bloc, Object event) {
    super.onEvent(bloc, event);
    print(event);
  }

  @override
  void onTransition(Bloc bloc, Transition transition) {
    super.onTransition(bloc, transition);
    print (transition);
  }

  @override
  void onError(Bloc bloc, Object error, StackTrace stacktrace) {
    super.onError(bloc, error, stacktrace);
  }
}

void main() {
  BlocSupervisor.delegate = SimpleBlocDelegate();
  final userRepository = UserRepository();

  runApp(
    BlocProvider<AuthenticationBloc>(
      create: (context) {
        return AuthenticationBloc(
          userRepository: userRepository
        )..add(AppStarted());
      },
      child: App(userRepository: userRepository),
    )
  );
}

class App extends StatelessWidget {
  final UserRepository userRepository;

  App({Key key, @required this.userRepository}) : super(key: key);

  @override
  Widget build (BuildContext context) {
    return MaterialApp(
      theme: ThemeData(
        primarySwatch: Colors.red,
        brightness: Brightness.dark,
      ),
      home: BlocBuilder<AuthenticationBloc, AuthenticationState>(
        builder: (context, state) {
          if (state is AuthenticationUnintialized) {
            return SplashPage();
          }
          if (state is AuthenticationAuthenticated) {
            return HomePage();
          }
          if (state is AuthenticationUnauthenticated) {
            return LoginPage(userRepository: userRepository,);
          }
          if (state is AuthenticationLoading) {
            return LoadingIndicator();
          }
        },
      ),
    );
  }
}




I have added the BlocDelegate and overridden the functions for debugging purposes. I know this is a slightly long post but I thought it will be good to include the whole application here for reference.

This shall be the last post in this series. You can also view the source code at this repo.

Links to previous posts:

Flutter signup/login application with Django backend #1

Flutter signup/login application with Django backend #2

Please feel free to drop a message at Ohuru in order to avail various development services offered by us.

Flutter signup/login application with Django backend #2

Amartya Gaur — Sun, 19 Apr 2020 06:17:59 +0000

Next steps

This post is in continuation of the other post that I wrote about the same topic. If you have not read that, please go ahead and give it a read so that you can follow along with this one. Flutter signup/login application with Django backend #1.
Coming to the point. I am kind of new to flutter and these are the posts I am writing as I am developing the application. I started by calling the freshly made APIs in the previous post with flutter. The problem is that even after defining port multiple times the application did not properly communicate with the localhost:8000. So, I decided to host this application and then call them.

Host your application

Since this series of posts will be a detailed step by step procedure that can be replicated to get the exact same application running, I thought it would be a good idea to include this step in the process.
When it comes to hosting Django application you have two options:

Host it on a VPS (you need to configure gunicorn and Nginx for it refer: this tutorial).
Host it on Heroku (or python anywhere etc. etc). Well, to be honest, the second method is relatively simple and we are going to do just that as our aim here is to just see how stuff works.

Hosting your application on Heroku

Changes required

Heroku provides a very easy method for hosting Django applications, the first step is to modify a few settings and add a few files in order to prepare our application for hosting.

We will be deploying using gunicorn so let us install that, run:

pip install gunicorn

We will be using django-heroku package to configure our setting.py file automatically. Thus, install that too:

pip install django-heroku

The first file we need to include is requirements.txt so that Heroku can recognize your application as a python application. Thus, activate your virtual environment and run:

pip freeze > requirements.txt

from HOME/

Create a Procfile

nano Procfile

and add :

web: gunicorn HOME.wsgi --log-file -

For the admin portal to run properly, we need static files' support, for that, we need to configure STATIC_ROOT property in our settings.py file, we also need to include django-heroku settings in the file. Thus, modify your HOME/HOME/settings.py and add the following:

import django_heroku # add in the beginning

... # denotes rest of the code in between

# add these lines at end
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

django_heroku.settings(locals())

Pushing our code to deploy it

We need to create an account on heroku and install heroku-cli. These instructions suffice for Ubuntu / Windows / Mac. I use Manjaro (yay as the installer) therefore just for reference, to can be installed by:

yay -S heroku-cli

It is the perfect time to log in to your heroku cli, run

heroku login

Next, we need to create an app, run

heroku create <app_name>

Now copy all your files from the HOME/ directory to this app_name/ directory we created just now.

Now is the time to run these magic commands (we are deploying using git make sure git bash is installed refer: here)

git add .
git commit -m "deploying to heroku"
git push heroku master

Now we need to migrate our changes on the heroku server as well. Thus run:

heroku run python manage.py migrate

Also, create a superuser with the same credentials as your local application (for consistency).

heroku run python manage.py createsuperuser

You should now be able to call your APIs with the URL you got here. Verify the functionality using POSTMAN before proceeding. Also, refer to my previous post for details about the endpoints, etc.

I will be publishing the next post in the series on Monday (20/04/20). Stay tuned.

Next Post: Flutter signup/login application with Django backend #3

Previous Post:
Flutter signup/login application with Django backend #1

Flutter signup/login application with Django backend #1

Amartya Gaur — Thu, 16 Apr 2020 05:43:21 +0000

Introduction

This series of posts intends to develop a flutter signup/login app working with API calls to Django backend. The idea is to use the DRF to create APIs which can be called by the flutter application. Most of the tutorials I have seen only talk about doing it with Firebase. Since I already will be using PostgreSQL for the Django, I intend on using that as my server DB and using SQLite with the Flutter Application to store just the Auth token and Username.

Django Rest Framework

I assume people reading this post already know about it but still just an introduction:

DRF allows for making powerful and flexible Web APIs.
DRF provides a lot of authentication schemes out of the box and even allows us to implement custom schemes.
DRF provides an easy way to serialize and deserialize JSON according to your DB Schema. (Model Serializers)

Flutter

Google Flutter is just awesome when it comes to cross-platform app development. The fact that it doesn't convert your code to machine compatible but instead is capable of directly rendering it on the machine is the best.

BloC design pattern

I intend to use BloC (Business Logic Component) which was announced by Google in GOOGLE I/O' 18. BloC is simply a state management pattern that works on sinks and streams. The widgets monitor the state changes and send them to the BloC using sinks and other widgets monitor those by subscribing to those streams where they expect the relevant data to arrive. The required sections are rebuilt per the state change. Thus Bloc stands in between data and UI and asynchronously maps events to states which rebuild the UI when required.

Let us Start

We will start by creating a new Django project to provide the required APIs for Login and Signup.
Go to your desired folder and activate your virtual environment. (You can refer here for the same)

Step 1

Create the project by:

django-admin startproject HOME

Note that I named my project as HOME, you can name it anything just be consistent in all the places where we use the names in the commands/code that follows.

cd into the project directory and create an app called api by:

python manage.py startapp api

I plan on using the auth system that Django provides by default. The User model has almost all the required fields that we may need for the Login and Signup. In case you need extra fields like phone number, address, etc. you can always define a new profile model that will have a one to one relationship with the User model. But for the sake of simplicity here, we are going to go with the User model itself. We have the fields first_name, last_name, username, email and password which we can use to create a new user.

Step 2

Install DRF by

pip install djangorestframework

and also add it to your installed apps

INSTALLED_APPS = [
    ...
    'rest_framework',
]

There are many auth systems available but I am going to go with Token Auth. We need to add that too in our installed apps:

INSTALLED_APPS = [
    ...
    'rest_framework',
    'rest_framework.authtoken',
]

We also need to add the following in our settings for us to be able to use the token auth:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
    ),
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ],
}

Migrate your changes by:

python manage.py migrate

Run it from HOME/

Step 3

Now that we are all set up and we have the default User model ready for us, we need to start creating our API

Create a file named serializers.py in HOME/api/ and the following code to it:

from django.contrib.auth.models import User
from rest_framework import serializers
from rest_framework.validators import UniqueTogetherValidator


class UserSerializer(serializers.ModelSerializer):

    def create(self, validated_data):
        user = User.objects.create_user(**validated_data)
        return user

    class Meta:
        model = User
        fields = (
            'username',
            'first_name',
            'last_name',
            'email',
            'password',
        )
        validators = [
            UniqueTogetherValidator(
                queryset=User.objects.all(),
                fields=['username', 'email']
            )
        ]

What we did here is we created a serializer in order to map JSON to our model and vice-versa. The unique together validator will allow us to verify that the combination of username and email is unique for any particular user.

edit your HOME/api/views.py and add the following code to it:

from .serializers import UserSerializer
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
from rest_framework.permissions import IsAdminUser
from django.contrib.auth.models import User


class UserRecordView(APIView):
    """
    API View to create or get a list of all the registered
    users. GET request returns the registered users whereas
    a POST request allows to create a new user.
    """
    permission_classes = [IsAdminUser]

    def get(self, format=None):
        users = User.objects.all()
        serializer = UserSerializer(users, many=True)
        return Response(serializer.data)

    def post(self, request):
        serializer = UserSerializer(data=request.data)
        if serializer.is_valid(raise_exception=ValueError):
            serializer.create(validated_data=request.data)
            return Response(
                serializer.data,
                status=status.HTTP_201_CREATED
            )
        return Response(
            {
                "error": True,
                "error_msg": serializer.error_messages,
            },
            status=status.HTTP_400_BAD_REQUEST
        )

What we are doing here is creating a get and a post method, the get method allows us to get a list of all the registered users with our application and the post method allows us to create a new user with the required info as specified in our serializer.
Also, I added IsAdminUser as the permission class, the reason is that we do not want anyone to spam our server by sending multiple POST requests and creating unnecessary accounts.

Last but not least we will add routes for access to the application. let us create the file HOME/api/urls.py and add the following code to it:

from django.urls import path
from .views import UserRecordView

app_name = 'api'
urlpatterns = [
    path('user/', UserRecordView.as_view(), name='users'),
]

We need to include the route for this and the token system in our main urls i.e. edit the file HOME/urls.py and add the following code to it:

from django.contrib import admin
from django.urls import path, include
from rest_framework.authtoken import views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include('api.urls', namespace='api')),
    path('api-token-auth/', views.obtain_auth_token, name='api-token-auth'),
]

We would need to create an admin user by:

python manage.py createsuperuser

Now enter the details asked and use them below.

Accessing our API

Thus, we have successfully created the required endpoint for the API.
We can now test it by the following:
I suggest you use POSTMAN which will make it easier for you.

Get authentication token

Endpoint: http://localhost:8000/api-token-auth/

Parameters required :

username, password
Sample body:

  {
    "username": "admin",
    "password": "password"
  }

Get / Post user

Endpoint : http://localhost/api/user/

GET
- parameters: auth token as header
- sample header:

  {
    "Authorization": "TOKEN <token>"
  }

sample response:

  {
    "username": "<username>",
    "first_name": "<first_name>",
    "last_name": "<last_name>",
    "email": "<email>",
    "password": "<encrypted string>"
   }

POST

parameters:
header: auth token as header
sample:

  {
    "Authorization" : "TOKEN <token>"
  }

body: username, email, first_name, last_name, password
sample:

{
  "username": "<username>",
  "first_name": "<first_name>",
  "last_name": "<last_name>",
  "email": "<email>",
  "password": "<encrypted string>"
 }

response (on succesful):

{
  "username": "<username>",
  "first_name": "<first_name>",
  "last_name": "<last_name>",
  "email": "<email>",
  "password": "<encrypted string>"
}

I will continue the project and write posts for each part. The end result will be a flutter app working with this API and SQLite DB.

Next: Flutter signup/login application with Django backend #2

Final Post: Flutter signup/login application with Django backend #3

Please feel free to drop a message at Ohuru in order to avail various development services offered by us.