DEV Community: Divyesh Aegis

How to Build Log Data Analytics using Apache Spark?

Divyesh Aegis — Fri, 11 Sep 2020 11:41:26 +0000

We are going to see how to process log data in Spark in this blog. Let us understand the log structure first and write a regular expression to match the log pattern and then extract the value and do some data analytics services on the log data.

First, let us see how the log data we are going to process looks like and understand it.

Sample log data:


83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET 

/presentations/logstash-monitorama-2013/images/kibana-search.png 

HTTP/1.1" 200 203023 

"http://semicomplete.com/presentations/logstash-monitorama-2013/" 

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

Now let us break this log data and understand what each of them represents:


83.149.9.216 – IPAddress
-            - ClientId
-            - userId
[17/May/2015:10:05:03 +0000] – dateTime
GET – Method
/presentations/logstash-monitorama-2013/images/kibana-search.png 
– endpoint
HTTP/1.1 – protocol
200 – responseCode
203023 – contentSize
"http://semicomplete.com/presentations/logstash-monitorama-2013/" – URL
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 

(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" – browser

Now let us write code to process this log data using Apache Spark Databricks environment.

I have a log file located in the below location.

Now let us read few lines of this log data using Spark and see how it looks like.

closest garage to me

Now let us write a case class to have all the fields we have in the logs and later we will create RDD of type Access Log.

Now lets us create a Pattern that matches the log data using a regular expression like below.

Now let’s write a Scala function that parses the log data and creates an object of case class that we can use to create an RDD.

Now let us read this log data and parse it to create RDD[AccessLog].

Now let us find out the top 10 response codes with the counts in descending order.

Logging Levels in Hibernate

Divyesh Aegis — Fri, 06 Mar 2020 07:00:20 +0000

Technology: Hibernate is most used framework for database related activities, using this database interaction will easily and quickly develop the application.

But if we get an exception and error in data either reading/inserting/updating/deleting the data from database we need to add some debugger logs, or we can enable hibernate logging so that we can easily know hibernate generated statement and parameters, what framework is passing to a database.

Hibernate provides different logging levels to log the statement.

Logging SQL: It is the basic level of debugging, we can log the Hibernate generated SQL statement, without actual parameter values.
Hibernate uses the category org.hibernate.SQL to log this information, so we can set the logger level to DEBUG for this package, so that generated statement will be logged.
Configuration for log4J:


<logger name="org.hibernate.SQL">
<level value="debug"/>
</logger>

Add these configuration either log4j.xml of log4j configuration file.

If we are using log4j2 then add the below in logger configuration:


<logger name="org.hibernate.SQL"value="debug"/>

If we are using logback, then add below line to configuration:


<logger name="org.hibernate.SQL"value="DEBUG"/>

And the generated SQL log will look like below:



2019-12-07 23:04:23 | DEBUG | [main] o.h.SQL:127 - insert into Employee (employeeNumber, name, title, id) values (?,?,?,?)
2019-12-07 23:04:23 | DEBUG | [main] o.h.SQL:127 - select employee0_.id as id1_0_, employee0_.employeeNumber as employee2_0_, employee0_.name as name3_0_, employee0_.title as title4_0_ from Employee employee0_

Using this information we only know the generated SQL statements, but it will not contain any parameter information.

Logging SQL Parameter Values: using the above approach, we know the generated SQL statement, in most cases, it is enough the understand the problem, but in some cases, we also need the parameters to understand the problem.

Hibernate logs the input parameters and retrieved results using the org.hibernate.type.descriptor.sql category with a log level of TRACE.

Log4j Configuration:


<loggername="org.hibernate.type.descriptor.sql">
<level value="trace"/>
</logger

Log4j2 Configuration:


<logger name="org.hibernate.type.descriptor.sql"level="trace"/>

Logback Configuration:


<logger name="org.hibernate.type.descriptor.sql"level="TRACE"/>

After adding these logging levels below logging will add logger output:



2019-12-07 23:04:23 | DEBUG | [main] o.h.SQL:127 - insert into Employee (employeeNumber, name, title, id) values (?, ?, ?, ?)
2019-12-07 23:04:23 | TRACE | [main] o.h.t.d.s.BasicBinder:64 - binding parameter [1] as [VARCHAR] - [001]
2019-12-07 23:04:23 | TRACE | [main] o.h.t.d.s.BasicBinder:64 - binding parameter [2] as [VARCHAR] - [John Smith]
2019-12-07 23:04:23 | TRACE | [main] o.h.t.d.s.BasicBinder:52 - binding parameter [3] as [VARCHAR] - [null]
2019-12-07 23:04:23 | TRACE | [main] o.h.t.d.s.BasicBinder:64 - binding parameter [4] as [BIGINT] - [1]
2019-12-07 23:04:23 | DEBUG | [main] o.h.SQL:127 - select employee0_.id as id1_0_, employee0_.employeeNumber as employee2_0_, employee0_.name as name3_0_, employee0_.title as title4_0_ from Employee employee0_
2019-12-07 23:04:23 | TRACE | [main] o.h.t.d.s.BasicExtractor:60 - extracted value ([id1_0_] :  [BIGINT]) - [1]
2019-12-07 23:04:23 | TRACE | [main] o.h.t.d.s.BasicExtractor:60 - extracted value ([employee2_0_] :  [VARCHAR]) - [001]
2019-12-07 23:04:23 | TRACE | [main] o.h.t.d.s.BasicExtractor:60 - extracted value ([name3_0_] :  [VARCHAR]) - [John Smith]
2019-12-07 23:04:23 | TRACE | [main] o.h.t.d.s.BasicExtractor:50 - extracted value ([title4_0_] :  [VARCHAR]) - [null]

As we see it will not only the parameters but it also logs the binding parameters, so we can enable and disable the previous logging level independent of this logging level.

Logging All Activity: To understand the problem even deeper, we can enable the whole hibernate logging to debug, to all activities.
Log4j Configuration:


<logger name="org.hibernate.type.descriptor.sql">
<level value="trace"/>
</logger>

Log4j2 Configuration:


<logger name="org.hibernate.type.descriptor.sql" value="trace"/>

Logback Configuration:


<logger name="org.hibernate.type.descriptor.sql" value="TRACE"/>

Hibernate Statistics: Apart from SQL and JDBC parameter values, Hibernate also logs the statistics, which will be useful for understanding the performance issues.

Hibernate uses the category org.hibernate.stat to log this information. But, Hibernate does not always generate these statistics because it can have a bad influence on the performance.
Hibernate provides one configuration property to specify the statistics logging.
true
Using this property hibernate will log all statistics for each SQL query.


2019-12-07 23:25:18 | DEBUG | [main] o.h.s.i.StatisticsInitiator:101 - Statistics initialized 
[enabled=true]
2019-12-07 23:25:19 | DEBUG | [main] o.h.s.i.StatisticsImpl:729 - HHH000117: HQL: 
from com.baeldung.hibernate.logging.Employee, time: 22ms, rows: 1
2019-12-07 23:25:19 | INFO | [main] o.h.e.i.StatisticalLoggingSessionEventListener:258 - 
Session Metrics {
    55600 nanoseconds spent acquiring 1 JDBC connections;
    178600 nanoseconds spent releasing 1 JDBC connections;
    2167200 nanoseconds spent preparing 2 JDBC statements;
    2426800 nanoseconds spent executing 2 JDBC statements;
    0 nanoseconds spent executing 0 JDBC batches;
    0 nanoseconds spent performing 0 L2C puts;
    0 nanoseconds spent performing 0 L2C hits;
    0 nanoseconds spent performing 0 L2C misses;
    47098900 nanoseconds spent executing 1 flushes (flushing a total of 1 entities 
    and 0 collections);
    0 nanoseconds spent executing 0 partial-flushes (flushing a total of 0 entities 
    and 0 collections)
}

Using these statistics we can easily analyze which query is taking more to execute and we can provide the solution based on this.

Conclusion: In the current tutorial we learned the different logging levels for Hibernate, how can a Java developer debug the exception/errors in data which executing the SQL statements. And finally, we learned the Hibernate statistics which are useful in understanding the time duration for each individual SQL query.

Introduction to Django

Divyesh Aegis — Tue, 18 Feb 2020 06:11:43 +0000

Introduction to Django

Django is a web-framework, developed in Python. It is free, open-source and used by millions of developers every year. Python Django development is very robust and can handle heavy websites easily. Some of the websites using Django are - Instagram, Pinterest, Bitbucket, etc.

Why Django

Extremely Fast - It was designed to help developers take applications from the ideation phase to the pilot project as quickly as possible.
Secure - Django takes security on priority and is designed to avoid concerns that developers generally miss.
Scalable - Few of the busiest sites on the internet are built on Django like - Disqus, Instagram, NASA, Spotify, Pinterest, etc.
All in One - It contains many extra features that can handle common Web-development tasks. It comes loaded with User authentication, RSS feed, sitemaps, etc.
Versatile - We can build many different types of web-application using Django like - Content Management Systems, Social networking, just-browsing websites, etc.

Now, let’s start with setting up Django in our system.

Installing Django Considering, Python 3.x.x and pip are already installed, run the following command to install Django.
```
pip install django
```
To check if version and if Django is properly installed, run the following command -

If any error is logged, check if Django is properly installed.

Creating a Virtual environment and starting the first app To keep our code and required libraries isolated, we will create a virtual environment. I have created an environment named venv. Run the following command to install the virtual environment.

sudo apt-get install virtualenv -y

Once it is installed, let’s create a virtual environment named venv.

virtualenv venv

After this, let’s start with the Django project.

We will run the following command to set up skeleton for django -

django-admin startproject mysite

Here, mysite is the name of my project. You will see that there is a directory created by the name of your project. Go into the directory and you must see a file named manage.py. This file will serve as the entry point to our server and all the commands will be sent through it.

Now, let’s check if the files are working correctly. Run the following command -

python manage.py run server

Your terminal will show that the server has started running your localhost on port number 8000.

Ignore the warning in red color for now and let’s check if the server actually started or not. Open your browser and open the URL - 127.0.0.1:8000.

Hurray! We just started a Django server.

Setting up a database and playing with migration

Now, in the previous command that we run, we saw the warning in red color regarding the migrations.
Just run the following command and see the magic -

python manage.py makemigrations

The terminal shows - “No changes detected”.
This means that there were no changes detected from the previous save. So let’s run the following command to send all the saved changes to the database.

python manage.py migrate

You must have got something similar to the above picture on your terminal. This means that your database just got set up and you must now see a file named db.sqlite3 in the folder.

Create an app in a project

Well, our project is now set and we can create apps within our project. I will create an app named demo for demo purposes.

python manage.py startapp demo

We will now be able to see a directory named demo in our folder.
Open the file named views.py in the demo folder and let’s add the following code to it.


from django.http import HttpResponse

def index(request):
    return HttpResponse("Hello, world. Your app is now ready.")

Now, as we have written a function, we need to map it to a URL to run it. So, we will now create a file named urls.py in the demo directory and write the following code there.


from django.urls import path
from . import views

urlpatterns = [
   path('', views.index, name='index'),
]

Now, our project does not know that some URL is defined for our app, so we need to create a URL pattern in our project directory as well. Copy-paste the following code in the urls.py file in the mysite directory.


from django.contrib import admin
from django.urls import include path

urlpatterns = [
    path('demo/', include('demo.urls')),
    path('admin/', admin.site.urls),
]

Now run the server to check if the app is running.

Open the following URL -

http://localhost:8000/demo

Voila! We just created a small Django app to start building a new project.

How to set up a REST Service or a Web Application in Django

Divyesh Aegis — Tue, 23 Jul 2019 06:04:46 +0000

Introduction:

Django is a very versatile framework, primarily used for developing web applications, but it is also widely used in creating mobile app backends and REST APIs, among other things. Here we will take a quick look at how to make these things possible using Django.

As you all probably know that REST stands for “Representational State Transfer”. Basically what happens is that a user (or some software agent on one end) provides some input (or performs some activity) and the result of those inputs (or activities) are sent to the server side using a protocol that allows a request to be sent to the server (in most cases HTTP protocol is used, but it doesn't need to be HTTP, as long as it can support a request/response scheme). The server, on receiving the request, makes appropriate changes to the state of the system (hence we call it “State Transfer”).

A Brief Discussion on How to Create a Django Project:

Django runs on HTTP protocol, and hence it is very handy in creating a REST API or a RESTful application. It allows the user to provide inputs on a web interface (or any interface that is capable of sending a HTTP request object to the Django application). In our examples in this document, we will be considering web application interfaces, but you can easily stretch them to a mobile app or any other app that can send a similar HTTP request.

In order to understand how to do these things, we need to know a bit of Django here. This is my first post in a series of 5 posts on the same topic, so in this document, I will acquaint you with certain Django files that are of utmost importance, along with the basic structure of a Django app. (Please note that 'Django' is pronounced as 'jango', NOT D-jango. The first character 'D' is silent.)

A 'Django project' is composed of a number of 'Django applications' that are stitched together with some logic. The first thing that you do when creating a Django project is to install Django in your python library. This is usually done by running the command “pip install Django==”. You may skip the version number part, in which case the latest version of Django that can be handled by your OS and your python distro installed on your system will be installed. Fair enough.

Once the installation is successfully complete, you will find a file named “Django-admin” in the path of your computer. (Run “which django-admin” to see where it exists, on my system it is in “/usr/bin/django-admin”). To create a project, you need to execute the following command in a directory of your choice. Preferably create a new directory, 'cd' into it and run the following command:

#> django-admin  startproject

Your project name has be to provide to create the Django project. For example, let us assume I am creating a project named “testyard”. In order to create my project, I need to execute the following command:


#> django-admin startproject testyard

The above command creates the following directory structure in the directory where you executed the above mentioned command.


testyard/
    manage.py
    testyard/
        __init__.py
        settings.py
        urls.py
        wsgi.py

A Discussion on the Files Created Above:

The first file, manage.py, is very important. This is the file that will eventually assist you to create applications, run the Django test web server, run Django management tasks (not covered in this document, but we will take look at it in a later article), and a host of other activities. The init.py file in the next level ensures that we will be working in an object oriented environment. Next comes the 2 most important files: settings.py and urls.py. We will discuss settings.py file first and urls.py next.

settings.py: This file contains the settings for the Django applications to work. Some of the more important config params in this file are the Database connection parameters (username, password, database name, DB host, port,Engine, etc.), static files location, Template Loaders, Middleware classes (we will be dicussing this in another article, but just keep in mind that these are the programs that interact with the request object before they reach the “view” functions, so they are capable of changing the request before it can be processed), installed apps, root urlconf (we will discuss this in this article later), etc.

You are also free to write your own application specific settings file in the same location. However, you need to put a different name to it and then import settings.py in it. There are a load of other parameters in settings.py and I would request you to go through a settings.py file in the location “https://github.com/supmit13/python-ff-ext/blob/master/urldbapp/settings.py” and figure out the parameters.

urls.py: This file contains the map of the path requested by a client (browser or a bot) to the specific view function that needs to be called when the given path is requested by an HTTP request. You will find a sample of such a urls.py file here: “https://github.com/supmit13/python-ff-ext/blob/master/urldbapp/urls.py”. This file is also known as the “urlconf”.

The file named wsgi.py exists to allow the applications you are creating to run on uwsgi and nginx (the production environment stuff). We will be taking a look at this file later in this article.

Once these files have been created, you start creating applications inside the project. But first, you need to figure out if the previous procedure is working fine with a server. So for that purpose, you need to run the development server of the application

To do this, you need to run the following command on the command prompt. Note that in order to run this, you need to be in the same directory where “manage.py” exists.

>python manage.py runserver 0.0.0.0:8080

This command will start the development server. Now, you should be able to go to a browser, type in localhost:8080/login/ , and you should be able to see the login screen if the urls.py has an entry for that url path. Alternatively, you may just type localhost: 8080/ to see if your Django server is running.

So that makes you capable enough to start creating a REST API using python/Django. In the subsequent sections we will demonstrate how to write code to create the REST API.

The views.py file: Django follows a variation of the MVC design pattern, and it is normally referred to as the MVT pattern. The 'M' stands for the model (the DB schema), V stands for views.py, which in an MVC framework is the controller part, while the 'T' stands for templates, which in MVC framework would be the view component.

Since we will be looking at the REST API first, we will concentrate on the views.py. First, have a look at the handlers.py at the following link: “https://github.com/supmit13/python-ff-ext/blob/master/urldbapp/savewhatyoulookedat/handlers.py”. In this program, the code that is to be in views.py has been put into handlers.py, but for all practical purposes, they work in the same manner. You may consider the code in handlers.py to be that of views.py in this example. (Actually, this was some code I wrote long back when I was just starting to dabbling in Django. Hence I made something to see if Django is flexible enough, and found that it sure was flexible).

The Structure and Behaviour of Django:

The “views.py” file will contain one or more functions (note that it may also contain classes, and we will take a look at classes in views.py in another post), and the functions will contain only one argument: “request”. The “request” is basically an instance of the HttpRequest class (defined by Django). All view functions in a Django App consume an instance of this HttpRequest class and return an instance of the HttpResponse class.

Why do we need to have multiple functions in the views.py file? This is because, you might want to serve multiple pages or responses, each of which has a different URL path. For example, you might want to serve a login page, a registration page, an activity to check the login credentials of the user who is trying to login into your app, and a dashboard page. Each of these pages will have a different URL path, say, http://mywebsite.com/login for login page, http://mywebsite.com/registration for registration page, and so on. So each of these URLs will need a separate function to handle the request. Hence, we need one function for each of these actions in the views.py file.

How do we associate each of the activities mentioned above with a specific views.py function? This is where the urls.py file comes into play. The urls.py has a map of each URL path to a specific function in views.py of a particular Django app (Remember we mentioned in the beginning that a Django project is composed of one or more Django apps. We will get to the apps part in a moment). An urls.py looks something like the following:


urlpatterns = patterns('',
    (r'^time/$', current_datetime),
    (r'^savewhatyoulookedat/login/$', userLogin),
    (r'^savewhatyoulookedat/register/$', userRegister),
    (r'^savewhatyoulookedat/$', saveURL),
    (r'^savewhatyoulookedat/logout/$', userLogout),
    (r'^savewhatyoulookedat/downloadPlugin/$', firefoxPluginDownload),
    (r'^savewhatyoulookedat/managedata/$', manageData),
    (r'^savewhatyoulookedat/commandHandler/$', executeCommand),
    (r'^savewhatyoulookedat/showimage/$', showImage),
    (r'^savewhatyoulookedat/search/$', searchURL),
    # (r'^admin/doc/', include('django.contrib.admindocs.urls')),
    # Uncomment the another line to  enable the authority:
    # (r'^admin/', include(admin.site.urls)),
)

Basically, as you can see, the mapping is actually between a regular expression that should match the URL path and a function in the views.py file. As you create more activity handlers in your views.py file, you keep adding an entry for each of them in the urls.py file. For a Django project with multiple applications, the urls.py file might also look like the following:


urlpatterns += patterns('',
  url(r'^%s$'%mysettings.REGISTER_URL, 'skillstest.Auth.views.register', name='newuser'),
  url(r'^%s$'%mysettings.DASHBOARD_URL, 'skillstest.views.dashboard', name='dashboard'),
  url("%s$"%mysettings.LOGIN_URL, 'skillstest.Auth.views.login', name='login'),
url(r'%s$'%mysettings.MANAGE_TEST_URL, 'skillstest.Tests.views.manage', name='managetests'),
url(r'%s$'%mysettings.CREATE_TEST_URL, 'skillstest.Tests.views.create', name='createtests'),
url(r'%s$'%mysettings.EDIT_TEST_URL, 'skillstest.Tests.views.edit', name='edittests'),
url(r'%s$'%mysettings.ABOUTUS_URL, 'skillstest.views.aboutus', name='aboutus'),
url(r'%s$'%mysettings.HELP_URL, 'skillstest.views.helpndocs', name='helpndocs'), url(r'%s$'%mysettings.CAREER_URL, 'skillstest.views.careers', name='careers'),
... ... ...

'Auth' and 'Tests' are the names of the applications in the “skillsets” project. Don't worry about the variables in uppercases – they are defined elsewhere and are of no consequence to our example here.

So, now let us see how to create an application inside a project. We do that by executing the following command:


python manage.py startapp

For example, if our app name is “Letsplay”, then we would run

python manage.py startapp Letsplay

The above command creates a directory structure like the following:


Letsplay/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

In the above structure, we will focus mostly on the views.py and the models.py files. However, we will also touch upon the others first.

The “admin.py” file is required if you want to customize your admin panel in Django. Normally, if you try to access the URL “http://localhost:8080/admin”, you would see an admin panel. This will display you all the models you have (we will discuss models in just a bit), the config settings of your Django project (in read-only mode, of course), etc.
The “apps.py” file allows the creator of the Django app to put in some application specific parameters. Each app in a Django project has its own apps.py file.

The “tests.py” file allows the app creator to write tests for the app. This file needs to conform to a certain structure. It needs to define a class for a specific test case. This class needs to have a method named “setup”, and then it should have the tests defined as methods in the class itself. Unless you are a Django purist, you won't use this file to define your tests. Normally, in the real life scenarios, we have an application created using Django, another component created using some other technology and several other components fitted together to work as a service. In such cases, we need to write tests to check the functionality of the entire scheme of things rather than just the Django part. Hence, it is almost customary to create tests as a different suite using python or some other language like Perl or ruby (or whatever the tester prefers).

By and large, any application you write (in python using Django or any other language and framework), you eventually end up interacting with a database somewhere down the line. All Django apps also tend to do the same. This is where the models.py file steps in. The “models.py” file basically provides you with an ORM (Object-Relational-Mapping) scheme in the Django app. Hence for every table in your preferred database, you have a class defined for it in the models.py file. It looks something like this:


from django.db import models
import os, sys, re, time, datetime
import inspect


"""
'Topic' is basically category or domain.
"""
class Topic(models.Model):
    topicname = models.CharField(max_length=150)
    user = models.ForeignKey(User, null=False)
    createdate = models.DateField(auto_now=True)
    isactive = models.BooleanField(default=True)

    class Meta: 
        verbose_name = "Topics Table"
        db_table = 'Tests_topic'

    def __unicode__(self):
        return "%s"%(self.topicname)


class Subtopic(models.Model):
    subtopicname = models.CharField(max_length=150)
    subtopicshortname = models.CharField(max_length=50)
    topic = models.ForeignKey(Topic, blank=False, null=False)
    createdate = models.DateField(auto_now=True)
    isactive = models.BooleanField(default=True)

    class Meta:
        verbose_name = "Subtopics Table"
        db_table = 'Tests_subtopic' # Name of the table in the database

    def __unicode__(self):
        return "%s (child of %s)"%(self.subtopicname, self.topic.topicname)

class Session(models.Model):
    sessioncode = models.CharField(max_length=50, unique=True)
    status = models.BooleanField(default=True) # Will be 'True' as soon as the user logs in, and will be 'False' when user logs out.
    # The 'status' will automatically be set to 'False' after a predefined period. So users will need to login again after that period.
    # The predefined value will be set in the settings file skills_settings.py. (skills_settings.SESSION_EXPIRY_LIMIT)
    user = models.ForeignKey(User, null=False, blank=False, db_column='userid_id')
    starttime = models.DateTimeField(auto_now_add=True) # Should be automatically set when the object is created.
    endtime = models.DateTimeField(default=None)
    sourceip = models.GenericIPAddressField(protocol='both', help_text="IP of the client's/user's host")
    istest = models.BooleanField(default=False) # Set it to True during testing the app.
    useragent = models.CharField(max_length=255, default="", help_text="Signature of the browser of the client/user") # Signature of the user-agent to guess the device used by the user.
    # This info may later be used for analytics.


    class Meta:
        verbose_name = "Session Information Table"
        db_table = 'Auth_session'
    
    def __unicode__(self):
        return self.sessioncode

    def isauthenticated(self):
        if self.status and self.user.active:
            return self.user
        else:
            return None

    def save(self, **kwargs):
        super(Session, self).save(kwargs)

The attributes in the classes are the fields in the respective tables in the database. The name of the DB table is defined in the “class Meta” of each of the Topic and Subtopic classes with the attribute named “db_table”. The database associated with these tables is defined in the settings.py file (remember when we discussed settings.py file attributes?). For the datatypes used in the models.py file, you need to look up the Django documentation as there are quite a few datatypes and relationships and they cannot be dealt with here. In fact, the documentation for them is quite substantial. However, we have used only a few of them above and they are quite self-explanatory.

Actually, Django is quite popular because of 2 reasons.

It provides the developer with all the boiler plate code, so the coder doesn't need to write all the boring stuff.
It provides the coder with the ORM, so retrieving or setting a value in a certain row of a specific table in the DB is quite easy. That is the “up side” of it. There is a “down side” too. When you use ORM, you do not use SQL statements, and hence if the operation is a little complex, the ORM can become quite inefficient. With SQL statements, you can do some optimization to make the statement run faster, but with ORM, there is no such possibility. For this reason, Django offers a way out. You can create “raw” SQL statements to query your DB, but this is rarely used by most developers. You should use “raw” SQL statements only when you see that the ORM way of manipulating the DB is distinctively inefficient.

Anyway, let us now move on to the final stages of this document. This happens to be the most important stage in the creation of a REST application. We will now take a look at the views.py file. Please refer to the example code below:


# User login handler
def login(request):
    if request.method == "GET":
        msg = None
        if request.META.has_key('QUERY_STRING'):
            msg = request.META.get('QUERY_STRING', '')
        if msg is not None and msg != '':
            msg_color = 'FF0000'
            msg = skillutils.formatmessage(msg, msg_color)
        else:
            msg = ""
        # Display login form
        curdate = datetime.datetime.now()
        tmpl = get_template("authentication/login.html")
        c = {'curdate' : curdate, 'msg' : msg, 'register_url' : skillutils.gethosturl(request) + "/" + mysettings.REGISTER_URL }
        c.update(csrf(request))
        cxt = Context(c)
        loginhtml = tmpl.render(cxt)
        for htmlkey in mysettings.HTML_ENTITIES_CHAR_MAP.keys():
            loginhtml = loginhtml.replace(htmlkey, mysettings.HTML_ENTITIES_CHAR_MAP[htmlkey])
        return HttpResponse(loginhtml)
    elif request.method == "POST":
        username = request.POST.get('username') or ""
        password = request.POST.get('password') or ""
        keeploggedin = request.POST.get('keepmeloggedin') or 0
        csrfmiddlewaretoken = request.POST.get('csrfmiddlewaretoken', "")
        userobj = authenticate(username, password)
        if not userobj: # Incorrect password - return user to login screen with an appropriate message.
            message = error_msg('1002')
            return HttpResponseRedirect(skillutils.gethosturl(request) + "/" + mysettings.LOGIN_URL + "?msg=" + message)
        else: # user will be logged in after checking the 'active' field
            if userobj.active:
                sessobj = Session()
                clientip = request.META['REMOTE_ADDR']
                timestamp = int(time.time())
                # timestamp will be a 10 digit string.
                sesscode = generatesessionid(username, csrfmiddlewaretoken, clientip, timestamp.__str__())
                sessobj.sessioncode = sesscode
                sessobj.user = userobj
                # sessobj.starttime should get populated on its own when we save this session object.
                sessobj.endtime = None
                sessobj.sourceip = clientip
                if userobj.istest: # This session is being performed by a test user, so this must be a test session.
                    sessobj.istest = True
                elif mysettings.TEST_RUN: # This is a test run as mysettings.TEST_RUN is set to True
                    sessobj.istest = True
                else:
                    sessobj.istest = False
                sessobj.useragent = request.META['HTTP_USER_AGENT']
                # Now save the session...
                sessobj.save()
                # ... and redirect to landing page (which happens to be the profile page).
                response = HttpResponseRedirect(skillutils.gethosturl(request) + "/" + mysettings.LOGIN_REDIRECT_URL)
                response.set_cookie('sessioncode', sesscode)
                response.set_cookie('usertype', userobj.usertype)
                return response
            else:
                message = error_msg('1003')
                return HttpResponseRedirect(skillutils.gethosturl(request) + "/" + mysettings.LOGIN_URL + "?msg=" + message)
    else:
        message = error_msg('1001')
        return HttpResponseRedirect(skillutils.gethosturl(request) + "/" + mysettings.LOGIN_URL + "?msg=" + message)
-------------------------------------------------------------------------------------------------------------------

# User registration handler
def register(request):
    privs = Privilege.objects.all()
    privileges = {}
    for p in privs:
        privileges[p.privname] = p.privdesc
    if request.method == "GET": # display the registration form
        msg = ''
        if request.META.has_key('QUERY_STRING'):
            msg = request.META.get('QUERY_STRING', '')
        if msg is not None and msg != '':
            var, msg = msg.split("=")
            for hexkey in mysettings.HEXCODE_CHAR_MAP.keys():
                msg = msg.replace(hexkey, mysettings.HEXCODE_CHAR_MAP[hexkey])
            msg = "%s"%msg
        else:
            msg = ""
        curdate = datetime.datetime.now()
        (username, password, password2, email, firstname, middlename, lastname, mobilenum) = ("", "", "", "", "", "", "", "")
        tmpl = get_template("authentication/newuser.html")
        #c = {'curdate' : curdate, 'msg' : msg, 'login_url' : skillutils.gethosturl(request) + "/" + mysettings.LOGIN_URL, 'register_url' : skillutils.gethosturl(request) + "/" + mysettings.REGISTER_URL, 'privileges' : privileges, 'min_passwd_strength' : mysettings.MIN_ALLOWABLE_PASSWD_STRENGTH, }
        c = {'curdate' : curdate, 'msg' : msg, 'login_url' : skillutils.gethosturl(request) + "/" + mysettings.LOGIN_URL, 'hosturl' : skillutils.gethosturl(request),\
             'register_url' : skillutils.gethosturl(request) + "/" + mysettings.REGISTER_URL,\
             'min_passwd_strength' : mysettings.MIN_ALLOWABLE_PASSWD_STRENGTH, 'username' : username, 'password' : password, 'password2' : password2,\
                 'email' : email, 'firstname' : firstname, 'middlename' : middlename, 'lastname' : lastname, 'mobilenum' : mobilenum, \
             'availabilityURL' :  mysettings.availabilityURL, 'hosturl' : skillutils.gethosturl(request), 'profpicheight' : mysettings.PROFILE_PHOTO_HEIGHT, 'profpicwidth' : mysettings.PROFILE_PHOTO_WIDTH }
        c.update(csrf(request))
        cxt = Context(c)
        registerhtml = tmpl.render(cxt)
        for htmlkey in mysettings.HTML_ENTITIES_CHAR_MAP.keys():
            registerhtml = registerhtml.replace(htmlkey, mysettings.HTML_ENTITIES_CHAR_MAP[htmlkey])
        return HttpResponse(registerhtml)
    elif request.method == "POST": # Process registration form data
        username = request.POST['username']
        password = request.POST['password']
        password2 = request.POST['password2']
        email = request.POST['email']
        firstname = request.POST['firstname']
        middlename = request.POST['middlename']
        lastname = request.POST['lastname']
        sex = request.POST['sex']
        usertype = request.POST['usertype']
        mobilenum = request.POST['mobilenum']
        profpic = ""
        #userprivilege = request.POST['userprivilege']
        csrftoken = request.POST['csrfmiddlewaretoken']
        message = ""
        # Validate the collected data...
        if password != password2:
            message = error_msg('1011')
        elif mysettings.MULTIPLE_WS_PATTERN.search(username):
            message =  error_msg('1012')
        elif not mysettings.EMAIL_PATTERN.search(email):
            message =  error_msg('1013')
        elif mobilenum != "" and not mysettings.PHONENUM_PATTERN.search(mobilenum):
            message = error_msg('1014')
        elif sex not in ('m', 'f', 'u'):
            message = error_msg('1015')
        elif usertype not in ('CORP', 'CONS', 'ACAD', 'CERT'):
            message = error_msg('1016')
        elif not mysettings.REALNAME_PATTERN.search(firstname) or not mysettings.REALNAME_PATTERN.search(lastname) or not mysettings.REALNAME_PATTERN.search(middlename):
            message = error_msg('1017')
....
....
    return HttpResponse(html)


The above code has 2 functions, and we will discuss them shortly. But before that, please take a look at the corresponding urls.py file for these 2 above functions:

urlpatterns = patterns('',
    (r'^savewhatyoulookedat/login/$', login),
    (r'^savewhatyoulookedat/register/$', register)
)

As you can see above, the 'login' function will be called when you try to access the following URL from your browser (or any other web client):
http://localhost:8080/savewhatyoulookedat/login/

The 'register' function will be called when you try to access the following URL:
http://localhost:8080/savewhatyoulookedat/register/

Note how the 'request' object has been used along with some other objects that are the product of Django ORM. For example, in the login function, there is an instance of the Session model. The DB table behind the Session model is named “Auth_session” and it is specified in the models.py file above. Thus, whenever a user hits one of the URLs mentioned above, the view runs some DB queries and figures out what response to send to the client. This is how a RESTful application should work, and as you can see, Django really makes it easy to develop one.

Conclusion:
Python Django development is a very extensive framework, and since we were discussing REST applications, I deliberately left out Django templates. I will be explaining templates in another post, but since REST apps do not always need an HTML interface, I am skipping it for now. Also, we have just touched on some of the concepts of Django, and there is not enough room to discuss all of them in detail here. I would suggest that you go through this post, try and understand as best as you can, and then take a look at the official Django documentation for more details on the topic.

Hope you find this useful.

The Role of Python as an IoT App Driver

Divyesh Aegis — Thu, 21 Jun 2018 07:22:09 +0000

Over the past decade or so the two technological phenomena is which have created a buzz are python web development and IoT. It is quite impressive to note the rise of IoT under the shadow of writing which has safeguarded its reach into public and relatively more comfortable development methods. IoT is considered as a sheer driving force that has helped IoT flourish and be more mature than ever before in recent times. Python is a lightweight and easy to learn programming language which is backed by one of the most significant contributing community which helps the language expand in all directions and that has helped the language cover IoT with full efficiency.

It would not be wrong to consider IoT as a substantial affair in the IT landscape. It is a fantastic piece of stat that has a 31.2% contribution share in helping IoT strengthen its roots in public and business solutions. Once there was a time when ITO was considered as something vague used by business giants to build content for the publicity matters, but Python has complimented the potential.

How is Python the real IoT app driver?

While there are numerous reasons for IoT to be in the driver seat of the technological Scenario, Brighton has been fuelling the adrenaline of tech-savvies in this regard. In the real sense, but I think can we consider to impart a living soul to every IoT device which has its programming bass, behavioral pattern, and an IP address to be identified across the network. Might sound easy but only a language as mature and extensive as Python can do. Bison is perfect for being the app driver as it can help you in bed quality functional code within the embedded systems and also back your functionalities with libraries for sensor networks, data analytics, machine learning, and several other domains of technology.

IoT and Python are generally clubbed up due to similar stride across various domains of technology. Siri encompasses almost all applications of technology while Python is steadily running to support all of them during the Python Software Development phase.

What does Python possess?

One of the most persuasive reasons for Baichung to be the language for IoT is its classic simplicity of syntax and how effortlessly one can learn to implement and deployed technology through this. Now, this has some serious benefits of its own. As my thing allows you to quickly code various technologies and complex functionalities, it reduces the time and material cost and also lets one move away from the importance of super-skilled human resources.

We all know that IoT has embedded systems of different kinds which are set up in environments that run on distinct mechanisms. Python offers numerous formatting standards and compiling options to let you build functionalities for your IoT devices.

Generally, Java is one of the clear examples which comes to anyone’s mind when we talked about and elaborated, mature programming language. The issue with such heavyweight languages is that they are difficult to code and require multiple dependencies to work together for binding into an application. Python is easily embedded, scalable, and entirely portable across any network of IoT devices. The language supports all primary single board computers which are extensively used in IoT solution development.

If IoT is one of the most bronze doubt free in a dense forest of technology, Brighton is the water to its roots. Attributes of Python like portability, scalability, and easy to code syntax encouraged IoT engineers to work out highly utilizable functionalities making it mature. The interesting combo of IoT with Brighton has a bright future ahead with the rising importance of data science and analytics in the recent time.

Follow These Java Best Practices for Ultimate App Performance

Divyesh Aegis — Fri, 18 May 2018 05:39:38 +0000

Java is a programming language that has evolved the most. From being birthed as an embedded devices' programming language, Java has evolved into a language that can be utilized to build robust and secure codes for all platforms. Java's impeccable versatility, robustness, and scalability render it suitable for a diverse range of applications. And, thus it is so that Java is the most popular programming language even to this day, throughout the world.

However, even when we have it good, we want it better. Such is the human nature. And, hire java developers have always been exploring how Java can be exploited for performance benefits on a number of platforms. With IoT and Data Analytics and Machine Learning, it has become increasingly important, to regard security and process perfection, as the highest aim.

Here are some of the best practices that Java developers can follow to improve the overall performance of their Java application development-

• Pair Programming – This methodology of writing code takes us back to our school days when one of us would write the code, and the other partner would read the code and think about the potential issues that could spring up from it. This is done with the help of two developers who share the infrastructure. While the focus of one of the developers is on writing the code, the other developer focuses on finding and analyzing the code for possible performance issues. This step makes sure that the code is improved upon just as it is being written. There are people who love to undertake pair programming for the simplicity and the effectiveness of the method.

• Premature Optimization – It is advisable to delay optimization until it is necessary. Premature optimization takes up a lot of time, and more often than not, developers need to do it again. Only after developers can prove that an optimization process is necessary, should the process be implemented. Premature optimization also makes the code hard to maintain and read, introducing unnecessary complexities in it. Why should you spend a lot of time, improving on the non-critical aspects of your application?

• Test the performance – By creating a performance test suite for your application, you can make sure there are no loopholes left in the application, and that no new ones have crept in. Building a performance test suite and running it on the application before and after executing it, can give you a fair idea of the window of improvement you have achieved. These test runs make sure that no other aspects of the app were impacted when you make a performance upgrade to your app.

• Follow Conventions – With coding conventions, Java developers can be sure that they have developed the best possible code for their application. Coding conventions help save the maintenance time and costs. These guidelines are prepared by the developers and carry the recommended style of programming and the methods for each individual aspect of the code. Everything including the file organization, indentation, spacing, comments, etc., is included in this document. This makes the code readable by another programmer because history has it that no project was solely started and finished by a single team. With a detailed convention guide, you can improve the performance of your application.

Reviewing the code is the next obvious step, to ensure that the performance of the application you are developing remains top-notch. With these guidelines and tips in mind, effective improvements can be made to the performance of any application.

Considerations for Customer Service Surveys

Divyesh Aegis — Thu, 29 Mar 2018 06:30:46 +0000

A considerable measure has been composed on performing client administration or consumer loyalty studies. I might want to offer only a couple of best practices to consider in light of my own experience being on the two sides of the overview.

Effective time management

Divyesh Aegis — Thu, 15 Mar 2018 05:51:40 +0000

However, by integrating following better time management tips, you can efficiently manage your time for things you want to get done.