DEV Community: Raynner Dourado Matos

Learning AI: Python foundation for Artificial Intelligence

Raynner Dourado Matos — Tue, 02 Jun 2020 02:08:20 +0000

Hello everyone!

Today we going to start a series of posts that will help us to learning Artificial Intelligence from the basic until the advanced concepts, in each post we will talk about a subject for we understanding how the Artificial Intelligence works and how we can manipulate it.

There are several programming languages that we can use to build a AI model, but we going to focus in Python, the most programming language used to Artificial Intelligence. In this post we going to learn the foundation of Python, your basic concepts and some tips that will help us to develop in this language.

It's recommended you have got knowledge about programming logic, the basics concepts like conditions, loops and other will not be explained in details in this series of posts.

Python foundation

The AI can be worked with several programming language, however the Python is the most language used to do this function nowadays, learn this language is essential, several pages in the Internet have got tutorials to teach how to develop something related with AI, Machine Learning or Deep Learning using the Python.

This language can be used for several other tasks, like:

To build an API;
To build a BFF (Backend for frontend);
To connect with a database;
To develop your business logic;
And beyond!

Python versions

Today we have got practically two popular versions of the Python, the Python 2 and Python 3. It's highly recommended — if you are starting a new project — to start using the version 3, with all news that this version brought to us, the developer will have the compatibility with the more recents packages available.

The Python 2 needs to be used when a project was started in this version and it's not possible, in this time, to upgrade for a newest version or if a specific package that was used is incompatible with the version 3.

There are several syntaxes differences between them, if you are using the old version and want to learn the new version, somethings that you was learn need to be forget.

Basic Concepts

Python is an interpreted language developed in 1991 by Guido van Rossum and it's based in object-oriented programming (OOP) concept, in other words, we can create class to represent objects and yours actions like any other OOP languages available.

As this language is multi-platform, we can run it in any popular operating system like macOS, Linux or Windows. This is important because we can write our code in a determined operating system and execute it in another, it's just necessary that the computer contains the Python package distribution installed.

Professionals who have got knowledge about other languages like C#, Java or Javascript can see some difference of syntaxe, but it's a little thing and is possible to learn easily and quickly. To we understand a difference of how we can create a class in Javascript (using Node.js) and Python I wrote a piece of code:

class Something {

    firstFunction() {

        return 0;
    }

    secondFunction() {

        return '';
    }
}

And now in Python:

class Something:
    def firstFunction():
        return 0

    def secondFunction():
        return ''

As we can see, it's not a monster, it's different, but we can learn easily. Python have got other facilities that increases the productivity of the developer, this language have got a huge amount of packages available in your community, these packages allow us to write, in few lines of code, things that in other languages we cannot do so easily. To understanding more deep the packages in Python, let's talk about the PIP!

The Python Package Manager

A modern programming language have got a package manager, like Node.js have got the NPM, .NET Framework have got the NuGet and Python have got the PIP.

With basic commands, the developer can install several packages that help your development, increase the security and other tasks that is very difficult to do by yourself. A developer community will support these packages that can be used by students, professionals or entire organisations, one of them, for example, is the Pandas package, this package allow us to manipulate the data more easily than if we used the common structures in Python, other packages like that can accelerate your development.

Let's see how we can install packages using the PIP:

pip install pandas

Above we can see an example of how we can install the Pandas packages in your more recent version, if is necessary to install a specific version of Pandas we need to do this:

pip install pandas==1.0.1

In this example, we set the version that we need to be installed, note when we install any packages, the installation is cached for all future projects, this can create a problem when we work with different projects with different requirements, for example, if a Project A needs a more recent Panda's version and a Project B needs one old version, we need to work with environments to allow us to use specific version for each project that we going to work.

And we have got this with another great resource developed with the same team of the PIP, the PIPENV.

NOTE: The PIP is included in Python installer and we will not need to installed separately, but PIPENV needs to be installed manually.

Managing your environment with PIPENV

For we understand what is PIPENV, we need to compare with another equivalent in market, we use the Node.js like our example. Node.js works a little bit different of the Python, all dependencies are described in package.json file inside a folder that contains all project's files, when we told to npm to install all dependencies, the Node.js creates a node_module folder that contains all references of this projects, in other words, all references stay inside a project's folder and not in a global folder.

In this way we don't have the problem of work in a two projects with requirement totally different. To resolve the problem of the versioning packages in Python, the PIP team created a tool that help us to manage correctly these dependencies. First we need to install the PIPENV, differently of the PIP, the PIPENV is not included in Python installer and need to be installed manually. Let's go:

pip install pipenv

As we can see, the PIPENV can be installed using the same PIP that we used to install other dependencies in examples above. After install, we need to understand how this tool works and how to configure it.
Our goal is the same what happens in Node.js, the dependencies versions in a different project cannot be conflicted each other, like a package.json file that indicates the project's dependencies. In PIPENV we going to create the requirements.txt file, we need to put the dependencies and your versions, the PIPENV will read it and install the right packages for each project that we work.

pandas==1.0.3
scikit-learn==0.23.1
nltk>=3.0

The example above is a requirements.txt file example, basically we have 3 dependences that we need to install when we execute the PIPENV command, for this project the Pandas package and Scikit-learn package need to have the exactly versions described in this file, but specifically in NLTK we can allow the PIPENV to install any available version greater than 3.0.

It is import to remember that this approach can generate error when a newest NLTK version is available to install, in other words, if a 4.0 version of the NLTK is available, your code probably will not prepared to use all new resources and some functions that you use in 3.0 version cannot be exists in the new version.

pipenv install

After create our requirements.txt, we just need to execute the above code in the same level of the requirements.txt, and after this, we can work with projects with different necessary requirements.

Conditionals in Python

Now that we know how to work the PIPENV, let's talk about conditionals in Python. Developers that have experience with other languages, like C style, could find strange for them of how we can use conditionals in this language, but it is easy to understand and more easily to write.

In this example below, we need to demonstrate the following conditions of the age variable:

An adult person, or;
A teenager person, or;
A child person.

age = 18

if age >= 18:
    print('You are an adult.')
elif age >= 13 and age < 18:
    print('You are a teenager.')
else
    print('You are a child.')

As we can see above, the syntax is different as Javascript, C# or Java conditions, but it is easy to understand how to work. The if and else is common like any language, but the else if is a little bit different. The Python syntax for intermediate conditions is elif, that means the same else if of other programming language.

We going to see another form to create conditional in Python, but for this, we need to understand another concept, the List Comprehensions.

List Comprehension

The List Comprehension is a way to create list in Python that have not similar in popular programming languages, this concept allow us to create new list more easily.

Below we have a list of numbers, of 0 to 9, and we have this List Comprehension that check what numbers is pair or odd:

number_list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

pair_list = [x for x in number_list if x % 2 == 0]
odd_list = [x for x in number_list if x % 2 != 0]

This is the easiest way to create a list in Python, in just one line of code we can create a conditional list, this concept can be useful in several algorithms of Machine Learning to decrease the complexity of our code.

To improve more the way to create a list, we can use List Comprehension together with another Python concept that I going to explain below: the Lambda operators.

Lambda operators

We going to use the same example above, but with some differences to apply the lambda concept in our code, the main objective is improve the quality and maintenance of code. The conditional that check if a number is pair or odd instead stay inside List Comprehension, we going to put inside a function and finally we use it together with List Comprehension to know if the number is the one we expected.

Let's take a example:

number_list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

is_pair_number = lambda number: number % 2 == 0

pair_list = [x for x in number_list if is_pair_number(x)]
odd_list = [x for x in number_list if not is_pair_number(x)]

The code is much similar with the another example, but in this, the verification if the number is a pair or an odd is done inside the lambda operator, it returns a True boolean value when is pair and False when is odd.

This Lambda code show to us that is simple to understand this functionality and how we can use it in our code, all of theses resources presented can be used to build your algorithm that work with Artificial Intelligence, Machine Learning and Deep Learning, these are bases of this language to you can build your models correctly and without great difficult.

Below we talk about of some packages available to work with AI, packages that manipulate our datasets, clean our data and more!

Packages to work with AI

In this section we talk about a little bit of the 3 packages that help us to work with Artificial Intelligence, they are very well documented if we need some more explanation about yours resources. In future post of Learning AI series we will detail in more deep yours functionalities and we will have real examples of how we can use them.

Pandas

This is the most basic package for all that want to work with AI and need to manipulate data. The Pandas has several practical functions when we can aggregate data, filter, remove null values, visualisation and more resources.

Data Scientist, AI Engineer and other professionals that work with dataset, can use this package to facilitate the process to understand and clean the data and other necessary process before use a Machine Learning or Deep Learning algorithm, this package is essential for all people who have this objective. Forums, GitHubs and other sources has a huge examples and code to you understand all functionality available.

In future post of this series we will use several times this packages to manipulate our data and transform them.

Scikit-Learn

For all people who needs to use out-of-box Artificial Intelligence models, the Scikit-Learn is the perfect choice, this package has AI models that we can apply in our dataset, to configure theses models the developer can easily configure the parameter and hyper parameters of models and test to evaluate the performance of the model that was chosen.

We work in future posts with classification models and we will use these classifications of the Scikit-Learn to process our dataset and we can understand deep and in more details some aspect of how to use it.

NLTK

Finally, who have familiar with Natural Language Processing probably know this package, because is so useful for this task, there are other packages to work with NLP, but NLTK is one of them more used. Depend of your language, this package can be a limitation, but if you use it with English language, all resource are fully used in your maximum.

This package is so important for people that need to work with Chatbots, all phrase wrote by user need to process with some NLP algorithm, need to do a pre-processing that remove the stopwords, applies lemmatizer, applies stemmizer and other techniques to process the user input.

Thank you to read this post, this series can help young professionals that want to discovery the wonderful world of Artificial Intelligence. In future posts we talk about in more specific details of how to prepare data, Machine Learning models, Deep Learning models, Computer Vision and other ways to use theses AI algorithm to build a great application that allow us to create a better experience of our applications and upgrade our knowledge.

References

[1] — https://en.wikiversity.org/wiki/Python_Concepts
[2] — https://realpython.com/what-is-pip/
[3] — https://pypi.org/project/pipenv/
[4] — https://www.pythonforbeginners.com/basics/list-comprehensions-in-python
[5] — https://pandas.pydata.org/
[6] — https://scikit-learn.org/
[7] — https://www.nltk.org/

Developing in Node.js using Typescript

Raynner Dourado Matos — Thu, 23 Jan 2020 02:47:55 +0000

Hello everyone!

In this first post we talk about how to develop a basic backend using Node.js and Typescript as main developer language, first of all we need to details the basics aspects of the Typescript and why is very interesting to use it as default language in your application.

Throughout in this article we can build together basic backend with routes, controllers and responses, in a next post we should continue to use this application to apply unit tests using Jest and you will learn how increase the coverage and how to use it with the Typescript.

Now let's go to discovery the benefits to develop using Typescript!

What is a Typescript?

The Typescript language is a typed superset of Javascript and allow us to create more cohesive and clean code, types errors can be discovered in development time, helping the team to understanding what kind of values need to pass in function's params that have no much knowledge about.

Let's take a little example in Javascript code and your equivalent in Typescript code:

function sum(num1, num2) {

    return num1 + num2;
}

Above we have a Javascript code that sum two values and returns the result, this is a simple function that can show to us the problem when you not working with typed language, let's take a call with this function with two different ways:

// Returns 4
sum(2, 2)

// Returns '22'
sum('2', 2)

When we call the sum function and the two parameters is numbers, the response will be 4, however, when some of this two parameters is a string, the Javascript will interpret that is a concat of two string and will response '22'.

This simple example show to us that non-typed language can be harder to understand by people that don't know how the code works. Because this problem, the Microsoft developed the open source language Typescript to mitigate things like that.

function sum(num1: number, num2: number) {

    return num1 + num2;
}

Above is the same code that was developed in Javascript, but with parameters types, all developers can now see the types and input the correct value to the function, another values will be generate an error.

// Returns 4
sum(2, 2)

// Error in development time
sum('2', 2)

The second call is not more valid to use the sum function, an error is generated and the developer will know that need to put two numbers, not strings.

Another good aspect to use Typescript is that you can use new features available in newest ECMAScript specification and have no problem to execute in old browsers or old Node.js versions, because the Typescript compiler it will transform all of your code to ECMAScript 5 specification.

Building a backend using Typescript

Now we go to build our basic backend using Typescript as the main language, the image below show us how the solution will stay:

Where:

classes - Is a folder that contains all generic classes that our application going to use.

controllers - Is a folder that contains all controllers of our application.

routes - Is a folder that contains all routes that we should define.

services - Is a folder that contains the integrations with another systems, like request for some external APIs.

To build our application we need some packages to allow us to start the app, the package.json below contains all necessary references that we the need to install before start the configuration:

{
    "name": "typescript-node-api",
    "version": "1.0.0",
    "description": "",
    "main": "index.js",
    "scripts": {
        "start": "npm run build && node dist/index.js",
        "build": "gulp scripts",
        "test": "echo \"Error: no test specified\" && exit 1"
    },
    "keywords": [],
    "author": "",
    "license": "ISC",
    "devDependencies": {
        "@babel/core": "^7.7.7",
        "@types/body-parser": "1.17.1",
        "@types/debug": "4.1.5",
        "@types/express": "4.17.2",
        "@types/morgan": "1.7.37",
        "@types/node": "13.1.4",
        "gulp": "4.0.2",
        "gulp-babel": "8.0.0",
        "gulp-typescript": "5.0.1",
        "typescript": "3.7.4"
    },
    "dependencies": {
        "body-parser": "1.19.0",
        "debug": "4.1.1",
        "express": "4.17.1",
        "morgan": "1.9.1"
    }
}

We should write our codes using Typescript, but the Node.js core need a javascript file to interpret the content and execute all commands, firstly we going to create and configure the Gulp file to transcript our Typescript to Javascript.

Below we have the basic configuration file, we need to create the gulpfile.js in the root of the application:

const gulp = require('gulp');
const babel = require('gulp-babel');
const ts = require('gulp-typescript');
const JSON_FILES = ['src/*.json', 'src/**/*.json'];

// Indicates to gulp the typescript configuration
const tsProject = ts.createProject('tsconfig.json');

const scripts = () => {

    return tsProject.src()
        .pipe(tsProject()).js
        .pipe(babel())
        .pipe(gulp.dest('dist'));
};

const watch = () => {

    gulp.watch('src/**/*.ts', scripts);
};

const assets = () => {

    return gulp.src(JSON_FILES).pipe(gulp.dest('dist'));
};

exports.scripts = scripts;
exports.watch = watch;
exports.assets = assets;

const build = gulp.series(gulp.parallel(scripts));
gulp.task('build', build);
gulp.task('default', build);

In this file we have some configurations, like where files will be transcripted and where will stay the javascript files created by this transcript.

The next step is define the tsconfig.json, this file contains the compile options to transcript yours Typescripts code to Javascript code, below we have the example of how this file is:

{
    "compilerOptions": {
        "target": "es6",
        "module": "commonjs",
        "sourceMap": true
    },
    "exclude": [
        "node_modules"
    ]
}

In this example, compilerOptions contains the information that indicates to compiler what version of ECMAScript will be used and others that is exampled below:

target - ECMAScript version that need to be transcripted.

module - Specify module code generation like "None", "CommonJS", "System", etc

sourceMap - If need to generate the .map files.

The exclude property is an array that contains all folders you don't need to compile when this process start, in this case node_modules is not necessary, because is a folder that contains our references.

Creating the Typescript files

The next step is start to code using Typescript, now we need to create the App.ts inside the "src" folder, in this file we have the configuration of middlewares, routes and we expose the express to use in others points of the system.

import * as express from 'express';
import * as logger from 'morgan';
import * as bodyParser from 'body-parser';
import { Utility } from './classes/helpers/utility';

class App {

    public express: express.Application;

    constructor() {

        this.express = express();
        this.middleware();
        this.routes();
    }

    private middleware(): void {

        this.express.use(logger('dev'));
        this.express.use(bodyParser.json());
        this.express.use(bodyParser.urlencoded({ extended: false }));
    }

    private routes(): void {

        this.setAllRoutes();
        this.setDefaultRoute();
    }

    private setAllRoutes(): void {

        const utility = new Utility();
        let arrayFileRoutes = utility.readRecursiveDirectory('routes');

        arrayFileRoutes.forEach(file => {

            let routeInstance = require(`./${file.replace(/\.[^/.]+$/, '')}`);
            let fn = `/api${file.replace('routes', '').split('\\').join('/').replace(/\.[^/.]+$/, '')}`;

            this.express.use(fn, routeInstance.default.getRouter());
            console.log(`Route ${fn} --> OK`);
        });
    }

    private setDefaultRoute(): void {

        this.express.get('/api', (req, res, next) => {

            res.status(200).json({
                title: 'API Test',
                version: '1.0.0',
                path: '/api/v1'
            });
        });
    }
}

export default new App().express;

In this project, the routes will be created based where the routes files is located, like this example below:

In this image we have the routes folder and inside it we have the v1 and v2 folders, this is a common pattern in API development to create folders to specify the version of api resource, work in this way allow us to modify the api behavior without create errors in applications that used this API before.

With this automatic defined route, the log when start the application will be this way:

All routes, based where the file is located in routes folder was automatically created by App.ts, the folder structure was read and was created all necessary routes.

And finally, let's to create the Index.ts that is the file that have the responsibility to start definitively our application:

import * as http from 'http';
import * as debug from 'debug';

import App from './App';

debug('ts-express:server');

const port = normalizePort(process.env.PORT || 3000);
App.set('port', port);

const server = http.createServer(App);
server.listen(port);
server.on('error', onError);
server.on('listening', onListening);

console.log(`Online in port ${port}`);

function normalizePort(val: number | string): number | string | boolean {

    let port: number = (typeof val === 'string') ? parseInt(val, 10) : val;

    if (isNaN(port))
        return val;
    else if (port >= 0)
        return port;
    else
        return false;
}

function onError(error: NodeJS.ErrnoException): void {

    if (error.syscall !== 'listen')
        throw error;

    let bind = (typeof port === 'string') ? 'Pipe ' + port : 'Port ' + port;
    switch (error.code) {

        case 'EACCES':
            console.error(`${bind} requires elevated privileges`);
            process.exit(1);
            break;

        case 'EADDRINUSE':
            console.error(`${bind} is already in use`);
            process.exit(1);
            break;

        default:
            throw error;
    }
}

function onListening(): void {

    let addr = server.address();
    let bind = (typeof addr === 'string') ? `pipe ${addr}` : `port ${addr.port}`;

    debug(`Listening on ${bind}`);
}

Routes, Controllers and Services!

Below we have the basic routes created to be the point for consults all client's information, new functions can be added to add more actions to your API's endpoints:

import * as express from 'express';
import { ClientController } from '../../controllers/v1/ClientController';
import { IRouterApi } from '../../classes/interfaces/IRouterApi';

class ClientRouter implements IRouterApi {

    public getRouter(): express.Router {

        const clientController = new ClientController();

        let router = express.Router();
        router.route('/').get(clientController.getClients);

        return router;
    }
}

export default new ClientRouter();

And our IRouterApi interface:

import { Router } from "express";

export interface IRouterApi {

    getRouter(): Router;
}

Now we have define the routes and your interface, we need to create the controller and service for we can test our endpoint and see the how the client's routes will return, let's take a look of Client controller:

import { NextFunction, Request, Response } from "express";
import { ClientService } from '../../services/v1/ClientService';

export class ClientController {

    public getClients(req: Request, res: Response, next: NextFunction): void {

        const clientService = new ClientService();
        res.status(200).json(clientService.searchClients());
    }
}

In this file, basically, we create a basic function that search all clients, we instantiate a ClientService and we returns this values in the response of our application.

export class ClientService {

    public searchClients(): Array<any> {

        return [
            {
                message: 'Client name'
            }
        ];
    }
}

Finally we have the ClientService that contains a function that returns an array with a simple object!

The test

After this journey to create our first basic backend using Typescript, we need to test if the response is in the way that we need, in our example, I will use the Postman, but you can use any of application that your preference.

And the response...

I'll be back!

In the next posts, we going to use this Typescript basic backend to apply other technologies and concepts. I hope that all of you have to like this post and I see you soon!

Bye!

References

[1] https://www.typescriptlang.org/
[2] https://ionicframework.com/docs/v3/developer-resources/typescript/
[3] https://medium.com/swlh/the-major-benefits-of-using-typescript-aa8553f5e2ed
[4] https://www.typescriptlang.org/docs/handbook/compiler-options.html