DEV Community: rodrigohfnogueiraOBatman

Let's talk about A.I.

rodrigohfnogueiraOBatman — Tue, 12 Mar 2019 01:32:19 +0000

Human brain

Before talking about real A.I. we need to try to understand what our brain does when making a decision. Let's imagine I have three cups, two cups made of glass and one made of plastic. I have one kid in the house, of course, I'll leave the plastic cup for childrens, in my memory I know that the cup in the children's hand has a bigger chance to fall on the floor.

Subconsciously I remember all the times a cup fell on the floor, and noticed a pattern. In my experience (for example), 80% of the times a cup has broken, it was on the hands of a children, and 60% of the times a plastic cup fell on the floor it didn't broke. So I found two facts, based on memories, that can help solve a problem.

So, we need to:

-Identify the entities of my problem.
-Identify the problem.
-Identify patterns coorelating the problem's entity and the result of my problem.
-Remember the possibilities of the patterns.
-Identify patterns that deny the effects of my problem.
-Use this pattern knowledge to solve future problems.

This is one simple kind of problem, the problem could be resolved by any different ways, proportional logic, trees, and other kind of organization our brain choose.

A.I.

Let's work the ideia inside the computer science field, using object-oriented programing.

We can first identify our entities to the problem:

-Cup
-Person

And then the problem happens when

-Cup.situation = broken;

And we know that a "Person" can be:

-Children
-Teeneger
-GrownUp

And we know that in our memory we have an inference motor wich will coorelate the facts, the rules and produce new memories with the learning. So, we know that after the lunch the inference motor will coorelate the final state of the cup and the person type to produce a memory (a fact). Let's understand a children has broke a cup, the fact will be "children - cup - broken".

After a number of lunchs we will have a combination of memories, like:

-"children - cup - broken"
-"Teeneger - cup - ok"
-"Teeneger - cup - broken"
-"GrownUp - cup - ok"
-"GrownUp - cup - ok"
-"children - cup - broken"
-"children - cup - ok"
-"children - cup - broken"
-"GrownUp - cup - ok"

So when I have to know for wich user will I give the thoughest cup, I just have to find all the percentage of broken devolutions of cups and choose the type of user wich has the biggest... In out example:

-Children {
count: 4,
broken: 3,
ok: 1,
brokenPercentage: 75%
};
-GrownUp {
count: 3,
broken: 0,
ok: 3,
brokenPercentage: 0
};
-Teeneger {
count: 2,
broken: 1,
ok: 1,
brokenPercentage: 50%
};

So the biggest chance to break a cup is to give it to a children.

Components

There are 3 main components, we have the rules, the inference motor, and the memory (facts).

The inference motor matches the rules that can be executed based on the facts avaible on the memory and process new memories.

So the components are:

-Inference Motor
-Facts
-Rules

The rules has the expert system conditions, it's where the expertise of your analysis is present.

Big Data

A basic introduction to big data is necessary in this posts because just as we can't remember everything in our life, the computer does not need to process everything to get to the correct answer in decisions.

Usually it's done using a percentage of the real memory size, we have, for example 10.000.000 registers. So we can just generate a random identifier in each register, and then just find, using statistics, a size of sample wich can give us the better margin of success with less computer processing and time.

When we select the sample to analyse we must order by the random identifier and take the firsts results, this way we can use a random sample to get the conclusion. If the statistic model has been done right, we will get the same result as if we processed all the registers in the database.

This is a simple and pratical exemple of big data's use for the AI.

Trees, Graphs and Neural Networks

There is a lot to talk about this topics so I will make one to each one of those, but for us in this posts is to understand how it can be used in de decision making process.

Another kind of decision is using graphs, where all the possibilities are considered using graphs and the decision is made using the DFS or BFS and others.

The trees are a lot similar to the graphs decisions, but more limited, it can be combined with linear algebra to get into the decision faster, it will be explain in another post.

Neural Networks had it's researchs stopped for something like 10 years, until the back propagation algorithm was developed, it has a lot of experience in the game learning, and the success point is the most important in the algorithms. It basically has neurons wich receive an input and execute a math formula to get the output, the math formula is influenced by the correction weight wich in each case of failure it auto corrects until gets success on the output. Knowing that, we have to understand there is input layers, hidden layers and output layers, and the backpropagation algorithm gives a way to send feedback with corrections in the formulas of the previous neurons.

Neural Network is a complex subject and will have a post just for it.

For now, that's enough to a "introduction" to AI.

Let's talk about webdrivers

rodrigohfnogueiraOBatman — Mon, 11 Mar 2019 19:14:20 +0000

The internet

The connections between your computer and a server (Google server, Amazon server) any of those is based on HTTP (or HTTPS) requests and "code" files as Html, CSS, and Javascript. As a client-server connection, your computer has a client software installed, wich can be Google Chrome, Firefox or any other. This software manage the communication between the client and the server.

So we can organize in this way: You (Web Browser) requests Server (online service) using HTTP(S) as communication protocol. When the files are delivered by the server the browser can design the web site and it's functions can be load.

Ok, this was a basic review.

A webdriver

We can now work a little further on the client's interpretations.

A webdriver pretends to be a web client making requests to the server as if it were a normal user. This way the software can access the Html of each page, capturing informations when needed.

I can give you usage examples, I'm from Brazil, and here we have some startups, and big companies that use this kind of system everyday, financial industry uses to capture information about their client's money situation, other companies can use to check company's money situation in government web sites.

How does it work

Inside the HTTP or HTTPS request we have the CRUD operations (Create, Read, Update Delete). In the HTTP it's called GET (Read), POST (Create), PUT (Update), DELETE (Delete). This is the first thing to know.

The HTTP request carries a body and a header, when it's a POST or a PUT request, it's expected to have a body content. When the request is a GET or DELETE, it usually does not have a body content, just headers and the URL.

We know that we need to pretend to be a browser, so we can receive the information that we want. But how will we know wich request and when we have to do? Usually you have to use a "sniffer" to track every request sent by your web client and then do the exactly same requests.

I like to use "Fiddler" (https://www.telerik.com/fiddler) it's very simple to use, you can just target the web client and track the requests of any navigation you want. Of course you have to target the web client and access the web site, the exactly path you want the webdriver to process.

The hardest part will be understand wich request is done, when and why, and of course where are the variables of each request.

Architecture

Everything is a state machine, your webdriver couldn't be different. I will show a model of architecture, wich can be good or not for your usage, but I have used many times this architecture in my webdrivers.

This is a really simple architecture that will be represented without the fixation of a specific language.

Goals:

-Map all the pages of the navigation.
-Parameters for each request and each page.
-Cookies (Parameters and navigation).
-Identify the informations of interest in each page (Regex).

So in your code you will need...

Enum (example):

    -Page 1.
    -Page 2.
    -Page 3.
    -Page 4.

Extract (example)

    -Url
    -Dictionary<string, string> query
    -List<string> regexs
    -Dictionary<string, string> header
    -object body

Dictionary <Enum, Extract> (example):

    -Page 1, Extract.
    -Page 2, Extract.
    -Page 3, Extract.
    -Page 4, Extract.

And the main function (example):

-Loop (navigation.situation != End):

    -Execute the functions NavigateTo(currentPage).
    -Get the response content (Html, JS and CSS).
    -Execute the regex of the page.
    -Get the parameters (including cookies, and Html         
    variables) for the next page navigation.

-End Loop

And for the end the extract module, wich can be done in markup languages, databases, tables, and whatever you would want to.

This was a introduction to webdrivers and it's architecture, of course it's a complex subject, and this is not even a half of it. What is usually done is make frameworks to deal with Html readings (the Regex for example), and even a automation of the navigation. Using XML tags to describe regex, urls, query url, and the others properties, it can just be uploaded to a generic webdriver, wich will just use the XML uploaded to navigate whatever web site you need.

Let's make a framework

rodrigohfnogueiraOBatman — Tue, 24 Jul 2018 13:32:59 +0000

Hello guys, this is my first post, so I'm a little nervous.

I would like to talk about making a framework.

First things first. In this post I would like to make a code to database interaction framework using MongoDB.

Using MongoDB with C#, we already have a good framework: mongodriver.
Let's make a abstraction to some operations of mongodriver, so we can code de business faster in other layers of code.

Let's start with the Configuration.

Using mongodb, we used to link the database with a "MongoUrl" class, wich we can build dynamically using a Configuration class to abstract.

Let's code (using C#) a simple configuration class to our framework of MongoDB:

public class Configuration
{
    public MongoUrl mongoUrl { get; set; }
    public string dbName { get; set; }
}

Let's remember the format of a simple mongodb Url to localhost:

MongoUrl mongoUrl = new MongoUrl("mongodb://localhost:27017/CONTROLE_OD_DB")

In this simple configuration example, we will not care to build dynamically or abstract the MongoUrl, this is not the point of the post.
Now we will get in the real point of this post, the art of write a generic framework, even a small one.

Let's make a class with the generic operations.

Again, first things first. Let's make the configuration.
I will name it class Context.

So:

public class Context
{
    #region Configuration
    public static Configuration _configuration { get; set; }
    static MongoClient client;
    static IMongoDatabase db;

    public static void Initializer(Configuration configuration)
    {
        _configuration = new Configuration();
        _configuration = configuration;
        client = new MongoClient(_configuration.mongoUrl);
        db = client.GetDatabase(_configuration.dbName);
    }
    public static void Dispose()
    {
        client = null;
    }
    #endregion
}

Let's understand the post.
We want to make our life easier making a framework.
First thing is to identify the boring parts of writing the code using just the old framework, mongodriver. The first boring part is to deal with the configuration. We had to get the database, make the mongoUrl every time, so we made here a "Context" class to save the configuration in a static variable, so it will not need a new instance all the time.
We will only get the database one time and get the IMongoDatabase one time. Better than that is the fact that we will not worry about them anymore.

So let's start to build a basic "collection" operations region.

    #region Collections

    public static void CreateCollection(string collectionName)
    {
        db.CreateCollection(collectionName);
    }
    public static IMongoCollection<T> GetCollection<T>(string collectionName)
    {

        return db.GetCollection<T>(collectionName);
    }
    public static IMongoCollection<T> GetCollection<T>()
    {
        return GetCollection<T>(typeof(T).Name + "_COLLECTION");
    }
    #endregion

We can use generic types to get whatever collection we want and set the collection using the type of the "T" class of our method. So, let's remove the problem of set the string name of a collection, and make the framework set the name of the collection using the class name. Just for organization we can write a "_COLLECTION" at the end, so it will be better to read.

Now we have this class.

public class Context
{
    #region Configuration
    public static Configuration _configuration { get; set; }
    static MongoClient client;
    static IMongoDatabase db;

    public static void Initializer(Configuration configuration)
    {
        _configuration = new Configuration();
        _configuration = configuration;
        client = new MongoClient(_configuration.mongoUrl);
        db = client.GetDatabase(_configuration.dbName);
    }
    public static void Dispose()
    {
        client = null;
    }
    #endregion
    #region Collections

    public static void CreateCollection(string collectionName)
    {
        db.CreateCollection(collectionName);
    }
    public static IMongoCollection<T> GetCollection<T>(string collectionName)
    {
        return db.GetCollection<T>(collectionName);
    }
    public static IMongoCollection<T> GetCollection<T>()
    {
        return GetCollection<T>(typeof(T).Name + "_COLLECTION");
    }
    #endregion
}

Here we have the database creation, and the collections creation, just what we need to start to write a operations region, CRUD (Create, Read, Update, Delete) region.

Let's make the insert:

    #region CRUD
    public static T Insert<T>(T document)
    {
        var collection = GetCollection<T>(typeof(T).Name + "_COLLECTION");
        collection.InsertOne(document);
        return document;
    }
    public static List<T> Insert<T>(List<T> documents)
    {
        GetCollection<T>(typeof(T).Name + "_COLLECTION").InsertManyAsync(documents);
        return documents;
    }
    public static List<T> Insert<T>(string collectioName, List<T> documents)
    {
        GetCollection<T>(collectioName).InsertManyAsync(documents);
        return documents;
    }
    public static T Insert<T>(string collectioName, T document)
    {
        GetCollection<T>(collectioName).InsertOne(document);
        return document;
    }
    public static List<BsonDocument> Insert(string collectioName, List<BsonDocument> documents)
    {
        GetCollection<BsonDocument>(collectioName).InsertManyAsync(documents);
        return documents;
    }
    public static BsonDocument Insert(string collectioName, BsonDocument document)
    {
        GetCollection<BsonDocument>(collectioName).InsertOneAsync(document);
        return document;
    }

In mongodriver we have always to choose between insert many, or insert one, insert one async, insert bson, insert object, I don't want to worry about that anymore, let the framework choose each one to use.
For all the inserts methods (and all the operations we will make of the CRUD sequence) we will use the database already declared and the getcollection method.

We are choosing each things not to worry, and making generic methods to abstract them.

Going on...

Let's make a the R (Read) of the "CRUD":

    public static List<T> GetDocuments<T>(FilterDefinition<T> filter)
    {
        return GetCollection<T>().FindAsync(filter).Result.ToList<T>();
    }

    public static List<T> GetDocuments<T>(string collectionName, FilterDefinition<T> filter)
    {
        return GetCollection<T>(collectionName).FindAsync(filter).Result.ToList<T>();
    }

    public static T GetDocument<T>(FilterDefinition<T> filter)
    {
        return GetCollection<T>().FindAsync(filter).Result.SingleOrDefault<T>();
    }
    public static T GetDocument<T>(string collectionName, FilterDefinition<T> filter)
    {
        return GetCollection<T>(collectionName).FindAsync(filter).Result.SingleOrDefault<T>();
    }

A good thing to do is get everything returning in the type of object I want to use, I don't want to convert c# collections into lists for example, I want every return of each method right how I want to use them, in this example, it would be a List or a simple T.

Let`s make the D (Delete) and the U (Update) :

    public static void Update<T>(MongoDB.Driver.FilterDefinition<T> filter, KeyValuePair<string, object> property, bool updateOne, bool IsUpsert)
    {
        var update = Builders<T>.Update.Set(property.Key, property.Value);            
        if (updateOne)
            GetCollection<T>()
                .UpdateOneAsync(filter, update, new UpdateOptions { IsUpsert = IsUpsert });
        else
            GetCollection<T>()
                .UpdateMany(filter, update, new UpdateOptions { IsUpsert = IsUpsert });
    }
    public static void Update<T>(MongoDB.Driver.FilterDefinition<T> filter, Dictionary<string, object> properties, bool updateOne, bool IsUpsert)
    {
        foreach(var property in properties)
            Update<T>(filter, property, false, true);
    }
    public static void Update<T>(MongoDB.Driver.FilterDefinition<T> filter, Dictionary<string, object> properties)
    {
        foreach(var property in properties)
            Update<T>(filter, property, false, true);
    }
    public static void Update<T>(MongoDB.Driver.FilterDefinition<T> filter, Dictionary<string, object> properties, bool updateOne)
    {
        foreach(var property in properties)
            Update<T>(filter, property, false, true);
    }
    public static void Update<T>(MongoDB.Driver.FilterDefinition<T> filter, KeyValuePair<string, object> property)
    {
        Update<T>(filter, property, false, true);
    }

    public static void Update<T>(MongoDB.Driver.FilterDefinition<T> filter, KeyValuePair<string, object> property, bool updateOne)
    {
        Update<T>(filter, property, false, true);
    }

    public static void Delete<T>(FilterDefinition<T> filter, bool DeleteOne)
    {
        if (DeleteOne)
            GetCollection<T>().DeleteOne(filter);
        else
            GetCollection<T>().DeleteMany(filter);

    }
    public static void Delete<T>(FilterDefinition<T> filter)
    {
        Delete<T>(filter, false);
    }

So we can go on to the beautiful part of the post.
Let's make more new methods, to get everything even easier.

Everything by now was changing the Context class, or making a basic Configuration class. Now let's create a Base class.

I will create a new class named "Base".

public class Base<T>
{
    public T Insert(T T)=>Context.Insert<T>(T);
    public void Upsert(FilterDefinition<T> filter, T entity)
    {
        if(List(filter).Count > 0)
        {
            Dictionary<string, object> props = new Dictionary<string, object>();
            foreach (var responsibleProp in entity.GetType().GetProperties()) props.Add(responsibleProp.Name, responsibleProp.GetValue(entity));
            Context.Update<T>(filter, props);        
        }
        else Context.Insert<T>(entity);    
    }
    public T Find(FilterDefinition<T> filter)=>Context.GetDocument<T>(filter);
    public List<T> List(FilterDefinition<T> filter)=>Context.GetDocuments<T>(filter);
    public void Delete(FilterDefinition<T> filter)=>Context.Delete<T>(filter);
}

One of the things I don't want to worry is if we are making a Insert or a Update, so we can create a Upsert method. If the filter get's any result in the query, it will update these results with the entity, else it will insert a new entity in the database.

"Find" with a simple filter, and "List" returning a "List"(List of generic objects), and then the update.

Now we can configure the framework into a new software: let's create a ConfigDB class :

public class ConfigDB<T> : Base<T>
{
    public ConfigDB()
    {
        Configuration configuration = new Configuration();
        Connection conn = new Connection();
        configuration.dbName = conn.GetDatabaseName();
        configuration.mongoUrl = new MongoUrl(conn.GetUrl());
        Context.Initializer(configuration);

    }
    ~ConfigDB()
    {
        Context.Dispose();
    }
}

And let's make a simple agent class, just to use the base operations :

public class Agent<T> : ConfigDB<T>
{

}

Now we can create our business class, I will name it UserBusiness :

public class User
{
    public string login {get;set;}
    public string password {get;set;}
    public string curriculum {get;set;}
    public string aboutMe {get;set;}


    [BsonId(IdGenerator = typeof(StringObjectIdGenerator))]
    public string ID { get; set; }
    public bool deleted { get; set; } = false;
    public bool active { get; set; } = true;
    [BsonDateTimeOptions(Kind = DateTimeKind.Local)]
    public DateTime creationDate { get; set; } = DateTime.Now;
}


public class UserBusiness
{
    public User InsertUser(User user) => new Agent<User>().Insert(user);
    public User findUserByID(string ID)=> new Agent<User>().Find(Builders<User>.Filter.Where(u => u.ID == ID));
    public User findUserByLogin(string login)=> new Agent<User>().Find(Builders<User>.Filter.Where(u => u.active == true && u.login == login));
}

This was just a small example of how to think when building a framework.

We need to worry, now, about what we don't want to worry about in the future.

This was my message, hope you like it.