DEV Community: viola kinya kithinji

INTERMEDIATE DATA ENGINEERING CONCEPTS.

viola kinya kithinji — Thu, 15 Sep 2022 13:15:29 +0000

In this article am going to discuss a number of key data engineering concepts that will help understand data engineering career path.
Big data is a term used to describe large, complex datasets that are difficult to process using traditional computing techniques. Big data often includes data sets.

Business intelligence (BI) is defined as the collection of processes and strategies for analyzing data to generate insights used to make business decisions.

Data architecture involves the process of designing, constructing, and maintaining data systems. Data architecture includes the design of data models, database management systems, and data warehouses. Data engineers often work with data architects to design and implement data systems, but they can also work independently.

Data lakes in data architecture.

Data architecture in data marts and Olap cubes.

Containerization is the process of packaging an application so that it can run in isolated environments known as containers. Containerization allows for better resource utilization and portability of applications. A containerized application encapsulates all of its dependencies, libraries, binaries, and configuration files into containers. This allows an application to run in the cloud or on a virtual machine without needing to be refactored.
Docker has become synonymous with containers and is a suite of tools that can be used to create, run, and share containerized applications.

Docker for data engineering courtesy of google pics

Kubernetes, or k8s, is a portable, open-source platform for managing containerized applications.

Kubernetes for data engineering. courtesy of google pictures.

Cloud computing is a model for delivering IT services over the internet. Data engineers often use cloud-based services, like Amazon S3 and Google Cloud Storage, to store and process data.

Databases are collections of data that can be queried. Relational databases, such as MySQL, Oracle, and Microsoft SQL Server, store data in tables and have existed for over four decades. Now, there are many different types of databases including:
Wide-column stores* such as Cassandra and HBase
Key-value stores such as DynamoDB and memcachedb
Document databases such as MongoDB and Couchbase
Graph databases such as Neo4j

Data accessibility is the ability of users to access data stored in a system.

Data compliance and privacy is the act of following laws and regulations related to data. Data privacy is the act of protecting data from unauthorized access.

Data governance is the process of managing and governing data within an organization. Data governance includes policies and procedures for managing data.

Data marts are subsets of data warehouses that contain only the data needed by a specific group or department.

images courtesy of google pictures.

Data integration platforms are tools that help organizations combine data from multiple sources. These typically include features for data cleaning and transformation.eg iPaaS- is a set of automated tools that integrate software applications that are deployed in different environments.

Data infrastructure components can include virtual machines, cloud services, networking, storage, and software. These components are necessary for data systems to function.

Data pipelines encompass the process of extracting data from one or more sources, transforming the data into a format that can be used by applications further down the line, and loading the data into a target system. Data pipelines essentially automate the process of moving data from one system to another.

Data repositories or data stores are systems that are used to store data, as discussed earlier. Examples include relational databases, NoSQL databases, and traditional file systems.
Data sources are the systems or devices from which data is extracted. Examples of data sources include Global mental health data.

Data warehouses are centralized systems that store all the data organizations collect. Data warehousing involves extracting data from multiple sources, transforming the data into a format that can be used for analysis, and loading the data into the warehouse.

Data lakes are repositories that store all the data organizations collect, in their rawest form. Data lakes are often used for storing data that has not been transformed or processed in any way.

ETL and ELT processes are used for moving data from one system to another.

ETL (extract, transform, load) processes involve extracting data from one or more sources, transforming the data into a format that can be used by the target system, and loading the data into the target system.
ELT (extract, load, transform) processes involve extracting data from one or more sources, loading the data into the target system, and then transforming the data into the desired format. ETL processes are useful for data that needs cleaning in order to be used by the target system. On the other hand, ELT processes are useful when the target system can handle the data in its raw form, so ELT processes tend to be faster than ETL processes.

Data formats for storage include text files, CSV files, JSON files, and XML files. Data can also be stored in binary formats, such as Parquet and Avro.

Data visualization is the process of creating visual representations of data. These can be used to examine data, find patterns, and make decisions. They are most often used to communicate data to non-technical audiences.

Data engineering dashboards are web-based applications that allow data engineers to monitor the status of their data pipelines. These typically display the status of data pipelines, the number of errors in a pipeline, and the time it took to run a pipeline.

SQL and NoSQL databases : are two types of databases that are used to store data.

-SQL (structured query language) databases are relational databases, which means that data is stored in tables and can be queried using SQL.

-NoSQL (not only SQL) databases are non-relational databases, which means that data is stored in a format other than tables and can be queried using a variety of methods.

You would use SQL databases for structured data, such as data from a financial system, while NoSQL databases are best suited for unstructured data, such as data from social media. For semi-structured data, such as data from a weblog, you could use either SQL or NoSQL databases.

Introduction to python for data engineering.

viola kinya kithinji — Sun, 04 Sep 2022 17:08:32 +0000

What is python programming? Python is an interpreted, object-oriented, high-level programming language with dynamic semantics.

Python becoming a popular programming language globally. We commonly knew that python was common for data science, machine learning/deep learning. Surprisingly it is also used for data engineering because it easy and smooth. Using Python Pandas data frames allows data engineers to process data effectively. Additionally, using python programming for data engineering is an excellent approach to understanding the requirements of data scientists better. Python also helps data engineers to build efficient data pipelines as many data engineering tools use Python in the backend. Moreover, various tools in the market are compatible with Python and allow data engineers to integrate them into their everyday tasks by simply learning Python programming language.

Advantages of python for data engineering

1.The role of a data engineer involves working with different types of data formats. For such cases, Python is best suited. Its standard library supports easy handling of .csv files, one of the most common data file formats.

2.A data engineer is often required to use APIs to retrieve data from databases. The data in such cases is usually stored in JSON (JavaScript Object Notation) format, and Python has a library named JSON-JSON to handle such type of data.

3.The responsibility of a data engineer is not only to obtain data from different sources but also to process it. One of the most popular data process engines is Apache Spark which works with Python DataFrames and even offers an API, PySpark, to build scalable big data projects.

4.Data engineering tools use Directed Acyclic Graphs like Apache Airflow, Apache NiFi, etc. DAGs are nothing but Python codes used for specifying tasks. Thus, learning Python will help data engineers use these tools efficiently.

5.Luigi! a Python module that is widely considered a fantastic tool for data engineering.

6.Python is easy to learn and is free to use for the masses.

Python libraries used for data engineering

pandas Pandas is the Python library popular among data analysts and data scientists. It is equally useful for data engineers, who often use it for reading, writing, querying, and manipulating data. The advantage of using Pandas dataframes is they are extremely compatible with two popular data types .csv and JSON. Additionally, dataframe objects have many easy-to-use functions that data engineers can perform quick exploratory data analysis. They can also use it to fix common data problems, such as replacing null values with neighborhood averages, removing columns, etc. Thus, Pandas allows data engineers to transform it into a readable and organized form.

SciPy SciPy, as the name suggests, is a library in Python that offers various functions for quick mathematical computations. A data engineer can use this library to perform scientific calculations on their data for better analysis.

Beautifulsoup This is a well-known library used for data mining and web scraping. You will find data engineers using this to extract information from websites, dealing with JSON/HTML data formats, all for preparing their data.

Petl Petl is a Python package for extracting, modifying, and loading tabular data. Data engineers use this library for building ETL (Extract, Transform, and Load) pipelines.

Great Expectation While Pandas is an essential library for analyzing data; there is even a better method to draw relevant conclusions from your data. And that method is to use the Great expectations library. It makes it easy for data engineers to clean data equally and allows them to specify their expectations simply. The library takes care of the backend logic, and it does not matter whether your data belongs to a database or is stored in a dataframe. Additionally, it makes it convenient for data engineers to add production-grade validation to a given data.

So how do you learn python for data engineering
Inorder to grasp the concept well and understand you have to work on real world python projects for data engineers.

Data ingestion - refers to collecting data from the database for immediate use. A data engineer needs to learn various tools like SQL, Python, etc., to know how to connect to a database and retrieve data.

Data Manipulation - A data engineer deals with data of both types, structured and unstructured. Once they have sourced data from the warehouse, the next step is to implement mathematical operations on it for cleaning.

Data surfacing -Data Surfacing involves building insightful dashboards to help businesses make better and quicker decisions. As a data engineer is the one who prepares the input data for such dashboards, it will be beneficial for them if they know how such dashboards are built.

Data acquisition - Not always a business is aware of how to identify sources of data. This is where a data engineer comes into the picture, as he is expected to identify the sources, for example, obtaining a website's log data using APIs.

** Data pipeline** - All the steps that a data engineer performs are eventually automated with the help of data pipelines. Depending on the organization's requirements, these pipelines can be of the type ETL/ELT, depending on the organization needs.

Benefits of python

1.Reduces development time.
2.Object oriented language.
3.No compiling.
4.Supports dynamic data type.
5.Reduce code length.
6.Easy to use learn and use as developers.
7.Easy to understand codes.
8.Easy to do team projects.
9.Easy to extend to other languages.
10.It's free(open source).

What is most necessary with Python to become a data engineer?
To become a data engineer with Python, the most necessary part is to explore as many Big Data projects and tools as possible.

Conculsion
Currently data engineering is in demand and not flooded another advantage is that being a data engineer allows you to interact with Big data projects and tools more than data scientists and analysts do. Consistency is key... And you can be anyone you want to be as long as you have a growth mindset.

DATA ENGINEERING 101:INTRODUCTION TO DATA ENGINNERING.

viola kinya kithinji — Wed, 24 Aug 2022 14:50:00 +0000

When you hear about data engineering what comes in your mind? According to me its basically the act of coming up with systems to facilitate the gathering and making use data. The data collected is used to enable subsequent analysis and data science which mostly work hand in hand with machine learning.

what do data engineers do They develop and maintain the architecture used in various data science projects. They are responsible for ensuring that the flow of data between servers and applications is uninterrupted.

This question comes up, now that i know what data engineering is what do i need to become one?

1. coding- In data engineering coding is like food lol! you cannot do without it. It's an highly valued skill for data engineers. Examples of programming language are: python, Ruby, mat lab and Golang

2. Data warehousing - we now know that data engineers are responsible for storing and analyzing a huge amount of data. So they need to be familiar with data warehousing solutions like panopy/Redshift, which are crucial in data engineering role.

3. Knowledge of operating systems - Being an engineer you need to have a better understanding of operating systems like Linux, Windows, macOS and UNIX

4. Database systems - As an engineer you need to have a deep understanding of database management. Since Structures Query language (SQL) is the most widely used solution, Having a deep understanding of it is crucial.
5. Data analysis - Most employers expect you to have a strong understanding of analytics software's to be precise Apache Hadoop-based solutions like MapReduce, Hive and HBase.

6. Critical thinking skills - Are you a critical thinker? You need to evaluate issues and develop solutions that are both creative and effective. Why? Most of the time you will be required develop a solution that doesn't exist. Critical thinking is key.

7. Basic understanding of machine learning - ML is mostly used by data scientists, Basic understanding of ML will help you in building your knowledge of data modelling and statistical analysis to create solutions that are usable to your peers and by so doing you become outstanding.
8. Communication skills - Let go of the introverted mentality and network, you need to network in order to learn and share ideas and suggestions with people around you.

Along the way am sure you seen Machine learning, Data scientists and Data analysts and you're wondering who are these people?
well!

Data scientists Analyses and interprets complex digital data such as the usage of a website, especially in order to assist a business in its decision making.

Data analyst Gathers and interpret data in order to solve a specific problem.

Another question comes up is data scientist better than data engineer?

unfortunately no, Data scientists can interpret data only after receiving in it's appropriate format while a data engineer gets the data to the data scientists so data scientists need to be equipped with the necessary skills to become data engineers. As of 2022 Data engineers are more in demand than data scientists.

Parting shot
You can be anything you want to be. Don't say there is no time just Begin...Take action!
Happy learning!!

INTRODUCTION TO JAVASCRIPT.

viola kinya kithinji — Mon, 11 Jul 2022 19:03:08 +0000

What is Javascript JavaScript is a cross-platform, object-oriented scripting language used to make webpages interactive (e.g., having complex animations, clickable buttons, popup menus, etc.). There are also more advanced server side versions of JavaScript such as Node.js, which allow you to add more functionality to a website than downloading files (such as real-time collaboration between multiple computers). Inside a host environment (for example, a web browser), JavaScript can be connected to the objects of its environment to provide programmatic control over them. It's standard libraries of objects are :Arrays, Date and math, it also have a core set of language elements such as: operators, control structures and structures. Core JavaScript can be extended for a variety of purposes by supplementing it with additional objects; for example:

1:Client-side JavaScript- extends the core language by supplying objects to control a browser and its Document Object Model (DOM). For example, client-side extensions allow an application to place elements on an HTML form and respond to user events such as mouse clicks, form input, and page navigation.
2:Server-side JavaScript- extends the core language by supplying objects relevant to running JavaScript on a server. For example, server-side extensions allow an application to communicate with a database, provide continuity of information from one invocation to another of the application, or perform file manipulations on a server.
This means that in the browser, JavaScript can change the way the webpage (DOM) looks. And, likewise, Node.js JavaScript on the server can respond to custom requests from code written in the browser.

Getting started with Javascript
Getting started with JavaScript is easy: all you need is a modern Web browser. This guide includes some JavaScript features which are only currently available in the latest versions of Firefox, so using the most recent version of Firefox is recommended.

The Web Console tool built into Firefox is useful for experimenting with JavaScript; you can use it in two modes: single-line input mode, and multi-line input mode.

Single-line input in the Web Console

The Web Console shows you information about the currently loaded Web page, and also includes a JavaScript interpreter that you can use to execute JavaScript expressions in the current page.

To open the Web Console (Ctrl+Shift+I on Windows and Linux or Cmd-Option-K on Mac), open the Tools menu in Firefox, and select "Developer ▶ Web Console".

The Web Console appears at the bottom of the browser window. Along the bottom of the console is an input line that you can use to enter JavaScript, and the output appears in the panel above:

The console works the exact same way as eval: the last expression entered is returned. For the sake of simplicity, it can be imagined that every time something is entered into the console, it is actually surrounded by console.log around eval, like so:

console.log(eval('3 + 5'))

Multi-line input in the Web Console

The single-line input mode of the Web Console is great for quick testing of JavaScript expressions, but although you can execute multiple lines, it's not very convenient for that. For more complex JavaScript, you can use the multi-line input mode.

Hello world
To get started with writing JavaScript, open the Web Console in multi-line mode, and write your first "Hello world" JavaScript code:

(function(){
  "use strict";
  /* Start of your code */
  function greetMe(yourName) {
    alert('Hello ' + yourName);
  }

  greetMe('World');
  /* End of your code */
})();

Press Cmd+Enter or Ctrl+Enter (or click the Run button) to watch it unfold in your browser!
Remember to always include the (function(){"use strict"; before your code, and add })(); to the end of your code. The strict mode and IIFE articles explain what those do, but for now they can be thought of as doing the following:

Prevent semantics in JavaScript that trip up beginners.
Prevent code snippets executed in the console from interacting with one-another (e.g., having something created in one console execution being used for a different console execution).

VARIABLES
Are containers that store values. You start by declaring a variable with the let keyword, followed by the name you give to the variable:

let myVariable;

A semicolon at the end of line, indicates end of a statement. Its crucial when you want to separate when you want to separate statement on a single-line. There are rules on when you will use and not use semicolons.
You can name your variables anything but there are restrictions. JavaScript is case sensitive you have to be careful when it comes to naming and calling your variables. You can give your variable a value as following:

myVariable ='Nick';

You can do both these operations on a same line as follow:

let myVariable = 'Nick';

You get the value by calling the variable as follows:

myVariable

you can change the value of the variable after assigning by:

let myVariable ='Nick'
myVariable = 'Ndolo'

NOTE
Variables may hold values with different data types: Example

Strings -This is a sequence of text known as a string. To signify that the value is a string, enclose it in single quote Eg let myVariable = 'Nick';
Numbers -This is a number. Numbers don't have quotes around them. Eg let myVariable = 10;

Boolean -This is a True/False value. The words true and false are special keywords that don't need quote marks.Eg let myVariable = true;

Arrays -This is a structure that allows you to store multiple values in a single reference.Eg let myVariable = [1,'Nick','Ndolo',10];
Refer to each member of the array like this:
myVariable[0], myVariable[1], etc.

Object -This can be anything. Everything in JavaScript is an object and can be stored in a variable. Keep this in mind as you learn.Eg let myVariable = document.querySelector('h1');
All of the above examples too.

      **Comments**

Comments are snippets of text that can be added along with code. The browser ignores text marked as comments. You can write comments in JavaScript just as you can in CSS:

If your comment contains no line breaks, it's an option to put it behind two slashes like this:

/*
Everything in between is a comment
*/

If your comment contains no line breaks, it's an option to put it behind two slashes like this:

This is a comment

Operators
An operator is a mathematical symbol that produces a result based on two values (or variables). In the following table, you can see some of the simplest operators, along with some examples to try in the JavaScript console.

Addition(+)
Add two numbers together or combine two strings.Eg 'hello'+ 'world'

Subtraction, Multiplication and Division(-, *, /)
These do what you'd expect them to do in basic math. Eg 9 - 3;
8 * 2; // multiply in JS is an asterisk
9 / 3;

Assignment(=)
As you've seen already: this assigns a value to a variable. Eg
let myVariable = 'Nick';

Strictly Equal(===)
This performs a test to see if two values are equal. It returns a true/false (Boolean) result. Eg let myVariable = 3;
myVariable === 4;

Not, Does not-Equal
This returns the logically opposite value of what it precedes. It turns a true into a false, etc.. When it is used alongside the Equality operator, the negation operator tests whether two values are not equal. Eg For "Not", the basic expression is true, but the comparison returns false because we negate it:

let myVariable = 3;
!(myVariable === 3);

"Does-not-equal" gives basically the same result with different syntax. Here we are testing "is myVariable NOT equal to 3". This returns false because myVariable IS equal to 3:

let myVariable = 3;
myVariable !== 3;

Note
Mixing data types can lead to some strange results when performing calculations. Be careful that you are referring to your variables correctly, and getting the results you expect. For example, enter '35' + '25' into your console. Why don't you get the result you expected? Because the quote marks turn the numbers into strings, so you've ended up concatenating strings rather than adding numbers. If you enter 35 + 25 you'll get the total of the two numbers.

Conditionals
Conditionals are code structures used to test if an expression returns true or not. A very common form of conditionals is the if ... else statement. For example:

let iceCream = 'chocolate';
if(iceCream === 'chocolate') {
  alert('Yay, I love chocolate ice cream!');
} else {
  alert('Awwww, but chocolate is my favorite...');
}

The expression inside the if( ... ) is the test. This uses the strict equality operator (as described above) to compare the variable iceCream with the string chocolate to see if the two are equal. If this comparison returns true, the first block of code runs. If the comparison is not true, the second block of code—after the else statement—runs instead.

Functions
are a way of packaging functionality that you wish to reuse. It's possible to define a body of code as a function that executes when you call the function name in your code. This is a good alternative to repeatedly writing the same code. You have already seen some uses of functions. For example

let myVariable = document.querySelector('h1');

alert('hello!');

These functions, document.querySelector and alert, are built into the browser.

If you see something which looks like a variable name, but it's followed by parentheses— () —it is likely a function. Functions often take arguments: bits of data they need to do their job. Arguments go inside the parentheses, separated by commas if there is more than one argument.

For example, the alert() function makes a pop-up box appear inside the browser window, but we need to give it a string as an argument to tell the function what message to display.

You can also define your own functions. In the next example, we create a simple function which takes two numbers as arguments and multiplies them:

function multiply(num1,num2) {
  let result = num1 * num2;
  return result;
}

Try running this in the console; then test with several arguments. For example:

multiply(4, 7);
multiply(20, 20);
multiply(0.5,3);

Note The return statement tells the browser to return the result variable out of the function so it is available to use. This is necessary because variables defined inside functions are only available inside those functions. This is called variable scoping. (Read more about variable scoping.)

Events
Real interactivity on a website requires event handlers. These are code structures that listen for activity in the browser, and run code in response. The most obvious example is handling the click event, which is fired by the browser when you click on something with your mouse. To demonstrate this, enter the following into your console, then click on the current webpage:

document.querySelector('html').addEventListener('click', function() {
  alert('Ouch! Stop poking me!');
});

There are many ways to attach an event handler to an element. Here we select the element. We then call its addEventListener() function, passing in the name of the event to listen to ('click') and a function to run when the event happens.

Note that

document.querySelector('html').addEventListener('click', function() {
  alert('Ouch! Stop poking me!');
});

Is equivalent to

let myHTML = document.querySelector('html');
myHTML.addEventListener('click', function() {
  alert('Ouch! Stop poking me!');
});

It's just shorter.

The functions we just passed to addEventListener() here are called anonymous functions, because they don't have a name. There's an alternative way of writing anonymous functions, which we call an arrow function. An arrow function uses () => instead of function ():

Conculsion
In this article i have handled introduction to Javascript and javascript basics...Consistency is key.

DATA STRUCTURES AND ALGORITHMS 102:DEEP DIVE INTO DATA STRUCTURES AND ALGORITHMS IN PYTHON.

viola kinya kithinji — Sun, 26 Jun 2022 19:11:23 +0000

In my last article on Introduction to data structures and algorithms I walked you through what algorithms and data structures are, types, classification, properties, characteristics, operations and merits. To get more insights read the articles . here and this here.
In today's topic we will look at:
1.we will learn data structures, abstract data types and their implementation
2.Implementation of binary tree traversal technique in python,
3.Graph traversals techniques i.e. Depth first search and Breadth-first-search in python
4.Implementation of sorting Algorithms in python.
5.Enhanced analytical skill and efficiently use searching and sorting algorithms in real applications.
6.Implementation of searching algorithms in python.

Implementation of ADT(Abstract data type)

Abstract Data type (ADT) is a type (or class) for objects whose behavior is defined by a set of values and a set of operations. The definition of ADT only mentions what operations are to be performed but not how these operations will be implemented. It does not specify how data will be organized in memory and what algorithms will be used for implementing the operations. It is called “abstract” because it gives an implementation-independent view. The process of providing only the essentials and hiding the details is known as abstraction.

user of data type does not need to know how that data type is implemented, for example, we have been using Primitive values like int, float, char data types only with the knowledge that these data type can operate and be performed on without any idea of how they are implemented. So a user only needs to know what a data type can do, but not how it will be implemented. Think of ADT as a black box which hides the inner structure and design of the data type. Now we’ll define three ADTs namely List ADT, Stack ADT, Queue ADT.

List ADT
The data is generally stored in key sequence in a list which has a head structure consisting of count, pointers and address of compare function needed to compare the data in the list.
The data node contains the pointer to a data structure and a self-referential pointer which points to the next node in the list.
The List ADT Functions is given below:
get() – Return an element from the list at any given position.
insert() – Insert an element at any position of the list.
remove() – Remove the first occurrence of any element from a non-empty list.
removeAt() – Remove the element at a specified location from a non-empty list.
replace() – Replace an element at any position by another element.
size() – Return the number of elements in the list.
isEmpty() – Return true if the list is empty, otherwise return false.
isFull() – Return true if the list is full, otherwise return false.
Stack ADT
In Stack ADT Implementation instead of data being stored in each node, the pointer to data is stored.
The program allocates memory for the data and address is passed to the stack ADT.
The head node and the data nodes are encapsulated in the ADT. The calling function can only see the pointer to the stack.
The stack head structure also contains a pointer to top and count of number of entries currently in stack.
push() – Insert an element at one end of the stack called top.
pop() – Remove and return the element at the top of the stack, if it is not empty.
peek() – Return the element at the top of the stack without removing it, if the stack is not empty.
size() – Return the number of elements in the stack.
isEmpty() – Return true if the stack is empty, otherwise return false.
isFull() – Return true if the stack is full, otherwise return false.
Queue ADT
The queue abstract data type (ADT) follows the basic design of the stack abstract data type.
Each node contains a void pointer to the data and the link pointer to the next element in the queue. The program’s responsibility is to allocate memory for storing the data.
enqueue() – Insert an element at the end of the queue.
dequeue() – Remove and return the first element of the queue, if the queue is not empty.
peek() – Return the element of the queue without removing it, if the queue is not empty.
size() – Return the number of elements in the queue.
isEmpty() – Return true if the queue is empty, otherwise return false.
isFull() – Return true if the queue is full, otherwise return false.
Features of ADT:

Abstraction: The user does not need to know the implementation of the data structure.
Better Conceptualization: ADT gives us a better conceptualization of the real world.
Robust: The program is robust and has the ability to catch errors.

implementation of searching Algorithms

Search algorithm is the set of procedures used to locate the specific data from the collection of data. The searching algorithm is always considered to be the fundamental procedure of computing. And hence it is always said that the difference between the fast application and slower application is often decided by the searching algorithm used by the application. We have various types of search algorithms like linear, Fibonacci, binary, exponential etc. But today we are going look at linear search algorithm and binary search algorithm in depth.

Linear search algorithm
Linear search is also known as a sequential searching algorithm to find the element within the collection of data. The algorithm begins from the first element of the list, starts checking every element until the expected element is found. If the element is not found in the list, the algorithm traverses the whole list and return “element not found”. Consider the python program below:

def LinearSearch(array, n, k):

    for j in range(0, n):

        if (array[j] == k):

            return j

    return -1


array = [1, 3, 5, 7, 9]

k = 7
n = len(array)

result = LinearSearch(array, n, k)

if(result == -1):

    print("Element not found")

else:

    print("Element found at index: ", result)


Output
Element found at index:  3

Time Complexity of Linear Search

The running time complexity of the linear search algorithm is O(n) for N number of elements in the list as the algorithm has to travel through each and every element to find the desired element.

Applications of Linear Search

Used to find the desired element from the collection of data when the dataset is small
The searching operations is less than 100 items

Binary search
Binary search is used with a similar concept, i.e to find the element from the list of elements. Binary search algorithms are fast and effective in comparison to linear search algorithms. The most important thing to note about binary search is that it works only on sorted lists of elements. If the list is not sorted, then the algorithm first sorts the elements using the sorting algorithm and then runs the binary search function to find the desired output. We have two binary search methods, recursive and iterative method

Below is an iterative method using python program

def binarySearch(arr, k, low, high):

    while low <= high:
        mid = low + (high - low)//2
        if arr[mid] == k:
            return mid
        elif arr[mid] < k:

            low = mid + 1
        else:

            high = mid - 1
    return -1

arr = [1, 3, 5, 7, 9]

k = 5
result = binarySearch(arr, k, 0, len(arr)-1)

if result != -1:

    print("Element is present at index " + str(result))

else:

    print("Not found")


Output
Element is present at index 2

Below is recursive method using a python program

def BinarySearch(arr, k, low, high):

    if high >= low:

        mid = low + (high - low)//2

        if arr[mid] == k:
            return mid

        elif arr[mid] > k:
            return BinarySearch(arr, k, low, mid-1)

        else:
            return BinarySearch(arr, k, mid + 1, high)

    else:
        return -1


arr = [1, 3, 5, 7, 9]
k = 5

result = BinarySearch(arr, k, 0, len(arr)-1)

if result != -1:
    print("Element is present at index " + str(result))
else:
    print("Not found")


Output
Element is present at index 2

Time complexity of Binary Search

The running time complexity for binary search is different for each scenario. The best-case time complexity is O(1) which means the element is located at the mid-pointer. The Average and Worst-Case time complexity is O(log n) which means the element to be found is located either on the left side or on the right side of the mid pointer. Here, n indicates the number of elements in the list.

The space complexity of the binary search algorithm is O(1).

Applications of Binary Search

The binary search algorithm is used in the libraries of Java, C++, etc.
It is used in another additional program like finding the smallest element or largest element in the array
It is used to implement a dictionary

Implementation of binary traversal tree in python
Binary tree is special type of hierarchal data structures defined using nodes. Basically its extended version of linked list. Its a tree data structure where each node is allowed to have maximum two children node, generally referred as Left Child and Right Child. Hashing, routing data for network traffic, data compression, and binary search trees are some of its application.

Key terminologies used
Root:- The Topmost node
Height:- Total Number of edges from root node to last(deepest) node
Leaf:- Node with no children
Depth of a Tree: The number of edges from the tree’s node to the root is.
Internal Node:- Node having at least one Child.

Type of Binary Tree

Perfect Binary Tree
A Binary Tree with all the interior node (all nodes except leaf node) have two children and all leaf node has same depth

Balanced Binary Tree
Every tree where the maximum difference between right and left subtree height is 1.

Complete Binary Tree

All binary tree where every node is completely filled with 2 or 0 node .

Degenerate Binary Tree

Every binary tree, where every internal node has only single child.

Applications of Binary Tree

-Used in 3d Video Games.
-Highly used in router for tabling purpose.
-Scheduling processes in OS.

implementation of binary tree in python

implementing the code

class BinaryTree:

    def __init__(self, value):

        self.left = None
        self.right = None
        self.value = value

    def insert(self, value):

        if self.value:
            if data < self.value:
                if self.left is None:
                    self.left = BinaryTree(value)
                else:
                    self.left.insert(value)
            elif data > self.value:
                if self.right is None:
                    self.right = BinaryTree(value)
                else:
                    self.right.insert(value)
        else:
            self.value = value

    def PrintTree(self):
        if self.left:
            self.left.PrintTree()
        print( self.data),
        if self.right:
            self.right.PrintTree()

root = BinaryTree(100)
root.insert(50)
root.insert(55)
root.insert(60)
root.insert(20)
root.insert(52)


root.PrintTree()

Traversal operation
When, we print the values of every node here we are using preorder traversal where left first left most child is printed, then root, then the right child.

Traversal:-

PreOrder Traversal
InOrder Traversal
PostOrder Traversal
PostOrder Traversal:-

Here the Left child is visited first, then the Right child node and then Root node.

The minutes changes in the PrintTree() method is following

    def PrintTree(self):

        if self.left:
            self.left.PrintTree()

        if self.right:
            self.right.PrintTree()
        print(self.value)

PreOrder Traversal

Here the Root is visited first, then left child node and then right child node.

The minutes changes in the PrintTree() method is following

    def PrintTree(self):
        print(self.value)
        if self.left:
            self.left.PrintTree()

        if self.right:
            self.right.PrintTree()

Search Operation
Searching in a Binary Tree

Searching in a binary tree is a very simple step, as we have already discussed traversing a binary tree, so we can use the traversing technique to get all the elements in a tree and and find our required element. Here we are using preorder traversal, you guys can use anyone of them.

   def PrintTree(self,element):

        if self.left:
            self.left.PrintTree()
        if(self.data==element):
            print("Found",self.data)
            return(1)
        else:
            continue
        if self.right:
            self.right.PrintTree()

Deletion operation
Deleting a element from the binary tree


def delete_Node(root, key):
    if not root: 
        return root
    if root.val > key: 
        root.left = delete_Node(root.left, key)
    elif root.val < key: 
        root.right= delete_Node(root.right, key)
    else: 
        if not root.right:
            return root.left
        if not root.left:
            return root.right
        temp_val = root.right
        mini_val = temp_val.val
        while temp_val.left:
            temp_val = temp_val.left
            mini_val = temp_val.val
        root.right = deleteNode(root.right,root.val)
    return root

Explanation

The above programs explain the procedure to delete a particular element from the given binary tree. Here root represents the root node and key represents the element that needs to be deleted or has been ordered by the user.

Graph traversals techniques
Graphs are very useful data structures in solving many important mathematical challenges.

Depth First Traversal
Also called depth first search (DFS),this algorithm traverses a graph in a depth ward motion and uses a stack to remember to get the next vertex to start a search, when a dead end occurs in any iteration. We implement DFS for a graph in python using the set data types as they provide the required functionalities to keep track of visited and unvisited nodes.

class graph:
   def __init__(self,gdict=None):
      if gdict is None:
         gdict = {}
      self.gdict = gdict
# Check for the visisted and unvisited nodes
def dfs(graph, start, visited = None):
   if visited is None:
      visited = set()
   visited.add(start)
   print(start)
   for next in graph[start] - visited:
      dfs(graph, next, visited)
   return visited

gdict = { 
   "a" : set(["b","c"]),
   "b" : set(["a", "d"]),
   "c" : set(["a", "d"]),
   "d" : set(["e"]),
   "e" : set(["a"])
}
dfs(gdict, 'a')

Breadth First Traversal
Also called breadth first search (BFS),this algorithm traverses a graph breadth ward motion and uses a queue to remember to get the next vertex to start a search, when a dead end occurs in any iteration. Please visit this link in our website to understand the details of BFS steps for a graph.

We implement BFS for a graph in python using queue data structure discussed earlier. When we keep visiting the adjacent unvisited nodes and keep adding it to the queue. Then we start dequeue only the node which is left with no unvisited nodes. We stop the program when there is no next adjacent node to be visited.

Example

import collections
class graph:
   def __init__(self,gdict=None):
      if gdict is None:
         gdict = {}
      self.gdict = gdict
def bfs(graph, startnode):
# Track the visited and unvisited nodes using queue
   seen, queue = set([startnode]), collections.deque([startnode])
   while queue:
      vertex = queue.popleft()
      marked(vertex)
      for node in graph[vertex]:
         if node not in seen:
            seen.add(node)
            queue.append(node)

def marked(n):
   print(n)

# The graph dictionary
gdict = { 
   "a" : set(["b","c"]),
   "b" : set(["a", "d"]),
   "c" : set(["a", "d"]),
   "d" : set(["e"]),
   "e" : set(["a"])
}
bfs(gdict, "a")

Implementation of sorting Algorithms
Sorting is defined as an arrangement of data in a certain order. Sorting techniques are used to arrange data(mostly numerical) in an ascending or descending order.

Some of the real-life examples of sorting are:

1.Telephone Directory: It is a book that contains telephone numbers and addresses of people in alphabetical order.
2.Dictionary: It is a huge collection of words along with their meanings in alphabetical order.
3.Contact List: It is a list of contact numbers of people in alphabetical order on a mobile phone.The different types of order are:

Increasing Order: A set of values are said to be increasing order when every successive element is greater than its previous element. For example: 1, 2, 3, 4, 5. Here, the given sequence is in increasing order.
Decreasing Order: A set of values are said to be in increasing order when the successive element is always less than the previous one. For Example: 5, 4, 3, 2, 1. Here the given sequence is in decreasing order.
Non-Increasing Order: A set of values are said to be in non-increasing order if every ith element present in the sequence is greater than or equal to its (i-1)th element. This order occurs whenever there are numbers that are being repeated. For Example: 1, 2, 2, 3, 4, 5. Here 2 repeated two times.
Non-Decreasing Order: A set of values are said to be in non-decreasing order if every ith element present in the sequence is less than or equal to its (i-1)th element. This order occurs whenever there are numbers that are being repeated. For Example: 5, 4, 3, 2, 2, 1. Here 2 repeated two times.

Sorting Techniques
The different implementations of sorting techniques in Python are:

Bubble Sort
Selection Sort
Insertion Sort

Bubble Sort
Bubble Sort is a simple sorting algorithm. This sorting algorithm repeatedly compares two adjacent elements and swaps them if they are in the wrong order. It is also known as the sinking sort. It has a time complexity of O(n2) in the average and worst cases scenarios and O(n) in the best-case scenario. Bubble sort can be visualized as a queue where people arrange themselves by swapping with each other so that they all can stand in ascending order of their heights. Or in other words, we compare two adjacent elements and see if their order is wrong, if the order is wrong we swap them. (i.e arr[i] > arr[j] for 1 <= i < j <= s; where s is the size of the array, if array is to be in ascending order, and vice-versa).

Example

Here we sort the following sequence using bubble sort

Sequence: 2, 23, 10, 1

First Iteration

(2, 23, 10, 1) –> (2, 23, 10, 1), Here the first 2 elements are compared and remain the same because they are already in ascending order.

(2, 23, 10, 1) –> (2, 10, 23, 1), Here 2nd and 3rd elements are compared and swapped(10 is less than 23) according to ascending order.

(2, 10, 23, 1) –> (2, 10, 1, 23), Here 3rd and 4th elements are compared and swapped(1 is less than 23) according to ascending order

At the end of the first iteration, the largest element is at the rightmost position which is sorted correctly.

Second Iteration

(2, 10, 1, 23) –> (2, 10, 1, 23), Here again, the first 2 elements are compared and remain the same because they are already in ascending order.

(2, 10, 1, 23) –> (2, 1, 10, 23), Here 2nd and 3rd elements are compared and swapped(1 is less than 10) in ascending order.

At the end of the second iteration, the second largest element is at the adjacent position to the largest element.

Third Iteration

(2, 1, 10, 23) –> (1, 2, 10, 23), Here the first 2 elements are compared and swap according to ascending order.

The remaining elements are already sorted in the first and second Iterations. After the three iterations, the given array is sorted in ascending order. So the final result is 1, 2, 10, 23.

** Implementation of bubble sort**

# Python3 program for Bubble Sort Algorithm Implementation
def bubbleSort(arr):

    n = len(arr)

    # For loop to traverse through all
    # element in an array
    for i in range(n):
        for j in range(0, n - i - 1):

            # Range of the array is from 0 to n-i-1
            # Swap the elements if the element found
            #is greater than the adjacent element
            if arr[j] > arr[j + 1]:
                arr[j], arr[j + 1] = arr[j + 1], arr[j]

# Driver code

# Example to test the above code
arr = [ 2, 1, 10, 23 ]

bubbleSort(arr)

print("Sorted array is:")
for i in range(len(arr)):
    print("%d" % arr[i])

Selection Sort
This sorting technique repeatedly finds the minimum element and sort it in order. Bubble Sort does not occupy any extra memory space. During the execution of this algorithm, two subarrays are maintained, the subarray which is already sorted, and the remaining subarray which is unsorted. During the execution of Selection Sort for every iteration, the minimum element of the unsorted subarray is arranged in the sorted subarray. Selection Sort is a more efficient algorithm than bubble sort. Sort has a Time-Complexity of O(n2) in the average, worst, and in the best cases.

Example

Here we sort the following sequence using the selection sort

Sequence: 7, 2, 1, 6

(7, 2, 1, 6) –> (1, 7, 2, 6), In the first traverse it finds the minimum element(i.e., 1) and it is placed at 1st position.

(1, 7, 2, 6) –> (1, 2, 7, 6), In the second traverse it finds the 2nd minimum element(i.e., 2) and it is placed at 2nd position.

(1, 2, 7, 6) –> (1, 2, 6, 7), In the third traverse it finds the next minimum element(i.e., 6) and it is placed at 3rd position.

After the above iterations, the final array is in sorted order, i.e., 1, 2, 6, 7.

Implementation of Selection Sort

# Selection Sort algorithm in Python
def selectionSort(array, size):

    for s in range(size):
        min_idx = s

        for i in range(s + 1, size):

            # For sorting in descending order
            # for minimum element in each loop
            if array[i] < array[min_idx]:
                min_idx = i

        # Arranging min at the correct position
        (array[s], array[min_idx]) = (array[min_idx], array[s])

# Driver code
data = [ 7, 2, 1, 6 ]
size = len(data)
selectionSort(data, size)

print('Sorted Array in Ascending Order is :')
print(data)

Insertion Sort
This sorting algorithm maintains a sub-array that is always sorted. Values from the unsorted part of the array are placed at the correct position in the sorted part. It is more efficient in practice than other algorithms such as selection sort or bubble sort. Insertion Sort has a Time-Complexity of O(n2) in the average and worst case, and O(n) in the best case.

Example

Here we sort the following sequence using the insertion sort

Sequence: 7, 2, 1, 6

(7, 2, 1, 6) –> (2, 7, 1, 6), In the first iteration, the first 2 elements are compared, here 2 is less than 7 so insert 2 before 7.

(2, 7, 1, 6) –> (2, 1, 7, 6), In the second iteration the 2nd and 3rd elements are compared, here 1 is less than 7 so insert 1 before 7.

(2, 1, 7, 6) –> (1, 2, 7, 6), After the second iteration (1, 7) elements are not in ascending order so first these two elements are arranged. So, insert 1 before 2.

(1, 2, 7, 6) –> (1, 2, 6, 7), During this iteration the last 2 elements are compared and swapped after all the previous elements are swapped.

Implementation of Insertion Sort

# Creating a function for insertion sort algorithm
def insertion_sort(list1): 

        # Outer loop to traverse on len(list1) 
        for i in range(1, len(list1)): 

            a = list1[i] 

            # Move elements of list1[0 to i-1],
            # which are greater to one position
            # ahead of their current position 
            j = i - 1 

            while j >= 0 and a < list1[j]: 
                list1[j + 1] = list1[j] 
                j -= 1 

            list1[j + 1] = a 

        return list1 

# Driver code
list1 = [ 7, 2, 1, 6 ] 
print("The unsorted list is:", list1) 
print("The sorted new list is:", insertion_sort(list1))

Application of search and sorting in real applications

Searching is the process of locating a given object or data within a large set of data. Sorting, on the other hand, is the arrangement of data in a specific required order. The common application of these algorithms is in databases and other computer applications.

Conculsion
Have handled data structure and algorithms in depth. Am open to additions, corrections and questions. I believe we learn through each other. Happy coding

Introduction to Data structures and Algorithms.

viola kinya kithinji — Sun, 19 Jun 2022 19:50:10 +0000

Have heard a lot techies talk about data structures and algorithms, Honestly speaking data structures and algorithms is a challenging topic. It needs consistency ,resilience and a lot of practice to crack it. Let's dive in:

1: What is an algorithms A finite sequence of steps for accomplishing some computational task.
2: what is a data structure Is an arrangement of data in memory locations to represent values of the carrier set of an abstract data type.

Properties of an algorithm

-The inputs must be specified, that is algorithm should have some inputs.
-The outputs must be specified, That is atleast one output should be returned by the algorithm after the completion of the specific task based on the given inputs.
-Definiteness- Every statement of the algorithm should be unambiguous.
-Effectiveness- Writing an algorithm is a prior process of actual implementation of the algorithm. So I need to analyze the algorithm in a finite amount of time with a pen and paper to judge the performance for giving the final version of the algorithm.
-finiteness- No infinite loop should be allowed in an algorithm.

Characteristics of an algorithm

-It's a step by step procedure for solving any problem an algorithms recognizes.
-Being finite set of instructions, if followed it accomplishes a particular task.
-Being a sequence of computational steps it transforms inputs to valuable or required output.

Types of an algorithm

-recursive algorithm- Its the most interesting because it calls itself repeatedly until the problem is solved.
-Divide and conquer algorithm- Its divides the algorithm into two parts. The first parts divide the problem on hand into smaller subproblems of the same type. The second part the smaller problems are solved the added together to give the final solution.
-Dynamic programming algorithms- Solves complex problems by breaking them into multiple simple sub problems and the its solves each of them once and then stores them for future reference.
-Greedy algorithm- Used for solving optimization problems. Its not guaranteed to be able to find an optimal solution. Its components are:

1: The first one is a candidate set from which we try to find a solution.
2: A selection function that helps choose the best possible candidate.
3: A feasibility function that helps in deciding if the candidate can be used to find a solution.
4: Solution function that tells when we have found a solution to the problem.

-Brute force Algorithms- Its the simplest algorithms in the concept.it blindly iterates all possible solution to search one or more than one solution that may solve a function.

Data structures
A data structure is not language specific you can use any programming language. its a set of algorithm that we can use to structure the data in the memory. By so doing 'n' number of algorithm are proposed and now all these algorithms are termed as "Abstract data types" ADT gives us the blueprint while data structure provides the implementation part.

Types of data structures

-Primitive data structures- These are primitive data types. i.e. char, int, float, double and pointer are primitive data structures that can hold a single value.
-Non primitive data structures- Are divided into two types:
1. Linear data structures- its the arrangement of data in a sequential manner examples are: Arrays, linked lists, Stacks and Queues. A element is connected to only one another element in a linear form.
2. Non-linear data structures- This one element is connected to the 'n' number of elements. example: Trees and Graphs.

Data structures can also be classified as:

Static data structure- Here the size is allocated at the compile time, So the maximum size is fixed.
Dynamic data structure- Here the size is allocated at the run time, so the maximum size is flexible.

Major operations on data structures

Searching- we can search any element in a data structure.
Sorting- We can sort the elements of a data structure in an ascending or descending order.
Insertion- we can insert new elements in a data structure.
Updation- we can replace the element with another element.
Deletion- we can delete an element from the data structure.

Merits of a data structure

Efficiency- when the chosen data structure for implementing a certain ADT is good. Then its efficient in time and space.
Reusability-Multiple client programs can use the data structure.
Abstraction- ADT specifies a data structure that provides a level of abstraction, This means a client cannot see the internal working of the data structure, so it doesn't have to stress about the implementation part.

Conclusion

I bet by the end of the article you will have a glimpse of what data structures and algorithms are, implementation, types and how there are used. Cheers am open to comments and additions.

The Ultimate Guide to Getting Started in Data Science.

viola kinya kithinji — Sun, 03 Apr 2022 21:13:01 +0000

Data Science

Is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from noisy, structured and unstructured data, and apply knowledge and actionable insights from data across a broad range of application domains.

I will handle this topic using the following steps:

-Downloading and installing python and the text editors
-Learn the data science process.
-Visualization
-Attempt your first data science project.
-Databases.
-Statistical learning
-Data structures and algorithms.
-Machine learning
-Attempt an advanced project.
-Learn more python.
-Deep learning.

Downloading and installing python and text editors
As a data scientist you will need first to be conversant with excel this is because of data cleaning and sorting. Then go to your browser and download python, run and install. we have different environments to write your python code example pycharm, visual studio and python IDE. We also have online platform like the goggle colab and jupyter notebook.

Learn the data science process
1st revolution - mechanization led by the steam engine
2nd revolution - mass production driven by electricity and oil based power.
3rd revolution - Automated production supported by electronics and information technologies.
4th revolution - information technologies, internet of things, artificial intelligence, Big data, cloud, Cyber physical systems.

Data scientist - also known as data managers and statisticians. data scientist takes data projects from end to end. They can help store large amount of data, create predictive modelling processes and present the findings.
Data engineers - also known as database engineer and data architect, they use computer science to process large datasets. They focus on coding, cleaning up datasets and implementing requests that come from data scientists.
Data analysts - they help people from across the company understand specific queries with charts.

Visualization

The graphical representation of information and data.by using visual elements like charts, graphs and maps, data visualization tools provides an accessible way to see and understand trends, outliers and patterns in data. They use libraries like pandas, seaborn and numpy and matplotlib.

First data project

my first project was on data visualization I downloaded a mental health dataset from data world, sorted the data and cleaned. For data analysis and visualization I used jupyter notebook and google colab and python libraries like pandas and matplotlib. You can look for sample projects in areas that you are well polished and work with them. Or also hacker ranks.

Databases

We have relational and non relational database. Non Relational databases provide a mechanism for storage and retrieval of data that is modeled in means other than the tabular relation used in relational database.
relational database - structured to recognize relations between stored items of information.

statistical learning

In python we have a built in python library for descriptive statistics.it can be used if your data datasets are not too large or if you you cant rely on importing other libraries. Numpy is a third party library for numerical computing, optimized for working with single and multi-dimensional arrays. Here you choose and get started with python statistics libraries, calculating descriptive statistics, working with 2D data, visualizing Data.

Data structures and Algorithms

Data structures - are containers that organize and group data according to type, They differ based on mutability and order. Mutability refers to the ability to change an object after creation. we have two types built in(lists, tuples, sets and dictionaries) and user defined data structures(Stacks using arrays, The condition check, Queue using arrays).
Data algorithms- Is a sequence of steps executed by a computer that takes an input and transforms it into a target output.

Machine learning

This is making the computer learn from studying data and statistics. Machine learning is a step into the direction of artificial intelligence(AI).ML is a program that analyzes data and learns to predict the outcome.it is used in installing python, loading, summarizing and visualizing datasets, evaluates some algorithms and make some predictions.

Advanced project

If you are conversant with python libraries and some of the external libraries you should now know how to install external libraries and work with them. Below are some source code to use at an advance project:
-Digital clock GUI
-Get desktop notification with python.
-use your phone camera for computer vision.
-music player GUI.
-image converter GUI
-weight converter GUI.

Learn more python

Python evolves every time so you have to keep coding and discovering new techniques in python. One thing about coding languages they need consistency and committed. Don't be too comfortable in one area explore the whole language understanding every bit.

Deep Learning

Is a subset of machine learning that trains a computer to perform human like tasks, image identification and prediction making. It improves the ability to classify, recognize, detect and describe using data. It's role is to process both unlabeled and unstructured data. This learning method also creates more complex statistical models. With each new piece of data, the model becomes more complex.

CONCULSION

That was a brief description of the ultimate guide to get started in data science, in your free time go through the steps deeply and make sure you understand what in happens in each step precisely. Your mindset and consistency is key.

INTRODUCTION TO SEABORNS.

viola kinya kithinji — Sun, 03 Apr 2022 18:22:25 +0000

Seaborn is a library for making statistical graphics in Python. It builds on top of matplotlib and integrates closely with pandas data structures.
Seaborn helps you explore and understand your data. Its plotting functions operate on dataframes and arrays containing whole datasets and internally perform the necessary semantic mapping and statistical aggregation to produce informative plots. Its dataset-oriented, declarative API lets you focus on what the different elements of your plots mean, rather than on the details of how to draw them.
One has to be familiar with Numpy and Matplotlib and Pandas to learn about Seaborn.

Seaborn offers the following functionalities:
-Dataset oriented API to determine the relationship between variables.
-Automatic estimation and plotting of linear regression plots.
-It supports high-level abstractions for multi-plot grids.
-Visualizing univariate and bivariate distribution.

Using Seaborn we can plot wide varieties of plots like:

-Heat maps-The heatmap represents the data in a 2-dimensional form. The ultimate goal of the heatmap is to show the summary of information in a colored graph. It utilizes the concept of using colors and color intensities to visualize a range of values.

-Pair Plots -are used when we want to see the relationship pattern among more than 3 different numeric variables. For example, let’s say we want to see how a company’s sales are affected by three different factors, in that case, pair plots will be very helpful.

-pie chart and bar charts.Pie Chart is generally used to analyze the data on how a numeric variable changes across different categories.

-scatter plots-Scatter Plot is used when we want to plot the relationship between any two numeric columns from a dataset. These plots are the most powerful visualization tools that are being used in the field of machine learning.
-distributions plots-We can compare the distribution plot in Seaborn to histograms in Matplotlib. They both offer pretty similar functionalities. here use sns.distplot.

Regression plots-Regression plots create a regression line between 2 numerical parameters in the jointplot(scatterplot) and help to visualize their linear relationships.

How to learn Seaborn
1.Do a thorough research on seaborns, the basics, it's importance in data visualization and how it relates with other libraries and why they do.
2.Try out plotting using seaborn this way you will grasp more lets agree on this one, practice makes perfect. So first start by looking for a dataset.
3.Try reading some articles about seaborns and looking for a worked out problem to master the techniques lastly make use of galleries and YouTube.

Installation of seaborn
I find Jupyter notebook efficient in doing data visualization hope you like it too. for the ones using local machine you run this command on your terminal. If you are using google colab or jupyter notebooks worry not about installation.

pip install seaborn

Importing the library

import seaborn as sns

Loading dataset
for demonstration am using a dataset in my local machine and we will see how seaborn helps us to visualize the distribution of each feature.

This shows the relationship between matplotlib and seaborn.

Conclusion
Data visualization is fun to work with and you get to work with different python libraries and makes your plots and graphs more appealing. Go through seaborn attentively and make sure you practice using all those graphs and libraries. Consistency is key.

Relational and Non relational database.

viola kinya kithinji — Sun, 03 Apr 2022 15:03:19 +0000

DatabaseA collection of information organized to provide efficient retrieval.
Relational database management systems(RDBMS) SQl: PostgreSQL ,SQlite, oracle.
Non relational database management systems NOSQL: MongoDB, Redis, Aerospike, Couchbase.

CAP Theorem It comprises of relational and non-relational databases. CAP stands for consistency, Availability and partition Tolerant.
Relational database management systems There are organized and defined in structured sticker schemas.
-Normalized data is the process of storing data into columns and tables to reduce redundancy.
Reliable relational DBs are ACID compliant:
-Atomicity-modification falls on nothing rule.
-Consistency- Valid data is brought into the database.
-isolation- Requires multiple simultaneous transactions do not impact each other's execution.
-Duration-Ensures any transaction committed in the database are not lost.

Disadvantages of relational databases

-Scalability
RDMS are historically intended to be run on a single machine. This means that if the requirements of the machine are insufficient, due to data size or an increase in the frequency of access, you will have to improve the hardware in the machine, also known as vertical scaling.

This can be incredibly expensive and has a ceiling, as eventually the costs outweigh the benefits. Plus, there will potentially come a stage where you simply cannot get hardware capable of hosting the database. The only solution would be to buy a machine that supports better hardware but none of that is cheap.

-Flexibility
In relational databases, the schema is rigid. You define the columns and data types for those columns, including any restraints such as format or length. Common examples of constraints would include phone number length or minimum/maximum length for a name column.

Although this means you can interpret the data more easily and identify the relationships between tables, it also means that making changes to the structure of the data is very complex. You have to decide at the start what the data will look like, which isn’t always possible. If you want to make changes later, you have to change all the data, which involves the database being offline temporarily.

-Performance
The performance of the database is tightly linked to the complexity of the tables — the number of them, as well as the amount of data in each table. As this increases, the time taken to perform queries increases too.

Non relational databases
-information is stored in a dynamic ,non-normalized and more flexible manner.
-More designed to be distributed than relational.
-Follow the base system: basically Available, soft state, eventual consistency.
Major types of NoSQL
-Key value-stored in big hash tables pf keys and values.
-column oriented- two dimensional arrays whereby each key has one or more key/values pairs attached to it.
-Document stored-similar to key value but with structured and encoding of data.
-Graph based- are like trees with edges, nodes and properties for index-free adjacency.

Comparison between SQL and NoSQL
SQL
-Strict and structured schemas, tabular data(design).
-Light declarative language, standardized use even with subtly different syntaxes.(CRUD Queries)
-CAP,ACID compliance. Consistency enforced after each transaction.(Transactions)
NoSQL
-Schemaless, document oriented, dynamic and unstructured.(design)
-No standard interface, differs from system to system.(CRUD Queries).
-CAP,BASE systems, fault tolerant transactions for eventual consistency.
How to choose
First you will have to understand the kind of data that you have and how it is stored.
For NOSQL-if you will have a lot of unstructured data and aggregate information with nesting denormalized data.
For SQL- normalized, space efficient, but costly joins!

In summary what you need to know about relational databases/Non relational database:

Relational databases

-They work with structured data.
-Relationships in the system have constraints, which promotes a high level of data integrity.

-There are limitless indexing capabilities, which results in
faster query response times.

-They are excellent at keeping data transactions secure.

-They provide the ability to write complex SQL queries for data
analysis and reporting.

-Their models can ensure and enforce business rules at the data
layer adding a level of data integrity not found in a non-
relational database.

-They are table and row oriented.

-They Use SQL (structured query language) for shaping and manipulating data, which is very powerful.

-SQL database examples: MySql, Oracle, Sqlite, Postgres and MS-SQL. NoSQL database examples: MongoDB, BigTable, Redis, RavenDb, Cassandra, Hbase, Neo4j and CouchDb.

SQL databases are best fit for heavy duty transactional type applications.

Non relational database:

-They have the ability to store large amounts of data with little structure.

-They provide scalability and flexibility to meet changing business requirements.

-They provide schema-free or schema-on-read options.

-They have the ability to capture all types of data “Big Data” including unstructured data.

-They are document oriented.

-NoSQL or non-relational databases examples:MongoDB, Apache Cassandra, Redis, Couchbase and Apache HBase.

-They are best for Rapid Application Development. NoSQL is the best selection for flexible data storage with little to no structure limitations.

-They provide flexible data model with the ability to easily store and combine data of any structure without the need to modify a schema.

Git and Github

viola kinya kithinji — Thu, 31 Mar 2022 20:24:09 +0000

What is Git? It is a version control system that you download onto your computer. It is essential that you use Git if you want to collaborate with other developers on a coding project or work on your own project.

What is version control? The management of changes to documents, computers programs, large websites, and other collections of information.

Terms used.

-Directory-folder.
-Terminal/command line-interface for text commands.
-CLI-commandline interface.
-Cd-change directory.
-Code editor-word processor for writing codes.
-Repository-project, or the folder/place where your project is kept.
-Github-A website to host your repositories online.

Git Commands.

-clone-Bring a repo that is hosted somewhere like Github into a folder on your local machine.
add-Track your files and changes in Git.
-commit-saves your files in Git.
-push-Upload Git commit to a remote repo, like Github.
-pull-Downloads changes from a remote repo to your local machine,the opposite of push.

Let's get started.

You will first start by downloading Git here https://git-scm.com/downloads. After downloading and setting up you will create a Github account here https://github.com/. Bingo! you have done it.

How to create a repo.

From the conversation box on the image you can see some commands there you will select new repository or on the left there is an icon indicating new.

On the above picture is an indication of how it will look like.

How to clone your repo to your local machine.
[](

The picture above shows how to copy the link so as to clone. On the icon labelled "code" below is the link copy it.

Cloning.

This picture shows how to clone it on your local machine.

How to add, commit and push.

This picture shows how to track and saves my files in Git.

Git push.

.Upload Git commit to a remote repo, like Github.

Pull Request.

.A pull request (or PR) is a way to alert a repo's owners that you want to make some changes to their code. It allows them to review the code and make sure it looks good before putting your changes on the primary branch. .
Merging pull request.

You might see a big green button at the bottom that says 'Merge pull request'. Clicking this means you'll merge your changes into the primary branch. When you successfully merge the pull request make sure you delete your branch too many branches is a put off by doing so make sure yours commits were merged.

How it will look like after merging

When you're done,You can double check that your commits were merged by clicking on the 'Commits' link on the first page of your new repo.

. When you're done, you can double check that your commits were merged by clicking on the 'Commits' link on the first page of your new repo.

This image shows a list of commits in your branch.

How to update the changes on Github to the repo on my local machine.
After creating a pull request and merging pull request on your github repo a little differences comes in with the one in your local machine. In order to get the most recent changes that you or others have merged on GitHub, use the git pull origin master command on your Git bash "git pull" This shows you all the files that have changed and how they've changed.

Conculsions.
The more you use git, the more comfortable you'll... git with it. (I couldn't resist.).Github is useful because you're able to store your code online and allows you and others to work together on projects from anywhere.

Mental health

viola kinya kithinji — Fri, 04 Mar 2022 17:47:28 +0000

I don't know what drove me into writing this article but I believe through this article I will communicate to many and give hope to many.
How is your mental health right now? How are you feeling are you afraid, hopeless, anxious, suicidal.
Recently have been feeling anxious. Anxiety is that condition that's accompanied with fear and worry. from my research have found out that can be as a result of childhood trauma and also your upbringing. Anxiety makes you a slave of your thoughts you fear to try something because you're worried what will people say or what if I can't do it.

Suicide

What comes in your mind when you hear about suicide,I remember I lost a friend to suicide.Do you check on your family? family is anyone close to you.Some of us assume that this is a man he will maneuver around and come into his senses no please, let's stop giving men these superhero titles and thoughts.It does not make you less of a man if you share your problems in fact to me that's a hero.Do you know the largest population of depressed people is men and children.Men who is the most important person in your life? Wife, children, family?? Noo you're the most important person in your life work on your self growth and love.You can't love your family if don't love you.
Many are asking how are children depressed? How many of us parents fight infront of kids,Use abusive words Infront of our kids and also direct to the kids,Compare your kids and many more.are you among the group how are planning to change this inorder bring up functional generation?.
Any time you feel your mental health is at stake kindly seek help from professionals not from your religious leaders...A professional is trained to handle the cases and religious leader is trained to handle religious matters. Yes some religious leaders are trained too. what if am financially down? we have institutions that offer free service inquire around you.
Let's embark on this journey of healing and living a happy and stress free life.
Please if someone confides with you don't make them regret sharing .Be the light to the person.
A healthy mind is a happy family.

Python for everyone: Mastering python the right way.

viola kinya kithinji — Fri, 04 Mar 2022 16:20:11 +0000

python the right way
Huh the right way? Some might be wondering which is the right way. Truth be told I felt the same in the beginning and yes there is the right way. First a mindset how is your mindset like? It requires a positive mindset and a go getter. Secondly passion, are you passionate or you're doing it Because of you saw someone doing it?. Take for instance if you're not passionate and you face a challenge, wait yes there are challenges do you think you will maneuver?. And lastly the drive, what drives you to know more about python? .

What is python
Is an interpreted, interactive, object-oriented programming language. So what is python used for:

Data science: Analysis and Visualization

With a Python data visualization library, you can create a wide variety of plots and visual representations, such as:

-Lines, Bars, and Markers.
-Images, contours and fields.
-Subplots, axes and figures.
-Statistics (Box Plots, Bar Charts, and Histograms).
-Pie and polar charts.
-3D Plots.
and more!
You can add text, labels, annotations, color, shapes, collections, animations, and interactivity to your plots depending on the package or library that you choose to work with.

Libraries and Packages
Let's see some of the most popular packages and libraries to work with Python in data science:

Python for Data Analysis
-NumPy: this package is described as "the fundamental package for scientific computing with Python". According to the official website of this package, "nearly every scientist working in Python draws on the power of NumPy."
-Pandas: is "a fast, powerful, flexible source data analysis and manipulation tool."
Python for Data Visualization
-Matplotlib: is "a comprehensive library for creating static, animated, and interactive visualizations in Python."
Seaborn: is "a Python data visualization library based on matplotlib."
-ggplot2: is "a system for declaratively creating graphics, based on The Grammar of Graphics". According to its official website: "you provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details."
-Bokeh: is "an interactive visualization library for modern web browsers".
Pandas: this library has many tools for data visualization.

Machine learning

Python is an essential tool for every developer who wants to enter the fascinating area of Machine Learning. Let's see a brief introduction to Machine Learning.

What is Machine Learning?
Machine Learning is an area of Computer Science that creates systems that are able to learn on their own.

This type of system uses algorithms that are continuously improved based on input data that helps the system "learn". It learns how to respond autonomously to new scenarios by generating an appropriate output in new scenarios based on previous knowledge.

One of the most amazing things about these systems is that they are continually refined.

They are not like the programs that we typically write in a Python script where we specify every possible action that the program can take. In Machine Learning, the system is trained to "think" and make decisions based on previous knowledge.

This is why we say that machines "learn" from the data

-Neural Networks: The Building-Blocks of Machine Learning
-Neural networks are the processing units of the system. They try to simulate a real network of neurons of the brain. They virtual "neurons" receive input, learn how to process that input, and generate an output based on their previous knowledge.

This is very similar to what out brain does every single moment of every single day.

Thanks to neural networks, a Machine Learning algorithm can learn how to predict the expected output from a given input based on previous knowledge.

For example, when you see recommended videos on YouTube, those recommendations were generated by neural networks that predict what videos you might like to watch based on your previous patterns. Amazing, right?

Python and Machine Learning
you have this question in mind right?: what is the role of Python in this area? It is one of the most popular and powerful tools used to program this type of system.

One of the most popular libraries used by developers around the world to work with Python applied to Machine Learning is

-TensorFlow: It's a free open-source library developed by the Google Brain Team. This library is used for research and production at Google.
According to Jeff Dean, the lead of Google's Artificial Intelligence division:
Today it is used heavily in our speech recognition systems, in a new Google Photos product, Gmail, and Google Search. (source)
The best part is that developers all over the world can use this library to tackle real-world problems.

These are two other popular Python libraries used for Machine Learning:

-Keras – an open-source neural-network library written in Python.
-PyTorch – an open-source Machine Learning library used for developing and training neural networks.

Web development
Python is used widely by back-end developers to run servers for web applications and interacts with databases and APIs after after a client requests data from the app's front-end in the browser. It uses frameworks and libraries like Django, pyramid and flask that extends the language utility. The frameworks are use for security, scalabilty and convenience.

Computer science education
Python currently plays a key role in computer science education around the world. Let's see why.

Why Python?

Python is so widely used as a teaching tool because:

-It is easy to learn: its syntax is simple and it can be learned quickly. Students start diving into more advanced aspects of computer science much more quickly than with other programming languages.
-It is powerful: it is used in real-world applications, so students immediately start acquiring valuable skills for their careers.
-It is versatile: it supports various programming paradigms including imperative programming, functional programming, procedural programming, and object-oriented programming.

Desktop Applications

Python is use to build desktop applications. Example some Linux and open source applications used python. In addition cross platform applications are built with GUI libraries using python.

Business Applications
Tryton and Odoo are tools used for enterprise development they also perform accounting, inventory, customer relationship management and other tasks. Many businesses rely on python for heavy lifting.
Game development

python is also used in developing interactive games. We have libraries that provides functionality and a room for game development. Examples are Pysoy and Pygame.

Web scrapping
Python is used to pull large data from websites which is useful in real world activities. Python uses BeautifulSoup which pulls such data.

Computer Vision and Image Processing
Python is used for computer vision and image processing, fields that are expanding rapidly.

The goal of image processing is to process an image, apply transformations to it, and return a new version of the original image.

In contrast, the goal of computer vision is more complex because it tries to make the computer understand and interpret an image and its content.

Image Processing
Let's start with image processing. With a Python library, you can perform operations such as:

-Cropping, flipping, and rotating.
-Manipulating exposure and color channels.
-Detecting edges and lines.
-Adding filters and restoring images.
-Computer Vision

Now let's dive into computer vision. If you start researching this topic, you might be surprised by its current applications. Some of them are:

-Navigation.
-Object and Event Detection.
-Facial recognition.
-Image classification.

This scientific field is so important that Google developed a tool called Cloud Vision, which has a Python version for developers to incorporate this functionality into their programs.

According to the "Using the Vision API with Python" tutorial in Google Codelabs, the Google Cloud Vision API:

Allows developers to easily integrate vision detection features within applications, including image labeling, face and landmark detection, optical character recognition (OCR), and tagging of explicit content.
This set of tools provides functionality for face detection, landmark detection, logo detection, label detection, text detection, and more.

Python Libraries
These are some awesome libraries for computer vision and image processing:

-OpenCV: an "open source computer vision and machine learning software library". Its Python version is called OpenCV-Python.
-scikit-image: a "collection of algorithms used for image processing".
-NumPy: it can be used to process the pixels of an image as a 2D array.
-SciPy: the scipy.ndimage package "contains various functions for multidimensional image processing."

We have a wide variety of python application kindly try to go through the rest and get insights and drive to be a python guru.

Important note
As a beginner you come across practice exercises, please do not copy paste them try do them manually. The point of these exercises is to train your hands, your brains and your mind in how to read, write and see code. If you're copypasting you are cheating yourself out of the effectiveness of the lessons.
we also have websites such as Hacker rank you can use it to assess yourself after covering a certain topic by trying the challenges offered there to see if you're understanding and if YOU can solve problems.

learning websites
-codecademy.
-Udemy.
-coursera.
-Youtube.
-W3Schools.
-freecodecamp.

*Conculsion *
As a beginner we have the fear of unknown and also some tend to suffer from the imposter syndrome please don't allow these monsters stop you, face them. Success is not final, failure is not fatal: it is the courage to continue that counts. Python is fun to work with if your motivated, driven and passionate to know what it entails and its applications and advantages. I have written two articles on introduction to data structure and algorithms the second one is introduction to modern python kindly take a look.