DEV Community: Siddharth Chandra

Blockchain Fundamentals, Cryptocurrencies and Open Source

Siddharth Chandra — Wed, 29 Dec 2021 06:09:57 +0000

Overview

This article is divided into 4 sections, 1^st section is about blockchain fundamentals, 2^nd section deals with the key features of building a blockchain mobile app, 3^rd section dives into cryptocurrency wallets available for us to use and 4^th section outlines how the opensource is driving the blockchain revolution.

You can jump onto any section you like to read, the order of reading these sections are not related.

Blockchain Fundamentals

Blockchain technology is a constantly evolving, secure and shared record-keeping system in which each user has a copy of the data, which can only be modified if all parties involved in a transaction agree. This section examines the fundamentals of blockchain technology.

A Blockchain Transaction

To understand the fundamentals of the blockchain, we need to understand how a transaction can be added to the existing blockchain. This can be done by going through a few important steps, as illustrated in the below figure.

Assume that user 1 wants to send X amount of digital bitcoins to user 2. User 1 then initiates the transaction.
A transaction is initiated by user 1's node by first making it and then digitally signing it with its private key. A transaction in a blockchain can reflect a range of actions.
A peer-to-peer (P2P) network broadcasts the desired transaction to each individual computer (or node).
Individual nodes receive the request and attempt to validate the transaction using cryptographic techniques. The miner node is the final node in the validation chain. These miner nodes are compensated in Bitcoin.
The verified blocks are now added to the blockchain network. Hashing is used to connect these nodes.
The approved transactions are then recorded in a public ledger. Transactions are completed and the ledger is updated after the block is added to an existing chain.

Cryptography Techniques Used in Blockchain

Hashing, public-private key mechanisms, and digital signatures are the key cryptographic techniques used by the blockchain community. In this case, a hash feature is implemented to offer each user the opportunity to view the blockchain in its entirety. Blockchain characteristically uses the SHA-256 hashing algorithm. Public-private key pairs are used to gain access to the information and allow transactions. Digital signatures are also used for multi-signature contracts and digital wallets on the blockchain, as well as to ratify transactions by signing them securely (offline).

Working of A Blockchain Network

In a blockchain network, each node holds a complete copy of the distributed ledger. The four fields of a block in this network are the block number, the data field, the hash value, and the nonce. Miners adjust the value of nonce to make it a suitable number for hashing the value of a block. A nonce is a random whole number that is a 32-bit (4-byte) field that is changed by miners to make it an acceptable number for hashing the value of a block. In addition, the chain of blocks is generated by applying the hash of the previous block to the current block. The previous block's hash is stored in the field prior.

Four zeros are used to begin the hash value. This determines whether or not a block is valid. Each new hash value is created by combining the old hash value, the new transaction block, and a nonce. The hash cash difficulty factor is determined by the number of leading zeroes required in the hash output. The difficulty is tweaked so that block production stays consistent at about one block per ten minutes. The more difficult it is to alter the blockchain, the tougher it is to overwrite the blockchain and double-spend coins.

Winding Up

Blockchain technology is widely accepted by a variety of businesses. It has opened up a plethora of opportunities in the areas of healthcare, finance, accountancy and the online marketplace. The adoption of blockchain technology is expected to grow manifold in the coming years, offering quite a wide range of jobs for young aspirants.

Basics of Building a Blockchain App

A blockchain stores data in such a way that it is almost impossible to alter or hack it. In this section, we will go through the key features of the blockchain, look at blockchain development platforms, and view the steps in the blockchain mobile app development process.

Some Blockchain Apps

While blockchain development began with cryptocurrencies, it is no longer limited to them. Blockchain applications have now gone beyond cryptocurrency and into a variety of other sectors. As we all know, the main benefit of blockchain technology is that it ensures maximum data transparency and security, helping businesses and organisations save a significant amount of money on standard security methods.

The following are some useful blockchain apps.

1. Filament: This is an excellent example of a blockchain-based IoT app that uses blockchain data ledger systems to manage data processed by software and microchip hardware. Several linked devices can communicate information over the blockchain and keep in sync with the help of the app.

2. KYC-chain: For many firms, completing Know Your Customer (KYC) documentation is a must. This simple and effective blockchain-based program streamlines the entire process of handling client documentation across administrative and criminal databases in many nations.

3. BurstIQ: BurstIQ is healthcare software that uses Big Data and blockchain technology to allow patients and doctors to share and collaborate on vital medical information.

Development Platforms

There are more than two dozen blockchain development platforms. A few of them are described below.

1. Multichain: With this powerful platform, developers can create and deploy private blockchains that can be used by a variety of businesses.

2. Hyperledger: This distributed ledger technology is used to govern the connected ecosystem of things and supply chains. This advanced blockchain platform has received a lot of acclaims.

3. Ethereum: A versatile opensource blockchain technology, Ethereum is commonly used for decentralised apps, smart contracts and initial coin offerings (ICOs).

4. EOS: The EOS platform’s goal is to create a reliable smart contract function. It’s also appropriate for modern businesses’ need for highly scalable and decentralised storage and hosting.

Blockchain systems are available in a wide range of sizes and shapes. Determining which platform is better and more suitable for one's application needs might be confusing. The solution is contingent on the desired capabilities and use-cases.

Choosing the most appropriate platform: The key to your project’s success lies in determining the platform’s character. Platforms based on cryptocurrencies as well as those based on smart contracts are available.

Decide whether you need smart contracts or not: You need to determine if your app needs smart contracts or not.

Selecting the consensus protocol: The consensus protocol ensures that every new block added to the blockchain is the only version of the truth that all nodes in the blockchain agree on, such as proof of stake, proof of work, proof of elapsed time, and so on.

Choosing the cryptocurrency: Another challenge is determining whether or not a cryptocurrency is required.

Selecting between a public and private network: Based on the requirement, you need to decide whether a public or a private network is needed.

Process of Blockchain Mobile App Development

There are mainly five stages in the blockchain app development process.

1. Identify the problem that the blockchain is required to solve: To begin, thoroughly examine the problem description and create a full report on it. Consider whether you need to create a new app from scratch or if you can just integrate the blockchain into an existing one.

2. Select the best platform for the app: Select the most appropriate platform for your project. Determine if you need a smart contract app or a crypto-transaction app, as well as the type of network and consensus protocol you'll need.

3. Requirement analysis: Begin by doing the requirement analysis and evaluating the app idea’s requirements. Do more brainstorming around the idea and, if possible, fine-tune it.

4. Create a blueprint of the idea: Try to develop the proof of concept that will put the entire app concept into a visual structure.

5. Development of the application: Develop the application employing best practices. Ensure to build a robust blockchain app.

Winding Up

We’ve covered the basics of blockchain in this essay — its main characteristics, as well as several development platforms and stages for developing blockchain applications. If you find the subject interesting, you can delve deeper into it. There are a variety of resources available on the Internet.

Cryptocurrency Wallets

Cryptocurrencies are getting entrenched in the global financial system slowly and steadily, with certain companies using these to pay their employees too. Hesitancy with respect to their use is slowly ebbing, and you may run the risk of being behind the times if you don’t stay updated on the latest trends in this domain. This section lists the exchanges and wallets you should be aware of in case you want to buy or trade-in cryptocurrencies.

Trends in Cryptocurrencies

Due to the high potential for profit, cryptocurrency trading is considered extremely lucrative, particularly among the younger population. Traditional stock market trading has a lot of restrictions and rules. There are no such limitations in the case of cryptocurrencies, and big profits can be produced immediately. The 'crypto market,' in which 'crypto users' trade various types of cryptocurrencies, is also known as the cryptocurrency industry.

The price of a BitCoin (BTC) was less than one US dollar around 10 years back when it was launched. Today, its price is more than US$ 30,000. A few months ago, its price touched around US$ 60,000, proving it is a very attractive form of investment.

The market capitalisation of cryptocurrencies is around 338 million US dollars, with 13,951 BitCoin ATMs located in different parts of the world, as per a report from Statista.

Wallets and Exchanges

Currently, there are a number of exchanges in the world that offer platforms and wallets for buying and trading cryptocurrencies. These include Web-based environments as well as apps for Android, iPhone and other mobile phone platforms. Crypto users can buy and sell different currencies that are legal tender around the world with ease on these sites.

The major worldwide crypto exchanges in the world that provide wallets and Web platforms for cryptocurrency trading are listed in the below figure. In crypto exchanges, pairings refer to the pairs of cryptocurrencies that are used to buy and sell such coins.

In the crypto world, there are two types of exchanges and wallets — centralised and decentralised.

Centralized Exchanges

The centralised exchanges and wallets are owned by companies or corporate giants, which record the complete Know Your Customer (KYC) details either through video KYC or paper-based KYC. Users can open accounts in these centralised exchanges and then trade in any cryptocurrency.

The below figure lists some centralized exchanges.

Decentralized Exchanges

Decentralised exchanges are platforms on which the account is opened directly without any intermediate company, and trading is done directly by the crypto users.

The below figure lists some decentralized exchanges.

Winding Up

The jury is out on the use of cryptocurrencies because these are sometimes used to circumvent the national money transfer system. Some governments prohibit their use in order to prevent money laundering.

In order to incorporate the maximum benefits and features into a real-world scenario, the use of cryptocurrencies in any country demands legislation and procedures. The benefits to society and the corporate sector might be substantial if cryptocurrencies are used for real-time applications and e-governance.

Open Source Blockchain Revolution

Blockchain is a disruptive technology that is changing the world in a variety of ways. Cryptocurrencies are the most obvious example. This section compares and contrasts this technique with opensource, another revolutionary technology that has transformed software development.

Blockchain Characteristics

Immutability: A hallmark feature of blockchain is that it is incorruptible. Data once saved in a block cannot be changed without breaking the chain.

Decentralisation: Blockchain does not require a central authority to maintain the ledger. All nodes in the network have a copy of the ledger that gets updated with every transaction.

Enhanced security: Decentralisation and consensus mechanisms built into blockchain make it highly secure, as every transaction has to be verified with ‘proof of work’.

Distributed: The blockchain ledger is not a singular entity in the hands of a few; instead it is a shared entity that is accessible and maintained by everyone in the network.

Transparency: Every addition to the blockchain requires consensus from all nodes in the peer-to-peer network. This makes every transaction transparent and public.

Open Source Characteristics

Transparency: The founding principle of open source is to make the source code of the project public. This enhances the transparency of the project, as every detail that goes into its development can be freely accessed by anyone.

Enhanced security: Finding security loopholes in the software is way easier when the code is non-proprietary and is publicly available. This naturally enhances the security of the software or, at the very least, can help raise public awareness about the potential security-related pitfalls.

Collaboration: The primary purpose of open source software is to encourage other developers to collaborate and contribute to the development of the software. This makes open source development, by nature, a community experience.

Modifiability: Open source software can be modified and re-distributed to solve different problems. This is the case with Linux, as its modified ‘distributions’ are used in various different gadgets and computers.

Decentralisation: One can argue that open source software is not inherently decentralised as making changes to the main project still requires the approval of the project moderator(s) and the owner(s). But if perceived from a different angle, given that the source code can be modified and re-distributed under proper licensing, this gives the developer the autonomy to use the software as and how desired thus making the process decentralised.

Convergent Evolution

Opensource software and blockchain are two independent technologies with different themes and strategies for addressing different problems. When we compare the attributes of blockchain to open source, however, we discover certain similarities.

In evolutionary biology, the term 'convergent evolution' refers to the occurrence in which animals from distinct time periods or epochs acquire similar characteristics. This can be seen in molecular biology, animal morphology, and even plant morphology. This holds true for technical breakthroughs as well. We produce new technologies every day to address a wide range of problems; nonetheless, there are underlying patterns in the motivations that drive the creation of these technologies.

Winding Up

Blockchain is new, but open source technology has been around for a long time. In my perspective, the rise of open source technology has had a direct impact on the birth of blockchain. Both technologies are focused on transferring power from a small number of people to a big number of people. Both strive to increase product security and transparency while also promoting fairness and collaboration. While the applications of open source and blockchain are diverse, the motives for their creation are similar. It is safe to assume that open source and blockchain share many characteristics in common and are hence the product of technological convergence.

Make sure to follow me to get regular updates or subscribe so that you never miss my upcoming articles, ranging from Python to Computer Vision to Scala.
Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Big Data Open Source Frameworks

Siddharth Chandra — Wed, 29 Dec 2021 06:08:46 +0000

Big Data is a term used to define large scale data sets that are too complex to be manipulated with basic DBMS. Handling Big Data requires sophisticated hardware and software technologies.

Just as open-source has been the primary reason for the Big Data revolution, so has it been at the forefront of developing tools to counter the challenges that we face with the data. And while there are hundreds of popular open-source projects that are focused on Big Data management, in this article, we shall focus on the top three that caught my attention.

We shall discuss the Hadoop framework, the Spark analytics engine and the Kafka framework.

Is Big Data Hard To Handle?

Almost 90% of the Big Data we produce today is unstructured and there are no predefined models that it adheres to. And there is no quick and easy way to access and analyse unstructured data.
Not only is Big Data unstructured, but it is also extremely complex. This means traversing through it requires advanced algorithms that are being run on top of grade-A machinery and not everyone can afford that.
We have multiple sources from which we accrue data and there are no clean ways to integrate disparate data from so many sources.
With Big Data comes big mistakes. Storing massive and complex data in a secure manner is a really daunting challenge. It is also the need of the hour. Since the beginning of the pandemic, more people are stuck at home with their computers than ever before and cybercrime and data theft have gone up by 400%.

Open Source Softwares

Hadoop

Developed by the Apache Foundation, Hadoop is a framework for Big Data management. It uses distributed systems architecture, which has the ability to store and easily access vast amounts of data.

While there are many different types of Big Data management solution software, what makes Hadoop stands out are 3 components, HDFS, MapReduce and YARN.

HDFS (Hadoop Distributed File System)

The Hadoop Distributed File System (HDFS) is a powerful and dynamic tool that is efficient at handling Big Data. HDFS breaks down Big Data into blocks and the default size of each block is 128MB. This makes accessing data simple, as the HDFS only has to access the specific block of data that the user is searching for. HDFS is a resilient framework, thanks to its replication methodology. To maintain the accessibility of the data even when one of the distributed system blocks is down, HDFS replicates that block data in a three-fold fashion making it robust to avoid failures or inconsistency.

MapReduce

Storing and accessing Big Data, with all its complications and difficulties, is still the easy part. The real fun begins when we try to process Big Data. MapReduce is exactly what it sounds like, it maps the data followed by reducing it. MapReduce takes the data as input and splits it into different parts. It then maps each individual component within each part. Then the data is shuffled and sorted into clusters of homogenous data. The data is then reduced by integrating homogenous data and assigning IDs, which give the smaller final output.

YARN (Yet Another Resource Negotiator)

A Hadoop environment is constrained to the number of resources it has, namely, CPUs, memory units, storage space, etc. But since Hadoop is designed to be able to run multiple processes at the same time, a mechanism for allocating resources is necessary to make sure there are no deadlocks and every task is accomplished in its desired time period. Now, YARN comes into play, when a job requires the resources to run, the application manager sends a request to the node manager to allocate the physical resources. The node manager then sends the resources to the resource manager, which assigns the physical resource to the client process.

Spark

While HDFS, MapReduce and YARN are the three major defining components of Hadoop, the Hadoop ecosystem has more tools to offer when it comes to Big Data processing, one such tool is Apache Spark. Spark is one of the few tools in the industry that combines Big Data analytics tools with Machine Learning and AI. Spark is 100 times faster than MapReduce in processing data. Spark computations are done in memory itself, it has the ability to perform both batch processing real-time processing of data. Spark is also comparatively lightweight as most of it is written in Scala.

Following are the Spark components:

Spark Core
Spark SQL
Spark Streaming
Spark MLib
Spark GraphX

Kafka

Apache Kafka is a framework implementation of a software bus using stream processing. If you break down an information transaction on the Internet, it's essentially nothing but an exchange of information between 2 systems which we sometimes call producers and consumers. Each transaction can be thought of as integration between producer and consumer. In order to simplify complex integrations, Kafka acts as a broker between producer and consumer, which can be well integrated with Hadoop and Spark.

Kafka consists of Brokers and a Zookeeper.

Broker

A broker is a Kafka server that runs in a Kafka cluster. Each broker divides a unique transaction between the producer and consumer into a topic. Now, these topics within the broker are used to consume and broadcast the messages in an asynchronous fashion.

Zookeeper

In order to define topics and keep a metadata record of all the topics defined in the cluster, Kafka uses Zookeeper which is a replicated distributed log with a file system API built on top. It keeps a record of all the topics within all the nodes and also keeps track of which nodes are down and which are up and running.

Where To Go From Here?

The tools and frameworks discussed in this article are a high-level overview of the innovation and efforts taking place in the Big Data management industry. There are many more open source tools and frameworks that can be learned and used for Big Data management.

Google all the available resources and check which fits your Big Data management need to the point and become a Big Data Engineer one step at a time.

Books to check out more on Big Data Engineering:

Make sure to follow me to get regular updates or subscribe so that you never miss my upcoming articles, ranging from Python to Computer Vision to Scala.
Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Scala Vs Python Syntax Cheat Sheet

Siddharth Chandra — Wed, 29 Dec 2021 06:06:48 +0000

There are about 8 million Python developers around the world and it's also the top programming language people want to learn, but why is that? Why not any other language but Python? Let me tell you, Python offers a lean learning curve, with just 2-3 days of hands-on Python anyone can start developing applications (they might not be technically sound software initially), it's syntax is one of the easiest to learn due to its plain english like code, it offers flexibility from both domains either object-oriented or functional programming (not so much FP, but it has its benefits) and it is used in almost every field of software engineering.

The only drawback people face when working with Python is related to its speed of executing programs. But is there any other programming language as easy to understand and to get started to work with? Yep, there is, Scala comes to the rescue!

Python faces speed issues mainly due to its interpreted nature, whereas Scala is a compiled language that offers dynamic typing as well (close to 90%), many new features in Python like pattern matching are already present in Scala since the beginning, there are awesome syntactic sugars in Scala that allows us to write even less code when compared to Python and also Scala being a functional programming language does not shy away from object-oriented programming as it is run over JVM.

When it comes to the syntax of these programming languages, there are a lot of similar patterns and some differences as well, but overall they both offer a lean learning curve with endless opportunities in the software engineering world.

So, let's discuss those similarities and differences in the form of a cheat sheet, which not only helps in refreshing some concepts from Python/Scala but also acts as a guide to learn Python or Scala for someone coming from the Scala or Python background respectively.

This will be a to-the-point kind of article, with little to no explanation but with necessary links wherever needed.

So, let's get started!

How To Import?

In Python importing a library can look like this:

import math.*

# OR

import math.pow

# OR

from math import pow, sqrt

# OR

from math import sqrt as squareRoot

In Scala importing a library can look like this:

import scala.math._

// OR

import scala.math.BigInt

// OR

import scala.math.{BigInt, BigDecimal}

// OR

import scala.math.{BigInt => BInt}

How To Declare Variables?

Python offers dynamic typing so you don't need to declare variables beforehand like in C/C++, but you can just type out the variables and use them:

some_variable = 10

Scala also offers dynamic typing to some extent and infer that type during compile time:

val some_variable = 10  // infers it as Int type

How To Define Conditionals?

In Python and Scala defining conditionals are more or less similar:

v = 5
if v%2==1:
    print("Odd")
elif v==0:
    print("Zero")
else:
    print("Even")

In Scala:

val v = 5
if (v%2==1) println("Odd")
else if (v==0) println("Zero")
else println("Even")

The important thing to note in conditionals is that in Python conditionals are referred to as statements whereas in Scala these are expressions that evaluates to something or nothing.

How To Define Loops?

In Python a simple for-loop would look like this:

for i in range(10):
    # do something here

The same for-loop in Scala would look like this:

for {
    i <- 0 until 10
}   {
    // do something here
}

The difference in their respective working behind the scenes, in Scala, is the use of conjunction of map and flatMap methods whereas in Python it is simple iteration.

There is a while loop in Scala, but let's not discuss that, while coding in Scala it is not encouraged to use while loops.

Why are while loops not recommended in Scala?

How To Write Methods?

In Python, we can simply use the def keyword to define functions, which are similar to methods in Scala.

def some_method():
    # do something here

In Scala we can define a method like:

def someMethod() = {
    // do something here
}

To define functions in Scala, we can use Function objects or their respective syntactic sugars. To know more about functions and functional programming in Scala, do check this.

How To Define Classes and Objects?

In Python, we can define a class using the class keyword, the same keyword can be used to define a class in Scala.

Whereas objects in Scala are singleton instances and similar to what we call static class in other programming languages.

A class however in Scala can be instantiated using the new keyword, whereas in Python calling the class directly creates an instance for that.

In Python:

class Student:

    def __init__(self):
        # do something here

student = Student()     # creates an instance of class Student

In Scala:

class Student

val student = new Student()     // creates an instance of class Student

// use Object keyword for singleton objects

Object Student

val aParticularStudent = Student

To know more about classes and objects in Scala, I recommend checking out this article.

Miscellaneous

Scala offers 2 more advantages over Python, namely traits and tail recursion.

Python does not offer tail recursive capabilities to its functions and suffers from stack overflow errors whenever functional programming goes wrong. To know more about why python does not offer tail recursion, you can read this article. However, there is a work-around as described in this article using decorators.

Similarly, Python does not have concepts like traits. A trait is used to describe the quality of a class, whereas in Python you would have to do multiple inheritances to get certain qualities or have to define certain methods within the class definition. Trait allows loose coupling.

How to define a trait in Scala?

trait SomeQuality

class Student extends SomeQuality

Wrap Up

Well, that wraps up this simple Scala Vs Python syntax cheat sheet.

What else would you add to this cheat sheet? Please let me know in the comments!

If you are looking to sharpen your coding skills in Python or Scala, then do consider buying these awesome books that go in-depth to let you understand the semantics of the language.

For Python:

For Scala:

Make sure to follow me to get regular updates or subscribe so that you never miss my upcoming articles, ranging from Python to Computer Vision to Scala.
Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Scala For Beginners - Crash Course - Part 5

Siddharth Chandra — Wed, 29 Dec 2021 06:04:32 +0000

Welcome to the 5^th and last part of the Scala beginners crash course, here we will go through the concepts like collections, sequences and map, flatmap, filter in the Scala programming language. This will be a short article, so if you need much more in-depth information you can check out the recommended books at the end of this article.

It will be a no-nonsense, to the point kind of article (like part 1, part 2, part 3 and part 4 with all the necessary links wherever needed.

Without any further ado, let's get started.

Before We Start

Before we start coding out, first create a new object in the crashcourse package (please go through part 1 for more information on creating packages and objects).

Let's name it Collections like below:

Now, we can start coding out the examples on this Scala application.

Collections

Scala offers 2 sets of collections, one is mutable and the other one is immutable. Scala by default works with immutable collection objects.

But, we can import mutable collection objects whenever needed like:

import scala.collection.mutable

Also, if we want we can see all immutable collection objects in package scala.collections.immutable.

Following are some of the immutable collection objects provided in scala:

Similar collection objects are available in the mutable set. This link has all the mutable collection objects set.

The commonality between mutable and immutable collection objects is that both extends trait Traversable. It is a base trait for all collections and offers a great variety of methods.

Some of these collection objects are often sequenced in nature. Let's discuss sequences now.

Sequences

Sequences in scala are a general interface for data structures that have a well-defined order and can be indexed.

These support various operations like:

apply, iterator, length, reverse for indexing and iterating.
Concatenation, appending, prepending.
A lot of others - grouping, sorting, zipping, searching and slicing.

Let's see how to define a sequence:

Output:

Range is one of the sequences available, we can define a Range like this:

Output:

There is another type of sequence, LinearSeq which is an immutable linked list with the following properties:

Accessing head, tail and using methods like isEmpty are fast and takes O(1) time.
Most other operations like length or reverse takes O(n) time.

Arrays in scala are equivalent to java arrays, these can be manually constructed with predefined lengths. These can be mutated, have fast indexing and are interoperable with java's arrays.

Vectors are also present in scala, these are the default implementation for immutable sequences taking constant time for indexed based read/write, with fast append/prepend, shows good performance for large size vectors and are implemented using a fixed branched trie data structure.

Tuples are finite ordered kinds of lists that can contain a maximum of 22 elements. A tuple value data type is defined similarly to a function like TupleN[...].

There is one more collection object, Map, which is a key->value pair data structure present in scala. Following are some ways to manipulate a map:

Output:

More on Scala collections can be found here or you can check the books provided in the last section for the more detailed in-depth working of collections.

map, flatMap, filter and for

Finally, on the most important part of functional programming, that would become the daily need of a Scala programmer. I wanted to include these in the previous part but to show how these work I needed to show collections in scala first, now that we know enough about collections, let's move on to these functions.

Note: For the sake of examples, I will be showing these operations on the List sequence, you can try these on other collections as well.

First, define our list collection like this:

What is the map method? It allows a certain function to be applied to all the elements present in the given sequence and evaluates to a sequence of updated values.

What is the filter method? It filters out the elements which do not meet a certain criterion defined as another function and evaluates to a sequence of filtered values.

What is the flatMap method? This is identical to the map method, but the only difference is that in flatMap the inner grouping of an item is removed and a sequence is generated. So, if flatMap is applied to a list then the generated output will be a list of lists, changing the inner grouping of integer elements to a list and later flattening it.

What is the for loop in scala? Yeah, you read it right 'What is' and not 'How to', in scala, writing a for loop is just a fancy way of writing a coupled map and flatMap methods. A for loop in scala can be written as:

Output:

These are some important methods to remember while coding out in scala.

Wrap Up

Well, that wraps up part 5 and the end of this crash course.

I hope it was something worth it for you as a reader and made you a scala enthusiast at the end of this course.

You can pick up any of the below books to master the art of programming in scala:

To keep in touch, you can follow me on here or subscribe to get the updates of the new blogs that I will write in the future or follow me on Twitter where I share my thoughts not only on programming but also on personal finance.

Make sure to follow me to get regular updates or subscribe so that you never miss my upcoming articles, ranging from Python to Computer Vision to Scala.
Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Scala For Beginners - Crash Course - Part 4

Siddharth Chandra — Wed, 29 Dec 2021 06:01:37 +0000

Welcome to the 4^th part of the scala beginners crash course, here we will go through the concepts like options, exception handling and pattern matching in the scala programming language. This will be a short article, so if you need much more in-depth information you can check out the recommended books at the end of this article.

It will be a no-nonsense, to the point kind of article (like part 1, part 2 and part 3) with all the necessary links wherever needed.

Without further ado, let's get started.

Before We Start

Before we start coding out, first create a new object in the crashcourse package (please go through part 1 for more information on creating packages and objects).

Let's name it OptionErrorPattern like below:

Now, we can start coding out examples on this scala application.

Options

An option is a wrapper for a value that might be present or not.

As the name suggests it evaluates to an option of 2 values, either Some or None, when given an expression. In this way, when we are dealing with some unsafe APIs, we can always wrap that API calls within an Option object to keep our program safe and error-free that might otherwise result in a Null pointer kind of exception.

For example,

Output:

Some wraps a concrete value.
None is a singleton for absent values.
It is better to use Option in APIs that we are creating.

Handling Exceptions

In scala, exceptions are handled inside try-catch blocks, like below:

Output:

But, it comes with its drawbacks:

Multiple/Nested try-catch blocks make the code hard to read/follow.
We can not chain multiple operations prone to failure.

To overcome these limitations, scala introduced a Try object that can act as a wrapper to wrap an error-prone expression and evaluates to either a Success or a Failure as a result.

A Try is a wrapper for a computation that might fail or not. It wraps failed computations as Failure and successful computations as Success.

To use Try, Success and Failure objects, we have to import these first:

import scala.util.{Try, Success, Failure}

So, the above method someErrorProneFunc can be used with Try as follows:

Output:

We can check if a Try is a Success or Failure using isSuccess or isFailure methods respectively.

We can use Try to handle exceptions for avoiding runtime crashes due to uncaught exceptions or just to avoid an endless amount of try-catch blocks.

Pattern Matching

Pattern matching is a mechanism for checking a value against a pattern. A successful match can also deconstruct a value into its constituent parts. It is a more powerful version of the switch statement in Java and it can likewise be used in place of a series of if/else statements.

A match expression has a value, the match keyword, and at least one case clause.

For example,

Output:

Some important points to note on pattern matching:

It is used to decompose values.
Cases are matched in the order given.
What if there was no case that matched the pattern, then it would result in MatchError.
Pattern matching works really well with case classes.

Wrap Up

Well, that wraps up part 4 of this crash course.

I know, it was a very short read, my motivation was to keep it short and simple and be able to provide ample impetus to you to explore it further.

Take it as a challenge or homework, explore advanced pattern matching, how you can use Option in conjunction with Try (it can do wonders to the API you are working on) and in the comments, mention what you know about on these concepts.

You can check these books to get more insights on these topics:

In the next and last part of this crash course, we will discuss the collections available in scala and most important of all, the map, flatmap and filter methods in scala.

Make sure to follow me to get regular updates or subscribe so that you never miss my upcoming articles, ranging from Python to Computer Vision to Scala.
Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Scala For Beginners - Crash Course - Part 3

Siddharth Chandra — Wed, 29 Dec 2021 05:56:43 +0000

Welcome to the 3^rd part of the scala beginners crash course, here we will go through the functional way of scala programming language and concepts like higher-order functions, curries and anonymous functions!

It will be a no-nonsense, to the point kind of article (like part 1 and part 2) with all necessary links wherever needed.

Without further ado, let's get started.

What Is Functional Programming?

The below is an excerpt from the Functional Programming in Scala book.

Functional programming (FP) is based on a simple premise with far-reaching implications: we construct our programs using only pure functions, in other words, functions that have no side effects. What are the side effects? A function has a side effect if it does something other than simply return a result, for example:

Modifying a variable.
Modifying a data structure in place.
Setting a field on an object.
Throwing an exception or halting with an error.
Printing to the console or reading user input.
Reading from or writing to a file.
Drawing on the screen.

We discussed side-effects also in part 1.

Note: Functional programming is a restriction on how we write programs, but not on what programs we can express.

Before We Start

Before we start coding out, first create a new object in the crashcourse package (please go through part 1 for more information on creating packages and objects).

Let's name it FunctionalProgramming like below:

Now, we can start coding out examples on this scala application.

Creating Functions In Scala

Before moving on to create functions in scala, please keep in mind that scala works on JVM and for JVM it needs classes/objects to work with, hence everything in scala will be an object, even the functions are objects.

Note: Due to scala's syntactic sugar, we can still write functions in a more functional fashion.

Let's create a function in scala:

Scala has a collection of functions namely Function1 to Function22, which can be used out-of-the-box without any issues.

A function like Function1[String, Int] represents that it will take String as an input and evaluates to Int as an output. This function definition will be written in its apply method, as Function1 is an object and we need to override its apply method to make it callable.

A simple syntactic sugar for a function like Function2[Int, Int, Int] would be (Int, Int) => Int, which is more functional while reading the code.

We should also note that while working in IntelliJ, it tells us that the given Function1[String, Int] can be converted to its respective syntactic sugar, we can do so by hovering over the yellow line and simply clicking Replace FunctionN[A1, A1, ..., AN, R] with (A1, A1, ..., AN) => R.

Converted to syntactic sugar:

Important points to note:

All scala functions are objects, that is, instances of Function1, Function2 and so on.
JVM was designed for object-oriented programming, but using it for scala function we need something like objects (like Function1) to instantiate them.
Functions are traits with maximum parameters to be 22.

Anonymous Functions

The anonymous function is something we saw in the previous example of Function2[Int, Int, Int] syntactic sugar.

So, the below expression:

can be written as:

or as:

Anonymous functions are just another way of writing functions as supported by scala's compiler, behind the scenes everything is the same.

More ways in which we can write anonymous functions:

Note: An anonymous function is also known as a function literal. A function that does not contain a name is known as an anonymous function in general.

Higher-Order Functions and Curries

Higher-order functions have at least one of the following properties:

Takes one or more functions as parameters.
Returns a function as a result.

In scala, a general higher-order function declaration can look like this:

These are called Higher-Order Functions, which essentially takes one or more functions as input and/or returns a function as output.

Moving on to curries, currying means transforming a function that takes multiple arguments into a chain of calls to functions, each of which takes one argument. Each function returns another function that takes the subsequent argument.

A simple curry can be like this:

Functions with multiple parameter list also act as curries:

Wrap Up

Well, that wraps up part 3 of this crash course.

I highly recommend you to go through the books below to get a better and deeper understanding of functional programming:

In the next part, we will discuss some topics specific to scala, like options, handling errors in a unique style and pattern matching (the most awesome thing in scala).

Make sure to follow me to get regular updates or subscribe so that you never miss my upcoming articles, ranging from Python to Computer Vision to Scala.
Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Scala For Beginners - Crash Course - Part 2

Siddharth Chandra — Wed, 29 Dec 2021 05:52:25 +0000

Welcome to the 2^nd part of scala beginners crash course, here we will go through the object-oriented way of scala programming language and concepts like anonymous class, case classes and traits!

It will be a no-nonsense, to the point kind of article (like part 1) with all necessary links wherever needed.

Without further ado, let's get started.

Before We Start

Before we start coding out, first create a new object in crashcourse package (please go through part 1 for more information on creating packages and objects).

Let's name it ClassAndObject like below:

Now, we can start coding out examples on this scala application.

Classes And Objects

In scala, to define a class we use the class keyword:

We can directly use the Student class later in the program. That's it! that's how we can create a class in scala.

It is different from writing object Student as it is not instantiated right away, we have just defined it and are not using it anywhere. To instantiate a class we can use the new keyword.

A class can have parameters, like this:

Note: Class parameters are not class fields, which means we can not use these parameters with instantiated objects.

To make class parameters as class fields, we have to use the val keyword along with the name of the parameter, like this:

We can use class parameters within the class code block by using the this keyword if value name conflict occurs, like below:

If value name conflict does not occur, then the class method by default (implicitly) uses the this operator on class parameters values.

Some other important concepts:

Overloading occurs with only different signature methods. More on overloading in scala here.
Auxiliary constructor are generally used for default parameters. More on auxiliary constructor here.
Instances are fixed, whenever we want to do something with the given class, say creating/updating something, then we would need to instantiate a new class, it is the clause of immutability, useful in functional programming.

Conceptually, scala does not provide class level functionality, like a static class. However, scala provides something called an object, using which we can obtain class level functionalities.

Scala objects are singleton instances by definition. We can create an object like the below code:

We can create an object with the same name as that of a class in the same scope, creating such an object/class is called companion object/class, known as companions.

Companions can be used to make a factory method in objects, that can help instantiate a class in a certain way. Using companions does not require using the new keyword to instantiate a scala class, we can define a method named apply within the object to handle instantiation.

We can create a companion like below:

Well, that's about it on classes and objects in scala. You can check out official docs on scala classes here and on objects here.

Anonymous And Case Classes

To create an anonymous class we can directly define the class like this:

Things to remember when creating an anonymous class:

Pass proper parameters whenever required.
Anonymous class declaration works for abstract as well as for non-abstract classes. It works for traits as well. > Note: We will discuss abstract classes and traits in the next section.
Useful for on-spot declaration in some scenarios.

Using the case keyword in front of the class keyword will make it a case class, like this:

Case classes promote class parameters to fields, it has a predefined toString method that evaluates to a representation that is understandable.

Equality and hashcodes are implemented out-of-the-box with a handy copy method to copy instances. Case classes come with their companion objects predefined.

Things to remember when creating a case class:

Case class is a quick lightweight data structure with little boilerplate.
Case classes are serializable and used with the Akka framework.
Case classes have extractor patterns that can be used in pattern matching.
Case objects are the same as case classes but don't get companion objects as they themselves are objects.

Well, that's about it on anonymous class and case class in scala.

Inheritance, Abstract Class And Traits

Scala offers single level inheritance. Inheriting a class inherits public methods and protected methods can be used within the child class.

Inheritance example:

Important notes on inheritance overriding:

To override methods, use the override keyword. Override works for val, var, def or anything within in the class.
To prevent override, we have to use the final keyword.
Put a final keyword before class to altogether prevent the inheritance of a class.
Use the sealed keyword to restrict inheritance in different scopes of the program.

Moving on to abstract classes in scala, we have to use the abstract keyword to make an abstract class and an abstract class cannot be instantiated, however, we can create an anonymous class using the new keyword and defining all the methods within that anonymous code block.

Traits are similar to interfaces in java. They are also used to declare undefined or unimplemented methods. We have to use the trait keyword to define traits in scala.

Important points to note for abstract classes and traits in scala:

Abstract class can have both abstract and non-abstract methods. It is the same for traits.
Traits does not have constructor parameters.
Classes can inherit multiple traits, but not multiple classes.
Traits are used to define the behaviour of a class.
Abstract class is the type of thing itself.

Note: If you see ??? in abstract/trait methods, it means that method is not implemented and evaluates to Nothing.

Well, that's about it on inheritance, abstract class and traits in scala.

External Links

Below are some of the external links you can go through to get a more in-depth understanding of object-oriented programming in scala.

Wrap Up

Well, that wraps up part 2 of this crash course.

In the next part, we will discuss functions, HOFs, curries and much more related to functional programming in scala.

Make sure to follow me to get regular updates or subscribe so that you never miss my upcoming articles, ranging from Python to Computer Vision to Scala.
Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Scala For Beginners - Crash Course - Part 1

Siddharth Chandra — Mon, 24 May 2021 05:53:25 +0000

Welcome to this scala beginners crash course, here we will go through the basic syntax of scala programming language and some of the nuances available just for scala developers!

It will be a no-nonsense, to the point kind of article with all necessary links wherever needed.

Without further ado, let's get started.

What Is Scala?

Scala is a programming language just like any other that is available out there but with a lot of sweetness and awesome features that opens our world to functional programming with easy to understand syntax, running on JVM under the hood, having all functionality that Java language offers with popularity ranging from being used in major FinTech companies to FAANG companies and as one of the major language for data engineering applications.

Need to know more about what is Scala, visit here.

How To Install Scala?

Well, there are a couple of ways, in which the shortest and easy to follow is:

Install IntelliJ IDE from JetBrains. Check here.
Select IntelliJ IDEA | Preferences for macOS ( Ctrl+Alt+S ) or File | Settings for Windows and Linux.
Select Plugins on the left pane.
Search for Scala, install the plugin and restart IDE.
Now, after it restarts, select New Project and then Scala -> IDEA project, click Next, set project name and your project folder are ready with necessary scala features.

Installing Scala

Creating a New Project

Set Suitable Project Name

How To Start Working With Scala?

After successful installation and creating a scala project, you would be seeing a nice project structure with src folder.

We will create a package, inside that package we will continue our work. What is a package?

To create a package, right-click on src folder and select New | Package type, then fill in a name, whatever you like, for our case, let's name it crashcourse.

Create New Package

Now, right-click the crashcourse then New | Scala, here we are creating a scala object (we will discuss what scala object is in a minute). Let's name it BeginnersCrashCourse. It will create a standard template of scala object like below:

Select Scala Class

Select Scala Object

Well, now we have all we need to get started with some code!

What Is An Object?

Before moving to object, if you notice the created template have package crashcourse at the top, it means that whatever code we are going to write below this, will belong to this package and can be accessed by any code file within the same package or can be imported using import crashcourse._ in some other package. How to do imports in scala?

Now, what is object in scala?

Object is a class that has exactly one instance, it means it already is an instance with a defined class structure. What is the defined class structure for an object type? It is everything we write inside scala object.

Scala objects are singleton, which means if we say we have an object named BeginnersCrashCourse then it is an instance that is a singleton in nature and whenever I am accessing it in the same executing program then its behaviour is going to be the same at every place, containing same values, methods or functions.

Treat an object as something that has static class level functionality.

How to write an object? Simple, just use object keyword before writing an object, like below:

package crashcourse

object BeginnersCrashCourse extends App {

}

What is extends App? We extend App level functionality to a scala object whenever we want it to be able to run something inside it. It is like saying, we have a prebuilt public static void main(String[] args) whenever we extend it.

The above code is equivalent to:

package crashcourse

object BeginnersCrashCourse {
  def main(args: Array[String]): Unit = {

  }
}

Just to avoid writing the main method we can simply extend App to an object.

It is called a Scala Application if the main object (here BeginnersCrashCourse) extends App and is executable in nature. The main code goes inside an object that has App level functionality. Make sure you have a single main application while creating a service. Each service will have its main application to execute scala code.

Need more on scala objects, visit here.

Expressions, Types, Vars, Vals, Def, Print and Codeblocks

How to create a variable in scala?

Well, there are 2 ways in which we can create a variable in scala:

package crashcourse

object BeginnersCrashCourse extends App {
  var var1 = 1  //Mutable
  val var2 = 2  //Immutable
}

Writing comments in Scala is the same as Java.

The key difference between the 2 variables is that var1 is mutable and var2 is immutable and in functional programming, we will prefer to use immutable values.

val - values and var - variables, the name suggest itself that val will be a definite value whereas var are variables.

We can also use the semicolon ; at end of each line, but they are not needed. We can also define types to the variables, but for the above case, compiler does this work for us, although it is best to explicitly define the type of each value we use in our code. So below code is similar to the above:

package crashcourse

object BeginnersCrashCourse extends App {
  var var1: Int = 1;  //Mutable
  val var2: Int = 2;  //Immutable
}

There are several data types available in scala, some of them are as below:

package crashcourse

object BeginnersCrashCourse extends App {
  var var1 = 1;  //Mutable
  val var2 = 2; //Immutable

  val bool: Boolean = true  //or false
  val chr: Char = 'a'   //Make sure you use single quotes for character type value
  val str: String = "String"    //And double quotes for a string type
  val int: Int = 123    //An int value
  val shrt: Short = 123   //A short value
  val lng: Long = 123     //A long value
  val lng2 = 123L   //Make sure to put L to let compiler know that it is of type Long
  val flt: Float = 123.0f   //Make sure to put f at the end of float values
  val dbl: Double = 123.0   //Otherwise compiler will infer it as a type of double
}

Wrap Up

Well, that warps up part 1 of this crash course.

In the next part we will be discussing classes, more on objects, companions, case classes and much more related to object-oriented programming in Scala.

Make sure to follow me to get the update for the next part or subscribe so that you never miss my upcoming blog articles ranging from Python Programming to Computer Vision to Scala Programming.

Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

How To Convert Markdown To HTML Using Python

Siddharth Chandra — Mon, 10 May 2021 03:29:18 +0000

In this article, we are going to see how we can use python to convert GitHub flavoured markdown text to HTML. We will go through a python library, what are its benefits and how we can leverage it for automated conversion.

So, without any further ado, let's get started!

What Are We Going To Use?

We are going to use Python and one of its awesome 3rd party library, gh-md-to-html.

To install gh-md-to-html run the following command on terminal:

$ pip install gh-md-to-html
Collecting gh-md-to-html
  Downloading gh_md_to_html-1.11.4-py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 5.9 MB/s 
Collecting shellescape
  Downloading shellescape-3.8.1-py2.py3-none-any.whl (3.1 kB)
Collecting Pillow>=8.0.1
  Downloading Pillow-8.2.0-cp38-cp38-macosx_10_10_x86_64.whl (2.8 MB)
     |████████████████████████████████| 2.8 MB 4.8 MB/s 
Collecting webcolors
  Downloading webcolors-1.11.1-py3-none-any.whl (9.9 kB)
Collecting requests
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting beautifulsoup4
  Using cached beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
Collecting soupsieve>1.2
  Downloading soupsieve-2.2.1-py3-none-any.whl (33 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Installing collected packages: urllib3, soupsieve, idna, chardet, certifi, webcolors, shellescape, requests, Pillow, beautifulsoup4, gh-md-to-html
Successfully installed Pillow-8.2.0 beautifulsoup4-4.9.3 certifi-2020.12.5 chardet-4.0.0 gh-md-to-html-1.11.4 idna-2.10 requests-2.25.1 shellescape-3.8.1 soupsieve-2.2.1 urllib3-1.26.4 webcolors-1.11.1
$

As we can see, this library depends on a lot of other libraries and is not a pure library like pytube discussed here.

Advantages

Advantages include (from its official documentation):

Let's you specify the markdown to convert as a string, as a repository path, as a local file name or as a hyperlink.
Pulls any images referenced in the markdown files from the web/ your local storage and places them in a directory relative to your specified website root, so the resulting file structure is host-ready for static sites. Multiple arguments allow the customization of the saving locations, but the images will always be referenced correctly in the resulting HTML files. This is especially useful since it reflects GitHub's behaviour to serve cached copies of README images instead of linking to them directly, reducing tracking and possibly downscaling overlarge images in the process.
Creates all links as root-relative hyperlinks and lets you specify the root directory as well as the locations for CSS and images, but uses smart standard values for everything.
Supports inline LaTeX-formulas (use $-formula-$ to use them), which GitHub usually doesn't. gh-md-to-html uses LaTeX and dvisvgm if they are both installed (advantage: fast, requires no internet), and otherwise the Codecogs EqnEditor (advantage: doesn't require you to install 3 GB of LaTeX libraries) to achieve this.
Supports exporting to pdf with or without Github styling, using the pdfkit python module (if it is installed).
Tested and optimized to look good when using DarkReader (the .js-module as well as the browser extension). This is especially relevant considering that DarkReader doesn't usually shift the colours of SVG images, and the formulas added by gh-md-to-html's formula support are embedded as inline svg. gh-md-to-html ensured that the formulas are the same colour as the text, shifted in accordance with DarkReader's current/enabled colorscheme.
Supports umlauts and other non-ascii-characters in plain text as well as in multiline code blocks, which the github REST api usually doesn't.
Allows you to choose which tool or module to use at its core for the basic markdown to html conversion.
Styles its output with github's README-css (can be turned off).

Whilst using pandoc to convert from markdown to pdf usually yields more beautiful results (pandoc uses LaTeX, after all), gh-md-to-html has its own set of advantages when it comes to quickly converting complex files for a homework assignment or other purposes where reliability weights more than beauty. pandoc's md-to-pdf-conversion acts quite unusual when it comes to images, nested lists, multiline bullet point entries, or formulas, and gh-md-to-html does not.

Converting Markdown To HTML

Now, let's create a simple script that converts markdown text to HTML using gh-md-to-html.

Add the following code to a file, name it whatever you like, say mdtohtmldemo.py:

#./ mdtohtmldemo.py

import gh_md_to_html

html = gh_md_to_html.main('md-to-html/sample.txt')
print(html)

Where the markdown contents are as follows (notice that we are using it as a text file, but it is not necessary, we can use .md files directly):

# This is a sample markdown

Hello World!

> Feel free to add your own markdown here.

After running the above script for given markdown content, we get the below output:

<?xml version='1.0' encoding='UTF-8'?>
<link href="/github-markdown-css/github-css.css" rel="stylesheet"/>
<meta charset="utf-8" content="text/html"/>
<div class="gist">
<style class="formula-style">
        svg.gh-md-to-html-formula {
            fill: black;
        }
    </style>
<div class="gist-file"> <!-- This is the class that is responsible for the boxing! -->
<div class="gist-data">
<div class="js-gist-file-update-container js-task-list-container file-box">
<div class="file" id="article-sample">
<div class="Box-body readme blob js-code-block-container p-5 p-xl-6" id="file-docker-image-pull-md-readme" style="margin-left: 40px; margin-right: 40px; margin-top: 20px; margin-bottom: 20px">
<article class="markdown-body entry-content container-lg" itemprop="text">
<h1 id="user-content-this-is-a-sample-markdown">
<a aria-hidden="true" class="anchor" href="#user-content-this-is-a-sample-markdown" id="user-content-this-is-a-sample-markdown"><span aria-hidden="true" class="octicon octicon-link"></span></a>This is a sample markdown</h1>
<p>Hello World!</p>
<blockquote>
<p>Feel free to add your own markdown here.</p>
</blockquote>
</article>
</div>
</div>
</div>
</div>
</div>
</div>

If we notice, it created a CSS /github-markdown-css/github-css.css in our local directory where we ran the script similar to what github uses.

There is an output HTML file also created with the same name as sample.html, if we open it in a browser, we get the following output:

Well, that's it from me. If you followed this, you would be seeing an autogenerated HTML file for your markdown file.

Try different functions, methods and utilities it provides and use them with your projects as needed.

Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Download YouTube Videos Using Python - Your Own YouTube Downloader

Siddharth Chandra — Fri, 07 May 2021 05:40:06 +0000

Do you find it troublesome to always look for a YouTube video downloader with your privacy at risk? Downloading a YouTube video using an external extension might be maddening because of the mouse click ads and other issues while using that extension.

So, why not we create our own bot that can download any YouTube video with just simply running a Python script? Everything would be in our control then, awesome, right?

In this article, we are going to see how we can make a Python script to automate the YouTube video downloading activity.

After going through this article, you can say NO to all those annoying ads, privacy issues, authorisation to different extensions and can finally have total control.

Let's get started!

What Are We Going To Use?

We are going to use Python and one of its amazing 3rd party library, pytube.

To install pytube:

$ pip install pytube
Collecting pytube
  Downloading pytube-10.7.2-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 3.4 MB/s 
Installing collected packages: pytube
Successfully installed pytube-10.7.2
$

Get The Streams

First, we will make a simple script to get all the available streams for the given YouTube video URL. Open a file name it whatever you like with the .py extension, let's say for our case we name it pytube_demo.py, add the following code in it and run:

# ./pytube_demo.py

from pytube import YouTube

video_url = input("Enter YouTube Video URL: ")
youtube_obj = YouTube(video_url)
print(youtube_obj.streams)

After you hit run, when the terminal prompts you to enter a video URL, enter this https://www.youtube.com/watch?v=fibIyvTleqc, it is one of my favourite songs Daydreamer by AURORA.

Note: You can enter any YouTube video URL here, what we are going to explore further doesn't depend on which URL you enter.

Enter YouTube Video URL: https://www.youtube.com/watch?v=fibIyvTleqc
[<Stream: itag="18" mime_type="video/mp4" res="360p" fps="25fps" vcodec="avc1.42001E" acodec="mp4a.40.2" progressive="True" type="video">, <Stream: itag="137" mime_type="video/mp4" res="1080p" fps="25fps" vcodec="avc1.640028" progressive="False" type="video">, <Stream: itag="248" mime_type="video/webm" res="1080p" fps="25fps" vcodec="vp9" progressive="False" type="video">, <Stream: itag="136" mime_type="video/mp4" res="720p" fps="25fps" vcodec="avc1.4d401f" progressive="False" type="video">, <Stream: itag="247" mime_type="video/webm" res="720p" fps="25fps" vcodec="vp9" progressive="False" type="video">, <Stream: itag="135" mime_type="video/mp4" res="480p" fps="25fps" vcodec="avc1.4d401e" progressive="False" type="video">, <Stream: itag="244" mime_type="video/webm" res="480p" fps="25fps" vcodec="vp9" progressive="False" type="video">, <Stream: itag="134" mime_type="video/mp4" res="360p" fps="25fps" vcodec="avc1.4d401e" progressive="False" type="video">, <Stream: itag="243" mime_type="video/webm" res="360p" fps="25fps" vcodec="vp9" progressive="False" type="video">, <Stream: itag="133" mime_type="video/mp4" res="240p" fps="25fps" vcodec="avc1.4d4015" progressive="False" type="video">, <Stream: itag="242" mime_type="video/webm" res="240p" fps="25fps" vcodec="vp9" progressive="False" type="video">, <Stream: itag="160" mime_type="video/mp4" res="144p" fps="25fps" vcodec="avc1.4d400c" progressive="False" type="video">, <Stream: itag="278" mime_type="video/webm" res="144p" fps="25fps" vcodec="vp9" progressive="False" type="video">, <Stream: itag="140" mime_type="audio/mp4" abr="128kbps" acodec="mp4a.40.2" progressive="False" type="audio">, <Stream: itag="249" mime_type="audio/webm" abr="50kbps" acodec="opus" progressive="False" type="audio">, <Stream: itag="250" mime_type="audio/webm" abr="70kbps" acodec="opus" progressive="False" type="audio">, <Stream: itag="251" mime_type="audio/webm" abr="160kbps" acodec="opus" progressive="False" type="audio">]

So, we got a list of Stream objects, each stream object giving us information about different aspects of a video.

If we see res attribute of stream object, we see that we have 360p, 1080p, 720p, 480p, 240p and 144p with different vcodec attribute. This res represents the resolution of a video, we can select any stream of desired resolution to download.

Also, notice that we have only audio option streams available as well to download.

Next, we will create a filter for video resolution and to keep it simple we will not dive into audio downloadable streams.

Filter Stream Based On Resolution

Update the code in the pytube_demo.py file to the below code:

# ./pytube_demo.py

from pytube import YouTube

def get_stream_for_res(streams, res):
    """
    Filter on the basis of a given resolution. Return a list of filtered streams.
    """
    stream = list(filter(lambda x: x.resolution == res, streams))
    return stream

video_url = input("Enter YouTube Video URL: ").strip()
youtube_obj = YouTube(video_url)

video_res = input(f"Enter YouTube Video Resolution for {youtube_obj.title}: ").strip()
req_stream_obj = get_stream_for_res(youtube_obj.streams, video_res)
print(req_stream_obj)

Hit run and we can see following output in the terminal:

Enter YouTube Video URL: https://www.youtube.com/watch?v=fibIyvTleqc
Enter YouTube Video Resolution for AURORA - Daydreamer (Audio): 720p
[<Stream: itag="136" mime_type="video/mp4" res="720p" fps="25fps" vcodec="avc1.4d401f" progressive="False" type="video">, <Stream: itag="247" mime_type="video/webm" res="720p" fps="25fps" vcodec="vp9" progressive="False" type="video">]

First, we take the video URL and creates the YouTube object, then we take custom input from the user while displaying the title of the video using youtube_obj.title. After receiving the resolution, say 720p, we call the function get_stream_for_res that filters the given list of streams on the given resolution. At last, we can see the output, it contains 2 stream objects in a list, both objects have different values for some attributes but have the same resolution.

Without diving deep on what all other attribute means, let's just simply take the first element of the list and download the video.

Download Stream Based On Resolution

Now update the code to the following to download the YouTube video (complete script):

# ./pytube_demo.py

from pytube import YouTube

def get_stream_for_res(streams, res):
    stream = list(filter(lambda x: x.resolution == res, streams))
    return stream

video_url = input("Enter YouTube Video URL: ").strip()
youtube_obj = YouTube(video_url)

video_res = input(f"Enter YouTube Video Resolution for {youtube_obj.title}: ").strip()
req_stream_obj = get_stream_for_res(youtube_obj.streams, video_res)[0]

req_stream_obj.download()
print(f"YouTube Video {youtube_obj.title} Downloaded With Resolution {video_res}")

When we run the above script, we get the following output and a downloaded video:

Enter YouTube Video URL: https://www.youtube.com/watch?v=fibIyvTleqc
Enter YouTube Video Resolution for AURORA - Daydreamer (Audio): 720p
YouTube Video AURORA - Daydreamer (Audio) Downloaded With Resolution 720p

Now, if we check our directory where we ran the script, we can find the downloaded video.

Note:

You may notice that some streams listed have both a video codec and audio codec, while others have just video or just audio, this is a result of YouTube supporting a streaming technique called Dynamic Adaptive Streaming over HTTP (DASH).

In the context of pytube, the implications are for the highest quality streams; you now need to download both the audio and video tracks and then post-process them with software like FFmpeg to merge them.

The legacy streams that contain the audio and video in a single file (referred to as “progressive download”) are still available, but only for resolutions 720p and below.

The video downloaded from the given code has a Stream object whose attribute progressive is False. The given video URL does not have legacy streams available for 720p but if you notice there is one stream with progressive as True and res as 360p.

<Stream: itag="18" mime_type="video/mp4" res="360p" fps="25fps" vcodec="avc1.42001E" acodec="mp4a.40.2" progressive="True" type="video">

So, go ahead and experiment with it and don't forget to check out the progressive attribute.

What Next?

Well, that's it from me. If you followed this and ran the final script, you would be seeing a downloaded video in your working directory.

You can take it to the next level by creating a UI for this utility. You can host this as a service that would be strictly private to you.

What are you waiting for then? Use Tkinter, PyQT, Kivy, Flask or Django and make a UI with more functions like a select option for audio and video, for video resolution and for audio ABR, etc. and say NO to all those annoying ads, privacy issues, several authorisations to different extensions and have total control over your downloaded videos.

Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Python - List Vs Tuple Memory Management

Siddharth Chandra — Thu, 06 May 2021 05:10:24 +0000

This article is going to be a short read, we will focus on how memory is managed in Python for objects like list and tuple and what can be the key takeaways.

Let's get started!

List

Python list as we all know is a mutable dynamic array that behaves like a linked list as well. Let's look at some code for creating and updating a list:

# ./list_1.py

l = [1,2,3]     # A new list gets created
print('Initial List', l)

l.append(4)     # Updating list using in-built append method
print('Append 4, List updated to', l)

l.pop()         # Pop last item
print('Pop, List updated to', l)

l += [5, 6]     # Updating list by adding a new list to given list
print('Add new list, List updated to', l)

Following will be the output:

Initial List [1, 2, 3]
Append 4, List updated to [1, 2, 3, 4]
Pop, List updated to [1, 2, 3]
Add new list, List updated to [1, 2, 3, 5, 6]
>>>

Now let's see what happens in memory when we run the above script, here we will be using sys to help us out on viewing system-related information:

# ./list_2.py

import sys
l = [1,2,3]     # A new list gets created
print('Initial List', l)
print('Initial List Address', id(l))
print('Initial List Memory', sys.getsizeof(l))

l.append(4)     # Updating list using in-built append method
print('Append 4, List updated to', l)
print('Append 4, List Address', id(l))
print('Append 4, List Memory', sys.getsizeof(l))

l.pop()         # Pop last item
print('Pop, List updated to', l)
print('Pop, List Address', id(l))
print('Pop, List Memory', sys.getsizeof(l))

l += [5, 6]     # Updating list by adding a new list to given list
print('Add new list, List updated to', l)
print('Add new list, List Address', id(l))
print('Add new list, List Memory', sys.getsizeof(l))

Let's look at the output first:

Initial List [1, 2, 3]
Initial List Address 139688582615624
Initial List Memory 88
Append 4, List updated to [1, 2, 3, 4]
Append 4, List Address 139688582615624
Append 4, List Memory 120
Pop, List updated to [1, 2, 3]
Pop, List Address 139688582615624
Pop, List Memory 120
Add new list, List updated to [1, 2, 3, 5, 6]
Add new list, List Address 139688582615624
Add new list, List Memory 120
>>>

Following points we can find out after looking at the output:

Initially, when the list got created, it had a memory of 88 bytes, with 3 elements.
Later on, after appending an element 4 to the list, the memory changes to 120 bytes, meaning more memory blocks got linked to list l.
Even after popping out the last element the created blocks memory remains the same and still attached to list l. And when we add another list with 2 more elements to the given list l, we see that occupied memory is still the same 120 bytes. Hence, we can conclude that it is behaving more like a dynamic array, which updates the size of the list to contain more elements at once to avoid repetitive callbacks (to create and attach new memory address) at every insertion of the new element.
We also notice that the list address remains the same, even if we do append, pop or add a new list to it (as long as we don't assign it to a new variable). This advocates the nature of the mutability of python lists (it means, more or less, that at the same address we are able to update values or extend it further).

Now, let's look at the tuples.

Tuple

Python tuple as we all know are immutable and do not extend in the memory further after the initial declaration. Now, let's look at the above same scenario but for tuples:

# ./tuple_1.py

import sys
t = (1,2,3)     # A new tuple gets created
print('Initial Tuple', t)
print('Initial Tuple Address', id(t))
print('Initial Tuple Memory', sys.getsizeof(t))

t += (5, 6,)     # Updating tuple by adding a new tuple to given tuple
print('Add new tuple, Tuple updated to', t)
print('Add new tuple, Tuple Address', id(t))
print('Add new tuple, Tuple Memory', sys.getsizeof(t))

Output:

Initial Tuple (1, 2, 3)
Initial Tuple Address 140452527817280
Initial Tuple Memory 72
Add new tuple, Tuple updated to (1, 2, 3, 5, 6)
Add new tuple, Tuple Address 140452528572728
Add new tuple, Tuple Memory 88
>>>

Following points we notice after looking at the output:

Since, tuples are immutable, once created cannot be updated unless a new tuple gets added to the previous one, resulting in the creation of a new tuple.
Once a tuple gets created, its allocated memory will not be changed.
Updating a tuple after adding another tuple to it, updates the address of the resulting tuple, this advocated its immutability property.
We can see it allocates 72 bytes for 3 values and 88 for 6 values, it does not allocate the unnecessary amount of memory, only what is required is allocated.

Let's look at some miscellaneous points.

MISC

What happens when we create an empty list and empty tuple?

# ./misc.py

import sys

t1 = ()
t2 = ()

l1 = []
l2 = []

print('List l1 address', id(l1))
print('List l2 address', id(l2))
print('List l1 memory', sys.getsizeof(l1))
print('List l2 memory', sys.getsizeof(l2))

print('List t1 address', id(t1))
print('List t2 address', id(t2))
print('List t1 memory', sys.getsizeof(t1))
print('List t2 memory', sys.getsizeof(t2))

What did we get?

List l1 address 139766851756616
List l2 address 139766851761800
List l1 memory 64
List l2 memory 64
List t1 address 139766871547976
List t2 address 139766871547976
List t1 memory 48
List t2 memory 48
>>>

What did we find?

Creating 2 empty lists have the same amount of memory allocated, whereas the address of both lists is different. It makes sense because, list address will have to act as mutable and cannot be the same at a given time, which means 2 different lists will be different unless they are the same (not in value but in reference point of view).
Creating 2 empty tuples have the same amount of memory allocated and will point to the same address initially. It makes sense because of its immutability property. Consider integer 1 value in python that is immutable and always point to the same address unless changed to some other integer. Similarly, the initial empty tuple will always point to the same address unless changed to some other tuple.

Well, that's it from me.

We explored quite a few points on how memory is allocated and how it could be managed in Python.

To know more about memory management, I encourage you to explore their code available on python's repo.

Feeling it's difficult to go through the source code of python? Don't worry, we can always use available sys module methods and other functions like id to develop an understanding of what could be happening behind the scenes.

Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Shell Script Basics

Siddharth Chandra — Wed, 05 May 2021 10:54:06 +0000

The best thing about GNU/Linux is that it’s packed with utilities that improve productivity immensely. One such utility is a shell that can help accomplish complex tasks with just a few sequences of commands. Most of the time, users operate shell in an interactive way.

However, it becomes really productive when we leverage the scripting capabilities of the shell.

In this article, we are going to explore the basics of shell scripting, how simple echo command can be used in a shell script file, how to write comments, working with variables and conditionals.

Keeping it short and simple, let's get started!

Shell Script Printing

When we are working with shell scripts printing on the output screen is called echoing which can be done using echo.

$ echo "Hello Shell"
Hello Shell

Let's do this inside a shell script file. Follow the below steps:

Create a file, name it whatever you want with the .sh extension. Let's say echo_script.sh.
Add following inside the file - echo "Hello Shell", hit Save and open a terminal on the file location.
On the terminal, write - $ bash echo_script.sh and hit enter, you should be able to see - Hello Shell.

Shell Script Comments

Comments are used to improve the readability of the script. Shell uses the pound # symbol for comments. A line beginning with the pound # symbol is ignored by the shell interpreter during execution.

The script given below shows the usage of the comment:

# Display Hello Shell using echo command of shell, filename.sh
echo "Hello Shell"

Go to terminal and type - $ bash filename.sh (replace filename with whatever name you have given) and hit enter. You should be able to see - Hello Shell without printing anything given after the # symbol.

Shell Script Variables

We can define variables to store information, which can be accessed within the shell script. There are certain rules we need to follow while defining variables:

The variable name can contain any combination of letters from A-Z or a-z, digits from 0-9 or an underscore (_) character.
The variable name should start with a letter or an underscore character.
Variables are case sensitive.

By convention, shell variables are defined in the upper case. Given below is a simple script, which shows the usage of a variable within it:

$ cat hello.sh
NAME="CodeKaro"
echo "Hello $NAME"

cat command is used to display the content of a file given after that. Try your hands on it.

So, we have a file hello.sh which contains a variable NAME which we used in the echo statement as $NAME that is after the $ symbol. Here we saw variable declaration and how inside double quotes a variable can be used.

If you hit - $ bash hello.sh and hit enter, you should be able to see - Hello CodeKaro on the terminal.

Shell Script Conditionals

Like other programming languages, shell supports conditional expressions like – if, if-else and case. Let us understand this with simple examples:

if expression: In case we want to verify whether a file exists or not, we can use the if expression, as follows:

$ touch file.txt #this creates a file

$ cat file_existance.sh
if [ -e file.txt ]; then
echo “file.txt file exists”
fi

The script given above generates the following output when it is executed:

$ bash file_existance.sh
file.txt file exists

if-else expression: Let us modify the above script to generate output when the file does not exist:

$ rm file.txt
$ cat file_existance.sh
if [ -e file.txt ]; then
echo “file.txt file exists”
else
echo “file.txt file does not exists”
fi

Type following and hit enter:

$ bash file_existance.sh
file.txt file does not exists

case statement: Shell provides a switch statement-like functionality using the case statement. Let us understand this with the example given below:

$ cat case_statement.sh
read -p “Enter the day: “ DAY

case $DAY in
Mon) echo “Today is Monday”
;;

Tue) echo “Today is Tuesday”
;;

Wed) echo “Today is Wednesday”
;;

*) echo “Unknown day”
;;
esac

The script given above generates the following output when it is executed:

$ bash case_statement.sh
Enter the day: Mon
Today is Monday

$ bash case_statement.sh
Enter the day: Tue
Today is Tuesday

$ bash case_statement.sh
Enter the day: Invalid
Unknown day

In the case statement, an asterisk (*) denotes a default case.

Well, that's it from me.

We covered printing, variables, comments and conditionals, not too much, right?

Just starting your Open Source Journey? Don't forget to check Hello Open Source
Want to showcase your Python project or just out there to explore? Check Awesome Python Repos
Want to make a simple and awesome game from scratch? Check out PongPong
Want to ++ your GitHub Profile README? Check out Quote - README

Till next time!

Namaste 🙏

Image Credits - pch.vector