DEV Community: Gursimar Singh

The Blockchain Handbook

Gursimar Singh — Sun, 26 Mar 2023 18:54:02 +0000

Blockchain is a new and exciting technology that underpins Bitcoin and other cryptocurrencies.

But there are many other applications for blockchain outside the world of cryptocurrency. And that’s what you’ll learn about here.

This article is going to be a divided into three parts:

Introduction to Blockchain
Embracing Blockchain
How does Blockchain work?

Introduction to Blockchain

In the first part, I’ll introduce you to blockchain: what it is, what it is not, and why it is attracting business interest.

At the end of this part, you should be able to understand:

The Purpose of Blockchain, and Why It Is Attracting Interest
On a Basic Level, How Blockchain Works

Why Blockchain Technology is Useful

Today, many transactions require a trusted third party as a ‘go-between’. For example, users often use banks or organizations, such as PayPal or Western Union, when transferring money to each other.

Such a third party is often required for trust: both parties want to be confident that they will not be cheated, and everything is above board. Sometimes it is also required because of cultural, border, or compliance issues.

However, these third parties usually require a payment for their services and using these third parties can slow down the transaction. Many people would be interested in a solution that does not require such a trusted third party.

Today, many financial transactions pass through some sort of central control or oversight. For example, governments control their currency: how much is produced or circulated. Similarly, governments often control financial transactions such as the purchase of gold or securities.

Many people would likely be interested in a currency or solution that was not controlled by a single authority or government.

There are many applications that could benefit from a database where existing data could never be updated or deleted — only added to.

For example, a solution that traced an aircraft part from manufacture, through testing, and finally onto an aircraft. If existing data could never be updated, anyone using this solution would be confident that they had the correct data, and it had not been changed, corrupted, or deleted.

Blockchain is a technology that has the potential to address these issues and opportunities. But before you can find out how, you need to learn a little about what blockchain is, and what it is not.

What is the Blockchain?

In essence, a blockchain is a way of storing data: it’s like a database. But databases are normally centralized: the data is managed by one process or computer.

Blockchain is a distributed database that is stored on many different computers in a peer-to-peer network. There is no single controlling computer or ‘boss’. This may not seem logical. But one of the most exciting features of blockchain is how it allows this to work. You will see how shortly.

With blockchain, all data–the entire blockchain–is usually stored on all computers. Every computer, or node, communicates with others, broadcasting database changes and other information.

Such a structure provides resilience and reliability advantages: there is no outage if one, or even several, nodes fail.

In a regular database, any data can be changed, updated, or deleted. Blockchain is different: only new information can be added. By its design, existing information can never be updated or deleted.

This may seem strange, but you will see soon how this is actually one of the key security features of blockchain.

So how can a database that cannot be changed or deleted work? Blockchain is often referred to as a ledger-based system. Some call it Distributed Ledger Technology. Changes are recorded as new information.

Distributed Ledger Technology: A technique of storing data on many synchronized, geographically separate computers with no central controlling process. Often used in relation to blockchain.

Let’s look at an example.

Suppose you have an application that tracks the number of widgets in a warehouse.

When the first order of widgets is received, a normal database would create a new record holding the number of widgets.

In Blockchain, a new record is also created, recording that 5 widgets have been received. So, you know you have 5 widgets.

Suppose 1 widget is removed. In a normal database, the record could be updated with the new number of widgets: 4.

In blockchain, a new record would be added, recording that 1 widget had been removed. So, you can find out that 4 widgets remain.

In a regular database, the record could be deleted if all widgets were removed.

In blockchain, a new entry setting the number of widgets to 0 performs a similar function.

What Does Trustless Mean?

The blockchain features we’ve covered so far allow the blockchain to work as a data management system without trusting any of the computers storing the blockchain.

These computers can even exist on public networks with no centralized security management. For example, anyone can create a Bitcoin node and participate in running Bitcoin.

Malicious users cannot change or corrupt the contents of the blockchain unless they can control updates to the blockchain. We’ll discuss this in more depth later in this tutorial.

Removing the need for trusted nodes allows us to implement exciting applications over the internet without a controlling organization. These are sometimes referred to as permissionless. Bitcoin is an example of a permissionless blockchain implementation.

Although blockchain applications can work with a permissionless network, many applications will not want this. They will want to retain some control over their blockchain: who can join, what they can do, and being able to verify their identity.

These solutions will operate in a trusted network, with a security management system maintained by some kind of centralized authority. These are sometimes referred to as “permissioned.” Many blockchain frameworks such as Ethereum and Hyperledger are permissioned.

What does Permissioned / Trusted mean?

Access limited
Centralized security
Generally used on an intranet
Cannot be anonymous

What does Permissionless / Trustless mean?

Anyone can join
No centralized security
Can use internet
Maybe anonymous

Why has blockchain become such a hot topic?

In the past, commercial transactions have been controlled by a central, trusted authority. For example, banks, exchanges, and governments.

Blockchain offers an alternative where a trusted central authority is no longer required to manage or monitor transactions, even for commercial transactions or currency. Also, boundaries and borders are no longer an issue as blockchain applications can be used over the internet.

The first blockchain was proposed by an unknown computer scientist using the name Satoshi Nakamoto in a paper proposing a cryptocurrency, which is now known as Bitcoin.

Since then many different applications, including various cryptocurrencies, have been created based on this technology.

Blockchain is a decentralized, distributed data management system. Data can only be added to a blockchain. It cannot be changed or removed.
Blockchain offers excellent reliability and resilience.
Blockchain can operate over a public untrusted network, and maintain integrity.
Blockchain applications can perform functions such as commercial transactions without requiring a trusted third party or ‘boss.’ This is one of the major benefits of blockchain.
Blockchain was created in 2008, and first implemented for the cryptocurrency Bitcoin.

How Does Blockchain Work?

Blockchain is a very new technology. As such, there are few rules, standards, or best practices to follow when using blockchain. This is exciting, as new applications are developed to use blockchain technology in different ways.

So, the best way to see how blockchain works is to look at current examples. The best known implementation of blockchain is the cryptocurrency Bitcoin.

Clients and nodes:

Bitcoin users are connected over the internet, and are divided into two groups, clients and nodes. Clients are the users of Bitcoin. Nodes are the computers where the Bitcoin blockchain resides, and which participate in the execution of Bitcoin.

Note: A node can also be a client.

Wallet software:

Suppose a user, Jane, wants to transfer one Bitcoin to another user, Brian.

Jane will have a Bitcoin client program executing on her computer. This is called a wallet, and there are several to choose from, including Bither, Electrum, and Mycelium.

Jane will have previously used this wallet to open a Bitcoin account and obtain bitcoins. The wallet software records the bitcoins that Jane has in her possession.

Note that the wallet is not used by the blockchain network to store data. Records in the blockchain are used to determine how many bitcoins can be spent by Jane.

To transfer a Bitcoin, Jane will use the wallet software, specifying the amount — one Bitcoin, and the address of the recipient — Brian.

Every Bitcoin user has a unique address that is created when they create their Bitcoin account. So, Brian will also have a wallet, and will have opened a Bitcoin account.

Jane will send the transfer request. The wallet software will send this request to one or more Bitcoin nodes. The wallet will also sign this transaction with a private cryptographic key that only Jane’s wallet knows.

The transaction will be transmitted to all other nodes. Each node will validate the transaction. This validation will include checking the signature of the transaction with a public key that Jane’s wallet transmits to everyone. Anyone can validate a signature, but only Jane can sign a transaction.

Every node will combine this transaction with others into a block. Periodically, around every 10 minutes or so, every node will try to add this block of transactions to the blockchain.

Blockchain processing will select one node to add newly blocked transactions to the blockchain. How this node is selected can vary between different blockchain technologies. However, this is usually a different node every time.

This winning node will notify all other nodes in the blockchain that it is adding a block of new transactions to the blockchain.

Every node will add this new block to its local copy of the blockchain.

When Jane’s transaction has been added to the blockchain, it is confirmed. Brian’s wallet is notified that he has received one Bitcoin, and Jane’s wallet is notified that she has sent one Bitcoin.

Now we will look at the security of blockchain. All transactions are signed using a private key that only Jane knows. So, like car keys, providing Jane keeps this private key a secret, others cannot perform transactions in her name. No one can transfer bitcoins from her account.

Many nodes validate every transaction. A single node cannot fake transactions, or pass a transaction that is not valid.

Existing blocks of transactions cannot be altered. A node cannot change or corrupt existing transaction data.

Jane’s transaction has been completed securely, without trusting all of the nodes that perform Bitcoin processing. There is no central computer or controlling process — all processing is distributed among the nodes that perform Bitcoin processing.

Anyone in Bitcoin can create a node and start Bitcoin processing over the internet.

Steps:

A transfer request is sent to the blockchain nodes
The transaction is validated by the nodes
Once validated, the transaction will be combined with others to form a block
After a period of time, each node attempts to add the block to the blockchain
Once successful, every node adds the new block to its copy of the blockchain
The receiver is notified of the successful transfer

We have seen one example which is bitcoin. There are many more such examples to explore.

In the next part, we’ll explore blockchain’s effect on the world and how blockchain can be implemented.

Embracing Blockchain

As we’ve been learning so far, blockchain is a new and exciting technology that underpins the Bitcoin digital currency ecosystem and other cryptocurrencies.

Many organizations are only now just beginning to recognize blockchain’s potential and are starting to identify how they can benefit from this technology.

In the previous part, we learnt the basics of blockchain with the example of bitcoin. Now, we’ll explore blockchain’s effect on the world and how blockchain can be implemented.

We’ve just discussed how blockchain offers an alternative to the past where commercial transactions were performed using a trusted third party: a bank, a government, or a broker.

You also saw that blockchain technology can work over the internet, bypassing boundary and border issues. It also can be used for applications that benefit from data that cannot be removed or updated.

Many of the early blockchain applications were cryptocurrencies: online currencies that are not controlled by any country, bank, or similar entity. Bitcoin was the first cryptocurrency, but others include Ripple (XRP), Peercoin, Litecoin, and Ether, which is used by Ethereum.

Cryptocurrencies provide an alternative way for people and organizations to pay for goods and services: one that is not controlled by any central government, bank, or organization.

Cryptocurrencies have also demonstrated that a blockchain application using trustless computers over the internet can provide a secure commercial framework in the real world.

What is a Smart Contract?

Another area of interest in blockchain applications are smart contracts. Not a contract in the normal sense, smart contracts are a set of business rules that are stored and implemented on a computer system. They allow transactions to be automatically enacted without needing a third party, such as a lawyer or bank.

Smart Contract: A set of business rules stored in a computer system that is executed automatically by that computer system based on a set of logic or clauses. Can allow contracts to be agreed and enacted without a middleman such as a lawyer.

Let’s explore with an example:

Suppose a person, Charlotte, is about to rent an apartment for two weeks from a second person: Jackson. Suppose they use a blockchain application that has been created to manage apartment rentals as a smart contract.

The terms of the rental agreement are coded in the smart contract: rent, start date and other details.

Charlotte pays the rent in a cryptocurrency such as Bitcoin. Jackson provides the code to access the apartment in the smart contract.

At a defined date and time, the smart contract automatically verifies that all terms of the contract have been met: rent has been received, and apartment entry code provided.

Once all terms are met, the smart contract automatically forwards the rent payment to Jackson, and the entry code to Charlotte.

If any terms of the contract are not met — for example, the rent has not been received — the contract is not enacted. No entry code is forwarded to Charlotte, no rent is forwarded to Jackson.

The contract is stored on many computers, and cannot be deleted or changed. Only new information can be added. Jackson and Charlotte are confident of the integrity of the information in the contract, and that it can be trusted as a valid document.

This is all done without a trusted central authority, such as a rental agency, managing the contract.

Applications of the Blockchain

Blockchain is still a relatively new technology, but there are some interesting applications that have already been implemented across many industries.

Bitspark: Bitspark is a remittance platform. It allows people to transfer money to each other. There are several similar applications including Abra and Veem.

IBM Food Trust: IBM Food Trust is a blockchain-based solution to track and manage food products as they pass from source to customer. This allows retailers, wholesalers, and suppliers to improve efficiency, trace origins of products that may not match requirements, and reduce food loss and waste.

Boeing: Boeing has used blockchain technologies with the Internet of Things (IoT) for tracking and maintenance of aircraft parts. This allows them to predict maintenance events, optimize production operations, and extend the lifecycle of components.

Arcade City: Arcade City is a peer-to-peer ridesharing application that uses blockchain.

Storj: Storj is a cloud storage solution. Users can rent out unused space on their local disks, or rent cloud based storage using these local disks.

The two major features of blockchain — that it is distributed, and that data cannot be updated — provides many interesting opportunities for businesses. In many ways, blockchain technology can be used to decentralize and record almost any transaction or movement.

A couple more interesting examples include,

Publishing music: In 2015, the startup uj0 Music released the song Tiny Human by singer-songwriter Imogen Heap using a Proof of Concept blockchain-based distribution system. Using this system, users immediately downloaded and paid for the song. Royalties were sent using blockchain immediately to the songwriter and collaborators. This showed how music could be released without the current music industry labels and publishers.

Human trafficking: One-fifth of the world’s population does not have an official document to prove their identity. Without legal identification, these people are at risk of human trafficking.

The United Nations is exploring ways that blockchain solutions can resolve this. Blockchain solutions are also being developed for online identity management and verification.

There is no doubt that blockchain technology has generated more than just interest. Most large IT organizations, including IBM, have positioned blockchain as a strategic technology. Many large organizations are actively pursuing blockchain solutions and there are many startups working with blockchain.

Blockchain has captured the interest of many organizations, from small startups to large financial institutions.

Many claim that blockchain technology is about to revolutionize the way business is done, while others are more skeptical. But most technology commentators agree that blockchain is a valid and valuable technology that is here to stay.

Here are some comments from a couple big players:

Like any emerging technology, the potential implications of blockchain seem both vast and at the same time not fully comprehensible. Many in the popular press have speculated about blockchain-driven notions as diverse as the development of borderless global currencies to the creation of fraud-proof voting tools to help eliminate electoral corruption and fraud. We may not yet even fully comprehend blockchain’s “killer app.” For corporate finance professionals, however, the potential implications of blockchain are more tangible. — J P Morgan Chase

And,

Even though the initial hype surrounding blockchain, and the prolonged blockchain “winter” that followed, are beginning to settle, this is no time to rest on our laurels. IT leaders must prepare for the inevitable blockchain “spring” on the horizon, bringing with it core-enabling technologies and significant opportunities for digital business. — Gartner

Large organization projects:

Forbes has recently created the Forbes Blockchain 50: a list of 50 companies in the US with minimum revenues or valuations of one billion USD actively exploring blockchain.

Samsung: Samsung is using a blockchain-based solution to overhaul how its battery manufacturing subsidiary manages contracts.
Nestle: Swiss based Nestle have commenced working with IBM Food Trust to manage and trace the lifecycle of food products.

In summary so far,

Many of the early blockchain applications have been cryptocurrencies such as Bitcoin.
Smart contracts are another interesting feature of blockchain that is being investigated.
Although there is disagreement as to how blockchain will be implemented, and its long-term effect on the computing industry, most agree that blockchain is an important and valuable technology that is here to stay.
Existing blockchain applications include shared storage, cryptocurrencies, and payment processing systems.
Some examples of potential future blockchain applications include land registration, identity management, and tracking of physical items such as diamonds.
Large organizations are actively investigating and investing in blockchain.

How to Get Started with Blockchain

There are many blockchain frameworks and tools designed to allow organizations to create their own blockchain solutions without starting from scratch.

Ethereum: One of the first blockchain frameworks created in 2014. It’s an open-source framework including the cryptocurrency Ether. It’s currently one of the most popular frameworks.

Hyperledger: An umbrella project of open source blockchains. It was started in December 2015 by the Linux Foundation, with input from companies such as IBM and Intel.

Quorum: An open source private blockchain framework released by JPMorgan Chase & Co.

In this article, we will learn more in detail about Hyperledger.

Hyperledger Fabric is a framework for private blockchains. It is not designed to operate as an open, permissionless blockchain like Bitcoin.

To achieve this, all Hyperledger Fabric actors — including client applications, administrators, processes within the blockchain, and more — must have a valid digital identity: an X.509 digital certificate. One or more external Membership Service Providers (MSP) will create, manage, and authenticate these certificates. Hyperledger Fabric provides instructions on what is required from MSPs.

Each actor has a set of permissions defined: what they can and cannot do. The digital identity and permissions together are called principals. Hyperledger Fabric includes functionality to use and manage principals.

In addition to identity and permissions, Hyperledger Fabric provides rules on how to create a blockchain These rules are backed by code, APIs, and documentation.

How to identify a user
How to setup permissions
How to setup a network
How to create a smart contract
How to create and submit a transaction
How to respond to a transaction
How a client connects to the blockchain
How to handle errors

For example, a key function of Hyperledger Fabric is the creation and operation of smart contracts. Hyperledger Fabric provides a set of rules for creating smart contracts.

Smart contracts are programs written in a supported chaincode.

Hyperledger Fabric provides libraries and APIs that can be called to define the smart contract.

package com.blog.chaincode.example

import java.util.ArrayList;
import java.util.List;

import org.hyperledger.fabric.contract.Context;
import org.hyperledger.fabric.contract.ContractInterface;
import org.hyperledger.fabric.contract.annotation.Contact;
import org.hyperledger.fabric.contract.annotation.Contract;
import org.hyperledger.fabric.contract.annotation.Default;
import org.hyperledger.fabric.contract.annotation.Info
import org.hyperledger.fabric.contract.annotation.License;
import org.hyperledger.fabric.contract.annotation.Transaction;
import org.hyperledger.fabric.shim.ChaincodeException;
import org.hyperledger.fabric.shim.ChaincodeStub;
import org.hyperledger.fabric.shim.ledger.KeyValue;
import org.hyperledger.fabric.shim.ledger.QueryResultsIterator;

public class final Example implements ContractInterface {
...

Hyperledger Fabric is designed for enterprise-level blockchains that handle a large number of transactions that must be processed quickly.

Unlike other frameworks, such as Ethereum, Hyperledger Fabric does not provide its own cryptocurrency.

What is Hyperledger Fabric comprised of?

Hyperledger Fabric provides libraries, APIs, and documentation that can be used to implement a blockchain solution. These resources provide services including commands, RESTful interfaces, and performance metrics.

Hyperledger Fabric provides libraries that are installed onto each computer: clients and nodes.These can be on UNIX, MacOS, or Windows machines.
Hyperledger Fabric provides a suite of APIs to assist in creating and running a private blockchain.These APIs are included in software development kits (SDKs) for programming languages includingGO, Node.js, Java, and Python.
Hyperledger Fabric provides samples to allow users to learn about Hyperledger Fabric, and to get started in implementing it.
Hyperledger Fabric provides documentation and tutorials for administrators, operators, and application developers.These explain the rules that Hyperledger Fabric has for implementing a blockchain solution.
Hyperledger Fabric provides line commands that can be used to manage blockchain nodes and Hyperledger Fabric resources.
Hyperledger Fabric executables provide an HTTP RESTful operations API. This is designed to be used for operations: not administrators or users.
Hyperledger Fabric provides performance data that can be processed using Prometheus monitoring software.

Blockchain frameworks often include additional features not common in traditional blockchain solutions.

Hyperledger Fabric provides services allowing private, confidential transactions processed by the blockchain, to be hidden from some nodes, using private channels.

Hyperledger is an umbrella project of the Linux Foundation. It includes several blockchain-related projects, including five blockchain frameworks.

The features of each blockchain framework vary: some may have features that others do not.

There are tools available, including some within Hyperledger, that can be used with these blockchain frameworks.

Benchmark: Tools such as Hyperledger Caliper allow blockchains to be compared. These tools measure the performance of a blockchain implementation using pre-defined test cases.

Operation and management: Tools such as Hyperledger Cello and Truffe provide features to manage the lifecycle of a blockchain: creation, start, stop, removal. These often include monitoring, logging, health, and analytics features.

View and Query: Sometimes administrators must view individual blocks in a blockchain, and their associated transactions, network information, and more. Tools such as Hyperledger Explorer provide graphic interfaces to do this.

Performance: As networks get larger, blockchain solutions slow down. Some tools provide features to improve the performance of blockchain frameworks or solutions. For example, the Raiden Network attempts to improve the performance Of Ethereum blockchain solutions by avoiding consensus. This is done by performing some transactions outside of the blockchain.

Blockchains in the Cloud

There are some cloud-based blockchain frameworks available. Some of these are part of large cloud offerings, while others are standalone blockchain frameworks.

Some Blockchain as a service (BaaS) providers are:

Microsoft Azure
Oracle Blockchain Platform
Alibaba Cloud
IBM Blockchain Platform
Amazon Web Services

Summary so far:

There are currently no official standards or best practices for blockchain. But some are being developed by standards organizations such as ISO and W3C.
Although it is possible to build a blockchain solution from scratch, many will choose to use an existing framework such as Hyperledger Fabric or Ethereum.
There are other blockchain-related tools to do things such as manage blockchain solutions, improve their performance, view individual blocks and related resources, or benchmark their performance.
There are cloud-based blockchain frameworks: Blockchain-as-a-Service (BaaS).
There are many blockchain books and courses available. Open-source blockchain frameworks also provide documentation, and their source code can be viewed.

In the next and final part, we’ll be looking at what hashes and blocks are, and how they work in a blockchain. Also, we’ll discuss how multiple computers work together to store and manage a blockchain.

How Does Blockchain Work?

So far, I’ve introduced you to blockchain and described what it is, what it does, who uses it, and why it is fast becoming an important component in today’s business world.

This is the third and final part of the handbook where we’ll dive more into the technical details of how blockchain works.

Blockchain is an exciting technology that has the potential to revolutionize banking, electronic transactions, electronic contracts, and much more. It relies on technology to ensure that messages cannot be tampered with, are correct, and are sent by authorized people.

In this part, you will explore the actual technology in more detail.

By the end, you should be able to understand:

What Hashes and Blocks Are, and How They Work in a Blockchain
How Multiple Computers Work Together to Store and Manage a Blockchain

What are Hashes?

An essential tool used by blockchain is the hash. A hash is a string of numbers and letters that can uniquely identify some text or data. So, in many ways, it is like a fingerprint.

Hashes are sometimes called hash values, hash codes, digests, or simply hashes. There are several hashing algorithms, or ways of creating hashes.

Hash: An algorithm used to map data of variable size to a set of data of fixed size. Often used in computer applications for lookup tables, and in cryptography. SHA-256 is one of the algorithms.

Hashes used by blockchain are not simply data generated from other data. They have some distinctive properties that are very useful for blockchain.

Hash Properties:

Same length: A hash is the same length — regardless of the length of the original text. Even if hashing the entire Library of Congress, the hash will be the same length.
Different: Any change in the original data, no matter how small, must produce a completely different hash.
Unique: Like a fingerprint, a hash should be unique. This is not possible in practice, but hashing algorithms make it almost impossible for a collision — a case where a program will encounter the same hash for different data. The probability of a SHA-256 collision is 4.3 * 10–60: almost impossible.
Cannot get original data: It should not be possible for the original data to be reproduced from a hash value.

What are Blocks?

In blockchain, information is stored in blocks. Each block includes a hash of the block’s data or contents.

So, a receiver can confirm that the contents of the block are valid by recreating the hash from the block contents, and comparing it with the hash included in the block.

This is the first step in blockchain security.

A blockchain is exactly that: a chain of blocks, each with a hash of the block’s contents included.

This chain provides a second layer of security.

A block is the basic entity in a blockchain.
The number of blocks in a blockchain can be very large.
Every block in a chain includes the hash of the previous block.
The previous block’s hash is included in the data used when creating the hash for a block.
If the data in a block is modified by an unauthorized person, the block’s hash will no longer match it.
The unauthorized person could recreate the hash in the original block, however, this would not match the hash in the next block.
The unauthorized person could try to update the hash in the next block, but then the next block’s hash would not match.
Even if the unauthorized person updated the second block’s hash, it would not match the hash in the third.
Therefore, if an unauthorized person updated a block, they would also have to update every later block in the blockchain.

Because no block is ever updated or deleted from a blockchain, blockchains are often referred to as ledgers. This ledger system can be used as a database system.

For example, a banking solution could use a blockchain to record deposits and withdrawals from a bank account.

When referring to a block, blockchain solutions usually use the hash of the block. An alternative is to use the block height, sometimes called the block number.

Blocks in a blockchain can store anything required by the blockchain solution. For example, the cryptocurrency Bitcoin stores multiple transactions in a block.

For an unauthorized person to change data in a block, they would need to recreate the block holding the data, and every later block in the blockchain.

Blockchain solutions prevent this from happening by storing the blockchain on many different computers at the same time. So now an unauthorized person must change blocks on every computer storing the blockchain. Blockchain is a distributed solution, not a centralized one.

Many blockchain solutions use hundreds, or even thousands of different computers (referred to as nodes) to store the full blockchain. Each computer stores the entire blockchain. Because each node stores the entire blockchain, each must update its own copy of the blockchain every time a new block is added.

The nodes must work together, following the same rules. For example, when a block is added to the blockchain, all nodes must add the same block at the same height in the blockchain.

Blockchain implementations will all have a system or method that determines which node will add the next block to the blockchain. This prevents more than one node from adding a block at the same time.

One way of doing this is to slow down the rate at which blocks are added to a blockchain. For example, currently a new block is added to Bitcoin every 10 minutes, and Ethereum every 15 seconds.

To see how many computers work together in a blockchain, let’s look at an example: Bitcoin.

When a Bitcoin node wants to submit a transaction, it transmits that transaction to all nodes in the network.
Full Bitcoin nodes are nodes that hold a copy of the blockchain and are prepared to update it as necessary.
Every full node validates the transaction. If the transaction is valid, every full node adds the transaction to a transaction pool, or memory pool. This is a group of ready transactions that have not been added to the blockchain.
Every full node then adds transactions from the transaction pool into a candidate block, and attempts to add this block to the blockchain.
All full nodes are trying to add the next block to the blockchain, but only one can be successful. So, all full nodes compete to add the next block — it is a race to create the next block.
The node that wins the race adds the block to the end of its local copy of the blockchain. Transactions in this new block are removed from the transaction pool.
All other full nodes are notified that there is a winner. Each adds this new block to their local copy of the blockchain. All transactions in this new block are removed from all local transaction pools.

This race between Bitcoin nodes prevents two nodes adding a block at the same time to the end of the blockchain. How blockchain determines who will add the next block is called consensus. Consensus also prevents the same transaction from being in two different blocks in the blockchain.

Bitcoin nodes compete using a system of consensus called proof of work.

With proof of work, every block includes a value that is not used for anything other than the proof of work validation. This is called a nonce, and is set to zero when a block is created.

Before a block can be submitted, the hash of the block must meet restrictions set by the network. Normally, this is a target value that the hash must be below.

If the hash is not below the target, the computer must try again. It does this by increasing the nonce by 1, recalculating the hash, and trying again. This is repeated until a hash is found that is less than the target. This process is called mining.

The computer has provided proof that it has done some work. Now the block will be added to the blockchain in all computers in the network.

Blockchain networks deliberately make it difficult for a computer to find a hash that meets the target. So, it can take a computer several minutes, or even hours or days, before a suitable hash is found.

Because it takes so long, it is almost impossible that two computers will attempt to add a block to a blockchain at exactly the same time. Computers in the blockchain network simply accept the blocks in the order they are received.

Using a nonce is not the only way proof of work can be used. The blockchain network can set any puzzle to be solved by a computer, with the aim of slowing down the rate at which a computer can add blocks.

A major criticism of proof of work is that it wastes computer processing power — forcing computers to do millions of calculations for no reason except to slow the rate that blocks are added.

There are alternative consensus algorithms. One is proof of stake. This method is used by the Peercoin cryptocurrency, and is being developed for Ethereum.

Every node has a share or stake in the network. Some will have more, others less. For example, cryptocurrencies could base this on coins unspent for a period of time: the more unspent coins, the larger the stake.

The computer submits a block to the network. In this block, the computer provides a key to identify itself, and its stake in the network. The network will have a mechanism whereby the next computer that can add a block is determined by all computers in the network.

This mechanism will determine how often blocks can be added. This mechanism will also determine the next node that can add a block. The bigger the stake, the more likely a computer can add a block.

A computer will try to add a block. If unsuccessful, it can try again. If it is this computer’s turn, the block is added.

Proof of work and proof of stake are consensus solutions for trustless blockchain networks: networks where there is no central control, and anyone can join. Bitcoin and Ethereum are two examples of trustless networks.

Achieving security using a network of trustless computers is one of the major strengths of blockchain.

Resolving consensus for internal, or trusted, blockchain networks is much easier. For example, a simple solution could be to give each computer a turn, one after the other — like a round-robin.

In practice, such a round-robin consensus is not practical. However, other examples such as Raft are possible in a permissioned blockchain.

How does Raft consensus work?

Each node begins as a candidate: eligible to become the leader.
Each node votes for one node to become the leader.
One node wins and becomes the leader. All other nodes become followers.
The leader adds all blocks.
If the leader shuts down or fails, a new vote is taken to select a new leader.

Conclusion

We’ve finally come to the end of this handbook. I hope this has helped you understand and get started with blockchain.

I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorse me for relevant skills on LinkedIn.

Till the next one, stay safe and keep learning.

Locked Out of Your RHEL 9 System? Here's How to Reset Your Root Password and Regain Access in Minutes

Gursimar Singh — Sun, 19 Mar 2023 20:19:50 +0000

The root password is an essential aspect of any Linux system. It provides administrative privileges to the user, allowing them to modify system files, install software, and perform other critical operations. As such, it is essential to keep the root password secure and to remember it at all times. However, in some cases, users may forget their root password, or the password may be changed without their knowledge, leading to a locked-out system.

Fortunately, RHEL 9 provides a method for resetting the root password, even if you do not have the original password.

In this, we will be looking at the procedure to reset the password for the root user in a hands-on manner, with all the illustrations and explanations for the same. We will also look at how it differs from RHEL 8.

We'll cover the following

● Preface

● How to reset it?

● The procedure

● Explanation

● Conclusion

Preface

Let's take a look at an overview of the most important phases that are taking place in the Boot Procedure right now. That brings us to firmware, the first step in the process. The software that runs on your computer is called firmware. Additionally, the firmware will be the source from which the boot device is allotted. The hardware that allows your computer to start up is called the hard drive. And on the device that boots, you will find something called a boot sector, and then immediately after that, you will find the GRUB program, which is the most essential component. The bootloader that is being used is called GRUB. Additionally, the GRUB boot loader is responsible for loading the Linux kernel into memory. In addition, the Linux kernel is always accompanied by a companion program known as the initramfs or the initrd. Both of these terms may be found in the terminology; as they refer to the same thing, you can choose whichever one you like. It is at this point that everything will begin to take place, and here is where systemd will come into play. Everything is managed by Systemd. Nothing else matters. You will also discover an early stage in systemd; for the sake of simplicity, we will refer to this step as the early stage. After the early stage, you will arrive at your services. And after these services have been loaded, or even better, while they are being loaded, a shell will be dropped, and you will be able to log in to that shell once it has been dropped. Why is it necessary for you to be aware of this information in order to do problem-solving tasks?

How to reset it?

This comes in handy in the event that you need to access a server for which you do not know the root password. I am going to walk you through the process now.

The procedure

So how exactly does that function? So, let's go over the steps, shall we?

1. Enter the Grub menu while booting

2. Find the line that loads the Linux kernel and add init=/bin/bash to the end of the line

3. mount -o remount,rw /

4. passwd root

5. touch /.autorelabel

6. exec /usr/lib/systemd/systemd

Explanation

You will need to visit the Grub boot menu while the operating system is booting up in order to troubleshoot the forgotten root password.

Now that we've reached this point, let's have a look at the Grub boot menu. Simply pressing the letter “e” will bring about the desired modification.

After that, scroll down until you reach the end of the line, where it says Linux. This line is spread over three lines, and you need to be right here, just after the "lv=rhel/swap" part of the line. Next, you will enter "init=/bin/bash" which will really ensure that you start with bash as the init process rather than systemd. This will be done by preventing systemd from running when the command is executed.

Please take note that there has been a modification to the method. If you were using REHL 8, you would use rd.break at the end of the line, but doing so will prompt you for the root password. Therefore, there is no use in doing it. init=/bin/bash is the command that you should be using. That will provide access to the system, despite the fact that the root file system is read-only.

Then press the Control key and the X key simultaneously, to begin with this particular choice; you will then be placed in an empty shell.

The result of it, as you can see, is that you are dropped into a root shell. Let's see whether you're able to put pen to paper now. I can already tell you that you won't be able to do it. This is due to the fact that the root filesystem is only accessible in read-only mode during this very early stage of the booting process. That being the case, we need to ensure that it can be read and written.

We can make that read-writeable by using the command,

# mount -o remount, rw /

Now to reset the root password you can just use the command,

# passwd root

Ignore any messages that say there was a problem with the fields or the dictionary check. What you do need to see is that all authentication tokens are updated successfully.

The next thing that we have to do is make sure that SELinux is working properly. This is a very crucial step to perform since it will ensure that SELinux is aware of all that you have accomplished up to this point. As a result of SELinux, the next thing that you do is execute the touch command in order to produce an empty file with the name autorelabel. This is going to ensure that all of the security labels in SELinux are going to be successfully restored.

# touch /.autorelabel

After that, you will need to make use of the exec command, "exec /usr/lib/systemd/systemd". It is clear that numerous processes associated with systemd are already running, and the file /usr/lib/systemd/systemd is the one you require. And in case you are wondering, can't I just exit? Well, the thing is no. This is the only method to get you into a running system because, if you leave from /bin/bash, your system will crash since it doesn't grasp that it doesn't have an interface anymore. Because of this, exiting from /bin/bash is the only way to get into a running system. The systemd process is going to take the place of the one that is now running as a result of this change.

To proceed with the standard boot routine, you need to start the systemd service. Normally, whenever you write something into the shell, it will split off into its own process. This indicates that the new process will begin its life as a child process of the environment in which it is currently running. However, systemd has made it clear that it does not wish to be anyone's kid in any way. Systemd has to act as the parent process for all of the other processes. Because of this, systemd must be started in a rather unconventional manner by using the exec command: exec. " # exec/usr/lib/systemd/systemd". This will cause your currently running process to be discarded, and systemd will be installed in its stead in order to ensure that the boot sequence may proceed correctly.

Right now, the program selinux-autorelabel should be visible in the exact centre of the screen. That constitutes a significant portion. Now, selinux-autolabel is going to require a little bit of time. It shouldn't take more than a couple of minutes to finish up.

If the process is taking too long, you may just restart your virtual machine. Let's give it some time. As you can see, nothing went wrong, and everything turned out nicely. Well, did it? The system is now starting up, and although it may take a minute or two before it is fully operational again, a login prompt will be on the screen in a very short amount of time.

Just now, we have successfully changed the root password. It goes without saying that we still need to conduct tests. The non-root user is unaffected by the root password in any way. Additionally, the non-root user has sudo rights because they are a user with administrative access. Because "su -" prompts the user for the root password, we will not be using sudo this time in order to test; rather, we will be using su. This is due to the fact that sudo prompts for the password. In addition to that, I have access at the root level.

So mission accomplished. This is how we can change the root password.

Conclusion

In conclusion, forgetting or losing the root password in RHEL 9 can be a frustrating experience, but it doesn't have to be a catastrophic one. By following the steps outlined in this article, you can reset your forgotten root password and regain access to your system in just a few minutes. Remember to keep your root password secure and choose a strong and unique password to prevent any future access issues. With this comprehensive guide at your disposal, you can be certain that you'll be able to master RHEL 9 root password reset and maintain the security of your system.

Pervasive Encryption (Cryptography)

Gursimar Singh — Mon, 23 Jan 2023 04:26:55 +0000

In 2017, there were a number of documented data breaches, in which sensitive or confidential information was made available to the general public. According to the findings of the 2017 Cost of Data Breach Study conducted by the Ponemon Institute, there are approximately 58 data records that are stolen every second, with the average cost per record being $141 USD.

Encryption of data is a critical component of any comprehensive defense strategy against data breaches. Encryption of data is becoming an essential need for several compliance requirements, such as the Payment Card Industry Data Security Standard (PCI DSS) and the General Data Protection Regulation of the European Union (GDPR).

In the past, a popular strategy consisted of encrypting only the data that needed to be encrypted and only in the locations where it needed to be encrypted.

Therefore, you are only permitted to encrypt client data when it is being sent from system to system. Alternatively, individual files or databases containing financial information.

There is also an option known as ubiquitous encryption, which can be used instead of selective encryption. This implies that all of the data is encrypted both while it is stored and while it is being transferred. This includes data that is stored on physical media such as discs and tapes, as well as data that is stored in files and databases and data that is being sent across network connections.

The use of pervasive encryption is one solution that has been suggested as a method to improve compliance and data security.

Why use Pervasive Encryption?

There are a lot of benefits that come along with using pervasive encryption. For example, given that every piece of data is encrypted, there is no possibility that any critical information may be overlooked inadvertently and then either saved or transferred without encryption.

Another significant advantage of pervasive encryption is that compliance verification is simplified by the fact that the technology encrypts all of the data. Finally, malicious users cannot target encrypted data knowing that it contains sensitive data because the data is hidden from them.

If pervasive encryption is so good, why isn’t everyone using it?

In most cases, this is due to the fact that it is either difficult or prohibitively expensive. To encrypt different types of data, such as files, databases, physical discs and tapes, and network connections, for instance, different encryption algorithms are used by different types of systems. These things need to be configured independently, and frequently in a manner that is unique to each resource.

The performance of encrypted data can suffer, and it can lead to an increase in the amount of CPU time that is needed. Modifications to the application’s source code can be necessary for certain circumstances.

In conclusion, the majority of websites do not believe that it is necessary to encrypt everything.

In 2017, IBM announced the z14 IBM Z mainframe. As part of this announcement, IBM introduced its pervasive encryption solution. This solution proposed encrypting all data at-rest or in-flight using new and existing hardware and software features.

This solution enabled the encryption of data at the physical media, file or database, coupling facility, and network levels.

The IBM pervasive encryption solution uses existing network encryption

features such as SSL.

The IBM pervasive encryption solution uses existing encryption features of

disk hardware such as the IBM DS8000. This encryption protects data if a

third party gets access to the physical disk media. However, it does not

protect against applications or users accessing files.

Many database systems including Db2, IMS, and Oracle provide features

to encrypt individual databases, tables, and even columns. The IBM

pervasive encryption solution relies on these existing database encryption

features.

The IBM pervasive encryption solution uses existing encryption features of

tape hardware devices such as the IBM TSI 155. This encryption protects

data if a third party gets access to the physical tape. However, it does not

protect against applications or users accessing files.

IBM's pervasive encryption is not entirely a new offering, in that it utilizes several encryption features that have been available for some time. However, the IBM Z mainframe includes features that reduce the overhead of encryption. This together with other features makes pervasive encryption more viable.

Pervasive encryption aims to encrypt all data at-rest. Traditionally you might think of files, databases, disks, and tapes as such at-rest data. It also aims to encrypt data in-flight. Or in other words, data in networks. It also supports support standard network encryption including SSL, TLS, and IPSec.

IBM provides a free library to implement the IBM Common Cryptographic Architecture (CCA). This includes APIs for C and Java applications.

The Linux libica library provides APIs for programs requiring cryptographic services. This library is supported by the icatools command that can show cryptographic information and statistics.

The open source openCryptoki library can be used to implement the PKCS #11 cryptographic standard on Linux.

A special Logical Partition (LPAR) type called a Secure Service Container (SSC) can be created in IBM Z mainframe systems. SSC partitions contain an operating system — usually Linux, middleware, and applications. One SSC cannot access any resources in another. SSCs are defined and then deployed as standalone appliances.

SSC files are encrypted for at-rest security. No direct access to the SSC is possible. Access is via Remote APIs only. Diagnostic data, and dumps, are encrypted as well.

So, there are different options for encrypting data for at-rest and in-flight. But, will we use all? No. For example, we probably won’t use both SSL/TLS and IPSec.

So, what option should we choose?

Well, it depends. To be blunt, there’s no simple solution. No easy answer as to which encryption solution is best.

It’s difficult to recommend one or a combination. Here’s a toolbox,

• Full Disk/Tape Encryption • Dataset Encryption • Coupling Facility Encryption • SPOOL Encryption • Database Encryption • Other Encryption • Channel Encryption • SSL/TLS • AT-TLS • JSSE • IPSec • SSH • VTAM Encryption

It’s up to us which tools to utilize and when. So, here are some tactics when you’re starting to look at pervasive encryption, and need to figure out which of our tools to use.

First, ask yourself why you’re encrypting. For example, if you’re encrypting to satisfy a compliance requirement, then you may not need coupling facility encryption to achieve this.

Next, list the encryption options you have like dataset encryption. You may also have other products or features that can be used.

Third, triage your data: most sensitive/important to least sensitive. Some will be very important to encrypt, others not so much.

Now, you can choose your solutions to achieve the encryption you need. You’ll probably start with data that needs encrypting the most, and move out from there.

In summary,

Encryption is becoming essential for data security and compliance.
Most encryption is selective, encrypting only what is required and where.
Pervasive encryption encrypts all data, at-rest and in-flight.
Pervasive encryption provides added security, together with simplified compliance verification.

Serverless Kubernetes on Google Cloud Platform

Gursimar Singh — Sun, 22 Jan 2023 04:13:39 +0000

Kubernetes is gaining a tremendous degree of popularity and is being adopted by all organizations, despite the fact that some are far larger than others. Now, deploying Kubernetes necessitates the recruitment of qualified software developers. It is possible that organizations, particularly bootstrapped start-ups and small-scale ones, would view this as an unnecessary overhead expense. Let’s look at the several ways that this problem may be solved.

Serverless computing: why bother?

The concept of serverless computing has been a topic of considerable interest for some time, and it is not likely to go out of favour any time in the near future. When developers are relieved of the task of managing infrastructure, they are better able to focus their attention on developing and refining the product, which speeds up the process of bringing it to market.

Cloud Run is a serverless platform that is powered by Knative, a runtime environment that extends Kubernetes for serverless applications, and the Functions Framework. Cloud Run was developed by Google. Google is the company that was responsible for developing Cloud Run. Cloud Run enables us to package our already-existing code in a Docker container, in contrast to other serverless services, which require us to deliver code that was specifically created to operate as a function and be activated by events. This is because other serverless services require us to deliver code that was specifically created to operate as a function and be activated by events.

This container is able to work in the fully controlled serverless environment that Cloud Run provides, but because it utilises Knative, it can also run on Google Kubernetes Engine. This enables us to add pay-per-use, on-demand code to your current Kubernetes clusters and gives us the flexibility to do so. Cloud Run is a fully managed environment for serverless computing that you may access over the cloud. Even though it is not always a fully-fledged solution, it does make it possible to run serverless apps on Kubernetes.

In addition to a runtime environment, the open application programming interface (API) that Knative offers is also available. This enables us to run your serverless applications anywhere we see fit, including completely managed on Google Cloud, on Anthos on Google Kubernetes Engine (GKE), or on our very own Kubernetes cluster. Knative makes it easy to get started with Cloud Run, then switch to Cloud Run for Anthos, or to get started with our own Kubernetes cluster and then move to Cloud Run. Both of these options are available. These two courses of action are both up to consideration on our end. Because we are utilising Knative as the basis upon which everything else is built, we are able to migrate workloads between platforms without having to pay large switching charges. This is made possible by the fact that we are using Knative.

Some of the major benefits of Cloud Run:

Rapid autoscaling
Split traffic
Automatic redundancy
No vendor lock-in

Cloud Run requires the code to be a stateless HTTP container that creates an HTTP server within four minutes, or two hundred and forty seconds, of receiving a request, replies to the request within the request timeout, and is compiled for 64-bit Linux. It is necessary to set the listening port to 8080, however, the actual port number should not be hardcoded.

Take note that, by default, all Cloud Run services have a consistent HTTPS endpoint with TLS termination taken care of for us.

Some of the use cases include:

· Websites

· REST API backend

· Lightweight data transformation

· Scheduled document generation, such as PDF generation

· Workflow with webhooks

As we have covered the necessary ground to get started with Cloud Run, let’s move forward and see how to deploy a container to Cloud Run

Before getting started, make sure the appropriate Google Cloud Project is activated and billing is enabled for the Google Cloud Project.

There are multiple ways to deploy to Cloud Run:

Deploying a pre-built container
Building and deploying a container from source code

We can create both services and jobs using Cloud Run.

Deploying a pre-built container

Go to Cloud Run either from the navigation menu or from the search menu.

2. Click Create service

o Select Deploy one revision from an existing container image.

o We can either provide our container image URL or click Test with a sample container.

o In the Region pulldown menu, we need to select the region where we want the service to be located.

o Under Authentication, we will select Allow unauthenticated invocations. We can modify permissions as per our use case.

o Finally, click Create to deploy the container image to Cloud Run and wait for the deployment to finish.

Note: We can configure CPU allocation and autoscaling as per need. We can also specify the security settings along with referencing secrets.

Conclusion

Even though Kubernetes could appear to be complicated, serverless technologies like Cloud Run might be used to simplify and expedite the process of designing software applications.

Apache Kafka — The Big Data Messaging tool

Gursimar Singh — Sun, 22 Jan 2023 03:50:22 +0000

Apache Kafka was built for real-time. In programming we consider everything to be events. Now, events have states as well. But the primary idea is that the event is an indication in time that the thing took place.

Now it’s a little bit cumbersome to store events in databases. Apache Kafka was built for real-time. In programming we consider everything to be events. Now, events have states as well. But the primary idea is that the event is an indication in time that the thing took place.

Now it’s a little bit cumbersome to store events in databases. Instead, we use a structure called a log. And a log is just an ordered sequence of these events. An event happens, and we write it into a log, a little bit of state, a little bit of description of what happens.

Logs are really easy to think about, and they’re also easy to build at scale. Historically, this has not quite been true of databases.

Apache Kafka is the system that is responsible for maintaining logs. It refers to them as topics, which is a very traditional phrase. A theme is nothing more than an organized list of occurrences.

What precisely is Kafka?

Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can manage a high amount of data and that enables you to transmit messages from one end-point to another. These features allow you to pass messages from one end-point to another. The reading of Kafka’s messages can be done so either offline or in an online setting. In order to avoid any data from being lost, Kafka messages are stored on the disc and duplicated across the cluster. The synchronization service ZooKeeper serves as the foundation upon which Kafka was constructed. For the purpose of performing real-time streaming data analysis, it interfaces very well with Apache Storm and Spark.

Okay. Now, what is a Messaging system?

The sharing and transmission of data between applications is handled by a messaging system, which enables the programs to focus solely on the data itself rather than being sidetracked by the sharing and transmission of data. The core component of distributed messaging is message queuing that can be trusted. Client applications and the messaging system engage in asynchronous message queuing amongst one another. There are two distinct patterns for conveying information. The first type of messaging system is known as a point-to-point messaging system, while the second type of messaging system is known as a publish-subscribe, also knows as pub-sub, messaging system. The pub-sub architecture is utilised by the vast majority of messaging systems today.

What are the benefits?

Before delving further into Kafka, it is imperative that we have a solid understanding of the primary terms, such as topics, brokers, producers, and consumers. The important parts are broken down into their component parts and illustrated in the following picture.

Definitions

Topics: Messages that fall into a certain category are referred to as a subject, and the stream of messages that make up a "topic" is termed a topic. Topics are used to organize and store the data. The various subjects have been separated into their own individual sections. Kafka maintains at the very least one partition dedicated specifically to each subject. Every one of these partitions carries messages that are organized in an unchangeable sequential order. A partition is realized as a collection of segment files of consistent sizes.

Partition: Because Topics may have any number of partitions, it can store and process any amount of data. Each message that has been partitioned has its own distinctive sequence id, which is referred to as the offset.

Replicas: It’s important to note that replicas are not the same thing as backups of a partition. There is never any data reading or writing done on replicas. They are utilised in operations to ensure that data is not lost.

Brokers: Brokers are straightforward systems that are tasked with the responsibility of preserving the published data. It’s possible for each broker to have zero, one, or several divisions for each subject. Assume that there are N partitions in a subject, and N number of brokers. Each broker will have one partition, if this is the case.

Producers: Producers are individuals or organisations that publish messages to one or more Kafka subjects. Producers transmit data to Kafka brokers.

Consumers: Consumers are those who read the data that is provided by brokers. A consumer can subscribe to one or more topics and will then consume published messages by fetching data from the brokers.

Kafka Clusters: Clusters of Kafka are referred to as Kafka clusters, and they are characterised by the presence of more than one broker. A Kafka cluster may have more nodes added to it without experiencing any downtime. The management of the durability and replication of message data is handled by using these clusters.

History of Apache Kafka

Linkedin created Kafka as a result of the difficult architectural decisions the company was forced to make throughout the course of its existence in order to get to the point where it could create Kafka. In order to develop this project into something that is capable of supporting the greater than 1.4 trillion messages that go through the Kafka infrastructure at LinkedIn, they also needed to find solutions to several fundamental problems. The engineering team at LinkedIn needed to do a complete redesign of their infrastructure. They had previously made the transition from a monolithic application infrastructure to one that was built on microservices in order to support the growing number of users they had as well as the increasing complexity of their website. Because of this adjustment, the search, profile, and communication platforms, along with any other platforms, were able to grow more effectively. It also resulted in the introduction of a second set of mid-tier services to enable API access to data models and back-end services to provide consistent access to the databases. Both of these developments were brought about as a direct consequence of this event.

In the beginning, they constructed a number of unique proprietary data pipelines to handle the numerous streams and queues of data that they had. Use cases for these systems ranged from simple things like tracking site events like page visits to more complex tasks like compiling aggregated logs from a variety of different services. The queuing capability for the InMail message system, as well as other systems, was provided by other pipelines. These needed to scale along with the site as it became bigger. They decided to invest in the construction of a single, distributed pub-sub platform as opposed to managing and growing each pipeline on an individual basis.

Thus, Kafka was consequently brought into the world at this time.

Why Kafka?

Let’s look at an illustration to understand this in a simple manner.

Before:

After:

I believe the figures are self-explanatory and can be understood easily, so let’s not spend time reading an obvious explanation for the same.

Use cases

Some of the popular use cases for Apache Kafka include,

Messaging
Website Activity Tracking
Metrics
Log Aggregation
Stream Processing
Event Sourcing
Commit Log

Now let’s have a look at some of the companies that make use of Apache Kafka. The list contains some recognisable names, such as the following:

Uber
LinkedIn
Twitter
Netflix
Pinterest
Airbnb

Features

Quick: A single Kafka broker can manage the reads and writes of clients at a rate of up to 100Mbps.
Scalable: Data streams are partitioned and disseminated over a cluster of machines to allow for data streams that are bigger than the capacity of any one machine.
Durable: Messages are stored permanently on a disc and duplicated throughout the cluster to ensure that data is never lost. Each broker has the capacity to process gigabytes of messages without a noticeable decrease in performance.
Distributed: Kafka is designed to be distributed, and its architecture is contemporary and cluster-centric; this configuration provides high fault tolerance and excellent durability.

Architectural Flow for Pub-Sub Messaging

Kafka provides a solitary consumer/client abstraction that summarises both Queuing and Publish-Subscribe at the same time. The process of Pub-Sub Messaging may be broken down into the following steps:

Producers will deliver messages to a subject at predetermined intervals.
The Kafka broker saves all messages in the partitions that are specified specifically for the subject in question. It guarantees that the messages are exchanged in an equitable manner between the divisions. In the event that the producer delivers two separate messages and there are two partitions, Kafka will save the first message in the first partition and the second message in the second partition.
The consumer registers their interest in a certain subject.
After the consumer has subscribed to a topic, Kafka will give the consumer with the current offset of the subject while simultaneously saving the offset in the Zookeeper ensemble.
The consumer will send a request to Kafka at certain intervals (for example, every 100 milliseconds) in order to get newly published messages.
As soon as Kafka is in possession of the messages that have been sent to it by producers, the company then transmits these messages to the users who are consumers.
The message will be delivered to the consumer, who will then process it. When the messages have been completely digested, the consumer will then send an acknowledgement to the Kafka broker.
After Kafka has been given an acknowledgement, it will alter the offset such that it corresponds to the new value and will then update the information in the Zookeeper. The consumer is able to appropriately read the next message even in the event that the server has an outage. This is because the offsets are kept in the Zookeeper.
This cycle, which has been described above, will continue until the customer cancels the request.
The consumer can rewind or skip to the appropriate offset of a subject at any moment and view all of the subsequent messages. This functionality is available round-the-clock.

Installation

Step 1: Install Java/Verify Java Installation

With any luck, you already have Java installed on your computer at the moment; if so, all you need to do is run the following command to confirm that it is indeed installed.

$ java --version

Please visit the following URL and download the most recent version of JDK if you have not already downloaded Java. If you have already downloaded Java, you may skip this step.http://www.oracle.com/technetwork/java/javase/downloads/index.html

Step 2: Install Zookeeper Framework

The Kafka brokers and the consumers communicate with one another using ZooKeeper, which acts as the coordination interface.

Visit the following link to obtain the most recent version of ZooKeeper, which you will need in order to install the ZooKeeper framework on your system.http://zookeeper.apache.org/releases.html

Extract tar file using the following command,

$ cd opt/

$ tar -zxf zookeeper-3.x.x.tar.gz

$ cd zookeeper-3.x.x

$ mkdir data

Open Configuration File named conf/zoo.cfg using the command vi “conf/zoo.cfg” and all the following parameters to set as starting point.

$ vi conf/zoo.cfg

tickTime=3000  
dataDir=/path/to/zookeeper/data  
clientPort=2811  
initLimit=6  
syncLimit=3

Once the configuration file has been saved successfully and return to terminal again, you can start the zookeeper server.

$ bin/zkServer.sh start

After executing the above command, you will get a response as,

$  JMX  enabled  by  default 
$  Using  config:  /Users/../zookeeper-3.x.x/bin/../conf/zoo.cfg 
$  Starting  zookeeper  ...  STARTED

Now, let’s run the CLI

$ bin/zkCli.sh

After executing the command, you will be connected to the zookeeper server and will get the response as,

Connecting to localhost:2811 
................ 
................ 
................ 
Welcome to ZooKeeper! 
................ 
................ 
WATCHER:: 
WatchedEvent state:SyncConnected type: None path:null 
[zk: localhost:2811(CONNECTED) 0]

After connecting the server and performing all the operations, you can stop the zookeeper server using the command,

$ bin/zkServer.sh stop

Now, as you successfully installed Java and ZooKeeper on the machine, let us look at the steps to install Apache Kafka.

Step 3: Install Kafka

To install Apache Kafka on the machine, you can visit the official Apache website and download the tar file link,

Extract the tar file,

$ tar -zxf kafka_x.x.x.x.x.x tar.gz

$ cd kafka_x.x.x.x.x.x

Step 4: Run the server

Now that you have downloaded Apache Kafka on the machine, you can start the server,

$ bin/kafka-server-start.sh config/server.properties

After the server starts, you would see a response on the screen,

 $  bin/kafka-server-start.sh  config/server.properties 
  INFO KafkaConfig values: 
  request.timeout.ms = xxxxx 
  log.roll.hours = xxx 
  inter.broker.protocol.version = x.x.x.x 
  log.preallocate = false 
  security.inter.broker.protocol = PLAINTEXT 
  ……………………………………………. 
  …………………………………………….

Produce and consume few messages

Kafka is a platform for distributed event streaming that enables users to read, write, store, and process events (sometimes referred to as records or messages in documentation) over a large number of machines.

Payment transactions, geolocation updates from mobile phones, shipment orders, sensor measurements from internet-of-things (IoT) devices or medical equipment, and a great many more types of events are examples of events. These occurrences are categorised and saved under their respective topics. At its most basic level, a subject may be compared to a folder inside a file system, and the events that occur within that folder can be compared to the files that occur within that folder.

Launch a new session of the terminal and type in:

$ bin/kafka-topics.sh --create --topic demo-events --bootstrap-server localhost:2990

All of Kafka’s command line tools have additional options: run $ kafka-topics.sh command without any arguments to display usage information.

Open two terminal windows,

Producer: $ bin/kafka-console-producer.sh --topic demo-events --bootstrap-server localhost:2990

Consumer: $ bin/kafka-console-consumer.sh --topic demo-events --from-beginning --bootstrap-server localhost:2990

Position the windows for the production terminal and the consumer terminal such that they are side by side. To continue, type a few more messages into the producer terminal, and then observe how those messages are shown on the consumer terminal.

How to stop and exit the Apache Kafka environment?

When you are finished playing around with Kafka, you should follow these procedures to get out of the Apache Kafka environment:

Stop the consumer and producer clients using Ctrl+C
Stop the Kafka broker using Ctrl+C
Stop the ZooKeeper server using Ctrl+C
Run the following command for cleaning up:

rm -rf /tmp/kafka-logs /tmp/zookeeper

Conclusion

We have covered the basics including a hands-on demo for getting started with Kafka. There is a lot more that goes into Apache Kafka. I hope you’ve enjoyed it and have learned something new.

I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

Till the next one, stay safe and keep learning.

Annotations in Kubernetes

Gursimar Singh — Wed, 04 Jan 2023 17:00:08 +0000

Just what are these "annotation things" anyway? Annotations are not fully explained in the Kubernetes documentation, which is otherwise quite useful.

Similar to how labels are made up of key-value pairs, annotations are also made up of strings. Annotations, however, were initially meant to serve no actual use and are now useless because they cannot be utilised in any queries; the original goal was to offer extensive metadata in an object. Information regarding licencing, the project's maintainer, and other such details are examples of such trivia. Newly-introduced Kubernetes resources, however, may make use of annotations to describe their functionality in greater detail.

To a lesser extent, annotations are unimportant. Labels, on the other hand, will require a lot of human attention.

Annotations and labels share certain similarities. However, for objects, Kubernetes uses labels rather than annotations. Annotations are not used by Kubernetes to apply any changes to the clusters. Annotations work more like comments for human readability and extra details. Annotations help provide context.

Note: Annotations include non-identifying metadata in contrast to labels which hold identifying metadata

Syntax

Annotations can be described in two ways:

Declarative i.e., using the CLI command

Syntax: kubectl annotate [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--resource-version=version]

Example: kubectl annotate --overwrite pods example description=this is an example'

We can both add and remove annotations using the CLI as well.
We have multiple options are available such as –-user, --cluster, --kubeconfig, --context and so on.

Imperative i.e., specifying in the manifest (yaml) file.

For the manifest files, they are defined in the metadata section of the files.

...
metadata:
  annotations:
    for.example/url: "https://www.k8s-blog.com/annotation"
...

Annotations are a way for other programs driving Kubernetes via an API to store some opaque data with an object. Annotations can be used for the tool itself or to pass configuration information between external systems.

Character constraints

There is an overlap between annotations and labels. Annotations have some extra characters allowed which are not allowed by labels.

Valid annotation keys have two segments: an optional prefix and name, separated by a slash (/). The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between. The prefix is optional. If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (.), not longer than 253 characters in total, followed by a slash (/).
If the prefix is omitted, the annotation Key is presumed to be private to the user. Automated system components (e.g. kube-scheduler, kube-controller-manager, kube-apiserver, kubectl, or other third-party automation) which add annotations to end-user objects must specify a prefix.
The kubernetes.io/ and k8s.io/ prefixes are reserved for Kubernetes core components.

How do we check annotations?

We can either use the “kubectl describe pod” command which displays all the info including the annotations or we can simply output only annotations.
However, this command’s output might not be very clear in terms of human readability.

Use cases

They are primarily used while rolling deployments. However, there are various other use cases where annotations are used such as,

Alerting about a specialized policy for a scheduler
Incorporating information about the resource's most recent updating tool and its update process
Including data that isn't meant for labels, such as build and release info.
Permit the Deployment object to maintain a record of the ReplicaSets it is responsible for maintaining throughout deployments.

Note: Some Kubernetes resources and third-party applications might use Annotations so it’s imperative to be careful while using annotations.