DEV Community: Arnab Chatterjee

Ethereum vs Hyperledger Fabric - A Deep dive Comparison - Part 2

Arnab Chatterjee — Sun, 14 Feb 2021 10:50:29 +0000

Introduction

Thank you for the overwhelming response to the first blog post on the rationale exploration for Blockchain Platform Architecture. In the first part, I discussed the basics of Distributed systems and their unreliability. However, when dealing with any system, architectural characteristics play a vital role in determining the system's quality. The ISO 25010 sets the standards of the quality characteristics that a software exhibits.

Performance and Scalability are two of the keystone qualities that make any large scale systems usable. Remember, if we had a large computer with infinite resources, there would not be any distributed systems. So, distributed systems, being so, must always be performant and should scale so as not run out of business.

However, a performant and scalable distributed system is anything but easy. There is a myriad of techniques to imbibe Performance and Scalability, each one always costing you something. Hybrid approaches do work but at the cost of complexity and manageability.

Enter the Block

Scale and Performance in a Blockchain no matter how complex and diverse a topic can always be started with the Block. Why?
Because it is the fundamental unit in a Blockchain that packs transactions. Imagine it like a medium to perform State Machine Replication of the database. It is similar to the economics of flying in business class versus economy. The difference is in the number of round trips when transactions need to be synced across nodes. Lesser the round trips, the more economical it becomes. The Block can contain transactions in the form of state changes. Each state change can be thought of as a set of key-value pairs proposed by users packed into a logical entity called a transaction.

Stitch 'em in a Chain

So why do we need a chain of Blocks? A chain of Blocks is a way to determine the integrity of the data on the network. We all are familiar with the dynamics of how a Merkle hash tree works. It works on the magic of trapdoors and randomness of Hash Functions. These characteristics are incredibly crucial in how we perceive the characteristics of Blockchain. In the real sense, the chain signifies the permanence of data built using a form of a singular thread of Blocks that is always growing outwards.

Ultimately, the hash's purpose is to establish a singular path from the leaf with the root ever-growing outwards.

To give Blockchain's its immutable essence, every Block should be immutable. The hash function captures that. We all know that every Block contains the hash of the previous Block and how difficult it is to alter the Block's hash. But wait; is that all? How do we take the hash? Just pile up all the data and transactions (concatenate them) and take the resultant hash?

Option A - hash all the data concatenated together
Option B - Use Merkle trees

While both seem feasible, but virtually all the Blockchain platforms use option B. You may be wondering why?

Merkle Trees...The name seems familiar.

While Blockchains are disgraced for encumbering nodes with all data from inception, it is not always the case that Blockchains are purely fueled by full nodes. Like big brothers, full nodes carry the load of the network by downloading all data from the network and validating transactions. But light nodes (of Ethereum and Bitcoin) work on the mechanism of Merkle Proofs.

While piling up all the data side by side may work for immutability, it is not efficient when it comes to light clients.
Light clients work on a concept of Merkle Proofs that leverage incremental build-up of the hash of N elements' to determine the presence of a transaction inside a Block. Merkle proofs provide a lightweight mechanism that enables the light clients (who do not store the whole Blockchain data) to verify transactions with a tiny fraction of the actual Block data. This enables the light clients to run in-sync with the full nodes while verifying Blocks only with headers, obviating the need for full-scale verification.

Merkle proofs are based on a neat scheme devised by the famous computer scientist Ralph Merkle. The scheme involves creating a binary tree of data present at the leaf level, where each leaf is hashed to give the next level of nodes. Here onwards, a pair of data is concatenated and hashed to form its parent. This is carried on until there is a single root called the Merkle root. The Merkle root forms the single hash representing the entire data.

To verify the presence of a single data element, one does not need all the leaf nodes to calculate the root. This mechanism enables lightweight verification of the element by selecting a unique path called the Merkle proof corresponding to a leaf node (or element in question to be verified). Just select all the non-leaf nodes: the ommer (parent's sibling) nodes starting with the element, and then recursively until you reach the root. The nodes that path followed is the Merkle Proof of the element to be verified.

So imagine if there are N elements to be stored, the proof would be as long as the depth of the tree, i.e. log(N).

Vitalik Buterin has explained this concept in this article. I recommend reading it highly.

Although things in the Fabric world are not different in using Merkle Trees, Fabric does not have a concept of light clients. The reason Merkle Trees why Fabric adopts Merkle trees are for optimizing data storage on-chain. For example, range queries supported by fabric chaincode use Merkle trees to optimize storage of a range of keys read in a transaction.

So, now that we have questioned a few fundamentals about Blockchain. Let us discuss State and why it is so crucial to the Blockchain.

Blockchain was enough. Now, why this State?

The State is the snapshot of the Ledger from inception till now. Let us come back to the famous example of chess. The Ledger is the list of all the moves you and your friend made from the game's start. But the State is how the board looks since the last move was played. The State gives a way to interpret the present, considering all the transactions that happened since inception. Without the State, to know Alice's balance, one has to get the books rolling afresh. The State can be thought of as a set of key-value pairs at their latest condition; checkpointed using a version number.

The State Juxtaposition

Before we probe further into why Blocks tick the Blockchain and contrast the ledger platforms' Block structure, let us tour the mystery on how and why the State of Ethereum differs from that of Hyperledger Fabric.

The Merkle Patricia Trie - Magically storing State of Ethereum

Let us again go back to the Design Philosophy of Ethereum and Hyperledger Fabric. Ethereum is a public ledger and must store all details of all transactions that happened since inception. But do we really need to know how the past looks like? Even if we had to, we could always replay the transactions that appeared in each Block. Right? So, does it make sense to store all the history as State again?
Yes, and, Ethereum does store history State. The rationale is discussed later in this post. But think for a moment, it is a gargantuan task to store the history since inception - like recording every single moment of your life in a camcorder. How does Ethereum manage to store the entire history? 🧐

Ethereum... History of the State... 🤯 How?

The State of any system can be represented as a set of key-value pairs. In simple terms, keys can be account addresses, and value can be the balance. There are numerous mechanisms to store key-value pairs persistently. But storing the History state means storing a part of the State which was identical in the last Block and didn't change in the current Block, N times over. This is highly inefficient, and Ethereum researchers had a nifty trick up their sleeve. Most Blocks would only store redundant information that didn't change since the last Block. To ensure tamper resistance of the states, the researchers tied the Trie structure and Merkle Tree into a hybrid structure called Merkle Patricia Trie.

The Merkle Patricia Trie (MPT) isn't ordinary data structure. It is meant to solve multiple challenges at once.

Save Space by reusing and referencing old state elements (KV pairs).
Solve the problem of proving the non-existence of an element in the State - which is not possible with a regular Merkle Tree
Quick search of keys. Max number of traversals is the tree's height (which is the length of the keys in this case).
Establish provenance of state changes by hashing state elements into an N-ary tree (where N = 17) into a single root. (N is 17 is because each node can branch into 16 different hexadecimal elements, and one place is reserved for a value.)

Although somewhat cryptic, I will reuse the fantastic original visual representation of the Merkle Patricia Trie from Stack Exchange.

Although intimidating, the MPT is a relatively simple concept to understand. There are three essential components:

Keys - The key is the reference to find the value. In Ethereum, Keys are User or Contract addresses. Addresses have specific values. But a lot of Keys(addresses) have the same prefix. To avoid too many branches, keys with the same prefix are grouped together like in a Trie.
Values - A value can be a number like a balance, the code of a smart contract or transaction receipts. The smart contract internally maintains the State of contract in a separate Merkle Patricia Trie called the storage trie. We discuss the smart contract storage in a separate post.
Nodes - The unit of the tree. Each node is either an Extension node,Branch Node or a Leaf node.
- Extension nodes - This is the divergent path node when key prefixes do not match. Each nibble (four bits) that correspond to the hexadecimal characters that match the Key prefix are stored as shared nibbles. In the example, a7 is the shared prefix for all the keys. So two nibbles a and 7 are stored at the root node a type of extension node. The next node is again a reference to a node, which can be either of an Extension, Branch or Leaf node.
- Branch Node - This is where the nibbles start getting really wiggly and differ significantly. There are 16 references. Each of them points to a different node whose nibble matches the single corresponding sequence in the key.
- Leaf Node - The ultimate of the trie, which usually ends in one or more nibbles in the keys. It contains a value.

The punchline is that if you follow each nibble in an extension node, each reference in a branch node, and the leaf node key-end nibbles - the path you trace is the Key nibble by nibble. The value is stored in the leaf node. You may end at an intermediate node - and if there is a value in any of them, then that corresponds to the key traced by the nibbles so far.

When a new Block is mined, and the current State is changed by the transactions in the Block, a new state root is formed. The new trie is constructed with the keys that changed in the transactions and reconstructing the entire state trie is avoided. Only the keys that change fork off to create a new part of the trie with new nodes, while part of the trie corresponding to the nodes where the keys did not change still refers to the old trie. In each Block, the State of the trie is a snapshot using a KECCAK256 hash function and stored as the state root in the Block header. This makes a lot of sense since there are millions of accounts in Ethereum that can change and reconstructing the entire State would be inefficient in terms of storage and computation.

The above diagram shows how the Merkle Patricia Trie changes from Block 654 when Dorothy(a77d379) transfer 0.01 ETH to Emily (a7ff377) who has just joined the network.

Ethereum History State, but why 😲?

In simple words, Ethereum faced vicissitudes of forks and understands the importance of decentralization. In case of a mishap (a fork), it would be a gigantic task to recompute the State of a historical state from inception. Keeping old states means that the chain can quickly switch back to a relatively sane State. But would it make sense to store all the State since its inception? Of course not, that would mean keeping the provision to rollback to the genesis Block in case of a catastrophe. 😅 Of course, this would not happen because that would mean Ethereum prices would crash overnight. There would be a mass exodus of believers of Ethereum. So, Ethereum stores the last 128 states as history and prunes the rest of them.

Can I query the history state?

Technically, one cannot query history. Still, events emitted by the Ledger are also stored on the Ledger and can be queried through Web3 APIs. However, querying the history state inside the smart contract itself is not possible. Ethereum does not allow the querying of the smart contract state because it prunes all historical States but the latest 128 Blocks.

Relax with the Couch State in Hyperledger Fabric

Unlike Ethereum, Hyperledger Fabric's State is quite simple. It uses a Couch database to store its ledger state. Couch DB is a document database where each document can be uniquely referred to by a unique key. Hyperledger Fabric allows the Smart contract developer to store keys (user-defined). These keys themselves form the State of the Ledger. Fabric also keeps track of the changes to a key. The approach is straightforward. It maintains a separate database that keeps track of all the keys changed and in what Block. It is then possible to query the history of that key using a Smart Contract API. Note that it is not the history state but a track of where the key has changed over time.

The Couch database, apart from offering a key value pairs also offer a rich NoSQL querying interface to query Key-Value pairs using Mango Queries. Those familiar with SQL like queries can relate and imagine life without them. Mango Queries helps you find the relevant documents (Key-Value pairs) of interest with a querying language. This speeds up the querying process without having to traverse through every key in the database.

The Block Recipe

Now that we have grasped an overwhelming lot about the State, I thank you for your patience and support for still reading. Let us now explore the Block of the Ledger. When talking about the fundamental structure of a Block, there are a few basics that all Block based ledgers follow. The Block would essentially consist of three things:

The list of transactions
The hash of the previous Block
The Block number

In addition to this basic structure, each Block holds unique information about accountability. Accountability metrics answer the WHO of the Block.

Who sent the transactions?
Who proposed the Block or the Block creator?

These metrics for accountability play a key role in the security of the Blockchain. The sender of the transaction needs to be identified for obvious reasons. However, the Block proposer needs to be identified for incentivizing or penalizing the Block creator. In the last post, we discussed the need for stakes. In a Block creation mechanism, there needs to be a leader, identifying the leader is imperative to maintain the sanity and security of the network.

Block Structure Compared

Now that we know the lifeblood of the Block, let us trek further to understand the fundamental elements of the Ethereum and the Fabric Block.

Ethereum Block Structure

When it comes to Ethereum, being a public Blockchain platform has a lot of Block data. The header of the Block in Ethereum contains three essential elements.

the essentials - This comprises what every Block in a Blockchain should have, i.e., the parent Block hash and the Block number.
Block creator data - This contains the details about the Block creator, the reward information, the total gas expenditure, the total gas limit imposed on the Block, the total gas used and the timestamp at which the miner mined the Block. This is to essentially know who has mined the Block for its accountability and establish the traceability of the new ethers introduced into the Blockchain. Without this information, there would be no provenance of where the new ethers came from.
proof-of-work data - This section essentially contains the proof that the miner has indeed mined the Block. This is the endorsement of the miner's work to mine new ethers into the system. Proof of Work (PoW) enables anyone to verify quickly and with ease - that sufficient computational energy has been spent to find the nonce that when hashed with the Block header hash (without the nonce) yields a value lower than a target. In other words, the miner did not introduce new Ethers out of thin air into the system with negligible or zero cost. This mechanism is detailed in the EthHash documentation.
data related to possible alternate chains - Since Ethereum rewards competing miners who have solved the PoW challenge unlike Bitcoin, it is also necessary to keep accountability and provenance of the money introduced through the route of alternate chains. However, there is a caveat that Blocks that are introduced by Ommers (or uncles) can be referenced only till six Blocks from the Block of introduction. The reason for this is simple.
Since Ethereum chooses the main chain by using GHOST (Greedy Heaviest Observed Sub Tree) protocol, there can be multiple parallel chains. If these chains grow too long, it would be extremely inefficient for the Blockchain to search and reference data during validation time. Imagine the Blockchain turning into a giant tree with ever-growing depth. It would be inefficient in traversing an unbalanced tree which is too deep to search. Along with the GHOST protocol restriction, the six-Block reference imposes an incentive to mine only on the longest chain.
state-related data - Ethereum keeps track of the State as well to ensure that states are consistent and non-repudiated across all Blocks. There are types of states.
- State Root - This is the MPT root of all the state changes in the current Block. This includes changes in State of user and contract accounts.
- Transactions Root - The MPT root of the transactions that were confirmed in the Block. Again MPT to the rescue to optimize storage and depth of the Merkle Tree.
- Receipts Root - The MPT root of the transaction receipt (the receipt you get after every transaction in Ethereum).

The rest of the blocks just contain signed and validated transactions that the miner has accumulated and proposed as the block content. Each transaction holds information about the sender and the receiver of the transaction. In all cases, the sender (from) is the User address, the receiver (to) is user in case of direct Ether transfer or a contract in case of a contract invoke or a call. Contract invokes write data on the ledger, while calls just read. The value is the amount of ethers the user wants to send, while the data is applicable for contract calls to store calldata.

Hyperledger Block Structure

The Hyperledger Fabric Block looks different but contains some similiar characteristics as that of Ethereum.

Block Header - Let us start with the Block Header. It contains the essentials - previous Block hash, the Block number and the data hash. The data hash is principally the hash of the transactions in the Block wrapped in an Envelope. The Envelope is an overarching concept of the transaction that we will explore soon.
Block Data - The Block data is a collection of Envelope; each wrapping a transaction. An Envelope contains more than the transaction proposed. But why enclose more data? Remember Part 1 of the article and how Fabric doesn't need to vet the transaction with all the participants in the network. To guarantee accountability and correctness, the transaction is send first as a proposal to interested parties in a channel for verification. And these participants are a part channels, the channel information is stored as well. So in a nutshell, the Envelope contains.
- Header - Information about the channel and the creator of the transaction along with the signature.
- Data - Information about the transaction proposal, the endorsed response, along with the endorsements (signatures) of the stakeholders who endorsed the transaction proposal response.

The endorsements capture the consent of the stakeholders in the transaction's outcome (validation). This furthers the accountability of the stakeholders in the transaction. We will discuss the details on Transaction flow including endorsements in a future article.

Well, that is all. Thank you for reading so far in the journey to explore the Architectural rationale of Blockchain Platforms. We discuss Identity management of Blockchain Platforms and the rationale behind decisions in the next article. Thanks again and stay tuned.

Ethereum vs Hyperledger Fabric - A Deep dive Comparison - Part 1

Arnab Chatterjee — Thu, 29 Oct 2020 17:33:32 +0000

Caveat Emptor - If you are concerned about the technical complexity of the article. Do not worry. I have tried to keep it jargon-free. So go ahead, enjoy and share your feedback.

Credits: Thanks to Prateek Reddy Yammanuru for insights and reviews.

In this article, we deconstruct some of the similarities and differences between two of the most popular Blockchain Platforms - Hyperledger Fabric v2 and Ethereum v1 (as of 10th Oct 2020). The motive is simple. We have much material explaining the workings of each of these ledger platforms. However, there still does not exist a deep-dive comparison of these ledger platforms. 🧐

Most ledger platform comparisons that I have stumbled discuss core features of the platforms like Consensus, Permissioning, smart contract language, and the likes. Personally, as a technology geek and a person who likes to dive deep and ask a lot of questions, these articles do not quench my thirst. 😅🍺

So, as an aspiring Software Architect, I decided to make an attempt to understand these platforms deep and explore the fundamentals that make these Platforms tick and what claims the fame of these ledger platforms.

I attempt to explore the architecture in four distinct areas. The overall functions that these ledger platforms provide, the architectural characteristics (also known as the non-functional requirements (NFRs)/ Quality Attributes / "ilities") that these functions are expected to exhibit, the architectural topology or simplified view, and finally some of the critical architectural decisions that were considered.

This will be a six-part series of articles where I will attempt to take each of the fundamental concepts of a Blockchain Platform and attempt to expound them as it is architected in each of the platforms with the rationale behind it. I will also try to provide an essence of the key architectural considerations made while designing the platforms.

I will try to explore the following areas, and in each, I try to cover three fundamental aspects. First is the need, second is key architectural considerations and finally the comparison. My goal is to be objective and non-partisan. For this, I will share the references to the sources from where I have obtained and compiled the information.

Comparison Areas

Area	Covered In
Distributed Systems 101	Part 1
Overall Architecture	Part 1
Block Architecture	Part 2
Ledger & Storage	Part 2
Identity Management	Part 3
Transaction Lifecycle	Part 4
Smart Contracts & Ledger State	Part 4
Consensus Process	Part 5
Security	Part 6
Confidentiality	Part 6

Distributed Systems 101

In this section, I attempt to break down the fundamental cogs that drive the Fabric Blockchain Operating system from the Ethereum Blockchain Operating system. Although mouthful it may sound, both of these platforms are irrefutably distributed in nature first - then decentralized. Hence all principles of distributed systems apply here. I believe it is fundamental to reason from the first principles of these.

If you are not too familiar with distributed system principles, I cannot recommend this article enough for a quick reckoner on Distributed systems.

But let me take a quick plunge into the fundamentals of distributed systems:

Distributed systems are composed of nodes or computers that are capable of computing (calculating), storing, and transmitting data.
These computers are Unreliable and so are the communication channel(s) that connect them. Any or both can crash or malfunction at any point in time.
Distributed systems do not have a global clock. Ever tried setting two electronic clocks precisely to the second (do it with two computers to the millisecond).
Distributed systems are complicated. One does not design a distributed system if there was a monolith computer with infinite compute and storage.
All systems are eventually intended to cater to Functional and Non-Functional needs. Non-Functional needs a.k.a. Quality Needs. Basic quality needs of any distributed system include Scalability, Performance, Availability, and Fault Tolerance.
For reliability and availability, one needs to replicate data. Where there is replication involved in distributed systems, the famous CAP theorem applies. It says that you cannot have all three properties in a distributed system (which replicates data) at the same time, i.e. Consistency, Availability and Partition Tolerance. Try achieving any two, and you have to trade-off the other.

Let us start with the needs of a Blockchain Platform, discuss the rationale behind the architectural characteristics.

Contrast of Overall Architecture

Need: State Machine Replication

Any blockchain system is fundamentally a distributed database in the most simple terms. It needs to be consistent, that means all data in all the computer nodes has to be the same exactly. Note here; I am not emphasizing on time here. Consistency can be immediate or eventual. To replicate the state of the database, State Machine Replication is used. There are two ways to do replication. Active Replication and Passive Replication. Active replication is where every database server processes the request in a deterministic way (i.e. everyone produces the same output on a single input). To guarantee that all servers produce the same output, it has to be ensured that all servers receive all the inputs in the same order. This is ensured by Atomic Broadcast. Atomic Broadcast says that the input should be received either by all servers or none.
Passive replication is more of a dictatorial in nature, which means that there is one master server executing the request and updates other servers with the execution result without blocking the client request.

Need: Smart Contract Execution or Agreement on the outcome of a business logic

Second-generation blockchain platforms work on the outcome of Smart Contracts, which is the custom business logic for determining the outcome of the transaction. However, there are subtleties involved here, and they are:

Who needs to execute the smart contracts and why is there a need to agree on the outcome?
What goes inside the ledger?
Is the agreement on the smart contract outcome different from the State Replication itself?

Let us try to answer these fundamental questions subjective to each ledger platform.

Who needs to execute the smart contracts and why is there a need to agree on the outcome?

Users who do not usually trust each other on the outcome of a transaction executes the smart contract. This helps one to verify the outcome is genuinely computed and has not been tampered. This dates back to the fundamental underpinnings of Bitcoin and cryptocurrency. Bank serves as the trusted arbiter to hold account balances for everyone. All banks universally operate on the same set of rules to maintain account balances. But if a nation or government or bank goes rogue, then what? This where secure computation comes into the picture. Smart contract execution ensures that every bank (hypothetically) executes the same logic and matches the outcome.

Note the subtlety in the last line? Every bank needs to execute the contract to agree on the outcome. So, does this mean that a civilian like you or me should compute the outcome for every other transaction? Probably no. Why should you be concerned about a transaction happening between other parties? But Bitcoin and Ethereum enforce this. It mandates that all participating nodes must agree on the outcome of the transaction, even if one is not involved in the transaction.

Moreover, there is not any need to bring store the transaction outcomes as well. This is the inception of the concept of Permissioned and private Blockchain platforms. So, if we delve into this a bit deeper, we would realize that:

Not everyone needs to store all transactions, and others would not be comfortable with sharing the data.
Even a lesser proportion of the population would require actually to execute a transaction and agree on the outcome.
Comparing 1 & 2, we can realize that there would be people who would be observers who are interested in storing the agreed outcome but not executing the transaction itself.

So we can deduce that the state machine replication should be separated from the agreement on the outcome of the transaction by Smart Contract execution.

What goes inside the ledger?

The ledger should be composed of all the outcomes of transactions checkpointed using a specific mechanism called blocks, unlike databases, where checkpoints are granular to the level of transactions itself. Checkpoints in Blockchain are Blocks which comprises the transactions.

The ledger then should contain all the delta updates (as transactions) that happened to the database checkpointed as blocks. Nodes in the network can either choose to store an irrefutable record of data and consent on the outcome of transactions (by executing smart contracts on the ledger data) or choose to store data as an observer, like a bank computer vs a bank vault. The ledger is merely the bank vault.

Is the agreement on the smart contract outcome different from the State Replication itself?

This is a question related to the previous one. And the answer is yes. Everyone interested to agree on the outcome of the transaction needs to have an irrefutable copy but so do people who are just interested in a copy of the ledger. So, the State Machine Replication is what guarantees that all interested parties would get the replica of the ledger.

Now, that we have established some fundamentals, we must realize that when we perform a transaction, we want both the steps to happen atomically (all or none). That means that relevant parties should agree on the outcome of the smart contract execution, and the appropriate parties should get a replica of the outcome of the transaction.

Different perspectives of Design Philosophy

As we know, Ethereum is famous for both dApps (i.e. it's the capability to host decentralized apps, leveraged by smart contracts) as well as it's acclaimed cryptocurrency. The backbone of trust in Ethereum is its backed cryptocurrency that allows to codify game-theoretic security and innoculate trust in the dApps. Likewise, the backbone of trust in Hyperledger platform is the consortial identity binding.
In other words, the idea of trust is backed by some stake (like money, reputation, something that is dear to someone). In a public blockchain network which aims to be secure and conserve the privacy of the individual, money is at stake. In contrast, in a permissioned (i.e. known environment), the identity or the reputation is at stake.

So, in a public network, to maintain the cryptocurrency or to enable cryptocurrency backed trust, all participants must host the ledger and run the base cryptocurrency codebase. And since the cryptocurrency is not bounded, its scope mandates every participant to run all the smart contracts ever hosted on the ledger. This essentially means that if you want to do transactions on the ledger, backed by the crypto trust; you have to vet not only who owns how much cryptocurrency but the transactions that alter the balances of the cryptocurrency as well - in this case, smart contract transactions.

But in a permissioned network, participants are known to each other (i.e. the identity and reputation are at stake). So a rogue actor would think twice before ruining its own reputation.
This gives a differentiating (edge) factor to permissioned chains. In the early days, communities used to serve the function of law and order. This is because the individuals of a community were tightly knit within that community, and the transactions were strictly within the community. As Globalization became more prevalent and transactional boundaries widened, central law and order became the need of the hour. A similar concept applies to consortial and public blockchain platforms.

Let us explore how these distinct requirements shaped the architecture of both the ledger platforms.

Order-Execute vs Execute-Order-Validate

This title "Order-execute vs Execute-Order-Validate" is borrowed from the official Hyperledger Fabric paper. Although I do not fully agree with the semantics, I still choose to borrow it for better relatability.

Let us break down each of these steps agnostic to any platform, and understand what these mean (not necessarily in that order).

Order
This is the fundamental step that tells which transactions should come first and what comes later. As a matter of fact, this is a really difficult (maybe even impossible) problem to solve. Remember distributed system fundamentals? If we have to find an order, we have to find a way to measure order. Time is out of the equation first because there is no global clock. In Distributed systems (especially Decentralised systems) it is not possible to decide on the order of N transactions taking consent of all the stakeholders (or at least the majority). It would be extremely inefficient to get everyone to agree on the order of transactions. So who decides the order? Well, someone dictates, and everyone agrees. But who dictates and the legitimacy of the ordering comes at a cost (at least, and incentive in some instances) which we would discuss more in our Consensus discussion.

Execute
To decide on the next state of the ledger, one has to execute the transactions and derive on the next state unanimously. So, we saw that the order step has to order the transactions in a block. After executing the set of transactions in the next proposed block, everyone arrives at the next state (or the next block). Note that this transaction is sequential, which impacts the overall throughput. However, this can hardly be cited as a disadvantage because of the universality of the transactions and the ledger state.

Validate
This step encompasses the order and the execution step. When the transaction first is constructed, it has to be verified of the semantics, account balance, transaction fee paid (if applicable). Likewise, when the transaction is executed on the current state after the block is propagated, it has to be checked for proper state validations like dirty or phantom reads, race conditions, and header validations.

Now let us contextualize each of these steps for Ethereum and Fabric by examining the transaction flow of each of the platforms and earmarking each of the above steps in the overall flow.

Ethereum - Transaction Flow

The forward transaction flow involves the flow of the transaction starting from the client (here a mobile dApp). The immediate full node validates the transactions for necessary checks like checking the transaction nonce, checking sufficiency of the gas price (transaction fees), transaction semantics, etc. The transactions continue to move through the network in the transaction pool (with its brothers) from full node to full node until it stumbles across a miner. Keep a note that the transaction has not been executed.

Like this, the transaction reaches other competing miners too who also want to compete to mine the block. So one lucky miner who solves that PoW puzzle first gets to propose the block. I will discuss this in detail in the Consensus architecture post. However, three things happen at the miner and in order:

Order - The miner orders the block based on a predefined strategy. Mostly this strategy is one that yields him the maximum revenue per block.
Execute - The miner executes the transactions to ensure the transactions correctly apply against the current state (as perceived by the miner) of the ledger, and it generates logically correct the next state for each account (this includes Externally Owned Accounts and Contract Accounts). If some transactions do not generate logically correct next state, the transaction is dropped.
PoW and Broadcasting - The final set of transactions obtained in the previous step is put in a block, the PoW nonce is computed, and the block is constructed with all relevant details and broadcasted to the nodes.
Validate - Each transaction is again vetted against the current perceived state in each full node before it accepts the block. Conceptually, the full node has the liberty to reject the block, but this does not happen in practice, because the miner spends stake (compute) to mine a block, it is not rational for him to mine a block which won't be accepted by other nodes.

Why do I say perceived state? Because each full node can run independently on a different blockchain. But ultimately the chain merges using a protocol called GHOST (Greedy Heaviest Observed Subtree).

So if you observe the overall flow is not just "Order > Execute" but is "Validate > Order > Execute > Validate" considering all the nuances.

Hyperledger Fabric - Transaction Flow

Execute
Like we did for Ethereum's transaction flow, the transaction flow for Fabric can be decomposed into two parts. Before ordering (mining) and after ordering. In the first half, the client selectively broadcasts the transaction to organizations for endorsement (endorsing peers).

Few things to take note here. First, unlike the ocean of full nodes in Ethereum, the nodes who execute the transaction is known upfront. This means the rule to determine a transaction as valid, i.e. who is needed to endorse or approve a transaction is decided upfront. Parties not interested in the endorsement but just the transaction can only receive the block. So postponing the execute till the last is not an option. But the validation can be delayed because it is the ultimate step.

Each of the endorsing peer validates the Transaction Execution Proposal with authenticity and authorization checks. Then the transaction is executed on the current state of the ledger. Each endorsing peer generates signed "Read-Write Sets" as a response. The endorsing peers have the liberty not to respond too. The "Read-Write set" is that delta that adds on the current state of the ledger to determine the next state of the block. The onus of executing the actual transaction is only to the endorsing peers who want to execute and compute the next state of the ledger is actually on who is interested. So this sort-of-voluntary mechanism eliminates the need for any gas or transaction fees which was otherwise introduced in Ethereum to cater to the Halting Problem. However, the halting problem is still not fully solved. We will discuss more how the halting problem is further catered in the Consensus article. (sneak-peek: Fabric encourages each organization to host its own source code for the smart contract, thereby eliminating the need to trust smart contracts written by others. So, the more complex or bad code your organization develops, the more compute you spend).

In the end, the client collects these endorsement responses and sends them to the orderer along with the original transaction proposal to the orderer.

Order
The client sends the transaction to a predetermined orderer or miner. Why is this the case? This is because the ordering is a crucial task. There is more accountability than reward here. Because of such high responsibility, Fabric introduces a separate concept (and status) of Orderer Organization who is identified as the organization authorized to order transactions of a consortium. Although we may argue about the incentive and stake in such a private blockchain network, we park that discussion for a different thread. When the transaction and other transactions reach the orderer, the transactions are ordered (based on the orderer consensus algorithm, like Raft) block can be formed based on some set parameters like Block size, block time and the number of transactions in a block.
Once again, the temporality relation of these transactions is entirely subjective to the orderer.
Next, these transactions are broadcasted to all peers, including ones who did not execute the transaction (committing peers). An endorsing peer is a committing peer too.

Validate
The transaction is then validated to check for sufficient endorsements (if the required stakeholder organizations have responded with the "Read-Write Set"), version checks against the current state (done through Multiversion Concurrency check) and most importantly, if all the signed endorsement responses (Read-Write Sets) are the same. This is the consensus on the next state of the ledger. There is no scope for inconsistent of ledger states here.
This is because, in a private environment, where you have already predetermined who would endorse the transaction and prevalidated the transactions, there is no second scope of perceiving different states of the ledger.
Once validated, the block is either appended if correct or rejected in full if the validation fails.

Moreover, since this entire process happens in a microcosm of the ledger called the channels, the peers, orderers and data can be scoped out based on the privacy needs of the consortium. Channels being the microcosm greatly facilitate the confidentiality and the performance of the ledger. This means every channel hosts its own smart contract, requires a subset of the endorsing peers which means unrelated parallel operations is possible.

Summarising, the overall flow is "Validate > Execute > Order > Validate" considering all the nuances.

Well, if you made it this far, I thank you and congratulate you for reading this article.

Thanks. Stay tuned for Part 2 where we dive deep into Block Architecture and Ledger Storage.