Some clarification
The purpose of mining is probably a little confusing at first. I will keep the Bitcoin blockchain as an example throughout this article. Mining is NOT about creating new bitcoins. Mining is the mechanism that allows the blockchain to be a decencentralized security. It secures the bitcoin system and enable a system without a central authority. Do not confuse the rewards given to miners ( new bitcoin ) with the process itself.
Mining in Bitcoin
Miners validate new transactions and record them on the global ledger ( blockchain ). On average, a block ( the structure containing transations ) is mined every 10 minutes. Miners compete to solve a difficult mathematical problem based on a cryptographic hash algorithm. The solution found is called the Proof-Of-Work. This proof proves that a miner did spend a lot of time and resources to solve the problem. When a block is 'solved', the transactions contained are considered confirmed, and the bitcoin concerned in the transactions can be spend. So, if you receive some bitcoin on your wallet, it will take approximately 10 minutes for your transaction to be confirmed.
Miners receive a reward when they solve the complex mathematical problem. There are two types of rewards: new bitcoins or transaction fees. The amount of bitcoins created decreases every 4 years ( every 210,000 blocks to be precise ). Today, a newly created block creates 12.5 bitcoins. This number will keep going down until no more bitcoin will be issued. This will happen around 2140, when around 21 millions bitcoins will have been created. After this date, no more bitcoin will be issued.
Miners can also receive rewards in the form of transaction fees. The winning miner can 'keep the change' on the block's transactions. As the amount of bitcoin created with each block diminishes, the transactions fees received by the miner will increase. After 2140, the winning miner will only receive transaction fees as his reward.
The amount of bitcoin issued with each block is divided by 2 every 210 000 blocks. So we can calculate the maximum amount of bitcoin with some code.
// First 210 000 blocks reward
const start_reward = 50
// The reward is modified every 210000 blocks
const reward_interval = 210000
const max_btc = () => {
// 50 BTC = 5000000000 Satoshis
// Satoshis are the smallest denomination in bitcoin
let current_reward = 50 * 10 ** 8
let total_btc = 0
while( current_reward > 0 ){
total_btc += reward_interval * current_reward
current_reward = current_reward / 2
}
return total_btc
}
console.log(`The maximum amount of BTC created will be ${max_btc()} Satoshis, or ${max_btc() / 10**8} BTC`)
// The maximum amount of BTC created will be 2100000000000000 Satoshis, or 21000000 BTC
So, yeah, 21 millions will be probably the maximum amount of bitcoin issued in the end!
How does it work, anyway?
The question is, how can everyone in the network agree on an universal 'truth' about bitcoin ownership, without having to trust anyone in said network? The blockchain is not created or managed by any central authority. Every node has access to the public ledger of transactions that acts as an authoritative record. Somehow, every node in the network, acting on those insecure informations, come up to the same conclusions and are able to replicate the same ledger. Let's try to explore how this works.
It will probably help if we take a real life example. Meet block #502426. We'll follow the lifecycle of this block from its construction to its final validation. Let's say the winning miner was called Joe.
The previous block
In the world of Bitcoin, it takes approximately 10 minutes to validate a new block. Our miner Joe was competing to validate the block 502425, the previous one. Unfortunately, someone else solved the problem before him. But, the end of one block's competition means the beginning of a new one. As soon as the block 502425 was mined, Joe updated his local copy of the blockchain and starts to create a candidate block, the block 502426. While Joe's computer ( or node ) was searching for the Proof of Work for the previous block, it was also listening for new transactions. Those new transactions are added to the memory pool or transaction pool. This is where transactions wait until they can be included in a new block and validated.
Creating the candidate block
When Joe's node is notified that the current block has a valid Proof of Work, it starts constructing a candidate block by gathering the transactions in the transaction pool. It removes the transactions already present in the previous block, if there are any. The block is called a candidate block because it doesn't have a valid Proof of Work yet.
So, we can see that block #502426 has 3189 transactions inside it. This was the number of transactions present in Joe's transaction pool when he created his candidate block.
Coinbase transaction
The first thing Joe's node does is creating the coinbase transaction. Very simply, this is his reward for mining the block. This transaction says => Pay Joe's wallet address xxx BTC to reward him for finding a valid Proof of Work. This transaction is different from the other ones because, as I explained earlier, the bitcoins in the reward are created from nothing. They do not come from someone's wallet. Joe's node also calculates the transaction fees in the block.
Joe's reward = Reward for mining block + transactions fees
In this case, we can see that the block reward is 12.5 BTC ( Block Reward in left column ) and the transactions fees are equal to 4.86507997 BTC ( Transaction fees in left column ).
12.5 + 4.86507997 = 17.36507997 BTC
You can see the details of this transaction in the list below.
You can see No Inputs (Newly Generated Coins). Like I said, coinbase transactions do not come from anyone's wallet, so they can't have any inputs. You only have the winning miner's wallet address here.
Constructing the block header
In a previous article, I described what was contained in a block. Joe's node has the responsability to create a proper block header for the block he is mining. In the article, I mostly focused on the merkle tree, a summary of transactions and mentioned that there are three different sets of data in a block header: the previous block hash, the merkle tree root and data for the mining competition. We'll dig deeper in the last one.
Data fields
This metadata set contains:
- The Version: This is a version number to track the software and/or protocol upgrades.
- Timestamp: Seconds from Unix Epoch. When the block was created.
- Target: Proof of Work algorithm target for this block
- Nonce: Counter used for the Proof of Work algorithm
When the block #502426 was mined, the version number was 2. It becomes 0x20000000 when converted in little-endian format in 4 bytes. ( Version in the left column )
Next, we get the 4-byte timestamp. This is the number of seconds elapsed since the January 1, 1970. We see the timestamp for this block is 2018-01-03 21:12:39 ( Timestamp in left column ). If we convert this in seconds, we get 1515013959 seconds.
The target field defines the required Proof of Work to make this a valid block. To put it simply, the target is generated by the network and defines what makes a block's hash valid or not. If your hash is higher than the target, it is not valid. This is what is used to calculate the difficulty. In our block, the difficulty is 1,931,136,454,487.72. Take a look at the block's hash:
00000000000000000020c60222099aaebc6e7795784f74628ec640b223d3d339
There are 18 leading zeros. This is our difficulty. Every hash with less than 18 leading zeros is invalid ( every hash with 17 leading zeros and less would be lower that the required target ).
The last field is the nonce. It is initialized to zero.
Everything is now ready for Joe's node to mine the block. The goal is to find a value for the nonce that will results in hash lower than the target. So, the mining node will try billions or trillions of nonce values before it gets a valid hash.
Mining
In a picture, mining is:
As you can see, mining is like a lottery. There is no way to predict which nonce will solve the problem.
In a previous article, I implemented a simple blockchain that demonstrates the concept of a nonce and how it changes the hash created.
In the case of Bitcoin, the hash function used is called SHA256. A hash algorithm always produces the same arbitrary length data given the same inputs. It is impossible to compute the same hash with two different inputs ( collision ). It is also impossible to predict the output of any given data in advance.
SHA256 always produces an output 256 bits long. Mining is finding the nonce, the only input that changes every time we run the hash function. It is very easy to prove that the nonce found indeed produces a valid hash. All the informations are available, everyone can run the hash function and confirm if the hash is valid or not. Because it is also impossible to predict what the nonce will be, it also act as a proof that the miner worked to get a valid hash ( Hence => Proof-of-Work ).
Adjusting the target
In Bitcoin, a block is mined every 10 minutes or so. The difficulty is calculated so that it never deviates too much from this limit. If the difficulty stays the same in the long term, while computer power increases, it will take less and less time to mine a block. To make sure this doesn't happen, the Proof of Work target is a dynamic parameter. In the Bitcoin world, the target is adjusted every 2016 blocks. Then, we check the amount of time it took to mine those 2016 blocks. It should have taken 20160 minutes ( 2016 * 10 minutes ). The difficulty increases or decreases depending on the time it took to mine those blocks.
Success !!
Joe's node is getting to work and starts hashing the block's header. After approximately 10 minutes, it discovers a valid hash. We can see the nonce used was 2469953656 ( Nonce in left column ).
Joe's node immediately transmits the block to all its peers. They need to validate the new block before propagating it to its peers. This is the part where a dishonest miner can be found out. If the data is invalid, the miner would have wasted his time and computing power. Valid data includes:
- Block header hash is less than the target
- Block size is within acceptable limits
- Block timestamp is less than two hours in the future.
- The first transaction is a coinbase transaction ( and only the first )
- The coinbase transaction has a valid reward.
- All transactions within the blocks are valid ( also have a checklist on their own )
Every node validates independently new blocks following the exact same rules. This assures that miners cannot cheat. This is a key component of the decentralized consensus.
If the block is valid, the other miners will update their own copy of the blockchain with the new block 502246. Joe's block hash is now used by all the miners to mine the block 502247.
Latest comments (14)
Hi Damien, Thanks for the amazing articles. I just started to learn blockchain, you make the blockchain concept so clear to me. I still have a few dumb questions here. All the questions are based on Bitcoin Blockchain.
How to calculate the transaction fees? And where is this fee from? Are they just like bitcoins, which are generated from nothing, since there is no input for it? You mentioned after 2140, there will no new bitcoins generating, but the transaction fees will still be generated?
For each candidate blocks, do they use the same set of transactions in the transaction pool except the coinbase transaction to generate the merkle tree? If not, do they use the same number of transactions, or they use different number of transactions? What is the rule to choose the transactions from the transaction pool to be included in the candidate block?
At the end of the article, you said a valid block should include "All transactions within the blocks are valid ( also have a checklist on their own )". As I understand, the transactions in the transaction pool are not confirmed. How to check the transactions within the blocks are valid? What is the definition of the valid transaction? What does the checklist mean?
A little bit more, I have read the original paper, it mentioned simplified payment verification (SPV). To verify a transaction, we use the merkle tree, bloom filter and the method you mention in the article what is in the block, is this right? However, what if a transaction is still in the transaction pool, not in the block yet? Do we think it is not verified?
I am confused whether there are intermediate transaction hashes in the block. In the previous article you mentioned it in the Q&A, but not very clear to me? So if we have transactions 1,2,3,4. Do we have Hash_1, Hash_2, Hash_3,Hash_4, Hash_12, Hash_34 (where Hash_1 is the hash value of transaction 1)? In this case, Hash_1234 is the merkel tree root, right?
I think I have so many dumb questions. If you could help me, I will very much appreciate it.
I thought maybe I could help you,
Hi Damien,
Thanks for the information.
Can you help me understand few basic questions:
1) If a block is verified, does it mean that the transactions in it are also verified? If yes, how does it make the system record authentic transactions (not exactly from a bitcoin perspective) ? Eg If a block has 20 pending transactions out of which 10 are authentic, is there a way to decline them?
2) What exactly is validation? Does every node (not every mining node) validate the block/transaction for it to be logged in a block? If yes, what happens if all the concerned parties do not authenticate- does this not increase the time in approving transactions?
Another basic question :-) What is the mechanism for resolving the winner if there are two valid blocks submitted at the same time by two miners that include overlapping but non-identical set of transactions? Is the block timestamp used to choose the winner? If so, does it create an incentive to use older timestamps?
I believe, and I may be wrong about this, that the block who gets the most confirmations first wins. I think there are a number of confirmations to get to make sure that the block has been propagated and validated.
Isn't the longest sub-block chain wins? Just making sure!
I believe it depends on what blockchain we are talking about. For Bitcoin, the block that receives the needed confirmations is added to the blockchain. Because it takes 10 minutes to mine a block, there are no conflicts between different blockchains states.
In Ethereum however, blocks are mined a lot faster, so there can be some conflicts in this regard. I don't know much about it, but it is called the GHOST protocol ( Greedy Heaviest Observed Subtree). This is basically what you said in your comment. The longest sub-block chain wins and becomes the truth.
It all depends how long it takes to mine a block. If it's low, you will need a system to resolve conflicts like Ethereum does.
Thanks Damien!
I've been loving your recent series of articles on the blockchain / blocks / mining. I had a few questions I was hoping you might be able to flush out:
I understand that in your example the difficulty was
1,931,136,454,487.72and that the valid hash was00000000000000000020c60222099aaebc6e7795784f74628ec640b223d3d339(18 leading zeros). I have two questions.If I understand correctly, all miners are using basically the same information in attempting to generate a valid hash (previous block hash, merkle tree root, target, timestamp, and version.) The only thing that "changes" miner-to-miner is the nonce. Is that right?
Maybe I'm misunderstanding, but it seems like a miner would benefit from trying random nonces in order to take a contrarian approach that potentially rewards randomness > sheer computing power.
Thanks again for any explanation you can provide. I've really found these articles extremely helpful.
1) My explanation wasn't clear at all on this point. Let me try again.
First, the target. I'll keep the same block example. In the left column, you have a bits field, with the value 402690497. This is the target bits. This number expresses the Proof-of-Work target as a coefficient/exponent format.
If we convert this number in hexadecimal, we get this: 0x180091C1.
The first two hexadecimals digits ( 18 ) are the exponent, the next six ( 0091C1 ) are the coefficient. Here is the formula used to calculate the target from this:
So by using the hexadecimal value above, we can calculate the target:
Which in decimal is =>
Back to hexadecimal (and a 64 length string) is =>
This represents the higher limit the hash must have. If the hash is lower than this target, it is valid ( here => 18 leading zeros ).
As for the difficulty, we divide the highest difficulty possible by the target found:
00000000FFFF00000000000000000000 /
00000000000000000091C100000013DE
= 1C1A0B3CF57
Which in decimals => 1,931,136,454,487
This means that compared to the first block every mined ( where the difficulty was 1 ), this block was 1.9 trillion times more difficult to solve. I may have skipped the decimals in this, but I hope you get the idea.
2) Well, I suppose the Bitcoin core developers are responsible for this. You can find the code used for the calculation here. They take the last 14 days worth of blocks and adjust the difficulty based on their findings.
3) Well, not every miner is using the same informations. I may not have explained this very well. Once a block is mined, everyone has access to the same informations and can make sure that the data is valid.
But, they won't be mining the block using the same informations. First, the merkle root will be different for everyone. The first transaction in the block ( coinbase transaction) sends bitcoins to the miner personal wallet. Each wallet address will be different, therefore the merkle root will be different too. It is also very unlikely that they will use the same timestamp for the block. So Joe finds the proper nonce at 2469953656 with its data, but other miners may not have a valid hash with this nonce because their timestamp and/or their merkle root was different.
As for the randomness of the nonces, I want to say you are right. Although I'm not quite sure how to prove it. Mining is absolutely a lottery.
If we have 2 dices, and the target is 2 ( the lowest possible ), we have only one throw possible to reach the target ( 1 on each dice ). This would mean that 1 throw out of 36 possible solves the problem, which is 2%. So, if I manage to hit the target with my dices, it doesn't mean I've thrown the dices 36 times, but the probability should be enough to prove that I've put the effort.
I'm not sure if that last paragraph has a lot to do with what you mean :D. I was just freelancing.
Sorry for the long answer. Let me know if anything isn't clear enough.
1) Amazing. This sent me on another rabbit hole to learn more about hexadecimals but was very educational.
3) Okay, I had gotten a bit confused about how the list of transactions was generated in a miner's candidate block. I now understand that they're drawn from the transaction pool and added to the candidate block at the miner's discretion. The specific transactions drawn, the order they're in, and the initial coinbase transaction will ensure the merkle tree is different for everyone. Additionally, as you point out, the timestamp would also be different.
The concept of fees and reward is really interesting. This is a block with zero transactions where the miner simply went after the reward.
I've held a fascination about blockchains for a while, but have found it really un-approachable and confusing. Your articles and explanations have been awesome. Thanks for the reply to my questions!
Happy to help :) These questions force me to learn on a more deeper level. Always welcomed.
It's an ingenious system when you think about it! But let me ask you a couple of dumb questions I couldn't figure out from the article;
Thanks again for this great series, Damien!
This is one of the problem with this Proof of Work mechanism. It has to consume a lot of electricity to solve the problem. If you don't win a block's race, the energy used during those 10 minutes you tried to solve the problem is wasted. This is one of the reasons why the blockchain mechanism doesn't scale so well.
I believe you can't compete as an individual today against mining farms. I know you can 'rent' some mining powers. Like, you pay 20 000 € to generate 0.00000001 BTC each second. As far as I know, you need to have a powerful installation to make a profit from this kind of activity.
I believe you are confusing the Coinbase plateform and the coinbase transaction. Yes, Coinbase is a plateform that allows you to buy and sell crypto currencies. You're right, it does look like a central bank now, because so much transactions go through plateforms such as this, which kind of defy the purpose of a decentralized currency.
The coinbase transaction is just a transaction that the miner includes in the block he is mining. The miner indicates that he is paying himself. In the article, our miner Joe said to the network that he calculated his reward ( 17 or so BTC ) and gave his wallet's address. When the block is validated, the miner receives the reward in its wallet.
It is called a coinbase transaction, but it has nothing to do with the Coinbase plateform.
Thank you for the kind words :)