DEV Community: thunderbiscuit

PGP for the Working Dev (2)

thunderbiscuit — Thu, 10 Jun 2021 02:29:11 +0000

The goal of this post is to get you up to speed on two of the main uses of GPG: cryptographically encrypting and decrypting data.

Note: this post is the second in a 2-part series aimed at developing a functional use of GPG for the day-to-day tasks of anyone who cares about software. Check out Part 1 to learn about signing and verifying signatures.

One important aspect of cryptographic keys is that they are simply numbers. They are often really, really big numbers, but simple integers nonetheless. Another important aspect of cryptographic keys as used in GPG is that they come in pairs: a public key and a private key. Knowledge of these numbers (the keys) allows us to perform many tasks, two of which we'll tackle here:

Encrypt a message in a way that only a person with the knowledge of a specific key could decrypt.
Decrypt a message that was encrypted for a private key you own.

There is much more to learn about privacy, encryption, and GPG than I cover in this series; for information on who invented this whole thing in the first place, check out Phil Zimmermann's Wikipedia page; for more on the difference between GPG and PGP, check out this article; to dig deeper into the specifics of GPG, check out their docs.

Priors

If you are coming to this article without having read Part 1, you should go and read the Priors section there. If you have already set up gpg, created a key, exported it, and imported other people's keys, you can safely move on to Task 3 on encrypting messages.

Task 3: Encrypting Messages

You can encrypt a message addressed to a specific recipient using their public key if it is in your keyring. For information on how to export your public key in order to share it with others as well as information about how to import other people's public keys, check out Task 1 from Part 1 of this series.

To encrypt the file message.txt in a way that only Morpheus will be able to decrypt it, we first check that his key in our keyring, then use the --encrypt command like so:

# check for the recipient's info in your keyring
# the second key here is our friend Morpheus' key we just imported
$ gpg --list-keys
pub   rsa1024 2019-09-14 [SC]
      0E94A26389C660DFE8C2F31705D085D9893479A7
uid           [ultimate] MrAnderson <MrAnderson@email.com>
sub   rsa1024 2019-09-14 [E]

pub   rsa1024 2019-09-14 [SC]
      5F357B756A571BF8A88567AF06898DFB48E3D899
uid           [ full ] Morpheus <morpheus@email.com>
sub   rsa1024 2019-09-14 [E]

# produce the encrypted file message.txt.asc
$ gpg --encrypt --armor -r Morpheus message.txt

Always make sure that the public key you use to encrypt is genuine! Call the person, verify on multiple channels, or message them on another encypted platform first before using their public key to encrypt a message. Note that you can enrypt a message to multiple public keys, but that will show up when people decrypt it (in other words you could not fake that a message was encrypted only for someone when in fact it was also encrypted for someone else at the same time).

Task 4: Decrypting Messages

Upon getting the encrypted file, our friend Morpheus could decrypt its content using

# print message to console
gpg --decrypt message.txt.asc

# send message to file
gpg --decrypt message.txt.asc > decryptedMessage.txt

And that's it. There is much more to this little piece of software than explored in these two articles, so go ahead and type gpg --help!

PGP for the Working Dev (1)

thunderbiscuit — Thu, 10 Jun 2021 02:29:05 +0000

Anyone who cares about software should learn how to produce cryptographic signatures on data (documents, messages, binaries, git commits) and how to verify other people's signatures on those types of data. It turns out it's rather easy, and all you need is a terminal and maybe half an hour of time!

The goal of this article is to get you up to speed on two of the main uses of the GPG cli software: cryptographically signing and verifying signatures on data.

Note: this post is the first of a 2-part series aimed at developing a functional use of GPG for the day-to-day tasks of anyone who cares about software. Check out Part 2 for the lowdown on how to encrypt and decrypt messages.

GPG stands for GNU Privacy Guard and is a small open source piece of software that allows you to manage keys for cryptographic purposes. GPG implements the OpenPGP standard maintained by the Internet Engineering Task Force. In practice, the terms GPG, OpenPGP, and PGP are often used interchangably even though they probably shouldn't be. You can think of OpenPGP as email and GPG as one implementation of the email protocol (like Hotmail or Gmail). PGP is the protocol, and the command line tool we use to perform tasks using this protocol is called gpg. Confusing I know.

Signing a piece of data in a way that only a person knowing a specific key could have done.
Verify the validity of a signature from someone else on a specific piece of data.

GPG does much more, but this post is really about those two basic tasks and their related options.

There is much more to learn about privacy, encryption, and GPG than I cover in this series; for information on who invented this whole thing in the first place, check out Phil Zimmermann's Wikipedia page; for more on the difference between GPG and PGP, check out this article; to dig deeper into the specifics of GPG, check out their docs.

Priors

Installing the Software

First, make sure you have GPG installed on your computer. You can get it on MacOS using homebrew (brew install gpg), on Linux using any of the package managers, or on Windows by downloading the binaries directly from the GnuPG official website. You will also need a bash terminal to interact with the software.

# if this returns a version number, you're good!
$ gpg --version

Creating Keys

The first thing we want to do is create a key set (a public and a private key pair). I recommend creating an set of test keys with a simple password to use for the rest of this tutorial, and deleting it afterwards:

$ gpg --full-generate-key  # generate a new key

The process of creating a new key pair requires that we answer some question about key type and size, as well as associate some information with the keys. If you answer the few questions required to build keys, you should soon find that you now have a key set available in your GPG keyring. To see what keys are available, enter:

$ gpg --list-keys
pub   rsa1024 2019-09-14 [SC]
      0E94A26389C61705D08560DFE8C2F3D9893479A7
uid           [ultimate] MrAnderson <MrAnderson@email.com>
sub   rsa1024 2019-09-14 [E]

We now have a key pair available to us for the tasks below. The --list-key command outputs some metadata about the key, as well as a fingerprint—the 0E94A26389C61705D08560DFE8C2F3D9893479A7 part. The fingerprint is a hash of the public key, and can therefore be used for identification.

Exporting and Importing Keys

To verify the validity of your signature on data or encrypt messages destined for you, other users will need to have your public key information. We can export our public key to a file using the following command, and then simply share the file with anyone we wish:

gpg --export --armor MrAnderson > MrAndersonPubKey.asc

Once you have someone's public key file, you'll need to import it into your keyring before being able to use it:

gpg --import MorpheusPubKey.asc

ASCII-armor Format

You can share your keys in binary format, or, often more conveniently, in a text format called ASCII-armor. You can print your public key to the console using the name or the email associated with the key as the last argument as in gpg --export --armor MrAnderson, or generate a file with your key in it using the following commands:

# generate public key file in binary format
$ gpg --export MrAnderson > myBinaryPubKey.gpg

# generate public key file in ASCII-armor format
$ gpg --export --armor MrAnderson > MrAndersonPubKey.asc

# print content of key file in ASCII-armor format to console
$ cat MrAndersonPubKey.asc
-----BEGIN PGP PUBLIC KEY BLOCK-----

mI0EXX0oXQEEAK/NyXBQxd1s9s3SYSVXMXfW0a3XR6JGwzf3ocfCu8OP12hVMvfo
BQfdj2WllNsWWpzNRJdeK2QCtLudykPNKtY7BuAk5bIWbGuVsNkWU/UmJZqijE0Q
aA+g2K1DFST5h4N12vLyLnN0On7RaJsTizR083E2QATsU/3VjP3Y3LUnABEBAAG0
IU1yQW5kZXJzb24gPE1yQW5kZXJzb25AZW1haWwuY29tPojOBBMBCAA4FiEEDpSi
Y4nGYN/owvMXBdCF2Yk0eacFAl19KF0CGwMFCwkIBwIGFQoJCAsCBBYCAwECHgEC
F4AACgkQBdCF2Yk0eadEqgQAm/LeK4BK77OPhy2jiouhmyhM922sATp07uP0s+WY
lJZZOlLSELc7FMn+iSzgsoBic432UFZVfJ5FZukn9oSSrvJagw7nx4Y3rIyUNDc+
0Dcw2lKH8WDqXXoANgWSewPb6P+kBq+ihVanaWgsG9a1nvb4/BoubMMoccAj3EaH
0Yq4jQRdfShdAQQArXj8/Ch3y//xeSPDXXu/HlylYT56s9gTOs8PzW/yeR8XAzGG
+YThFvMsRdfr8VYwr8fsNT+fS12IadaiwOX6ejs63LnE9WnyfnYdVTQO5ZTh61eU
zB1q97/YevWXIwTYAP/0h+tASqirffamdLZHctx54iozfb/M3QuSU2c4i5fAEQEA
AYi2BBgBCAAgFiEEDpSiY4nGYN/owvMXBdCF2Yk0eacFAl19KF0CGwwACgkQBdCF
2Yk0eacnRgP7BKkW5dMJn8aHKxSGmhcpeGnfI1Rp4MorfjcwCz/gqgtoiqOY99SK
maij8J92E5sKpdMpZXhQfjZjVDb91oAW0TdUEXbVVBSjZ4Ku33QwFCm+Qzud/xaT
lQDrnYx05SITsBAAmOrwfBRo464WfU/0rlXziE2E4wDY4U6IYtbrAcQ=
=aNWV
-----END PGP PUBLIC KEY BLOCK-----

Both the binary format and the ASCII-armor format are representations of the same data, but you'll find that the ASCII-armor is much easier to share (for example you could copy/paste it in an email or a text message).

Task 1: Sign a document (without encrypting it)

Signing a document (or any piece of data) creates proof that the document was signed by the owner of a specific public key, and that the document was not tampered with after the signature. Note that the signature is completely agnostic to the data it is signing, meaning that we can use GPG to sign sound data, text data, image data, or any other type of data we want.

The goal here is really to sign an open document. There is no encryption of the document in this procedure. As such, the document is usable by anyone whether or not they can verify its signature.

One of the common objectives of signatures is to attest that a specific version of the data is being stored or sent across a network. Once a piece of data has been signed, changing a single bit in the data will render the signature invalid. This is why, for example, the lead maintainer of the bitcoin core software will sign the downloadable releases of the software. Once downloaded, you can test for yourself whether his or her signature is valid on the download you just performed. If the signature is valid, you are in posession of the exact same piece of code that he or she signed. If the signature is invalid, the download was tampered with on its way to you. You can use this as an assurance that the software you download is trustworthy without having to trust the network you download it on. Note that it does not prevent the signator from writing bugs or malware into the software at all—it only attests to the fact that the version in your possession is the same as the version that was signed, bit for bit.

Let us suppose you have a file called message.txt in your current directory with the content hello, world, and that you wish to sign it.

gpg --sign message.txt

outputs a new file in the same directory called message.txt.gpg that contains the signature and the original message. The message is not encrypted. Anyone can retrieve the message from the file, whether they verify the signature or not.

Notice that the signed file is in .gpg format, a binary format. This can make it inconvenient to look at or share (try cat message.txt.gpg to see what I mean). Another very common way to save the file is in ASCII-armor format using the --armor switch like so:

gpg --sign --armor message.txt

The output file will be message.txt.asc, and will look something like this:

----------BEGIN PGP MESSAGE-----

owGbwMvMwME4vbHMPzN7+SPGNbJJ3LmpxcWJ6al6JRUlsVVLizNSc3LydRTK84ty
UjqOsDAwcjDoiSmyuPmV/RI73XDE80h8BUw7KxNIg4BMSX5JYo5Dem5iZo5ecn4u
AxenAExJ2E7mf/orZmwoeTJ1vspC35XrYvfoisoilj0k3uqnmdkfHX2t0eW0NeQN
TNi39keSVeD2utSvOzfZaoXsLUlekmUt7DGhg2lCRZmm53sHM58PFitPqmXNqOsU
+cjRljPzQHXgcfUbjDOmr3nvkbApSlp3yo53XS/e/0vgEFzFq+Y1oUwx4OqrJ+/9
eQA=
=c82p
----------END PGP MESSAGE-----

You can then copy and paste this anywhere to share or store the signed message. For example you could post it publicly on a website, send it as a text, or even write it down on a piece of paper.

Task 2: Verify the signature of a document

Let us assume someone has sent us a signed document (message_from_Morpheus.txt.asc) and we wish to verify its signature. We can use the --verify option to do so:

$ gpg --verify message_from_Morpheus.txt.asc
gpg: Signature made Sat 14 Sep 18:55:38 2019 EDT
gpg:                using RSA key 5F35BF8A88567B756A5F06898DFB48E3D8717A99
gpg:                issuer "morpheus@email.com"
gpg: Can't check signature: No public key

Note that if you simply have the file but not the public key of the signator in your keyring, the --verify command will state that the signature could not be verified as valid.

To validate the signature, we'll first have to import the public key of the signator on our GPG keyring. In this case let's assume your friend Morpheus just sent you his public key in ASCII-armor format (he knows how to export his key because he read Task 1 of this post), and the file is saved under MorpheusPubKey.asc. You can import his key in your keyring using:

$ gpg --import morpheusPubKey.asc
gpg: key 06898DFB48E3D899: public key "Morpheus <morpheus@email.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Now if you list your keys, you'll find you have 2 keys in your keyring:

$ gpg --list-keys
pub   rsa1024 2019-09-14 [SC]
      0E94A26389C660DFE8C2F31705D085D9893479A7
uid           [ultimate] MrAnderson <MrAnderson@email.com>
sub   rsa1024 2019-09-14 [E]

pub   rsa1024 2019-09-14 [SC]
      5F357B756A571BF8A88567AF06898DFB48E3D899
uid           [ unknown] Morpheus <morpheus@email.com>
sub   rsa1024 2019-09-14 [E]

When first importing a key, it won't be certified by default (GPG does not assume you really know that the key is the key from the person who it says it belongs to). The key could have been tampered with on its way to you, and anyone can create keys with anyone else's name. GPG does not, in any way, verify that you are the person who's name you are using to create a key.

It is recommended that you check that the key in your keyring belongs to the person you think it does using multiple sources, or that you take the time to call or text the person to verify. Once that is done, you can sign that key yourself with your own private key, adding a layer of protection to the validity of your friend's public key in your key ring.

gpg --sign-key Morpheus

This will identify the key as [full] instead of [unknown] in our keyring. Notice that we do not need to sign a public key to use it in signature verification; it simply is an added layer of certainty when verifying signatures.

Once we have the public key of the signator of the piece of data we wish to verify in our keyring, we can verify the signature on any file in one line of code. GPG will print the result of the signature verification on the console. In the case of a valid signature, it would look like this:

gpg --verify message_from_Morpheus.txt.asc
gpg: Signature made Sat 14 Sep 18:55:38 2019 EDT
gpg:                using RSA key 5F357AF0687B756A571BF8A885698DFB48E3D899
gpg:                issuer "morpheus@email.com"
gpg: Good signature from "Morpheus <morpheus@email.com>" [full]

To see the content of the message, you'll want to either print it to the console or create a new file with the original message in it.

$ gpg --decrypt message.txt.asc
what's up dog?
gpg: Signature made Sat 14 Sep 18:55:38 2019 EDT
gpg:                using RSA key 5F357AF0687B756A571BF8A885698DFB48E3D899
gpg:                issuer "morpheus@email.com"
gpg: Good signature from "Morpheus <morpheus@email.com>" [full]

which will print it to the console, as well as print the result of the signature verifcation. You could have used --decrypt on a message without the public key in your keyring and GPG would still have printed the message, but it would have told you that it could not verify the signature the same way as it did above with the --verify command.

Anyone in posession of the file can also output the original data to a file using

$ gpg --decrypt message_from_Morpheus.txt.asc > Morpheus_message_only.txt
gpg: Signature made Sat 14 Sep 18:55:38 2019 EDT
gpg:                using RSA key 5F357AF0687B756A571BF8A885698DFB48E3D899
gpg:                issuer "morpheus@email.com"
gpg: Good signature from "Morpheus <morpheus@email.com>" [full]

# the cat command prints to console the contents of a file
$ cat Morpheus_message_only.txt
what's up dog?

This will print the verification of the signature to the console as the file is created, but the signature verification will not be included in the newly created file; the new file is now an exact replica of the original file.

Extras

Clearsign

Note that the signature file build in ASCII-armor format in Task 1 does not contain a human readable version of the original data (in this case the string hello, world). The file is not encrypted, but you'll need gpg installed and a command to show its content.

# create a signature file named message.txt.asc
$ gpg --sign --armor message.txt
$ cat message.txt.asc

----------BEGIN PGP MESSAGE-----

owGbwMvMwME4vbHMPzN7+SPGNbJJ3LmpxcWJ6al6JRUlsVXfJmak5uTk6yiU5xfl
pHQcYWFg5GDQE1NkcfMr+yV2uuGI55H4Cph2ViaQBgGZkvySxByH9NzEzBy95Pxc
Bi5OAZiSV+3M/11+bLr0h3OFrsyTxPfv3irrfRX0m/U3aOXlwwd/KCju8Od59+5J
uGP84n0KOQbvjGbnvv31R7Nh1tz3b1r3aQks4Fglndb581JBIMN0M/dPxaKejOzH
61tOVRS3Kepk1L77tnv/XkeftjcHA1mrm0WCrS9sOlX2YEOg2Kt3HuY5nop7g9Xn
nwMA
=/jWM
----------END PGP MESSAGE-----

Using the --clearsign command on ASCII-armor format to sign a document (rather than the --sign command) will include the original data in the signed file. The resulting file keeps the signed message at the top, making it easy to parse, whether we verify the signature or not.

$ gpg --clearsign --armor message.txt
$ cat message.txt.asc

----------BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

hello, world
----------BEGIN PGP SIGNATURE-----

iMQEAQEIAC4WIQRGTnb6FsuAxEnEX3iXgXZPaWun4gUCXXr2FxAcdG90YWxAZ21h
aWwuY29tAAoJEJeBdk9pa6fi038D/1dXYUkFlKak1jRH369BjU5UaSmjiJvoWM8j
tQW78RVuYlOosL7bRSzgXJK1YH4hiz4rj3UWlbwRmF6s+UEu9wfDisHssHEiijru
LqCZpsWEDkzihjVwKSRz1UNIm1Egg4jyEV2ZY32dJnveN6wCiB1ABMe1zvB3G2cI
72Y3KOLr
=nKbM
----------END PGP SIGNATURE-----

Detached Signatures

A popular way to sign software is through the use of detached signatures. A detached signature is a small file that sits external to the actual softare or piece of data being signed.

Detached signatures have one great advantage over normal signatures: because they are separate, they are not mandatory in order to use the data. The signatures for popular software (say, Python, for example), are downloaded separately, and therefore do not interfere with one's ability to simply download and start using the software. On the other hand, verifying the validity of such signatures is just as easy as it is with regular signatures.

We create a detached signature using the following command, which will produce a small file called mySoftware.dmg.sig:

$ gpg --detach-sign mySoftware.dmg
$ ls
mySoftware.dmg
mySoftware.dmg.sig

Now let us assume we have downloaded a piece of software and its associated signature (python.dmg and python.dmg.sig). Provided the files are in the same directory and have the same name, we can verify the signature like so:

$ ls
python.dmg
python.dmg.sig

$ gpg --verify software.dmg.sig
gpg: assuming signed data in 'python.dmg'
gpg: Signature made Sat 14 Sep 19:28:09 2019 EDT
gpg:                using RSA key 0E49C5D08660D9A2638FE8C2F31705D9893479A7
gpg:                issuer "mranderson@email.com"
gpg: Good signature from "MrAnderson <MrAnderson@email.com>" [ultimate]

That's it! Check out part 2 for how to encrypt and decrypt data using GPG.

Deconstructing a bitcoin transaction

thunderbiscuit — Thu, 06 May 2021 00:38:55 +0000

Here is a hex dump of the raw bitcoin transaction e778e8765fdbb60f62e267de4705789f526a5fe9bb0c0f5e56ab4f566c5240eb:

You've seen bitcoin transactions expressed in this form if you've ever tried to dive into the more technical bitcoin books, or have just been lurking around in the bitcoin community for long enough. They seem arcane and of impenetrable complexity, but it turns out that that's not the case at all; the above transaction, for example, is composed of 15 small building blocks, each one easy to find and interpret.

Studying the structure of bitcoin transactions in their "true" form is a valuable quest for all bitcoiners. This article is written with the curious bitcoiner in mind looking to get a high level understanding of what exactly gets passed around from node to node on the network. Let's get to it.

Priors

To really understand all pieces of a transaction, a basic grasp of 4 foundational, non-bitcoin concepts are important: bytes and hexadecimal notation, endianness, variable-length fields, and varints. I address all 4 of those in the sections below, but if you are familiar with computer science in general you most definitely won't need this kind of entry level review. If you are not familiar with them, however, I think you'll find them enlightening, and they will most definitely serve you well on your journey into bitcoin.

1. Bytes and Hexes

You might have seen "raw" bitcoin transactions printed in hexadecimal format (the transaction above is an example of that). But of course computers only speak the language of bits (0s and 1s). A bitcoin transaction in it's computer-understandable form is therefore a string of binary digits. Moreover, those 0s and 1s are always kept in small groups of 8 bits, called bytes. Here is an example of a byte: 1101 0110.

The problem with binary code is that it's not easy for humans to parse. The transaction above (and it's not even a big one) written in binary format is exactly 1,800 digits (10101011100101000101100000110100000011101...).

But binary numbers are still numbers; if we write them for humans we can write them in any we want that works best. Decimal notation (our regular number system), Roman numerals, Korean characters. etc.

Decimal notation would be an obvious candidate, but it turns out that it is not very convenient for working with bytes. Take for example the bytes 1101 0110 and 0000 0010. In decimal notation, the first number is 214, whereas the second number is 3. In fact every byte, when written in decimal notation, will take between 1 and 3 digits. That's not convenient, because then you'd never know when a byte ends and when the next one starts—how many bytes is 21431042? There can be multiple interpretations.

Instead, a preferred numbering system for writing bytes for human consumption is the hexadecimal number notation. A full explanation is beyond the scope of this post, but you should know that the number of possible arrangements of 4 bits is 16, and that the hexadecimal notation system has exactly... 16 digits. They map out nice and tidy with half a byte.

Notice that we can now represent our bytes as two hexadecimal digits, for a clean notation. Each byte is two hexadecimal digits, like so:

When you're looking at a big hexadecimal string like the one at the beginning of this post, you're really just looking at a neat little representation of a bunch of bytes, with each block of two characters representing one byte.

2. Big-Endian and Little-Endian Ordering

Endianness refers to the order of bytes within a representation of a number. You can think of it as the "direction" in which bytes should be read for meaning, while understanding that either direction does not influence the ultimate meaning of the bytes.

Take for example the number one thousand five hundred and ninety two. You are probably used to reading a number like this in the following way: 1592. At the same time, if I told you that I had this weird habit of always writing my numbers from right to left, you would still know that 2951 means one thousand five hundred and ninety two, because you'd know me and understand my weird habit.

It turns out that some computer architectures are more efficient when working with numbers if they are stored with the least significant byte first (the equivalent of reading right to left), and so a lot of the numbers we use when communicating with computers are "translated" to that format. We call this computer version of a number little-endian, because it starts with the little end. When shifting from big-endian to little-endian, it's the bytes that we shift. This means shifting the last two characters (remember that one byte equals two hexadecimal characters) for the two up front, and so on.

The number 220,000 in hexadecimal notation is written 03 5b 60, but expressed in little-endian it becomes 60 5b 03.

A real life example would be our transaction id for this article, which if referred to within a bitcoin transaction will be written in its little-endian form, but if you try to look it up in a block explorer, you'll need it's "human", big-endian version:

3. Variable-Length Fields

You'll notice that the bytes in a transaction are all glued together in one big continuous blob. How does the software know where an input start and where it ends? How does it know if a certain byte belongs to the number of bitcoin transmitted or to the receiving address? The flexibility offered by bitcoin transactions implies that there are an extremely rich number of combinations possible, and that the scripts required for unlocking utxos vary greatly in length.

One way to deal with this uncertainty would be to give every field a set length in bytes. This is what the version field does, for example: the version number of a transaction is always written in the first 4 bytes of a transaction (see the 01000000 number that starts the transaction below). In a lot of cases, however the length needed to transmit the necessary data differs widely between transactions: unlocking scripts for a simple Pay to Public Key hash might be 106 bytes long like in the transaction we are using in this post, but they can easily be 5 times that size on complex multisig scripts. Giving a fixed length to that data section would not only be inefficient (a lot of transactions would not need that much space at all), it would also be limiting, because scripts would have to stay under that size.

A better way to deal with this and keep both flexibility and efficiency is by using a small marker at the beginning of a variable-length section that will give the software an indication of the lenght of the section to follow. Here is an example of a series of 6 bytes: 05 e7 78 e8 76 5a. If we knew that the first byte was a indicator byte for the length of the section, a plain-english reading of this would then look like this:

byte 1: 05 >>> the following section is 5 bytes long
bytes 2 to 6: e7 78 e8 76 5a >>> data

Bitcoin transactions use a mix of fixed-length fields and variable-length fields. I'll make note of which ones use which in the description of each parts.

4. Varints

Variable Integers (varints) are a way to write a very wide range of numbers in a way that minimizes their cost on transaction space.

To understand how they accomplish this, first note that the bigger the number we need to write down, the more bytes it requires. Three is written as 00000010 (1 byte), whereas two million twenty nine thousand five hundred and twelve is written as 000111101111011111001000 (3 bytes).

The problem we are faced with is that all bytes are glued together in one long string of 0s and 1s, and the software needs to know exactly where each of the fields pertinent to a transaction start and end. One way to deal with this is to give fields a never-changing length, so that we always know when they end. This is what the version field does, for example: the version number of a transaction is always written in the first 4 bytes of a transaction (see the <code 01000000 number that starts the transaction below). The problem with this approach is that if we sometimes need to accommodate numbers of great size, we will need to give the field a length with the ability to accommodate all of those numbers (say 8 bytes dedicated to a particular field). But if in most cases we only use the field for very small numbers, then a lot of those bytes are just wasted, because those small numbers only need 1 byte. If this type of field is required in multiple places in a transaction, all that waste starts to add up. Rather, we need a solution that will use only the space required for the number we wish to write. This is what varints do; they use 1 byte for most of our use cases, and up to 9 bytes for the really big numbers we don't expect often.

The way this is achieved is simple. A single byte can normally be used to represent the numbers 0 to 255. If the number we need represented (say, the number of inputs in a transaction) is below 253, we write it in the first byte, and the software knows that that's all there is to it. If the number is big enough that it needs a few more bytes to write, we instead write 253 in that first byte, which will be interpreted by the software as "read the next two bytes as the actual number I need to communicate". If the number is even bigger, we use 254 instead, meaning "read the next 4 bytes for the actual number", and if our number needs even more, we use 255, which implies the next 8 bytes are the actual representation of our number. Easy and efficient; most varints used in transactions never need to be more than one byte, but they can all grow to accommodate incredibly large numbers.

A Legacy Bitcoin Transaction

A "legacy" bitcoin transaction is the name we give a transaction that does not implement Segregated Witness, a newer form of transaction in which the "witness" data (fields 5 and 6 below) are put into their own special section (we say they are segregated, hence the name).

These legacy transactions are perfectly valid bitcoin transactions, but they are being used less and less because of the efficiency gains made by the segregated witness approach resulting in lower fees, as well as its fix of the transaction malleability bug, enabling, among others, the creation of lightning channels. Legacy transactions are easy to identify because they involve unlocking utxo(s) belonging to addresses starting with a 1.

This article breaks down a typical legacy transaction where one utxo is used and spent into two: one payment to a payee, and one payment to a change address. It contains 15 different fields, and I describe each of their purpose below.

[1] transaction version number

This field specifies the type of transaction being transmitted. It is of fixed-length (4 bytes) and is little-endian.

In our example:

01000000

The field indicates that this transaction is of version 1.

[2] number of inputs

This field expresses how many inputs will be unlocked by the transaction. Each of those inputs will need to be identified (here with fields 3 and 4), and unlocked (here with fields 5 and 6). In our case there is only one input, and so we only need to go through this loop once, but in the case where there are many inputs, we repeat the fields 3 to 6 as many times as there are inputs. This field is a varint, is little-endian, and can grow up to 9 bytes.

In Our Example:

01

The byte indicates that the transaction unlocks only one UTXO.

[3] previous transaction id

This field expresses the transaction which contains the output to be unlocked by the unlocking script in the coming fields 5 and 6. It is of fixed-length (32 bytes), and is little-endian.

In Our Example:

04dde43b0e4724f1e3b45782a9bfbcc91ea764c7cb1c245fba1fefa175c3a5d0

Because the field is little-endian, if you wish to search for that transaction in a block explorer you'll need to convert it to big endian first: d0a5c375a1ef1fba5f241ccbc764a71ec9bcbfa98257b4e3f124470e3be4dd04. A look at that transaction will reveal that there were 2 outputs to it. Which one of those two is unlocked by the signature script is defined in the next field.

[4] ouput number in previous transaction

Defining the transaction an output comes from is not precise enough—there might be more than one. We need to know which output from that transaction is being unlocked, and this field expresses that. It is of fixed-length (4 bytes), and is little-endian.

In Our Example:

01000000

This is the number 1 written in little endian, indicating that the output being unlocked is the second one in transaction d0a5c375a1ef1fba5f241ccbc764a71ec9bcbfa98257b4e3f124470e3be4dd04. It is valid for 300,000 satoshis, or 0.00300000 bitcoin.

[5] size of the unlocking script

This field indicates the number of bytes taken by the unlocking script, the field that follows it. It is a varint, and can take up to 9 bytes.

In our example:

6a

This byte is the hexadecimal representation of 106, meaning our unlocking script (field 6) will be 106 bytes long (212 hex characters).

[6] Unlocking script

You can think of the unlocking script as the key that unlocks the utxo. If any of the unlocking scripts fail for any of the input utxos, the whole transaction fails. If all unlocking scripts succeed, the signer has proven they have ownership of the coins, and the transaction can move forward to the next steps. This field is of variable length.

In our example:

4730440220519f7867349790ee441e83e545afbd25b954a34e0733cd4da3b5f1e5588625050220166730d053c3672973bcb2bb1a977b747837023b647e3af2ac9c15728b0681da01210236ccb7ee3a9f154127f384a05870c4fd86a8727eab7316f1449a0b9e65bfd90d

The unlocking script is written in a language called Script, a language unique to bitcoin. It is beyond the scope of this article to look at the exact unlocking script used in this transaction, but we know it was a valid script, since the transaction was indeed propagated by the network, and later on mined.

[7] sequence number

The sequence number is a field initially designed for a purpose it never fulfilled. Nowadays it is often disabled by setting it to ffffffff. It can used to signal that a transction is replace-by-fee enabled as per BIP 125, by setting the field equal to any number below ffffffff -1. In some cases, the field is used to set timelocks (to enable this, verion 2 of a transaction must be declared in field 1). It is of fixed-length (4 bytes) and is little-endian.

In our example:

ffffffff
The field is disabled in this transaction.

[8] number of outputs

This field expresses how many outputs the transaction will create. It is a varint.

In our example:

02

The transaction has two outputs.

[9] amount going to output 0

This field expresses the amount of bitcoin being locked in output 0, expressed in satoshis. It is a fixed-length field of 8 bytes, and is little-endian.

In our example:

5d36010000000000

Output 0 locks in 79,453 satoshis.

[10] size of locking script for output 0

This field expresses the size of the locking script for output 0. It is a varint.

In our example:

19

This byte is the hexadecimal representation of 25, meaning our locking script will be 25 bytes long (50 hex characters)

[11] locking script for output 0

This field is the locking script for output 0. It is a variable-length field.

In our example:

76a91478364a559841329304188cd791ad9dabbb2a3fdb88ac

We can think of this field as a of lock we put on output 0. It is written in Script, bitcoin's own programming language.

[12] amount going to output 1

This field expresses the amount of bitcoin being locked in output 1, expressed in satoshis. It is a fixed-length field of 8 bytes, and is little-endian.

In our example:

605b030000000000

Output 1 locks in 220,000 satoshis.

[13] size of locking script for output 1

This field expresses the size of the locking script for output 1. It is a varint.

In our example:

19

This byte is the hexadecimal representation of 25, meaning our locking script will be 25 bytes long (50 hex characters).

[14] locking script for output 1

This field is the locking script for output 0. It is a variable-length field. This is a variable-length field.

In our example:

76a914064e0aa817486573f4c2de09f927697e1e6f233f88ac
We can think of this field as a of lock we put on output 1. It is written in Script, bitcoin's own programming language.

[15] nLocktime

nLocktime field allows for a transaction to be unspendable until a certain point in the future. If the field is set to 00000000, the transaction is spendable right away. If they field is any number below 500 million, it is intepreted as a block height. If it is above 500 million, it is interpreted as a Unix timestamp. Transactions with locktimes on them will not be propagated by nodes if they are not valid at the time a node see it, hence the sender must wait until the transaction is valid before broadcasting.

In our example:

00000000

In our example the nLocktime field is such that the transaction is spendable right away.

Some notes on the above transaction

The example used here is a type of transaction known as a Pay to Public Key Hash, or P2PKH. It is the simplest form of transactions we see nowadays. The transaction hex has 450 characters, and the transaction is therefore 225 bytes in size.

TXID

The txid (transaction identifier) is derived from hashing the transaction data twice using SHA256. You can test this with our example transaction right in your shell. The following command basically takes the hex dump of the transaction, converts it to binary, hashes it, then converts that result to binary again, and hashes it once more. It is then printed to console in little-endian, hex format. Notice that you'll need to convert it to big-endian if you want to use it in a block explorer!

echo 010000000104dde43b0e4724f1e3b45782a9bfbcc91ea764c7cb1c245fba1fefa175c3a5d0010000006a4730440220519f7867349790ee441e83e545afbd25b954a34e0733cd4da3b5f1e5588625050220166730d053c3672973bcb2bb1a977b747837023b647e3af2ac9c15728b0681da01210236ccb7ee3a9f154127f384a05870c4fd86a8727eab7316f1449a0b9e65bfd90dffffffff025d360100000000001976a91478364a559841329304188cd791ad9dabbb2a3fdb88ac605b0300000000001976a914064e0aa817486573f4c2de09f927697e1e6f233f88ac00000000 | xxd -revert -plain | sha256sum | xxd -revert -plain | sha256sum

On Signing and Broadcasting

An often overlooked aspect of bitcoin transactions is how creating them and broadcasting them are two completely separate tasks, and that they can be done independently of each other. We mostly use wallets that construct and sign transactions and also broadcast them for us, but it does not have to be so.

This is what projects like TxTenna and the Lightning Network are leveraging.

That's it!

I hope this article proved to be an interesting way to peel the first layer on bitcoin transactions if you had not seen them this way before. More to come!