DEV Community: Peter Tyonum

Generating and Working With ScriptPubKeys in Bitcoin Transactions

Peter Tyonum — Tue, 27 Feb 2024 18:11:12 +0000

Introduction:

Bitcoin transactions involve locking funds in scripts, which can only be spent if those locking conditions are met. The part of the script that expresses these locking conditions are called ScriptPubKeys. On the other hand, the part that provides unlocking scripts to satisfy the locking conditions is referred to as ScriptSig for legacy transactions, and ScriptWitness for SegWit Transactions. These scripts are evaluated by a stack-based language called Script. This article will mainly focus on ScriptPubKeys.

As mentioned previously, ScriptPubKeys are scripts that embody the locking conditions of bitcoins in a Bitcoin transaction. They are a crucial element of a Bitcoin transaction that specifies the requirements that must be met before an unspent transaction output (UTXO) is used. The ScriptPubKey, combined with the Amount, creates an output of a Bitcoin transaction. Like every other script, ScriptPubKeys are made up of operators and data. During the encoding of a transaction, both data and the operands are represented in hexadecimal format. You can find the hex representation for all operands used in Bitcoin Scripts here.

When encoding transaction data, ScriptPubKeys and ScriptSig are typically preceded by the length of the script. To represent the length of a script whose length ranges between 1 to 75 bytes, a single byte is used, with hexadecimal values ranging from 0x01 to 0x4b. If the script's length is between 76 and 255 bytes, it is preceded by a PUSHDATA1 operand. Similarly, if the length is between 256 and 520 bytes, it is represented by PUSHDATA2.

During the evaluation of transactions, ScriptSigs are evaluated first, followed by ScriptPubKeys. If both evaluate to true, the value is transferred to a new destination. So how are ScriptPubKeys derived?

Deriving ScriptPubKeys

One of the simplest ways to obtain the ScriptPubKeys is through the Bitcoin Command Line Interface (CLI). This method is ideal for development and testing purposes. Once you generate an address using the bitcoin-cli getnewaddress command or you have an address available, you can check the address information to obtain the ScriptPubKey value using the bitcoin-cli getaddressinfo <address> command. This command will produce various outputs, including the ScriptPubKey value, highlighted in the screenshot below.

Now, let’s look at how we can derive ScriptPubKeys outside the fancy world of Bitcoin cli for all types of standard scripts.

Pay to Public Key (P2PK)

Although the P2PK scripts have been depreciated in Bitcoin core, standard P2PK ScriptPubKeys are derived using the following script combination:

<publickey> OP_CHECKSIG

The above script combination is then encoded in hex format. When using ECDSA-compressed public keys, the above may likely be 35 bytes long. This is because the public keys are 33 bytes long, a single byte for OP_CHECKSIG (ac in hex) and another byte to indicate the total length of the P2PK ScriptPubKey, resulting in a total of 35 bytes long script.

Pay to Public Key Hash (P2PKH)

P2PKH used in P2PKH transactions are now considered legacy transactions. They have been replaced by newer Segwit P2WPKH outputs that are non-malleable and smaller in size. Nonetheless, the P2PKH scripts have some advantages over P2PK. For instance, the public key hash is shorter than the bare public key, resulting in lower storage requirements. To encode a P2PKH script, the below procedure is followed:

OP_DUP OP_HASH160 <pubkey hash> OP_EQUALVERIFY OP_CHECKSIG

To obtain the public key hash, take SHA256 of the public key, and then RIPEMD160 of the resulting hash (HASH160(publickey)).

Pay to Multisig (P2MS)

The P2MS standard enables multiple parties to manage their assets collectively by enlisting public keys (n) and the minimum number of signatures (m) required to unlock funds locked in its ScriptPubkey. This reduces the risk of a single point of failure. Although P2MS produces the longest script, it ensures safety by requiring multiple signatures. The P2MS standard is defined as follows:

<m> <n pub keys> <n> OP_CHECKMULTISIG

Here, 'm' represents the required signatures, 'n pub keys' refers to all the public keys, and 'n' denotes the total public keys.

Pay to Script Hash (P2SH)

P2SH enables complex conditions to be embedded in a script. The script is commonly used in multisig transactions where several parties provide public keys used in deriving the script hash. P2SH ScriptPubKeys are generated using the following formula:

OP_HASH160 <script Hash> OP_EQUAL

To unlock the above, you need to present the custom secret which was hashed to obtain the script hash (Redeem script).

Pay to Taproot (P2TR)

P2TR is a Bitcoin script that locks bitcoins to a script that can be unlocked using a public key or a Merkelized Alternative Script Tree (MAST). P2TR scriptpubkeys often take the shape of a single Schnorr public key. They are however derived from a combination of other public keys. Outputs locked in a P2TR are spent either by a ScriptSig for the public key or any of the scripts within the Merkle tree. To construct a locking condition for P2TR, a few steps are required. First, Schnorr public keys are derived, which are 32 bytes long. Next, P2PK scripts are derived using the formula:
<len> + <pubkeys> + OP_CHECKSIG
After that, Tapleaves are calculated, and then their Taptweak is computed. If you would like to learn more about how to derive P2TR ScriptPubKeys, you can find more detailed information here.

Segwit?: You may be wondering if Segwit implementation has caused any changes in the ScriptPubKeys of the standard scripts mentioned above. The answer is no, Segwit did not introduce any new structure to the scripts. Instead, it relocated the unlocking scripts to the witness section of the transaction data.

There is a unique ScriptPubKey known as NULL DATA, used primarily to store data on the blockchain. The format for this type of ScriptPubkey is:
OP_RETURN <data>
when the Script execution comes across OP_RETURN, it will halt and return false.

Validating ScriptPubkeys

During development, you might want to check that a derived ScriptPubKey is valid. The easiest way to do this by using the Bitcoin-cli. Simply pass the script to the decodescript <script> command and it will return the details of the script.
Example below:

Pls note that in the above example, bcr is an alias for bitcoin-cli -regtest
In the above screenshot, we are encoding a P2PK script as displayed on the type. It has a pubkey of 039ecd8fee6e102eae1a71f99bc14694a51bba8938c6d5785c81012bd4a364a901 and an OP_CHECKSIG which conforms to the P2PK stated above.

How to use ScriptPubKey in a Bitcoin Transaction

After constructing a ScriptPubkey using any of the above standard scripts, you might want to include it in a transaction. You do this by calculating its length and prefixing it in hex format.

Summary: This article discussed various methods to express standard locking conditions in a transaction. We covered how to derive ScriptPubKeys using the Bitcoin command line by providing an address. Additionally, we explored different standard scripts that can be implemented when working on a transaction. We also learned how to verify the accuracy of a derived script and concluded by examining how these scripts can be utilized when constructing a transaction. Thank you for reading and I welcome your feedback.

References:

BIP 141: Segregated Witness
Scripts: Scripts
Chaincodelab: Transaction Tutorial

SETTING UP BTCD FOR A DEV. ENVIRONMENT

Peter Tyonum — Thu, 16 Feb 2023 16:15:30 +0000

Bitcoin is made up of computing devices connected with each other on a peer-to-peer (P2P) network running the Bitcoin software. These computing devices participating in the network are called nodes. A Bitcoin node has a blockchain database, a routing mechanism to receive and propagate transactions and blocks, a mining function to validate and create new bitcoin, and a wallet to manage user keys. A node can have all (full node) of these features or selected features according to the needs of the user.

The first and most popular full-node client implementation is the Bitcoin core. It is the original client that was released by Satoshi Nakamoto. It is open-source and actively maintained by the Bitcoin community. Installable binaries exist on bitcoin.org for any platform of choice.

Another well-known client implementation is the BTCD by Conformal Systems. It is also an open-source project that has already been used in production as far back as 2014 and is actively maintained. Btcd implements most of the core functionalities of a node except wallet and chain functionality. The team has instead unbundled the wallet functionality into a separate wallet application called btcwallet.

btcd client also has minor differences from bitcoind such as enabling TLS for RPC connections by default and accepting both HTTP requests and Websockets. You can read their blog post here about why they were separated.

btcWallet is a hierarchical deterministic (HD) bitcoin wallet client for a single user. It acts as both an RPC client to btcd and an RPC server for wallet clients and legacy RPC applications. It uses an HD path for address derivation (BIP0044) and encrypts both private keys and public data. You can read more about the wallet on their official repository here.

Having been using Bitcoin core for some time, I wanted to see how well I can work with btcd client. So the first thing is to compile btcd from the source. The following sections document how I was able to set up the btcd node.

COMPILING BTCD

One prominent thing I have noticed while compiling BTCD is that a lot of details are abstracted. I was wondering where to place the configuration file, what is the command line tool, ports for RPC calls, etc. I will attempt to answer these questions in this article.

btcd is said to provide a compatible RPC interface with Bitcoin core, I am assuming that most features provided in bitcoind aside wallet and chain are found in btcd.

Following the instructions on their official repo, first, install go and check the version with go version, root path, and path with go env GOROOT GOPATH.

Next, cd $GOPATH and create the subdirectories mkdir -p src/github.com/btcsuite/btcd

Clone the repo into the directory: git clone https://github.com/btcsuite/btcd $GOPATH/src/github.com/btcsuite/btcd

Then install: GO111MODULE=on go install -v . ./cmd/...

List installed btcd and its utilities: ls $GOPATH/bin/

We have addblock, btcctl, btcd, findcheckpoint, gencerts, lncli, lnd

btcctl - is a command line utility used to query and control btcd via RPC.

gencerts- is another command line utility used to generate TLS certificates for a specified host

lncli is a command line tool for querying lnd via RPC.

lnd Lightning Network Daemon is a lightning network node implementation by Lightning labs.

There are other Bitcoin-related Go packages released by the team, you can check them out here

BTCD NODE CONFIGURATION USING btcd.conf FILE

Next, we will create our configuration file. Now that btcd has been successfully installed, run the command btcd —help to see where we can place our configuration option.

For the macOS platform which I am using, the default /Users/{username}/Library/Application Support/Btcd/btcd.conf.

Navigate to the Btcd directory and create a config file: cd /Users/{username}/Library/‘Application Support’/Btcd && touch btcd.conf && code btcd.conf

Set the following options in the config file:

Network settings: simnet=1 or testnet=1 depending on the test network you want to use.
RPC Server: rpcuser={username}, rpcpass={password} set a strong password
Indexes: txindex=1
Debug: debuglevel=info

Summary of config file:

rpcuser="username"
rpcpass="password"
simnet=1
txindex=1
debuglevel=info

Save the file and return to the terminal and start btcd by running btcd.

SETTING UP BTCCTL COMMAND LINE TOOL

Next, we will configure btcctl to interact with btcd similar to bitcoin-cli for bitcoind.

Create a btcctl.conf file inside the default directory for your platform. For macOS, it is at /Users/{username}/Library/Application Support/Btcctl/btcctl.conf, check the default location for your platform by running btcctl -h.

Set the credentials to be same as those used for btcd: rpcuser="username", rpcpass="password", and network: simnet=1

You should be able to query btcd now.

You can use btcctl -l to list all available command options or visit this documentation page here for further information.

Thank you.

References:

Btcd official repo: https://github.com/btcsuite/btcd
Btcd documentation: https://btcd.readthedocs.io
Conformal blog post: https://blog.conformal.com/btcd-not-your-moms-bitcoin-daemon

BITCOIN TRANSACTION VALIDATION: P2PKH

Peter Tyonum — Mon, 30 Jan 2023 17:37:13 +0000

Part 1:

In Bitcoin, transactions transfer value from the sender to the receiver. They consume unspent transaction outputs (UTXO) and produce new UTXOs. In this article, we will examine how the Bitcoin script's execution engine validates transactions thereby permitting value to be moved from one user to another.

On a high level, transactions are made up of inputs (except coinbase transactions) and outputs. Inputs contain unlocking scripts, transaction id(s), locktime, the recipient address as well as a change address provided there is a change.

As stated earlier, inputs contain transaction id(s) (txid) and they are references to UTXOs. When referencing a UTXO, their output index(vout) is also specified. A single transaction can reference more than one txid which sums up to the amount of bitcoin being sent. The sum of these inputs must be greater than or equal to the sum of output(s) for the transaction to be accepted as valid.

Transaction amounts are represented in units called satoshis and one unit of bitcoin is 100,000,000 satoshis. The difference between the sum of inputs and that of outputs is collected as fees by the miners.

Each txid is stated alongside its unlocking scripts that must satisfy the locking conditions placed on the output. Depending on the type of output, the unlocking script can be constructed from a pair of private and public keys or it may require more than a pair of keys to construct the unlocking scripts.

Transaction inputs also contain locktime which specifies when the transaction will be valid and can be broadcasted to the network. In most transactions, the locktime is set to zero to indicate immediate validity. However, if the locktime is not zero and is below 500 million, it is interpreted as the block height but if it is above 500 million, it is interpreted as Unix Epoch timestamp (seconds since Jan. 1st, 1970) all stating when the transaction becomes valid.

Outputs on the other hand state the value of bitcoin sent and the locking conditions to spend them. Similar to inputs, a transaction can have more than one output each having a value, an index, and a locking script.

For a transaction to be accepted as valid and mined into a block, it has to conform to the consensus rules. These rules are set out in the Bitcoin node's software. Some of these rules in addition to those stated above are that the UTXOs have not been spent, and the unlocking scripts must satisfy the locking scripts amongst others.

There are many types of Bitcoin transactions: Pay-to-public-key-hash (P2PKH), Pay-to-Script-Hash(P2SH), Pay-to-Witness-public-key-hash(P2WPKH), Pay-to-Witness-Script-Hash(P2WSH) and Pay-to-TapRoot(P2TR). The most common type amongst them is P2PKH, which requires only a pair of keys to sign and unlock outputs.

To spend a P2PKH output, the sender has to present an unlocking script called ScriptSig which comprises the public key and digital signature created from the senders' private key. This is validated by the Bitcoin script execution engine according to the consensus rules.

Let's examine a P2PKH transaction below to understand how the locking and unlocking scripts are evaluated by the stack execution engine.

Below is a deserialized P2PKH transaction output:

Input:
Previous tx: f5d8ee39a430901c91a5917b9f2dc19d6d1a0e9cea205b009ca73dd04470b9a6
Index: 0
scriptSig: 304502206e21798a42fae0e854281abd38bacd1aeed3ee3738d9e1446618c4571d10
90db022100e2ac980643b0b82c0e88ffdfec6b64e3e6ba35e7ba5fdd7d5d6cc8d25c6b241501

Output:
Value: 5000000000
scriptPubKey: OP_DUP OP_HASH160 404371705fa9bd789a2fcd52d2c580b65d35549d
OP_EQUALVERIFY OP_CHECKSIG

Let's consider the locking script in the above output called scriptPubKey. 404371705fa9bd789a2fcd52d2c580b65d35549d is the hexadecimal representation of the hash of the recipient public key without base58 encoding otherwise called Bitcoin address. For simplicity, let's replace it with the shorthand <PKH> meaning Public Key Hash in the scriptPubKey. We, therefore, have the scriptPubKey as:
OP_DUP OP_HASH160 <PKH> OP_EQUALVERIFY OP_CHECKSIG.

Let's assume Bob is the recipient of the above transaction output. To spend this output, he will construct a transaction that references this transaction id, provide a public key and signature (unlocking script), and the address to send to (these are all handled by his wallet application).

When his wallet application broadcasts this new transaction, each full Bitcoin node that receives the transaction will validate it to ensure it is indeed a valid transaction. The below diagram shows how the transaction will be evaluated:

P2PKH Transaction validation

As can be seen in the above diagram, the unlocking (scriptSig) produced by Bob's wallet is first placed on the stack, followed by the locking scripts, and executed on a Last-in-First-Out (LIFO) basis, from left to right.

Bitcoins' script language operators (OP_DUP, OP_HASH160, etc) push or pop one or more parameters from the stack, act on them and push the result onto the stack. If at any point the execution fails, then the transaction is invalid and rejected. However, if at the end of the execution, the result on the stack is TRUE, then the transaction is valid.

Following the above diagram, OP_DUP will duplicate Bob's Public key and push the result back to the stack. OP_HASH160 operand takes one of Bob's duplicated public key, acts on it, produces a Public Key Hash (PKH), and push the result - PKH- back to the stack.

OP_EQUALVERIFY comprises two operators - OP_EQUAL and OP_VERIFY. OP_EQUAL will take the two PKHs and compare them and return TRUE or FALSE back to the stack, while OP_VERIFY will terminate execution if it is FALSE that is found on the stack otherwise it will remove TRUE from the stack. Then OP_CHECKSIG will take the pubkey and signature and check if the signature matches the pubkey, then it will return TRUE marking the transaction as valid otherwise it will return FALSE marking the transaction as invalid. You can read more about OP_CODES used in Bitcoin here.

In conclusion, a P2PKH is a type of transaction that can be spent by presenting a public key and a digital signature from a single pair of public and private keys. If TRUE remains after the Bitcoin execution engine executes the combined script, then the transaction is valid otherwise it is invalid. You can read more about Bitcoin transactions in Mastering Bitcoin by Andreas Antonopoulos. Thank you.

References:

Mastering Bitcoin by Andreas Antonopoulos.
Transaction Bitcoin Developer Guide
Transactions and Scripting Saylor Academy
OP_CODES Used in Bitcoin Script

Bitcoin Transaction Signature Types (SIGHASH)

Peter Tyonum — Mon, 04 Jul 2022 15:05:19 +0000

Introduction

Bitcoin is digital money that uses a distributed public ledger to record its transactions. To own bitcoin is to be able to spend it. Wallets provide the means and convenience to manage bitcoin assets by constructing, signing, and sending transactions to bitcoin addresses. During signing of Bitcoin transactions, SIGHASH flags are used to indicate which part of the transaction data is included and signed by the user's private key. In this article, we will look at SIGHASH flags in detail.

Signatures are used to signify the authenticity and integrity of messages and are a function of private key and the message (digest). In Bitcoin, signatures provide cryptographic proof that the sender is in control of the private keys required to authorize spending certain UTXOs. Importantly, the sender never has to reveal their private key(s), the signature can be verified using just the public key.

When talking about signatures, there are two important actions. One is crafting the signature, the other one is verifying it. The former make use of an algorithm to create a signature using a private key (the signing key) from a message (the transaction data). The later action uses an algorithm that allows anyone to verify the signature, given the message and a public key. For Bitcoin, the public key is usually part of the UTXO script that is publicly accessible on the ledger. When a user signs transaction data using their private key, anyone can use the corresponding public key to verify that it was indeed signed by the user with the private key and that it has not been modified.

Signatures are an essential part of the Bitcoin protocol. Most transactions in Bitcoin uses the Elliptic Curve Digital Signature Algorithm (ECDSA). ECDSA signatures are cryptographically secure digital signatures based on elliptic-curve cryptography (ECC). It is used by the script functions OP_CHECKSIG, OP_CHECKSIGVERIFY, OP_CHECKMULTISIG, and OP_CHECKMULTISIGVERIFY.

Another signature algorithm used in Bitcoin is the Schnorr signature soft forked in since Taproot. They offer better privacy to multisig transactions and reduces the amount of signature data for such transactions.

In Bitcoin transactions, signatures are located in the scriptSig field of the input for non-segwit transactions and in witness field for segwit transactions. They consist of encoded parts r and s values and a flag called SIGHASH flag, which specifies which part of the transaction the signature signs.

ECDSA SIGNATURES: SIGNING AND VERIFICATION

The ECDSA algorithm takes a message msg and a private key privKey which is the user's private key and produces a signature as output which contains a pair of integers (r,s). The r and s values are both 256 bit (32-byte) integers.

The ECDSA algorithm is as follows:

𝑆𝑖𝑔=𝐹𝑠𝑖𝑔(𝐹ℎ𝑎𝑠ℎ(𝑚sg),privKey)

where:

privKey is the signing private key
msg is the transaction (or parts of it)
Fhash is the hashing function
Fsig is the signing algorithm
Sig is the resulting signature consisting of r and s.

Once the values r and s are calculated, they are encoded into a byte-stream using the Distinguished Encoding Rules (DER) encoding scheme. SIGHASH flags are then appended to the encoded data. They are usually between 71-73 bytes long. Unlike ECDSA signatures, Schnorr signatures are not DER-encoded and are between 64 to 65 bytes long.

In verifying a signature, the ECDSA algorithm takes the signed message msg and the signature {r, s} produced from the signing algorithm and the public key pubKey, corresponding to the signer’s private key, and returns true or false.

SIGNATURE HASH (SIGHASH) FLAGS

The SIGHASH flag is a single byte that is attached at the end of the signature. SIGHASH flags are used to signify which parts of the transaction data are being signed (see table below). Every signature in a Bitcoin transaction requires a SIGHASH flag for signing and verification purposes, and the flag can be different for each input. Schnorr signatures may omit the SIGHASH flag, defaulting to SIGHASH_DEFAULT.

Each transaction input may contain one or more signatures in its unlocking script. As a result, a transaction that contains multiple signatures may include signatures with different SIGHASH flags that commit to different parts of the transaction.

Flag	Modifier	Inputs Signed	Outputs Signed
SIGHASH_ALL	-	All	All
SIGHASH_ALL	ANYONECANPAY	One	All
SIGHASH_NONE	-	All	None
SIGHASH_NONE	ANYONECANPAY	One	None
SIGHASH_SINGLE	-	All	One at same index as input
SIGHASH_SINGLE	ANYONECANPAY	One	One at same index as input

SIGHASH Flags summary table

Currently, there are three standard types of SIGHASH flags SIGHASH_ALL, SIGHASH_NONE, and SIGHASH_SINGLE, and a modifier flag ANYONECANPAY.

SIGHASH_ALL (0x01) signifies that the signature covers all the transaction inputs and outputs.
SIGHASH_NONE (0X02) signifies that the signature covers all inputs and none of the outputs, while the
SIGHASH_SINGLE (0x03) signifies that the signature applies to all inputs but only a single output with the same index number as the signed input.

The modifier flag ANYONECANPAY can be combined with any of the above flags, as illustrated in the above table, to limit the signature to apply to a single input. Below are the combinations of ANYONECANPAY flags.

SIGHASH_ALL|SIGHASH_ANYONECANPAY flag (0x81) signifies that the signature applies to one input and all the outputs. The user can construct the transaction and allow others to amend it by adding inputs to the transaction.

The SIGHASH_NONE|ANYONECANPAY flag signifies that the signer commits to one input and none of the outputs. The common use case for this flag is dust collection. Dust represents tiny UTXO in the wallets and users cannot be able to construct valid transactions with such amounts as the transaction fees are often bigger than their value. Using this signature type, dust can be donated and collected as the user commits to only one input and anyone can claim the output.

The last flag combination is SIGHASH_SINGLE|ANYONECANPAY which signifies that the signature is for a single input and the corresponding output with the same index number. That is, if the input is at index 0, then the signature also covers the index 0 output. This enables multiple inputs from different signers to combine their transactions into a single transaction that pays each user their corresponding output.

Schnorr signatures have an extra SIGHASH_DEFAULT flag available to them, which is implicitly applied if no other SIGHASH flag is present. Signatures with this implicit flag are 64 bytes long. SIGHASH_DEFAULT functions the same as SIGHASH_ALL, committing to all inputs and outputs. However, if a Schnorr signature is 65 bytes long, it can not have a SIGHASH_DEFAULT flag to protect against malleability.

Each SIGHASH and any flags are checked by the script interpreter against these enums in Bitcoin Core.

Conclusion

As we have seen above, signatures are generated from a private key and a message and can commit to some or all inputs and outputs of a transaction, which allows for flexibility in transactions (see table above). Although most consumer wallets make use of SIGHASH_ALL which signs all inputs and outputs, it is useful to know that transactions are not limited to this type of signature.

Thank you for reading.

Reference:

Bitcoin Developer Guide: Transactions
Practical Cryptography for Developers by Svetlin Nakov: Digital Signatures
Wikipedia: ECDSA
Cryptobook: Elliptic Curve Signatures

Routing Payments in Lightning Network using HTLCs

Peter Tyonum — Wed, 08 Jun 2022 14:47:36 +0000

Recall that in this article, we discussed how peers on the Lightning Network (LN) establish a channel to make numerous payments without announcing them to the network. However, if every payment requires setting up a channel between nodes on the LN, then there will be delays in the first payment as the transaction establishing a channel has to be broadcasted and confirmed on-chain. Also, there are on-chain transaction fees and the challenge of managing multiple channels. Alternatively, nodes may choose to route payment across the network to a destination node. In this article, we will discuss how a payment can be routed through other nodes in LN to a destination node in the same trustless manner using Hash Time Lock Contracts (HTLCs).

The first step in routing a payment using HTLCs is for the receiver node to generate a payment invoice and send it to the payer. The invoice contains the amount, a payment hash, the receiver identity on the LN, an optional description, timelock, request expiration, and an on-chain fallback address.

The most vital data on the invoice is the receiver node identity and the payment hash as this is all the information the sender is given about the identity of the recipient. The identity is a secp256k1 public key of the node, which uniquely identifies the node on the network.

The receiver node generates the payment hash by first generating a 32-byte random number called a preimage. The preimage remains a secret. It is then hashed using the SHA256 hash algorithm to generate a payment hash for the invoice. Recall that, SHA256 is a one-way hash function whose input cannot be calculated from the result.

The recipient node may also optionally include routing hints in the payment invoice. These hints enable the sender to include unannounced channels while constructing a route to the recipient. Also optionally included in the invoice is the timestamp which enables the sender to know how old the invoice is, and also permits the receiver to enforce the expiration of the payment invoice.

Alongside other earlier stated information, the payment invoice is signed by the receiver node. BOLT11 invoice can also include the recipient pubkey as a tagged field which is covered by the signature. However, it's also possible to omit the recipient pubkey and to perform pubkey recovery on the signature itself to extract the pubkey it was signed with.

As we will see later in the article, payment invoices are meant to be used once and never to be reused, though new LN specifications exist that make it possible to reuse invoices (those are not covered in this article).

The signed invoice is then communicated to the sender. This is often done outside the LN through emails and QR codes or fetched via a webserver e.g. during a checkout process, though new specifications include using LN onion messages.

Once the sender node receives the invoice, it first computes a path to send the payment to the recipient. This path is constructed by examining the information the sender node has about the network. (Recall that, when channels are created, they are announced to the network and nodes keep this information to represent the state of the network called the network graph).

Provided the sender and recipient nodes are connected with channels that have enough capacity, the sender node will create a list of possible paths, and attempt to deliver payment in a trial and error loop until a path is found that can deliver the payment. Whilst the network graph includes total channel capacity for each channel, nobody except the two nodes in each channel knows how that capacity is distributed within the channel, i.e. what the balance at each side of the channel is.

ROUTING A PAYMENT

When the sender node identifies a path to deliver payment to the recipient, it will encrypt a payment instruction containing the invoice’s payment hash in nested layers (like an onion) and send it to the first (intermediary) node on the path (hop). The intermediary node will unwrap the outermost layer of the payment instruction, identify and send it to the next hop along the path. This approach is called source-based onion routing (SPHINX), and each hop can only decrypt one layer at a time to reveal the next hop to forward the payment.

Intermediate nodes only gain knowledge of previous_hop and next_hop relative to themselves, so if there is more than 1 hop in the path then nobody other than the sender will ever know the complete path. The final recipient will only ever know previous_hop and then that the payment was destined for them.

As illustrated below, let’s assume Alice wants to send a letter to Chan through Bob (i.e, Alice is connected to Bob who is then connected to Chan). She will write the letter and seal the letter in an envelope addressed to Chan which only Chan can open. But because Alice is not connected to Chan directly, she will put the first envelope and the instruction inside another envelope and address it to Bob. When Alice delivers the entire packaged letters, Bob will open the first outer envelope which reveals the instruction TO CHAN. Bob will then forward the letter to Chan, the intended receiver.

Developing on the above illustration further, the intermediary (Bob) might decide to charge a fee for forwarding the payment instruction to the next hop on the path (in this example Chan). So Alice will pledge to pay Bob an additional 100sats for forwarding the instruction. However, the payment will only be made if and only if Chan will share a secret number (the preimage discussed earlier) with Bob that can be hashed to result in the payment hash stated on the payment invoice.

With this pledge, Bob will also pledge to Chan that he will pay Chan an amount less than 100sats if Chan will present a preimage that Bob can hash that will result in the payment hash. Once Chan reveals the preimage, Bob will verify by hashing the preimage and if the result matches the payment hash, Bob will then pay Chan the amount stated on the payment instruction. Bob will then share the preimage with Alice who pledged to pay Bob the amount and an additional 100sats if Bob shares the secret hashing to the payment hash. Once Alice verifies by hashing the preimage and it results in the payment hash, she will pay Bob the amount she pledged earlier, which is the invoice amount + 100sats.

The above illustration has a single hop between the sender and recipient. Whilst the specification currently allows for a maximum of 27 hops in the route, in normal usage more than 7 hops would be considered unusual. In computing a path, the sender often optimizes for fewer hops to reduce the chance for failure and routing fees.

So, how does Alice (our payment sender) knows what fee to pay to each hop (fee_base_msat) and whether a channel exists between nodes capable of forwarding this payment? This is the information Alice sought during the path-finding phase. She is also aware of how long it will take for Bob to pay Chan the agreed amount (cltv_expiry_delta). All this information aids the sender to construct a payment instruction that will pass through each intermediary with minimal chance of failure.

Also, as illustrated in our example, the sender starts constructing the payment instruction from the destination backward to them. Each layer of the onion is encrypted in such a way that only the node meant to read it can be able to read. The sender achieves this using the intermediate node’s public key and the Elliptic Curve Diffie–Hellman (ECDH) on Bitcoin’s secp256k1 curve. And the message can be checked that it has not been modified. Also, the sender generates a temporal key for the entire session so that no other node can identify her node as the sender of the payment instruction. All these measures ensure the integrity of messages and the privacy of the nodes.

If for any reason, the payment instruction could not go through at any point, it will fail and the failed status will propagate backward along the same path so that any pledged payment will be canceled.

As we have seen, sending payments this way on the LN is both beneficial and safe for everyone involved (the sender, the intermediary nodes, and the recipient). Intermediary nodes earn fees for forwarding payment instructions. The sender pays the recipient within a short time and with a minimal fee. The recipient node receives its payment within a short time without risk. This enables payments to be made on the LN without establishing channels between nodes for each payment.

References:

Mastering Lightning Network: By Andreas M. Antonopoulos (@aantonop), Olaoluwa Osuntokun (@roasbeef), Rene Pickhardt (@renepickhardt).
Public Key Recovery question on Stackoverflow explanation here
Bolt11 Invoice for Lightning Payment Specifications.

Bitcoin Multisig Addresses

Peter Tyonum — Sun, 15 May 2022 18:39:33 +0000

For one to be able to transact on the bitcoin network, they need an address. A bitcoin address can be likened to a bank account in the traditional banking system. However, bitcoin addresses differ from bank accounts in so many aspects. In this article, we will look at bitcoin addresses and how to derive a multi-signature bitcoin address.

Bitcoin Addresses

A bitcoin address could be a bare public key (P2PK), but is more commonly the hash of a public key (P2PKH) or the hash of a locking script (P2SH). The most common type of bitcoin address is derived by cryptographically hashing a public key into a format that is easy to copy and share, can be error-checked, and verified using Base58 encoding. This type of address has a prefix of 1 and represents a pay to public key hash (P2PKH).

Bitcoin addresses can also be derived from scripts instead of public keys. These category of addresses have a prefix of 3 signifying a pay-to-script-hash address (P2SH), commonly referred to as multi-sig addresses. There are also bitcoin addresses that represent Segwit upgrade in Bitcoin, and comprise of public key hash (P2WPKH) and script hash (P2WSH) main types. They have a prefix of bc1 and are longer than the first two types as they use bech32 encoding.

Recently we have addresses that represent Taproot upgrade. Pay-to-taproot (P2TR) addresses start with bc1p and use bech32m encoding. When used with the key-path spending route, P2TR enables multi-key schemes like 3-of-5 to appear the same on the bitcoin network as though a single party pay-to-public key address, giving good privacy benefits. P2TR addresses are bare public keys, like P2PK.

Bitcoin Addresses hold Unspent Transaction Output (UTXOs) which represents the value of bitcoin in satoshis. To spend those UTXOs, one needs to prove ownership by using the private key associated with the address to create digital signatures that unlock the value stored in the UTXOs.

Multisig Bitcoin Addresses

For addresses derived from more than a single public key, depending on the number of minimum signatures set at address creation, they must be provided to unlock such UTXOs. These types of addresses are called multi-sig addresses. This is similar to a corporate/joint bank account that requires more than a single signature to sign a cheque to withdraw funds from the account.

Multi-sig addresses offer interesting use cases such as escrow transactions, collective management of assets, and minimizing the risk of losing a single private key among others. The most common downside of using multi-sig addresses and transactions is that they require technical knowledge to manage them.

Multi-sig transactions are of the form m-of-n where m is the number of signatures required to spend funds and n is the number of maximum public keys. For instance, a 2-of-3 transaction will require at least 2 signatures for that transaction to be valid out of a possible 3. In the following section, we will look at how a multi-sig address can be generated from 3 public keys and how we can set that at least two of the possible 3 private keys must be provided to unlock funds sent to the address.

How to Generate 2-of-3 Multisig Bitcoin Address

We will be using a bitcoin-core daemon to create a P2SH 2-of-3 multi-sig address. Here's a link to how you can set it up. Check that the bitcoin core is running on testnet. You can start the daemon by running bitcoind.

We will be using this library to generate private/public key pairs to use in generating our addresses. Follow the instructions on the readme to generate 3 pairs of private and public keys:
Using the library, we have the following public keys in hex format:

"03150176a55b6d77eec5740c1f87f434cf416d5bbde1704bd816288a4466afb7bb"
"02c3b2d3baf90e559346895b43253407fbb345c146910837b61f301f4c9a7edfe5"
"02c6e3e94f7ff77457da9e76cf0779ca7c1e8575db064a2ea55400e6a9d8190225"

Now we will create our multisig address with the above given public keys.
Using the bitcoin-cli, let's generate a multisig address as follows:

bitcoin-cli -testnet createmultisig 2 '["03150176a55b6d77eec5740c1f87f434cf416d5bbde1704bd816288a4466afb7bb", "02c3b2d3baf90e559346895b43253407fbb345c146910837b61f301f4c9a7edfe5", "02c6e3e94f7ff77457da9e76cf0779ca7c1e8575db064a2ea55400e6a9d8190225"]'

See cover image for result of the above command.
The parameter 2 tells bitcoind to create an address that requires the signature of at least two private keys. The fact that we are using three public keys makes it a 2-of-3 multisig address. You can use any number of keys as long as it is equal to or greater than the "minimum required to sign" parameter.

You can confirm that the generated address is valid by using the command:

bitcoin-cli -testnet validateaddress '2MyxShnGQ5NifGb8CHYrtmzosRySxZ9pZo5'

See cover image for result of command.
It is worthy to note that our generated P2SH address starts with a 2 instead of 3 as stated earlier because we are using testnet (an alternative chain for testing purposes).

You can send testnet coins to the generated address and construct a transaction to spend the transaction UTXOs using the key pairs. That is beyond the scope of this write-up. You can read more about bitcoin addresses and transaction in Chapter 4 of Mastering Bitcoin by Andreas Antonopoulos. Thank you for reading.

Understanding How Payment Channels Work in Lightning Network

Peter Tyonum — Mon, 25 Apr 2022 12:43:52 +0000

Recall that in our previous article, we discussed how Lightning network make payments in Bitcoin faster, cheaper and more private. This article aims to give a high level understanding of how payment channels work in Lightning network covering ways payment is sent, how funds are protected and delivered within the network.

On the lightning network, there are popularly two ways by which a payer can send payment to a recipient. A payer can choose to create a path that connect them directly to the recipient peer called a channel or the recipient can create an invoice which contains the amount, their identity on the network amongst other things. Both approaches does not require that you trust the other party and are secure. Let’s look at the former in detail.

Payment channels are suitable for multiple payments. Channels are created between peer nodes (connected nodes) on the Lightning network. To open a channel, the two nodes exchange series of messages, negotiate the time funds can be locked, amongst others. Thereafter, both parties will create both a public/private keys and exchange the public keys (funding public keys). Then the payer (wallet) will derive an address called Pay-to-Witness-Script-Hash (P2WSH) Bitcoin address from the public keys, then construct a a 2-of-2 multisignature transaction called the funding transaction that spends to the address, which effectively place the funds in an output that requires both the payer and the recipient to jointly sign a transaction to spend from its output. When this transaction has been transmitted and recorded on the Bitcoin network, it form the basis of a channel. Now, what happens if the recipient decline to sign any transaction that spends from the funding transaction? Is the payer’s money locked up? Let’s consider this scenario closely.

In a decentralised network like Bitcoin and Lightning network, transacting parties are not required to trust each other for them to transact. They are governed by the cryptographic protocol which ensure that no one party is cheated. Before the channel funding transaction is broadcasted to the Bitcoin network, the payer creates another transaction signed both by themself and the recipient that spends all the output back to the payer’s address. This is a prerequisite step that protects the payers’ funds. This way, if the recipient who is a channel partner no longer cooperates, attempt to cheat or disappears, the payer can recover all their funds by simply broadcasting the signed transaction to the bitcoin network. This same protective principle is applied when the channel becomes operational.

When the channel has been established, the payer can pay the recipient by simply constructing a transaction (referencing the funding transaction id as input) that send the required number of sats to their channel partner (recipient), and the balance back to their address. These transactions are called commitment transactions, and any of the channel partner can send a payment to the other. They can exchange as many commitment transactions as they want without fees. Because these commitment transactions are not broadcasted to the bitcoin network, they are incredibly fast. Use cases exists where payments can be made per bit of streaming services. You might be wondering what happens if one of the channel partners broadcasts a prior state transaction which is not the latest commitment transaction in the channel to the Bitcoin network?

Since commitments transactions are valid Bitcoin transactions, once they are broadcasted to the bitcoin network, they cannot be stopped, censored or cancelled. Lightning network works in a way that punishes the party who broadcast a previous transaction. It does this by locking the part of the transaction output that pays the party that broadcasts the transaction and pay immediately the channel partner’s balance. This timelock (delay) can be the number of blocks or the duration of time (negotiated by the channel partners) from when that transaction was confirmed on the blockchain. Also, the outputs to the party are not just timelocked, they contain a condition that permits the channel partner to spend that output using a revocationpub key. The key permits the party to construct a transaction that claim the entire balance immediately. This key is exchanged between the channel partners per a previous commitment whenever they acknowledge, sign and commits to a new commitment transaction. This is similar to giving the other party a means to punish them if they attempt to broadcast the previous transaction.

As we have seen above, it is not in the best interest for anyone to attempt to cheat in a channel. So how can they close the channel and each party claim their due balances? when any of the channel party wants to close the channel, they will initiate a closing transaction that both parties will negotiate and agree. This closing transaction pays each party their balance immediately. The closing transaction does not have timelock and penalty revocation keys.

In summary, in this writeup, we have looked at how payment channels permits a large number of transactions to be carried out in a secure, cheap and fast way. In another article, we will look at how a node can send payment to another node when they are not in a channel. Thank you for reading. I look forward to your comments.

Lightning Network: Your Gateway to Bitcoin Payments

Peter Tyonum — Wed, 20 Apr 2022 03:33:54 +0000

Amongst other things, bitcoin was introduced to facilitate borderless payments regardless of the amount, bring financial services to the unbanked while preserving users privacy without a central authority. Functionally, it has achieved all of these but has not been a preferred payment method for many probably because of the time it takes for payment to be confirmed and the transaction fees. In this article, we will look at how lightning network handles transactions faster, cheaper and more privately.

Lightning network is a second layer technology built on top of Bitcoin that has made making payments faster, cheaper and more private. Making payments (transactions) on Lightning networks are secure and does not require users to trust the other party similar to Bitcoin. But what makes lightning network more exciting? Let’s look at it more closely.

On the bitcoin blockchain, when a transaction is constructed, it is broadcasted to the entire bitcoin peer to peer network, accepted and mined into a block. It is then confirmed (confirmation here implies adding more mined blocks after the block that has your transaction in it and the recommended amount of confirmations are 6). However, in lightning network, transactions are not broadcasted to the network (except opening and closing transactions), but instead sent directly to a person (a node) on the Lightning peer to peer network.

You might be wondering if payments on the lightning network are not broadcasted and confirmed on the bitcoin blockchain network, how then are they secure? are there no chances someone is going to be cheated? Lightning network makes use of the game theory otherwise called fairness protocol that serve as a deterrent for would-be cheaters. In Lightning network, payment can be made in a channel (a financial relationship between two parties on the LN network) or be sent through (routed) the network to a node. Let’s look at each scenario and how it is uninteresting and financially punishable for any party to attempt to cheat.

When two parties establish a channel on the Lightning network, they have a direct pathway to send payments to each other. These payments are built on the channel opening transaction called funding transaction which is confirmed on the bitcoin blockchain. Subsequent transactions (commitments) are no longer broadcasted but are valid transactions. These transactions simply update the channel state (the balance between the parties in the channel). See figure below.

adopted from Mastering Lightning Network.

As I have stated earlier, all the (commitment) transactions are valid transactions that are not just broadcasted to the bitcoin network. If one of the parties broadcast any of the prior commitment transaction to the bitcoin network in an attempt to cheat the other party, the other party stand a chance to revoke and claim the entire balance of the channel. This is because the transaction outputs are timelock for the party who broadcasted the transaction while the other party can claim those outputs immediately and also claim the balance of the party that broadcasted the transaction.

In situations where parties are not in a channel but there is need to send payment, they can simply generate an invoice which contain a payment hash, their node id on the network, the amount to be paid, an optional description and the invoice expiry date. The payer (their node) will find a path that requires the least number of hops and fees to send such transaction to the party. They will then route the payment to the recipient and receive acknowledgment. Payments made through this method either succeeds or fails, there is no in-betweens.

It is also worth noting that, payments made on lightning network remains private between the parties involved. No third party node has explicit knowledge of the amount, source and destination in a transaction and the nodes propagating such payments do not have knowledge of the origin or destination. These make such payments more private unlike bitcoin transactions where such transactions are public while the users are anonymous.

Fees in lightning network are very low because they are paid for routing payments unlike in Bitcoin. While miners in the Bitcoin network usually order transactions according to the highest fees per unit (bytes) of transaction weight, nodes on the lightning network with lower fees for routing are more attractive to route payments.

You may not necessarily understand the technical details of how lightning work but it exists to make to make it easier for you to make secure, cheap and faster transactions. There are many implementations of lightning wallets such as Eclair, c-lightning, lnd, muun amongst others that will make it easier for you to use bitcoin for your payments.

Thank you for reading and I welcome your comments.

How to Setup Bitcoin Core and Lightning Network Node Developer Environment

Peter Tyonum — Mon, 11 Apr 2022 08:20:02 +0000

Like any other developer environment, for one to build on Bitcoin Core and Lightning Network, you need to setup their (development) environments.

Note: This writeup is based on implementation on macOS. If you need help setting up for another platform, please reach out and I will be happy to assist.

Prerequisites:

Please check that you have the following softwares installed on your computer:

Git
python
Home Brew
Sqlite

We are going to start with Bitcoin. To build on Bitcoin, we are going to setup its client software called Bitcoin Core. Bitcoin Core consists of a full-node software for fully validating the blockchain as well as a bitcoin wallet. We will clone their Github repository and compile.

Required Dependencies

We will be following instructions found in the ‘doc’ directory of the repository to install all the required dependencies. Install the following dependencies:

install Xcode: xcode-select -—install
using home-brew, install automate, lib tool, boost, png-config, and libevent by running: brew install automake libtool boost pkg-config libevent
Ensuring that python is installed, you can install the deploy dependencies by running: pip3 install ds_store mac_alias.

Optional Dependencies

install berkeley-db@4 if you want to support legacy wallets brew install berkeley-db@4
install qt@5 if you intend to compile GUI: brew install qt@5
To build in QR support for the GUI, install qrencode: brew install qrencode
to support UPnP port mapping, install miniupnpc: brew install miniupnpc
to support NAT-PMP port mapping, install libnatpmp: brew install libnatpmp
to support for ZMQ notifications, install ZMQ: brew install zeromq

Building Bitcoin Core

Navigate to where you want to compile bitcoin core and clone the repo:

git clone https://github.com/bitcoin/bitcoin.git

Configure Bitcoin Core:
We will be configuring our Bitcoin Core to support Wallet (BDB + SQlite) without a GUI. cd into bitcoin, and run the following commands:

./autogen.sh
./configure --with-gui=no

Please note that, if you did not install berkeley-db@4 stated above, kindly install before running the above command to avoid errors.

Compile Bitcoin Core:
Inside the same directory, run the following commands:

- make
- make check (to run tests)

Setting up Bitcoin Core to use Regtest

If you are setting up Bitcoin core primarily for development, there is no need to connect to mainnet and download the entire block. Therefore, we will be configuring ours to use regtest.

Run the below commands to create Bitcoin directory, its configuration file bitcoin.conf and grant appropriate permissions.

mkdir -p "/Users/${USER}/Library/Application Support/Bitcoin"

touch "/Users/${USER}/Library/Application Support/Bitcoin/bitcoin.conf"

chmod 600 "/Users/${USER}/Library/Application Support/Bitcoin/bitcoin.conf"

open the Bitcoin configuration file in your editor and add the following:

# Daemon Options
server=1
daemon=1
fallbackfee=0.00072
txconfirmtarget=6
# Network Options
regtest=1
#signet=1
#testnet=1
[regtest]
# RPC Options
rpcport=18443
rpcuser=bitcoin
rpcpassword=bitcoin
# ZMQ Options
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333
zmqpubhashtx=tcp://127.0.0.1:28332
zmqpubhashblock=tcp://127.0.0.1:28332

save and close the above configuration. With the above in place, you can now run your bitcoin core with:

bitcoind

Next, we will be setting up our lightning network

Similar to Bitcoin, we are going to setup Lightning network node using Lightning Network Daemon (lnd). LND has several pluggable back-end chain services including btcd (a full-node), bitcoind, and neutrino (a new experimental light client).

Requirements:

install Go using home-brew brew install go

Navigate to any directory where you want to setup LND and clone the repo:

git clone https://github.com/lightningnetwork/lnd

cd lnd

make install

To check that lnd was installed properly run the following command:

  make check

Similar to our bitcoin core installation, we want to configure lnd to use regtest and connect to our Bitcoin core, so we'll create it's config file in the following directory with the below configuration:

mkdir -p "/Users/${USER}/Library/Application Support/Lnd"

touch "/Users/${USER}/Library/Application Support/Lnd/lnd.conf"

chmod 600 "/Users/${USER}/Library/Application Support/Lnd/lnd.conf"

paste the following configuration details in the configuration file:

[Application Options]
debuglevel=trace
maxpendingchannels=10

[Bitcoin]
bitcoin.active=1
bitcoin.node=bitcoind
bitcoin.regtest=1

Creating a wallet
In one of the terminal tab, run the command

lnd

and in another tab, create a new wallet with:

lncli create

This will prompt for a wallet password, and optionally a cipher seed passphrase.
lnd will then print a 24 word cipher seed mnemonic, which can be used to recover the wallet in case of data loss. The user should write this down and keep in a safe place.

If you encounter errors at any point during your setup or have any comments, kindly share in the comments section and I will be happy to reply. Thank you for reading.

File Storage and Symbolic Links in Laravel

Peter Tyonum — Tue, 30 Mar 2021 11:43:12 +0000

Storing files on the same server where your application lives is not a good practice but a simple application may not necessarily require using cloud storage such as AWS S3 and Cloudinary, and files may be stored locally. Storing files locally and displaying them may be a little tricky than we could imagine.

Saving files on the local server

Using the default filesystem configuration for Laravel (config/filesystem), files are saved in the storage/app directory. So when you upload a file using the storage facade, it is saved in this directory. For example:

Storage::put($filename, $file, 'public');

will save the file $file in the directory storage/app with name as $filename. i.e. app/storage/$filename and will have public visibility.

Displaying Files stored on the server:

As stated earlier, files are by default stored in the storage/app/ directory. This prevents files from been publicly accessible (that is, anyone assessing your files over the internet without needing permission).

So to display files in our application from the storage directory correctly, we will create a symbolic link to the public directory using the following artisan command:

php artisan storage:link

Using the helper function asset, we can display this file in our application. For example, to use this in the img tag,

<img src=“{{asset($filename)}}” alt=“{{$filename}}” />

so, what happens when you choose to store files outside this default directory say storage/app/myfiles ? Laravel offers the option of linking this directory to the public directory as well.

To create a symlink from this subdirectory to the public directory, open the config/filesystems file and add the link,

'links' => [
        public_path(‘myfiles’) => storage_path('app/myfiles’),
    ],

then run the command php artisan storage:link

this will create a symlink called myfiles in the public directory.

With this, we can be able to access files in this subdirectory with the asset helper as follows:

For example, say you want to access a file stored as display.jpeg in storage/app/myfiles/ subdirectory, all we need to do is call the asset helper as below:

<img src=“{{asset(‘myfiles/display.jpeg’)}} alt=“myimage’/>

this is applicable when you have subdirectories in the myfiles subdirectory. Something like storage/app/myfiles/subdirectory/display.jpeg, just specify in the url as follows:

<img src=“{{asset(‘myfiles/subdirectory/display.jpeg’)}}” alt=“my image” />

Also, remember if you did not add the storage directory to your git ignore and it’s pushed, you need to delete it on the server before running the storage link command.

Thanks for reading as I welcome your observations and comments.

Getting started with SSH Key-based Authentication For Servers

Peter Tyonum — Wed, 02 Oct 2019 17:32:32 +0000

When I got started with AWS, after creating an EC2 instance, I was given keys with instructions on how to log in from my local machine, and I was amazed at how everything worked. In this writeup, we will see how you can get your servers (web, email, files, database etc) running on your local system using Virtual Machines (VMs) and how you can create users, assign permissions and enforce SSH key-based authentication the same as found on AWS.

SSH key-based authentications are proven to be more secure and easy to use. They offer convenience as a user is not required to remember passwords to login. Let's configure. Okay, let's see how they work.

Step 1: Install and configure Virtual Box and Vagrant for your platform. Confirm that it is correctly installed by running vagrant -v.

Step 2: If you do not have one already, create a VMS directory for keeping all your boxes. Inside the VMS directory, create a sub-directory and initialize an Ubuntu VM in the directory using this command
mkdir keybased-auth && cd keybased-auth && vagrant init ubuntu/trusty64.
This should create a Vagrantfile in the directory. You can open the Vagrantfile and customized the IP address, size and ports depending on the server you are building. In the Vagrantfile, look for this line
# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
and uncomment it by removing the preceding #. This will simply allow us to be able to access our server on port 8080 from our system.

Step 3: Now it's time to bring up our box. Run the command vagrant up and wait for the machine to download and install Ubuntu 14 OS on the machine. If all went well, you should have your VM up and running. Confirm that by running vagrant status, it should show running.

Step 4: Log in to the VM and update packages. Let's log into the machine by typing vagrant ssh. This should log the default user vagrant into the machine. Your terminal should read something like this vagrant@vagrant-ubuntu-trusty-64:~$.

We can now update our packages by running sudo apt-get update && sudo apt-get upgrade.

Step 5: Now we will create a new user, assign permissions and log in with the new user. Using the currently logged in vagrant user, type sudo adduser <username> substituting <username> to any name you want your new user to bear, in my case, tvpeter. This will create a new user
and group tvpeter, add the user to the group and creates the users home directory. Next, you will be prompted to set the users password, full name, room number, home phone, and others. You can choose to leave it blank by hitting the return key successively.

At this point, our new user can log into the system but does not have permission to run administrative commands. Let's rectify that by adding the user to the subdoers group. Run the command sudo cp /etc/sudoers.d/vagrant /etc/sudoers.d/tvpeter to make a copy of the user vagrant's permissions as tvpeter in the sudoers.d directory. Let's modify the file using nano. Type sudo nano /etc/sudoers.d/tvpeter to open the file. On the second line, change vagrant to tvpeter (same as shown below), save ^o and exit the file ^x.

Next, log out the vagrant user by typing exit and log in with our new user by typing ssh tvpeter@127.0.0.1 -p 2222 or ssh <newUsername>@<privateIPAddress> -p 22 if you used a private IP address. Supply the password set for the user when prompted.

Step 6: Generating login keys and connecting our new user: As we stated in the introductory part of the writeup, using SSH key-based authentication improves server security and makes it easy to log into without having to remember passwords.

From your local system, run ssh-keygen and supply a file name to save the generated keys. This will generate two files with the supplied name one ending with .pub and saved in the .ssh subdirectory of your home directory /users/<username>/.ssh/. In my case, I used keyauth.

View the content of the file that ends with .pub using cat /users/<username>/.ssh/<fileName.pub>.

Back to where our new user is logged in on the server, we will create a new directory and a file in the directory. Run the command cd && mkdir .ssh && touch .ssh/authorized_keys to create authorized_keys file. Open the file using sudo nano .ssh/authorized_keys and copy the content from /users/<username>/.ssh/<fileName.pub> and paste in the authorized_keys file on the server.

Next, we will restrict other users from having access to the user's .ssh directory and authorized_keys file. Run the command chmod 700 .ssh and chmod 644 .ssh/authorized_keys. We are all done setting our keys. Type exit to log the user out. Next, we will re-login using the ssh keys.

Step 7: Login user using ssh keys and disable password-based logins: Having configured our server to use ssh keys to login, we will login with the user and disable logins into the server using passwords. login using the command ssh <username>@<ipAddress> -p 2222 -i ~/.ssh/<localKeyName> or -p 22 for private IP address, which in my case will be ssh tvpeter@127.0.0.1 -p 2222 ~/.ssh/keyauth.

Next, we will disable logins using passwords. Open the sshd_config file with nano using the command sudo nano /etc/ssh/sshd_config and search for PasswordAuthentication yes, change the yes to no (as seen below) and restart ssh service using sudo service ssh restart.

Your server is now configured to use only SSH Key-based authentication. You can then proceed to set up your setup as web, email, database or whatever purpose bearing in mind it is now more secure.

Your comments/observations are welcome. Thanks for reading.