DEV Community: Alec Buda

Introduction to the Bitcoin Network Protocol using Python and TCP Sockets

Alec Buda — Wed, 30 Oct 2019 07:02:47 +0000

Blockchain technology is built around consensus algorithms which allow distributed nodes to share a common ledger. A fundamental dependency of these algorithms is a common network protocol to enable communication between participating nodes. Today, let's write a Python program from scratch to interact with a real Bitcoin node.

This post will assume you're familiar with the fundamentals of blockchain technology. If you aren't, I would recommend checking out the Bitcoin White Paper by Satoshi Nakamoto.

Bitcoin nodes communicate with each other using the TCP protocol. Nodes will typically listen on port number 8333. For a detailed description of the bitcoin network protocol check out this resource.

Today, we are going to write a Python program to connect to a Bitcoin node and fetch the details of a specific transaction. Here is a diagram of the message flow that will be developed.

Before we start coding our program, we must make one point clear. Interacting with a Bitcoin node using raw TCP sockets is reinventing the wheel. This has already been done by python packages such as python-bitcoinlib.

If you want to write sophisticated applications you should definitely use the correct tool for the job. With that said though, programming with TCP sockets is a great way to improve your low level understanding of a network protocol.

To begin, let's import the dependencies our program will require.

#!/usr/bin/env python

# Filename:                     bitcoin-network-tutorial.py
# Command to run the program:   python bitcoin-network-tutorial.py

# Import dependencies
import socket
import time
import random
import struct
import hashlib
import binascii

Let's now define the methods required for constructing the "version" request message.

# Binary encode the sub-version
def create_sub_version():
    sub_version = "/Satoshi:0.7.2/"
    return b'\x0F' + sub_version.encode()

# Binary encode the network addresses
def create_network_address(ip_address, port):
    network_address = struct.pack('>8s16sH', b'\x01', 
        bytearray.fromhex("00000000000000000000ffff") + socket.inet_aton(ip_address), port)
    return(network_address)

# Create the TCP request object
def create_message(magic, command, payload):
    checksum = hashlib.sha256(hashlib.sha256(payload).digest()).digest()[0:4]
    return(struct.pack('L12sL4s', magic, command.encode(), len(payload), checksum) + payload)

# Create the "version" request payload
def create_payload_version(peer_ip_address):
    version = 60002
    services = 1
    timestamp = int(time.time())
    addr_local = create_network_address("127.0.0.1", 8333)
    addr_peer = create_network_address(peer_ip_address, 8333)
    nonce = random.getrandbits(64)
    start_height = 0
    payload = struct.pack('<LQQ26s26sQ16sL', version, services, timestamp, addr_peer,
                          addr_local, nonce, create_sub_version(), start_height)
    return(payload)

The struct module is used for packing binary data. The hashlib module is used for generating message checksums. For a full understanding of the code, you'll need to cross reference the data encoding with the protocol documentation.

Next, let's add a method for creating the "verack" request message. The verack command name is derived from "version acknowledge".

# Create the "verack" request message
def create_message_verack():
    return bytearray.fromhex("f9beb4d976657261636b000000000000000000005df6e0e2")

With the mandatory messages out of the way, we may now create our "getdata" method for retrieving the details of a specific transaction.

# Create the "getdata" request payload
def create_payload_getdata(tx_id):
    count = 1
    type = 1
    hash = bytearray.fromhex(tx_id)
    payload = struct.pack('<bb32s', count, type, hash)
    return(payload)

Please note that not all nodes will be able to return arbitrary transaction data; some will prune their history to save disk space.

We'll also create a method for printing TCP data to the terminal.

# Print request/response data
def print_response(command, request_data, response_data):
    print("")
    print("Command: " + command)
    print("Request:")
    print(binascii.hexlify(request_data))
    print("Response:")
    print(binascii.hexlify(response_data))

We may now add our main method which will connect to a bitcoin node and execute the desired message flow.

if __name__ == '__main__':
    # Set constants
    magic_value = 0xd9b4bef9
    tx_id = "fc57704eff327aecfadb2cf3774edc919ba69aba624b836461ce2be9c00a0c20"
    peer_ip_address = '104.199.184.15'
    peer_tcp_port = 8333
    buffer_size = 1024

    # Create Request Objects
    version_payload = create_payload_version(peer_ip_address)
    version_message = create_message(magic_value, 'version', version_payload)
    verack_message = create_message_verack()
    getdata_payload = create_payload_getdata(tx_id)
    getdata_message = create_message(magic_value, 'getdata', getdata_payload)

    # Establish TCP Connection
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((peer_ip_address, peer_tcp_port))

    # Send message "version"
    s.send(version_message)
    response_data = s.recv(buffer_size)
    print_response("version", version_message, response_data)

    # Send message "verack"
    s.send(verack_message)
    response_data = s.recv(buffer_size)
    print_response("verack", verack_message, response_data)

    # Send message "getdata"
    s.send(getdata_message)
    response_data = s.recv(buffer_size)
    print_response("getdata", getdata_message, response_data)

    # Close the TCP connection
    s.close()

We found the IP address of the node using Bitnodes. Details of the transaction we elected to query can be found on a block explorer.

Execute the program on a terminal with the command python bitcoin-network-tutorial.py. A sample output is provided below.

Command: version
Request:
b'f9beb4d976657273696f6e000000000064000000f4de76b762ea00000100000000000000c8c6ae5d00000000010000000000000000000000000000000000ffff68c7b80f208d010000000000000000000000000000000000ffff7f000001208d0f2f736a397699b60f2f5361746f7368693a302e372e322f00000000'
Response:
b'f9beb4d976657273696f6e000000000066000000fe4aee167f1101000d04000000000000c2c6ae5d00000000010000000000000000000000000000000000ffff68c7b80f208d0d040000000000000000000000000000000000000000000000000b63185e17ebcdb3102f5361746f7368693a302e31382e302fbc29090001'

Command: verack
Request:
b'f9beb4d976657261636b000000000000000000005df6e0e2'
Response:
b'f9beb4d976657261636b000000000000000000005df6e0e2'

Command: getdata
Request:
b'f9beb4d9676574646174610000000000220000007b00a9b50101fc57704eff327aecfadb2cf3774edc919ba69aba624b836461ce2be9c00a0c20'
Response:
b'f9beb4d9616c65727400000000000000c0000000d2f50d9ef9beb4d9616c65727400000000000000a80000001bf9aaea60010000000000000000000000ffffff7f00000000ffffff7ffeffff7f01ffffff7f00000000ffffff7f00ffffff7f002f555247454e543a20416c657274206b657920636f6d70726f6d697365642c2075706772616465207265717569726564004630440220653febd6410f470f6bae11cad19c48413becb1ac2c17f908fd0fd53bdc3abd5202206d0e9c96fe88d4a0f01ed9dedae2b6f9e00da94cad0fecaae66ecf689bf71b50'

That concludes the tutorial! Stay tuned for more.

ULTRA CONFIG GENERATOR

Have you heard of Ultra Config Generator? If you haven't, I highly recommend you check it out.

We designed the product to allow network engineers to generate and automate network configuration in a highly flexible, efficient and elegant manner. Our customers love the application and I hope that you will too.

Take care until next time!

Ultra Config

Python Automation on Cisco Routers in 2019 - NETCONF, YANG & Jinja2

Alec Buda — Mon, 02 Sep 2019 00:38:19 +0000

Before reading this post, I highly recommend that you check out our first NETCONF post we did if you are new to this subject. In this post I explained the fundamentals of NETCONF and YANG models which make for great background knowledge before getting into Python automation.

For todays exercise, we will write a Python program which will configure an interface on a Cisco CSR 1000v router. Our solution will use a Jinja template for the configuration data model and the NETCONF protocol for pushing our rendered configuration payload to the router - it is my hope that this post will shed light on how this technology can be implemented in a scalable manner by decoupling configuration templates from the python codebase.

To begin, let's spin up a fresh Cisco CSR 1000v router. We'll then need to configure credentials and enable NETCONF.

conf t
!
username admin privilege 15 secret admin
!
netconf-yang

We'll also configure an interface on the router for our NETCONF client to connect to.

interface GigabitEthernet1
 ip address 192.168.159.10 255.255.255.0
 no shut
 exit
!
end

Next, install Python on your workstation if you don't already have it.

With that done, let's open up a terminal and install the "ncclient" package. This package will provide us with a NETCONF client that we will use to manage our session with the router.

pip install ncclient

We will now create a new file for our python program - we entitled ours "cisco-automation-tutorial.py". At the head of the file, import the dependencies our program will utilize.

# Filename:                     cisco-automation-tutorial.py
# Command to run the program:   python cisco-automation-tutorial.py

# Import the required dependencies
from ncclient import manager
from jinja2 import Template

Next up, we'll establish the NETCONF session to our router using the "connect" method.

# Establish our NETCONF Session
m = manager.connect(host='192.168.159.10', port=830, username='admin',
                    password='admin', device_params={'name': 'csr'})

In order to configure our device with NETCONF we will first have to understand how to structure the configuration data within our RPC payload. Data structures in NETCONF are defined by YANG models and the CSR router actually supports many different options to choose from. For example, OpenConfig models and IETF models are both compatible with the IOS-XE software family.

For todays exercise, we will use the IOS-XE native YANG model.

To get a quick feel for the native model, add the python code below to your program.

# Create a configuration filter
interface_filter = '''
  <filter>
      <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
          <interface>
            <GigabitEthernet>
              <name>1</name>
            </GigabitEthernet>
          </interface>
      </native>
  </filter>
'''

# Execute the get-config RPC
result = m.get_config('running', interface_filter)
print(result)

This code is essentially the same as the CLI command "show running-config interface GigabitEthernet 1". When we execute the program using python cisco-automation-tutorial.py we get the output below.

<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="urn:uuid:f4241f31-5098-475c-9d01-bcf34df25643" 
  xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <data>
    <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
      <interface>
        <GigabitEthernet>
          <name xmlns:nc='urn:ietf:params:xml:ns:netconf:base:1.0'>1</name>
          <ip>
            <address>
              <primary>
                <address>192.168.159.10</address>
                <mask>255.255.255.0</mask>
              </primary>
            </address>
          </ip>
          <mop>
            <enabled>false</enabled>
            <sysid>false</sysid>
          </mop>
          <negotiation xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-ethernet">
            <auto>true</auto>
          </negotiation>
        </GigabitEthernet>
      </interface>
    </native>
  </data>
</rpc-reply>

You can see from the RPC reply that we have now revealed the IOS-XE data structure of an interface.

We may now go on to use this model as a template for configuring other interfaces. From the RPC reply, copy and paste everything inside the config tag to a new file entitled "interface.xml". This will become our Jinja template. Next, replace the IP address, subnet mask and interface index with Jinja variables using the double curly braces syntax. Once done, you should end up with a file like the one below. Ensure the file is placed in the same directory as your python program.

<config>
  <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
    <interface>
      <GigabitEthernet>
        <name xmlns:nc='urn:ietf:params:xml:ns:netconf:base:1.0'>{{ INTERFACE_INDEX }}</name>
        <ip>
          <address>
            <primary>
              <address>{{ IP_ADDRESS }}</address>
              <mask>{{ SUBNET_MASK }}</mask>
            </primary>
          </address>
        </ip>
        <mop>
          <enabled>false</enabled>
          <sysid>false</sysid>
        </mop>
        <negotiation xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-ethernet">
          <auto>true</auto>
        </negotiation>
      </GigabitEthernet>
    </interface>
  </native>
</config>

Returning to our python program, we may now configure any interface on our router with two steps.

Template Rendering - Render the Jinja template with desired variable values.
NETCONF Transaction - Send the rendered object as the payload of an edit-config RPC.

To do this in our python program, add the code below. For our example, we are configuring the interface GigabitEthernet 2 with an address of 10.0.0.1/30.

# Render our Jinja template
interface_template = Template(open('interface.xml').read())
interface_rendered = interface_template.render(
  INTERFACE_INDEX='2', 
  IP_ADDRESS='10.0.0.1', 
  SUBNET_MASK='255.255.255.252'
)

# Execute the edit-config RPC
result = m.edit_config(target='running', config=interface_rendered)
print(result)

After the execution of the program, you should see and "ok" RPC reply from the router indicating that the transaction completed successfully.

<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="urn:uuid:2db17593-b51d-4ab2-be28-a268441d6af1" 
  xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ok/>
</rpc-reply>

Finally, verify that the configuration actually exists on the router with the familiar CLI command.

Router#show running-config interface GigabitEthernet 2
Building configuration...

Current configuration : 129 bytes
!
interface GigabitEthernet2
 ip address 10.0.0.1 255.255.255.252
 shutdown
 negotiation auto
 no mop enabled
 no mop sysid
end

Although the example we showed today was quite trivial, I hope it inspires you to think of the possibilities of how this technology can be scaled. For example, you could build an API back-end with atomic Jinja templates for configuring absolutely anything and everything on a network.

Multiple front-end apps could then be built which consume this configuration service - these front end tools could be network orchestrators, change management software, order-to-activation tools, policy managers and much more. This technology truly does enable the digital transformation of companies part of the networking industry.

Finally, if you haven't heard of Ultra Config Generator I would highly recommend that you check it out. It is essentially an out of the box solution for the technology we just discussed which allows companies to rapidly digitize their configuration processes. We designed the product to allow network engineers to generate network configuration in a highly flexible, efficient and elegant manner. Our customers love the application and I hope that you will too.

Thank you very much for reading and if you like the content feel free to follow.

Take care until next time!

Alec

Refusing to give up on your dreams, Storytime

Alec Buda — Fri, 16 Aug 2019 04:04:39 +0000

Everyone has innate dreams and passions - people just find them at different stages of their life. For me, it happened when I was 22 years old. I'll never forget that moment - not only did I discover what I wanted to do with my life, I also discovered my identity.

Backtracking a few years earlier, I was always a pretty random guy. I got really into the guitar one year and jammed all day with friends. I spent a short period of time doing street magic. One month I'd have an emo fringe and the next a mohawk. So yeah - I was all over the place. The only consistency here was that I loved to be creative and learn new things.

Study-wise I was doing a masters in electrical engineering and loved mathematics and problem solving as much as I loved creating and building. It wasn't until after my degree though that I connected the dots to entrepreneurship as my path.

The day I connected the dots was within the same week I completed my degree. I was hanging out with my brother and his girlfriend at their house. I had a 3 month holiday ahead of me before starting a full time job as an Engineer. At one point, we chatted about what I was going to get up to in the 3 months I had off before starting the job. I made a joke about starting a business to sell an electronics gadget I came up with in my last couple of months at uni. I didn't think much of it at the time, but when I was heading home that evening on the train, watching the sun set across some beautiful wheat fields, that I had the realisation. I knew how to design an electronics gadget from my degree - what's the worst that could happen if I actually tried to create this business? The more I thought about it, the more excited I got.

Fast-forward 3 months and I'd created a prototype. Fast-forward another 3 months and I made my first sale on eBay. Go forward another 2 years and the business became a fully established online store with an annual profit on par with a full-time salary.

The story doesn't end there, unfortunately. You see, I never quit my full time job. And the truth is, I don't find fulfilment at all from my full time job. Entrepreneurs out there might know what I mean - some people just can't find fulfilment as an employee, irrespective of the job. In my opinion, there is nothing wrong with this, just as there is nothing wrong with someone who does find peace as an employee. It is just is the way things are.

With that said, I haven't quit my full time job because of money. Without getting into the details, I know that my first business won't provide the same profit in the long term that it does at present. It simply can't be the business that I go all in on. Also, money is a pretty big factor in my life as the cost of living in my city is high. I won't be able to provide for my family in the same way that my parents did for me without a good income. So, in order to achieve my dream as a full time entrepreneur, I had to start all over again from scratch. Outside of the 40 hours working on my full time job and the 10 hours working on my first business, I started working an additional 20 hours a week building a second business.

This is when I began studying ReactJS, MongoDB, ExpressJS and GatsbyJS. I had a grip on HTML, CSS and JavaScript but I had no experience with these modern frameworks when I began. On the first day of deciding I needed to build a second business I completed the famous ReactJS tutorial and was hooked. I developed a passion for the MERN stack and spent every night studying and coding a full stack software application. Jump one year forward and I launched the software application on AWS along with a website to market it on. For anyone whoever doubts their ability to become a great software developer, please don't. I myself am definitely not a professional coder by any means but I was still able to make it through perseverance, passion and a determined commitment to never give up on my dreams. I'll throw in a Jack Ma quote here that I love from 1999 in the early days of Alibaba.

"All of our brains are just as good as theirs. This is the reason we dare to compete with Americans. If we are a good team and know what we want to do, one of us can defeat ten of them. We can beat government agencies and big famous companies because of our innovative spirit. Otherwise, what is the difference between us and them?"

I haven't reached my dream yet. My new business is still in the early days of marketing. I can say with confidence, however, that it will succeed one day. It might not be this year or next year, but one day it will. I will not stop until it does.

That will be it for today! Thanks so much for reading. If you are interested in checking out the UI of my new website I'll link it below. I hope that if you have the desire to build your own website it will inspire you as one person truly can do so much.

https://ultraconfig.com.au

Take care!

Alec