How to implement Hyperledger Fabric External Chaincodes within a Kubernetes cluster

Pau Aragonès Sabaté — Tue, 21 Jul 2020 14:54:31 +0000

Recently released Hyperledger Fabric 2.0 was very welcomed as it introduced some interesting features on how to manage the chaincode with its new decentralized governance for chaincodes, private data enhancements and the ability to use the External chaincode launcher.

The last feature mentioned finally eliminates the docker dependency for launching chaincodes, which caused many difficulties in setting the blockchain networks in some environments, specially when not using the docker-compose setting that is used as tutorials on the Hyperledger Fabric official documentation.

Since the distribution of the blockchain components are done through Docker, many use cases have been developed using Kubernetes as the container orchestration platform to launch the entire or some part of the blockchain network, and the dependency of the Docker daemon in the peer leaded to some configurations that might not be acceptable in production environments, such as the use of the Docker Socket of the worker nodes in the peer to deploy the chaincode containers, hence making these containers out of Kubernetes domain, or the use of Docker-in-Docker (DinD) solutions, which required these containers to be privileged.

With these constraints and limitations, the newly released version enables the preferred way on how these chaincodes are deployed, and in this tutorial, jointly made with Laura Esbri, we are going to see how to deploy them using the External Chaincodes launcher.

Prerequisites

The following is a list of what we are going to need in order to make a simple Fabric network work within a dev Kubernetes cluster.

A Kubernetes cluster. It can perfectly be one cluster using simple tools such as minikube or a single-node kubeadm cluster. The latter is the environment we are going to showcase here. You will need the tools to manage a Kubernetes cluster such as kubectl. Setting up a K8s cluster is out of scope in this article.
Hyperledger Fabric 2.2.0 docker images. The images will be pulled when we are launching the deployments of the Kubernetes yaml descriptors.
Hyperledger Fabric 2.2.0 binaries. We will need them to create the crypto-config and the channel-artifacts. You can download them in this link.
You will find the repository with all the files used in this tutorial here: https://github.com/vanitas92/fabric-external-chaincodes

Install the binaries

Install the binaries required using the following instructions:

wget https://github.com/hyperledger/fabric/releases/download/v2.1.0/hyperledger-fabric-linux-amd64-2.2.0.tar.gz

tar -xzf hyperledger-fabric-linux-amd64-2.2.0.tar.gz

# Move to the bin path
mv bin/* /bin

# Check that you have successfully installed the tools by executing

configtxgen --version

# Should print the following output:
# configtxgen:
#  Version: 2.2.0
#  Commit SHA: 5ea85bc54
#  Go version: go1.14.4
#  OS/Arch: linux/amd64

Launching the network

Once we have the Kubernetes cluster up and ready, we will launch the network. But first we need to generate the basic crypto material necessary to establish the identity and the genesis block of the network. There has been some changes in the configtx.yaml file that are needed to be implemented in order to make the new chaincode lifecycle work, which are necessary to use the External Chaincode Launcher.

The blockchain network will consist of a RAFT orderer service with 3 instances, 2 organisations that have a peer for each one (org1 and org2) with a CA for each organisation. This is already encoded in the configtx.yaml and crypto-config.yaml and there is no need to modify these files.

There are a few new configuration options on configtx.yaml as the new lifecycle and endorsement policies. These options can be defined in this file to set which role is allowed to sign when an endorsement action is happening in the network. In this case, we have set this to the peers that belong to the organization MSP, with the Endorsement policy option:

Organizations:
    - &org1
        # DefaultOrg defines the organization which is used in the sampleconfig
        # of the fabric.git development environment
        Name: org1MSP

        # ID to load the MSP definition as
        ID: org1MSP

        MSPDir: crypto-config/peerOrganizations/org1/msp

        # Policies defines the set of policies at this level of the config tree
        # For organization policies, their canonical path is usually
        #   /Channel/<Application|Orderer>/<OrgName>/<PolicyName>
        Policies: &org1Policies
            Readers:
                Type: Signature
                Rule: "OR('org1MSP.admin', 'org1MSP.peer', 'org1MSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('org1MSP.admin', 'org1MSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('org1MSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('org1MSP.peer')"

    - &org2
        # DefaultOrg defines the organization which is used in the sampleconfig
        # of the fabric.git development environment
        Name: org2MSP

        # ID to load the MSP definition as
        ID: org2MSP

        MSPDir: crypto-config/peerOrganizations/org2/msp

        # Policies defines the set of policies at this level of the config tree
        # For organization policies, their canonical path is usually
        #   /Channel/<Application|Orderer>/<OrgName>/<PolicyName>
        Policies: &org2Policies
            Readers:
                Type: Signature
                Rule: "OR('org2MSP.admin', 'org2MSP.peer', 'org2MSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('org2MSP.admin', 'org2MSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('org2MSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('org2MSP.peer')"

        AnchorPeers:
            # AnchorPeers defines the location of peers which can be used
            # for cross org gossip communication.  Note, this value is only
            # encoded in the genesis block in the Application section context
            - Host: peer0-org2
              Port: 7051
            # - Host: peer1-org2
            #   Port: 7051

After that, we can set the application capabilities to set the signing policies when there is an endorsement happening. There are two rules to be defined, the LifecycleEndorsement and the Endorsement rules. We set them to the MAJORITY rule, which indicates that any endorsement must be approved by more than half of the network participants (50% + 1 signatures required):

Application: &ApplicationDefaults
    # Organizations is the list of orgs which are defined as participants on
    # the application side of the network
    Organizations:

    # Policies defines the set of policies at this level of the config tree
    # For Application policies, their canonical path is
    #   /Channel/Application/<PolicyName>
    Policies: &ApplicationDefaultPolicies
        Readers:
            Type: ImplicitMeta
            Rule: "ANY Readers"
        Writers:
            Type: ImplicitMeta
            Rule: "ANY Writers"
        Admins:
            Type: ImplicitMeta
            Rule: "MAJORITY Admins"
        LifecycleEndorsement:
            Type: ImplicitMeta
            Rule: "MAJORITY Endorsement"
        Endorsement:
            Type: ImplicitMeta
            Rule: "MAJORITY Endorsement"

Once you are all set with the configuration of the network, you can now generate the crypto materials and the genesis block of the blockchain network. We will use the fabricOps.sh script with the following command:

Note: Make sure that this script has execution permissions. Changing to 744 permision should be enough.

$ ./fabricOps.sh start

This will create all the crypto materials such as all the certificates of the peers orderers and CAs, as well as the genesis block of the channel.

We will first need to create a new namespace for the Hyperledger Fabric workload, create one with the following command:

$ kubectl create ns hyperledger

Create a folder in the VM for storing the persistent data for all the Hyperledger Fabric workload:

mkdir /home/storage

Once we have created the namespace, we are ready to deploy the workload:

Note: Check and modify if needed the hostpath of the volumes attached on the orderers’ deployment, the yamls use absolute paths and it is assumed here that the repository is in the /home folder.

$ kubectl create -f orderer-service/

Check that everything is running fine by issuing the following command:

$ kubectl get pods -n hyperledger

### Should print a similar output
NAME                        READY   STATUS    RESTARTS   AGE
orderer0-58666b6bd7-pflf7   1/1     Running   0          5m47s
orderer1-c4fd65c7d-c27ll    1/1     Running   0          5m47s
orderer2-557cb7865-wlcmh    1/1     Running   0          5m47s

Note 2: Again, check and modify if needed the hostpath of the volumes attached on the peer, CA and CLI deployments, the yamls use absolute paths and it is assumed here that the repository is in the /home folder.

Now create the org1 workload, which will deploy the peer and the CA of this organisation:

$ kubectl create -f org1/

Check that everything is running fine by issuing the following command:

$ kubectl get pods -n hyperledger

### Should print a similar output
NAME                          READY   STATUS    RESTARTS   AGE
ca-org1-84945b8c7b-9px4s      1/1     Running   0          19m
cli-org1-bc9f895f6-zmmdc      1/1     Running   0        2m56s
orderer0-58666b6bd7-pflf7     1/1     Running   0          79m
orderer1-c4fd65c7d-c27ll      1/1     Running   0          79m
orderer2-557cb7865-wlcmh      1/1     Running   0          79m
peer0-org1-798b974467-vv4zz   1/1     Running   0          19m

Repeat the same steps with org2 workload:

Note 3: Again, check and modify if needed the hostpath of the volumes attached on the peer, CA and CLI deployments, the yamls use absolute paths and it is assumed here that the repository is in the /home folder.

$ kubectl create -f org2/

Check that everything is running fine by issuing the following command:

$ kubectl get pods -n hyperledger

### Should print a similar output
NAME                          READY   STATUS    RESTARTS   AGE
ca-org1-84945b8c7b-9px4s      1/1     Running   0          71m
ca-org2-7454f69c48-q8lft      1/1     Running   0        2m20s
cli-org1-bc9f895f6-zmmdc      1/1     Running   0          55m
cli-org2-7779cc8788-8q4ns     1/1     Running   0        2m20s
orderer0-58666b6bd7-pflf7     1/1     Running   0         131m
orderer1-c4fd65c7d-c27ll      1/1     Running   0         131m
orderer2-557cb7865-wlcmh      1/1     Running   0         131m
peer0-org1-798b974467-vv4zz   1/1     Running   0          71m
peer0-org2-5849c55fcd-mbn5h   1/1     Running   0        2m19s

All workload above should appear right now in the hyperledger namespace.

Note 4: If there are CrashloopBackoff errors, check that the paths are correctly set and crypto files are in place and correctly set. Check the logs by issuing the kubectl logs pod_name -n hyperledger command, pointing to the faulty pod.

Setting up the network channels

Once all workload is deployed, we are ready to create the channel and join the peers to the channel. Enter in the cli pod of the org1:

$ kubectl exec -it cli_org1_pod_name sh -n hyperledger

Once inside the cli pod, execute the following command:

$ peer channel create -o orderer0:7050 -c mychannel -f ./scripts/channel-artifacts/channel.tx --tls true --cafile $ORDERER_CA

### Should print a similar output
2020-03-06 11:54:57.582 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-03-06 11:54:58.903 UTC [cli.common] readBlock -> INFO 002 Received block: 0

The channel mychannel is created and ready to be used. Next, join the peer of org1 to the channel:

$ peer channel join -b mychannel.block

### Should print a similar output
2020-03-06 12:01:41.608 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-03-06 12:01:41.688 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel

We are going to perform the same steps on org2 cli, but since the channel is already created by org1, we are going to fetch the genesis block from the orderer service. Enter the pod first:

$ kubectl exec -it cli_org2_pod_name sh -n hyperledger

Once inside the cli pod, execute the following command:

$ peer channel fetch 0 mychannel.block -c mychannel -o orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
2020-03-06 12:18:14.880 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-03-06 12:18:14.895 UTC [cli.common] readBlock -> INFO 002 Received block: 0

Then join to the channel from the genesis block:

$ peer channel join -b mychannel.block

### Should print a similar output
2020-03-06 12:20:41.475 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-03-06 12:20:41.561 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel

You can check anytime if any of the peers has joined the channel by executing the following command:

$ peer channel list

### Should print a similar output
2020-03-06 12:22:41.102 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Channels peers has joined: 
mychannel

Installing the External Chaincode

Now we are going to do the interesting stuff 😄. We are going to deploy the marbles chaincode as an External chaincode. You can find the original chaincode in the fabric-sample Github repository, but in order to deploy it as External Chaincode, we need to perform some changes in the imports and the init function to declare it as external chaincode server.

The new version of Hyperledger also comes with the new chaincode lifecycle process for installing a chaincode on your peers and starting it on a channel. In order to use the External Chaincodes feature, we have to use this new process as well.

The first thing we have to do is to configure the peer to process external chaincode. The external builders are merely based on buildpacks and so we have create 3 scripts, detect, build and release. These 3 scripts have to be defined inside the peer container. These scripts can be found in the buildpack/bin folder.

Note 5: Make sure that these 3 scripts have execution permissions in the buildpack/bin folder. The scripts are executed in the peer pod as root so changing to 744 permission should be enough. Failing to do so will cause the externalbuilder to fail its execution.

The definition of the builders can be found inside the org1 folder in the builders-config.yaml. This file has all the default options to configure the peer of the core.yaml except the core_chaincode_externalbuilders option, which has a custom builder configuration like the following.

            # List of directories to treat as external builders and launchers for
            # chaincode. The external builder detection processing will iterate over the
            # builders in the order specified below.
            externalBuilders:
              - name: external-builder
                path: /builders/external
                environmentWhitelist:
                   - GOPROXY

Because the Kubernetes deployment descriptors of the peers have environment variables to configure the option, these environment variables override the default option from the core.yaml. This was done this way because the environment variables do not accept array formats and this way is able to configure each peer with his own configuration but establish this builder. It is used by all organizations in this network.

The new lifecycle process packages the chaincode and installs it in a different way than previous versions. Since we are using the external chaincode feature, the chaincode code does not have to be compiled and installed within the peer pod itself, rather be on another pod. The only code to be installed in the peer process is the information required to be able to connect to the external chaincode process.

Note 6: We are going to perform the steps for the org1 peer chaincode, hence execute the commands on the org1 cli pod. These steps will be repeated for org2 peer but changing some configuration of the chaincode later.

To achieve this, we need to package the chaincode with some requirements. There has to be a connection.json file that contains the information of the connection to the external chaincode server. This includes the address, TLS certificates and dial timeout configurations.

{
    "address": "chaincode-marbles-org1.hyperledger:7052",
    "dial_timeout": "10s",
    "tls_required": false,
    "client_auth_required": false,
    "client_key": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----",
    "client_cert": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
    "root_cert": "-----BEGIN CERTIFICATE---- ... -----END CERTIFICATE-----"
}

This file needs to be packaged into a tar file called code.tar.gz. Achieve this by executing the following command.

Note 6: You can perform these commands inside the cli tools pod or outside the cli pod in your host machine where your chaincode hostpath storage points. In the end, the resulting tar files are needed in the cli pod to launch commands, so i recommend it doing it from the cli pod. If you are going to perform these commands in the cli pod, then go to this path: /opt/gopath/src/github.com/marbles/packaging

$ cd chaincode/packaging
$ tar cfz code.tar.gz connection.json

Once we have the code.tar.gz file, then we have to repackage it again alongside with another file called metadata.json, which includes information about the type of chaincode to be processed, the path where the chaincode resides and the label we want to give to that chaincode.

{"path":"","type":"external","label":"marbles"}

We package this two files with the following command:

$ tar cfz marbles-org1.tgz code.tar.gz metadata.json

When you got the tar file, you are ready to install it in the peer using the new lifecycle process.

$ peer lifecycle chaincode install marbles-org1.tgz

### Should print a similar output
2020-03-07 14:33:18.120 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 001 Installed remotely: response:<status:200 payload:"\nGdmarbles:e001937433673b11673d660d142c722fc372905db87f88d2448eee42c9c63064\022\006marbles" > 
2020-03-07 14:33:18.126 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 002 Chaincode code package identifier: marbles:e001937433673b11673d660d142c722fc372905db87f88d2448eee42c9c63064

Copy the chaincode code package identifier as we will need it later. Nevertheless, you can always retrieve it back by executing the following command:

$ peer lifecycle chaincode queryinstalled

### Should print a similar output
Installed chaincodes on peer:
Package ID: marbles:030eec59c7d74fbb4e9fd57bbd50bb904a715ffb9de8fea85b6a6d4b8ca9ea12, Label: marbles

Now we are going to repeat the same steps above for org2 but since we want to have a different pod to serve the same chaincode for org2 peer, we have to change the address config option in connection.json file. Modify the file address value like this:

"address": "chaincode-marbles-org2.hyperledger:7052",

Then repeat the same steps as before, executing the install command in the org2 cli pod:

$ rm -f code.tar.gz
$ tar cfz code.tar.gz connection.json
$ tar cfz marbles-org2.tgz code.tar.gz metadata.json
$ peer lifecycle chaincode install marbles-org2.tgz

### Should print a similar output
2020-03-07 15:10:15.093 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 001 Installed remotely: response:<status:200 payload:"\nGmarbles:c422c797444e4ee25a92a8eaf97765288a8d68f9c29cedf1e0cd82e4aa2c8a5b\022\006marbles" > 
2020-03-07 15:10:15.093 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 002 Chaincode code package identifier: marbles:c422c797444e4ee25a92a8eaf97765288a8d68f9c29cedf1e0cd82e4aa2c8a5b

Copy the chaincode code package identifier as before. It should have a different hash from the org1 as we changed the address for org2.

What happened in the peer while installing the chaincode? If there are some external builders or launchers defined in the peer, then they get executed before attempting the classic option of building the Docker container by the peer itself. As we have defined the external-builder, the following order of scripts are executed.

The detect script simply evaluates whether the chaincode that is getting installed has the type key of metadata.json as a value external. If that script fails then the peer considers that this external builder does not have to build the chaincode and iterates to other external builders until there are no more defined and fallback to the Docker building process.

#!/bin/sh

# The bin/detect script is responsible for determining whether or not a buildpack 
# should be used to build a chaincode package and launch it. 
# 
# The peer invokes detect with two arguments:
# bin/detect CHAINCODE_SOURCE_DIR CHAINCODE_METADATA_DIR
#
# When detect is invoked, CHAINCODE_SOURCE_DIR contains the chaincode source and 
# CHAINCODE_METADATA_DIR contains the metadata.json file from the chaincode package installed to the peer. 
# The CHAINCODE_SOURCE_DIR and CHAINCODE_METADATA_DIR should be treated as read only inputs. 
# If the buildpack should be applied to the chaincode source package, detect must return an exit code of 0; 
# any other exit code will indicate that the buildpack should not be applied.

CHAINCODE_METADATA_DIR="$2"

set -euo pipefail

# use jq to extract the chaincode type from metadata.json and exit with
# success if the chaincode type is golang
if [ "$(cat "$CHAINCODE_METADATA_DIR/metadata.json" | sed -e 's/[{}]/''/g' | awk -F"[,:}]" '{for(i=1;i<=NF;i++){if($i~/'type'\042/){print $(i+1)}}}' | tr -d '"')" = "external" ]; then
    exit 0
fi

exit 1

If the detect script succeeds, then the build script is called. This script should generate the artifacts or binaries if we were building the chaincode inside the peer, but as we want to have as external service, it is enough to simply copy the connection.json file to the build output directory.

#!/bin/sh

# The bin/build script is responsible for building, compiling, or transforming the contents 
# of a chaincode package into artifacts that can be used by release and run.
#
# The peer invokes build with three arguments:
# bin/build CHAINCODE_SOURCE_DIR CHAINCODE_METADATA_DIR BUILD_OUTPUT_DIR
#
# When build is invoked, CHAINCODE_SOURCE_DIR contains the chaincode source and 
# CHAINCODE_METADATA_DIR contains the metadata.json file from the chaincode package installed to the peer.
# BUILD_OUTPUT_DIR is the directory where build must place artifacts needed by release and run. 
# The build script should treat the input directories CHAINCODE_SOURCE_DIR and 
# CHAINCODE_METADATA_DIR as read only, but the BUILD_OUTPUT_DIR is writeable.

CHAINCODE_SOURCE_DIR="$1"
CHAINCODE_METADATA_DIR="$2"
BUILD_OUTPUT_DIR="$3"

set -euo pipefail

#external chaincodes expect connection.json file in the chaincode package
if [ ! -f "$CHAINCODE_SOURCE_DIR/connection.json" ]; then
    >&2 echo "$CHAINCODE_SOURCE_DIR/connection.json not found"
    exit 1
fi

#simply copy the endpoint information to specified output location
cp $CHAINCODE_SOURCE_DIR/connection.json $BUILD_OUTPUT_DIR/connection.json

if [ -d "$CHAINCODE_SOURCE_DIR/metadata" ]; then
    cp -a $CHAINCODE_SOURCE_DIR/metadata $BUILD_OUTPUT_DIR/metadata
fi

exit 0

Finally, once the build script completes, the release script gets called. This script is responsible for providing the connection.json to the peer by placing it in the release output directory. Hence, the peer now knows where to call the chaincode if some invocations of that chaincode are to be executed.

#!/bin/sh

# The bin/release script is responsible for providing chaincode metadata to the peer. 
# bin/release is optional. If it is not provided, this step is skipped. 
#
# The peer invokes release with two arguments:
# bin/release BUILD_OUTPUT_DIR RELEASE_OUTPUT_DIR
#
# When release is invoked, BUILD_OUTPUT_DIR contains the artifacts 
# populated by the build program and should be treated as read only input. 
# RELEASE_OUTPUT_DIR is the directory where release must place artifacts to be consumed by the peer.

set -euo pipefail

BUILD_OUTPUT_DIR="$1"
RELEASE_OUTPUT_DIR="$2"

# copy indexes from metadata/* to the output directory
# if [ -d "$BUILD_OUTPUT_DIR/metadata" ] ; then
#    cp -a "$BUILD_OUTPUT_DIR/metadata/"* "$RELEASE_OUTPUT_DIR/"
# fi

#external chaincodes expect artifacts to be placed under "$RELEASE_OUTPUT_DIR"/chaincode/server
if [ -f $BUILD_OUTPUT_DIR/connection.json ]; then
   mkdir -p "$RELEASE_OUTPUT_DIR"/chaincode/server
   cp $BUILD_OUTPUT_DIR/connection.json "$RELEASE_OUTPUT_DIR"/chaincode/server

   #if tls_required is true, copy TLS files (using above example, the fully qualified path for these fils would be "$RELEASE_OUTPUT_DIR"/chaincode/server/tls)

   exit 0
fi

exit 1

Building and deploying the External Chaincode

Once we have the chaincodes installed in the peers, we can now build them and deploy them in our Kubernetes cluster. Let’s look at the changes we need to implement to the chaincodes.

As the chaincodes do not vendor and include the modules needed currently, the imports have changed, so there is the need to import them and vendoring the modules.

import (
    "bytes"
    "encoding/json"
    "fmt"
    "strconv"
    "strings"
    "time"
    "os"

    "github.com/hyperledger/fabric-chaincode-go/shim"
    pb "github.com/hyperledger/fabric-protos-go/peer"
)

It is important to also define the go.mod file to establish the version of the modules before building the code. The latest builds in the shim modules are needed to enable the External Chaincode property.

module github.com/marbles

go 1.12

require (
        github.com/hyperledger/fabric-chaincode-go v0.0.0-20200128192331-2d899240a7ed
        github.com/hyperledger/fabric-protos-go v0.0.0-20200124220212-e9cfc186ba7b
        golang.org/x/net v0.0.0-20200202094626-16171245cfb2 // indirect
        golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4 // indirect
        golang.org/x/text v0.3.2 // indirect
        google.golang.org/genproto v0.0.0-20200218151345-dad8c97a84f5 // indirect
)

The other change is the Chaincode Server property which listens to a particular address and port that we want to establish. As we might want to change them by using Kubernetes yaml descriptors, we are going to use the os module to get the value of the address and the chaincode code package identifier (CCID). By doing this we can deploy the same image of the code but changing the parameters as needed.

func main() {

    server := &shim.ChaincodeServer{
        CCID:    os.Getenv("CHAINCODE_CCID"),
        Address: os.Getenv("CHAINCODE_ADDRESS"),
        CC:      new(SimpleChaincode),
        TLSProps: shim.TLSProperties{
                Disabled: true,
        },
    }

    // Start the chaincode external server
    err := server.Start()

    if err != nil {
        fmt.Printf("Error starting Marbles02 chaincode: %s", err)
    }
}

We are now ready to build the chaincode container using the following Dockerfile:

# This image is a microservice in golang for the Marbles chaincode
FROM golang:1.14.6-alpine AS build

COPY ./ /go/src/github.com/marbles
WORKDIR /go/src/github.com/marbles

# Build application
RUN go build -o chaincode -v .

# Production ready image
# Pass the binary to the prod image
FROM alpine:3.11 as prod

COPY --from=build /go/src/github.com/marbles/chaincode /app/chaincode

USER 1000

WORKDIR /app
CMD ./chaincode

The chaincode is built using a golang alpine image. Once the building stage is complete, the binary is transferred to a bare alpine image for a smaller and secure image. Execute the following command:

$ docker build -t chaincode/marbles:1.0 .

If everything goes right, you should have the image ready to be deployed. Modify the files of the chaincode deployment ( org1-chaincode-deployment.yaml and org2-chaincode-deployment.yaml) the CHAINCODE_CCID variable to the values you got before when installing the chaincode respectively.

#---------------- Chaincode Deployment ---------------------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chaincode-marbles-org1
  namespace: hyperledger
  labels:
    app: chaincode-marbles-org1
spec:
  selector:
    matchLabels:
      app: chaincode-marbles-org1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: chaincode-marbles-org1
    spec:
      containers:
        - image: chaincode/marbles:1.0
          name: chaincode-marbles-org1
          imagePullPolicy: IfNotPresent
          env:
            - name: CHAINCODE_CCID
              value: "marbles:d8140fbc1a0903bd88611a96c5b0077a2fdeef00a95c05bfe52e207f5f9ab79d"
            - name: CHAINCODE_ADDRESS
              value: "0.0.0.0:7052"
          ports:
            - containerPort: 7052

After that, deploy them:

$ kubectl create -f chaincode/k8s

The service and deployments will be deployed in the same k8s namespace as the other workload.

$ kubectl get pods -n hyperledger
NAME                                      READY   STATUS    RESTARTS   AGE
ca-org1-84945b8c7b-tx59g                  1/1     Running   0          19h
ca-org2-7454f69c48-nfzsq                  1/1     Running   0          19h
chaincode-marbles-org1-6fc8858855-wdz7z   1/1     Running   0          20m
chaincode-marbles-org2-77bf56fdfb-6cdfm   1/1     Running   0          14m
cli-org1-589944999c-cvgbx                 1/1     Running   0          19h
cli-org2-656cf8dd7c-kcxd7                 1/1     Running   0          19h
orderer0-5844bd9bcc-6td8c                 1/1     Running   0          46h
orderer1-75d8df99cd-6vbjl                 1/1     Running   0          46h
orderer2-795cf7c4c-6lsdd                  1/1     Running   0          46h
peer0-org1-5bc579d766-kq2qd               1/1     Running   0          19h
peer0-org2-77f58c87fd-sczp8               1/1     Running   0          19h

Now we have to approve the chaincodes for each organization. This is a new feature of the chaincode lifecycle process, each organization has to agree to approve a new definition of a chaincode. We are going to approve the marble chaincode definition for org1. Execute this command inside the org1 cli pod, remember to change the CHAINCODE_CCID:

$ peer lifecycle chaincode approveformyorg --channelID mychannel --name marbles --version 1.0 --init-required --package-id marbles:e001937433673b11673d660d142c722fc372905db87f88d2448eee42c9c63064 --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
2020-03-08 10:02:46.192 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [4d81ea5fd494e9717a0c860812d2b06bc62e4fc6c4b85fa6c3a916eee2c78e85] committed with status (VALID)

You can check the state of approvals throughout the entire network by executing the following command:

$ peer lifecycle chaincode checkcommitreadiness --channelID mychannel --name marbles --version 1.0 --init-required --sequence 1 -o -orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
Chaincode definition for chaincode 'marbles', version '1.0', sequence '1' on channel 'mychannel' approval status by org:
org1MSP: true
org2MSP: false

Now let’s approve for org2. Execute this command inside the org2 cli pod, remember to change the CHAINCODE_CCID:

$ peer lifecycle chaincode approveformyorg --channelID mychannel --name marbles --version 1.0 --init-required --package-id marbles:25a9f6fe26161d29af928228ca1db0c41892e26e46335c84952336ee26d1fd93 --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
2020-03-08 10:26:43.992 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [74a89f3c93c10f14c626bd4d6cb654b37889908c9e6f7b983d2cad79f1e82267] committed with status (VALID)

Check again the commit readiness of the chaincode:

$ peer lifecycle chaincode checkcommitreadiness --channelID mychannel --name marbles --version 1.0 --init-required --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
Chaincode definition for chaincode 'marbles', version '1.0', sequence '1' on channel 'mychannel' approval status by org:
org1MSP: true
org2MSP: true

Now that we have all the approvals of all the organizations, let’s commit the definition of this chaincode in the channel. You can do this operation on any of the peers:

$ peer lifecycle chaincode commit -o orderer0:7050 --channelID mychannel --name marbles --version 1.0 --sequence 1 --init-required --tls true --cafile $ORDERER_CA --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt

### Should print a similar output
2020-03-08 14:13:49.516 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [568cb81f821698025bbc61f4c6cd3b4baf1aea632e1e1a8cfdf3ec3902d1c6bd] committed with status (VALID) at peer0-org1:7051
2020-03-08 14:13:49.533 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [568cb81f821698025bbc61f4c6cd3b4baf1aea632e1e1a8cfdf3ec3902d1c6bd] committed with status (VALID) at peer0-org2:7051

Now we have the chaincode definition added to the channel and ready to be invoked and make queries against it! 😄

Testing the External Chaincode

We can test the chaincodes making invoke and query commands from the cli pod. These commands are not modified by the lifecycle chaincode process and can be called as the chaincodes in version 1.x of Hyperledger Fabric. First, let’s create some marbles in the ledger. Execute the following command on one cli pod, either org1 or org2:

$ peer chaincode invoke -o orderer0:7050 --isInit --tls true --cafile $ORDERER_CA -C mychannel -n marbles --peerAddresses 
peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer
0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt -c '{"Args":["initMarble
","marble1","blue","35","tom"]}' --waitForEvent

### Should print a similar output
2020-03-08 14:23:03.569 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [83aeeaac47cf6302bc139addc4aa38116a40eaff788846d87cc815d2e1318f44] committed with status (VALID) at peer0-org2:7051
2020-03-08 14:23:03.575 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [83aeeaac47cf6302bc139addc4aa38116a40eaff788846d87cc815d2e1318f44] committed with status (VALID) at peer0-org1:7051
2020-03-08 14:23:03.576 UTC [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 003 Chaincode invoke successful. result: status:200

This will initiate the ledger, the --isInit will call to the legacy init function to start the ledger. Now create the first marble:

$ peer chaincode invoke -o orderer0:7050 --isInit --tls true --cafile $ORDERER_CA -C mychannel -n marbles --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt -c '{"Args":["initMarble","marble1","red","45","tom"]}' --waitForEvent

### Should print a similar output
2020-03-08 14:23:40.404 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [8391f9f8ea84887a56f99e4dc4501eaa6696cd7bd6c524e4868bd6cfd5b85e78] committed with status (VALID) at peer0-org2:7051
2020-03-08 14:23:40.434 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [8391f9f8ea84887a56f99e4dc4501eaa6696cd7bd6c524e4868bd6cfd5b85e78] committed with status (VALID) at peer0-org1:7051
2020-03-08 14:23:40.434 UTC [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 003 Chaincode invoke successful. result: status:200

The first c

Retrieve information from marble1:

$ peer chaincode query -C mychannel -n marbles -c '{"Args":["readMarble","marble1"]}'

{"docType":"marble","name":"marble1","color":"red","size":45,"owner":"tom"}

There are many example commands you can execute with this chaincode, just check the source code of the chaincode for more examples.

You can also check the logs of the chaincode containers by executing the following command:

$ kubectl logs chaincode_pod_name -n hyperledger

### Should print a similar output
invoke is running initMarble
- start init marble
- end init marble
invoke is running initMarble
- start init marble
- end init marble
invoke is running readMarble

Bonus Track! ContractApi based Chaincodes with External Chaincodes feature

As a new addition that we have been playing lately, we can now deploy external chaincodes with the new ContractApi introduced in version 2.0, which makes it even more easy to develop chaincodes. We will deploy the fabcar chaincode, you can find the original here. The modified one would be pushed in the repo under the folder named fabcar.

We will need to do add the new shim api which allows to add the External Chaincode server interface that allows the external communication with the peer.

package main

import (
    "encoding/json"
    "fmt"
    "strconv"
    "os"

    "github.com/hyperledger/fabric-chaincode-go/shim"
    "github.com/hyperledger/fabric-contract-api-go/contractapi"
)

// SmartContract provides functions for managing a car
type SmartContract struct {
    contractapi.Contract
}

// Car describes basic details of what makes up a car
type Car struct {
    Make   string `json:"make"`
    Model  string `json:"model"`
    Colour string `json:"colour"`
    Owner  string `json:"owner"`
}

// QueryResult structure used for handling result of query
type QueryResult struct {
    Key    string `json:"Key"`
    Record *Car
}

// Main function to start the external chaincode server
func main() {
    cc, err := contractapi.NewChaincode(new(SmartContract))

    if err != nil {
        fmt.Println("Error starting a new ContractApi Chaincode:", err)
    }

    server := &shim.ChaincodeServer{
        CCID:    os.Getenv("CHAINCODE_CCID"),
        Address: os.Getenv("CHAINCODE_ADDRESS"),
        CC:      cc,
        TLSProps: shim.TLSProperties{
                Disabled: true,
        },
    }

    // Start the chaincode external server
    err = server.Start()

    if err != nil {
        fmt.Println("Error starting FabCar chaincode server:", err)
    } else {
        fmt.Println("Succesfully started new Fabcar Chaincode server with the new ContractApi")
    }
}

The go.mod file would need to import these two modules as well.

module github.com/fabcar

go 1.14

require (
        github.com/hyperledger/fabric-chaincode-go v0.0.0-20200424173110-d7076418f212
        github.com/hyperledger/fabric-contract-api-go v1.1.0
        golang.org/x/net v0.0.0-20200202094626-16171245cfb2 // indirect
        golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4 // indirect
        golang.org/x/text v0.3.2 // indirect
        google.golang.org/genproto v0.0.0-20200218151345-dad8c97a84f5 // indirect
)

And that would be all modifications needed in order to make it work! The instructions to install it works just the same but with the word fabcar instead of marbles. The only difference is that there is no need for --init-required or --isInit flag in the approveformyorg , commit and invoke commands.

peer lifecycle chaincode approveformyorg --channelID mychannel --name fabcar --version 1.0 --package-id fabcar:005c35f4f172c056723eca09d41e8048e0beaa2712d920c19af837640df7e2aa --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

peer lifecycle chaincode approveformyorg --channelID mychannel --name fabcar --version 1.0 --package-id fabcar:61ab817a6ad76098d340952e5d8e928d9c5ddff1a53341dc8d0c64b4345564b0 --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

peer lifecycle chaincode checkcommitreadiness --channelID mychannel --name fabcar --version 1.0 --sequence 1 -o -orderer0:7050 --tls --cafile $ORDERER_CA

peer lifecycle chaincode commit -o orderer0:7050 --channelID mychannel --name fabcar --version 1.0 --sequence 1 --tls true --cafile $ORDERER_CA --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt

peer chaincode invoke -o orderer0:7050 --tls true --cafile $ORDERER_CA -C mychannel -n fabcar --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt -c '{"Args":["InitLedger"]}' --waitForEvent

peer chaincode query -C mychannel -n fabcar -c '{"Args":["QueryAllCars"]}'

You would also find the original implementation of this same chaincode with the External Chaincode feature by the Hyperledger team in their github repo. Not many differences are between their implementation and ours, so we assume that this is the proper way to implement this contract based chaincodes with external feature.

Future work and improvements

There are some improvements that can be made and we are happy to receive feedback from the community to improve these External Chaincodes.

External Builders definition as environment variables rather than configmap. Although it works this way, we would like to define the External Builders in the environment variables as other options. We tried to define the external builders as environment variables but since the ExternalBuilders configurations is expected as array and the environment variables only accept strings, we were having many errors. UPDATE 2: Many of you have reported many problems with externalbuilder definition not being executed succesfully or properly detected. This was caused mainly for permissions on the scripts folder (execute permission required not enabled after cloning from GitHub). For ease of use, the buildpack scripts have now been included as a configmap and attached in the peer pods with greater kubernetes integration.
Endorsement Policy failures: We had to disable the Endorsement and the Lifecycle Endorsment policies due to many endorsement policy failures when committing the chaincode definition on the channel. As this might need to have some better understanding and definition of the policies in version 2.0, we did not have enough time to configure it properly. UPDATE 1: This has now been solved and the article has been updated with the new requirements for making the endorsement policies and lifecycle endorsement policies coded in the genesis block.
Manual changes on the address configuration of the chaincode. The address value of the connection.json file needs to be changed every time we have a new organization. And also can change if we have multiple endorser peers for each organisation. There could be some interesting pipelines to help support the deployment of these type of chaincodes.

Thanks for taking a look into it!

DEV Community: Pau Aragonès Sabaté

How to implement Hyperledger Fabric External Chaincodes within a Kubernetes cluster

Prerequisites

Install the binaries

Launching the network

Setting up the network channels

Installing the External Chaincode

Building and deploying the External Chaincode

Testing the External Chaincode

Bonus Track! ContractApi based Chaincodes with External Chaincodes feature

Future work and improvements