DEV Community: Joe Miyamoto

Building c-lightning plugin with .NET

Joe Miyamoto — Sun, 27 Mar 2022 10:53:58 +0000

Update 2022/05/20: Based on the methodology described in this article, I have created a package for building your own plugin. please check out DotNetLightning.ClnRpc

Recently, I have been modifying my own Lapps to work as a c-lightning plugin in order to make them work with not only with LND but also with c-lightning (What are Lapps?).

I couldn't find someone doing it with .NET/C#/F#, so I will leave here a guideline for how to.

I also made a sample application, so if you are more interested in the sample than the explanation, please go there. My application is in F#, but this sample is written in C#. The language is different, but the basic approach is the same in both cases.

In this case, we will assume that there is a single class named MyAwesomeRpcServer, that has method handlers for RPC invocation. It starts out like this


using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using StreamJsonRpc;

namespace HelloWorldPlugin.Server
{
  public class MyAwesomeRpcServerOptions
  {
    public string GreeterName { get; set; } = "World";
    public bool IsInitiated { get; set; }
  }

  public class MyAwesomeRpcServer
  {
    private readonly MyAwesomeRpcServerOptions _opts;

    public MyAwesomeRpcServer(MyAwesomeRpcServerOptions opts)
    {
      _opts = opts;
    }

    [JsonRpcMethod("hello")]
    public async Task<string> HelloAsync(string name)
    {
      return $"hello!! {name}! This is {_opts.GreeterName} !!";
    }
  }
}

The requirements are

We want to expose an rpc method called hello.
We want to return a message containing GreeterName passed as a command line argument
We want to return a message that includes the name passed as an RPC parameter from the User.
The number of exposed RPC may increase in the future.

1. Create a JsonRPCServer using StreamJsonRPC.

First, create an application that acts as an ordinary JsonRPC 2.0 server.
The standard way to do this is to use the StreamJsonRpc library authored by MS.

If you want to use it as a normal web server over HTTP, this sample is a good reference.
However, the c-lightning plugin communicates with lightningd itself via standard input/output, so this time we will use Console.In and Console.Out for transport.

It looks like this

var rpcServer = new MyAwesomeRpcServer();
var formatter = new JsonMessageFormatter();
var handler = new NewLineDelimitedMessageHandler(Console.OpenStandardOutput(), Console.OpenStandardInput(), formatter);
var rpc = new JsonRpc(handler);
rpc.AddLocalRpcTarget(rpcServer, new JsonRpcTargetOptions());
rpc.ExceptionStrategy = ExceptionProcessing.CommonErrorData;
rpc.StartListening();

The Json RPC 2.0 specification does not specify the way of separating each messages as well as transports, so we must choose a suitable one.
Since the c-lightning plugin uses newlines to delimit messages, we use NewLineDelimitedMessageHandler as a message handler.
Also, if the communication partner is not C# (which is not in our case), it is a good to specify ExceptionStrategy as ExceptionProcessing.CommonErrorData.

2. Make read/write operations thread-safe.

Normally, the RPC server runs in multi-threaded mode, but writing to stdin/output in parallel will result in message corruption.
The easiest way to deal with this problem is to process requests sequentially.

In this case, the easiest way is to use AsyncSemaphore, as described in the official guidelines.

In the case of plugins, performance issues related to rpc calls are rare, so we decided to limit writes by acquiring the semaphore for all methods.

3. Automatic generation of methods

The c-lightning plugin must expose two RPC methods: getmanifest and init.

The former is to pass information from the plugin to c-lightning, and the latter is to pass information other way around when c-lightning is launched.

Much of the processing logic can be generated automatically from other plugin method definitions, so we use reflection to keep things generic enough.

For example, getmanifest looks like this

    [JsonRpcMethod("getmanifest")]
    public async Task<ManifestDto> GetManifestAsync(bool allowDeprecatedApis = false, object? otherParams = null)
    {
      using var releaser = await _semaphore.EnterAsync();
      var userDefinedMethodInfo =
        this
          .GetType()
          .GetMethods(BindingFlags.Public | BindingFlags.Instance | BindingFlags.DeclaredOnly)
          .Where(m => !m.IsSpecialName && !String.Equals(m.Name, "initasync", StringComparison.OrdinalIgnoreCase) && !String.Equals(m.Name, "getmanifestasync", StringComparison.OrdinalIgnoreCase));
      var methods =
        userDefinedMethodInfo
          .Select(m =>
          {
            var argSpec = m.GetParameters();
            var numDefaults =
              argSpec.Count(s => s.HasDefaultValue);
            var keywordArgsStartIndex = argSpec.Length - numDefaults;
            var args =
              argSpec.Where(s =>
                !string.Equals(s.Name, "plugin", StringComparison.OrdinalIgnoreCase) &&
                !string.Equals(s.Name, "request", StringComparison.OrdinalIgnoreCase))
                .Select((s, i) => i < keywordArgsStartIndex ? s.Name : $"[{s.Name}]");

            var name = m.Name.ToLowerInvariant();
            if (name.EndsWith("async"))
              name = name[..^5];
            return new RPCMethodDTO
            {
              Name = name,
              Usage = string.Join(' ', args),
              Description = _rpcDescriptions[name].Item1,
              LongDescription = _rpcDescriptions[name].Item2
            };
          });

      return new ManifestDto
      {
        Options =
          // translate `System.CommandLine` to c-lightning compatible style.
          CommandLines.GetOptions().Select(CliOptionsToDto).ToArray(),
        RpcMethods = methods.ToArray(),
        Notifications = new NotificationsDTO[]{},
        Subscriptions = new string[]{},
        Hooks = new object[] {},
        Dynamic = true,
        FeatureBits = null
      };

init is as follows. This is more difficult to automate than getmanifest, and the process varies depending on the requirements of the application.

    [JsonRpcMethod("init")]
    public async Task InitAsync(LnInitConfigurationDTO configuration, Dictionary<string, object> options)
    {
      using var releaser = await _semaphore.EnterAsync();
      foreach (var op in options)
      {
        var maybeProp =
          _opts.GetType().GetProperties()
            .FirstOrDefault(p => string.Equals(p.Name, op.Key, StringComparison.OrdinalIgnoreCase));
        maybeProp?.SetValue(_opts, op.Value);
      }
      _opts.IsInitiated = true;
    }

init passes method arguments as objects.
The object has fields named configurations and options, So the method handler on the C# side must have a same name to automatically bind.

In this case, we only bind the startup options givin by the c-lightning side to _opts.
LnInitConfigurationDto's information on the c-lightning side (e.g. is an user using tor? What lightning feature bits are supported? ) can be used to freely perform the initialization process.

We set IsInitiated at the completion of the initialization process.
This is used later to suspend the server startup process until the init message is received.

4. JsonRPCLogger

The fact that c-lightning and the plugin communicate via standard input/output means that logs cannot be sent to standard output at will.

The existing c-lightning plugin sends a message to the c-lightning's log rpc method as an rpc notification and leaves the rest of the processing to the c-lightning side.
In this way, the logs are unified on the c-lightning side, making them easier to read.

.NET is bundled with ConsoleLogger, but this performs the writing process as a background task, so the aforementioned thread race may occur if we implement the logging logic in the same way.

To solve these problem, I have created my own ILoggerProvider.

5. startup process

c-lightning starts the plugin specified by the user at startup, passing the environment variable LIGHTNINGD_PLUGIN=1 when launching.

Thus, the presence or absence of this environment variable can be used to determine in the program whether the application should be run as a plugin or not.

In my case, I refer to this variable in two places
The first is where you configure the host.

Action<IHostBuilder> configureHostBuilder = hostBuilder =>
{
  var isPluginMode = Environment.GetEnvironmentVariable("LIGHTNINGD_PLUGIN") == "1";

  if (isPluginMode)
  {
    hostBuilder
      .ConfigureLogging(builder => { builder.AddJsonRpcNotificationLogger();})
      .ConfigureServices(serviceCollection =>
        serviceCollection
          .AddSingleton<MyAwesomeRpcServerOptions>()
          .AddSingleton<MyAwesomeRpcServer>()
      );
  }
  else
  {
     // configuration for running as a normal webserver.
  }
};

The other is during Host execution.
We delay the execution of IHost.RunAsync() as follows.

  var host = hostBuilder.Build();
  var isPluginMode = Environment.GetEnvironmentVariable("LIGHTNINGD_PLUGIN") == "1";
  if (isPluginMode)
    await host.StartJsonRpcServerForInitAsync(); /// initialize only rpc server and listen to `init` call.
  await host.RunAsync();

This will delay the startup of BackgroundService and log output until the init message is received from lightningd.

6. compile as a single binary

Finally, the plugin must be compiled as an executable single binary.
Therefore, we utilize the Single file Executable Compilation feature of .NET

And that's it!
You can specify the --plugin or --plugin-dir of c-lightning to launch the plugin.

publish as a library?

At first, I thought about converting what I have done this time into a library and releasing it to the public, but since it is not that hard to do it, and the init process is difficult to automate, I will just leave the document and examples here.

If you want to make it into a library, probably the best way is to use the Source Generator

Prototyping generic Bitcoin Smart contract

Joe Miyamoto — Fri, 11 Dec 2020 11:11:46 +0000

While ago, I’ve been building F# library for the Bitcoin Lightning Network (LN), DotNetLightning.
After (barely) finishing my first iteration, I’ve noticed that there is a more matured library called rust-lightning for doing the same for the rustlang, and I’ve switched my efforts to its .NET binding with C#
After working for both F# and C# for a decent amount of time, I noticed that it both has goodies and baddies for different domains.
One particular domain I think that F# is a clear winner is for prototyping a new idea.

So today, I will sketch out my recent idea about Bitcoin and how I use F# to organize my thoughts in my head.

But before that, I have to describe the overview about bitcoin smart contract and my thoughts about it.

What is a Smart contract?

1. Smart contract as a scripting language

The word “Smart contract” is mostly used to refer to a byte code which is executed on the blockchain as a part of the consensus mechanism, and sometimes including a programming environment to write that byte code.
One example of this is Solidity on Ethereum which compiles to the EVM byte code.

The bitcoin has a similar construction (bitcoin script) which is intentionally made less descriptive than the EVM byte code. and there is a proposal for a high level language which compiles down to the actual bitcoin script, namely Miniscript and its predecessor ivy-bitcoin BitML

However, at the same time at least in the Bitcoin community there is a consensus that writing a secure byte code is just a very little part of building the secure multi-party protocol.

2. Smart contract as a multi-party procedure

Recently, the trend is to bring the statefulness to outside of the blockchain, and use the blockchain as a “dumb” settlement layer. Which makes sense in terms of privacy and scalability.

In other words, the "smartness" of the contract is not in the blockchain but in the procedure among related party.

To note a few keywords for such trend, there is a Scriptless Scripts, Lightning Network (LN) , Discreet Log Contract DLC, etc.

All these contracts are defined as a set of procedure among multi-party (in most case, two-party)
For example something like

Alice and Bob shares specific transaction and some of those signatures
Both Verify signature and the shape of the transaction, if it is Ok Alice broadcast transaction XXX
If Bob sees the transaction on the blockchain before timeout, he sends an Adapter Signature with the same key with XXX and Alice verifies it and… otherwise ...

Such a way of defining the contract is secure and fine. But can we achieve more generality somehow? If we could make some kind of universal language to describe the “shape” of the contract, it would be a good mental model for protocol developers to build/communicate new Smart contracts.

3. Smart contract as a Transaction graph

At the same time, there is an old proposal by Jeremy Rubin called TXGL , which defines not only the script for the contract but also the shape of whole transactions which could be involved in the contract. This is great as a mental model for the developers, but in reality it is not enough. Because, some transactions are related with each other outside of the shape (e.g. by Adaptor Signature) and those relations will affect how we communicate with each other in course of the contract.

Objective

So, my hypothetical objective is

Could we make a declarative Universal language (or DSL) to describe the “shape” of the contract? and
Could we automatically compile that into an actual procedure describing what each party should do while executing the contract?

This is a very bold claim to make. And it might be impossible or overkill. But I have to try to make a PoC to see if it is really so.

The idea is affected by a proposal named Improv, which achieves something similar with the Miniscript but for a payment on the Lightning Network.
It is supposed to look similar to the Miniscript but compiles down to a multi-party-procedure instead of the byte code.

So, let’s first survey existing proposed contracts, and try to abstract properties of those.

Note for readers those familiar to the bitcoin protocol: I assume we are able to use Taproot and PTLC

types of on-chain contracts

It seems that most, if not every, on-chain contract has roughly three phases

Setup: exchange transaction graph and some of its signatures, adaptor signatures
Start: broadcast the root ancestor of those transactions (often called funding tx)
Act: react to events which happen on the blockchain or through communication with another party.

My purpose here is to define the language to describe 1 and 2, and deterministically generate 3 from it.
So, hopefully I can describe a contract with a set of transaction graphs. and the relation of the transactions in each graph.

With that in mind, let’s make a graphical model for Atomic swap with an adaptor signature. I won’t go into the details about the protocol.. The model will be something like

Similarly, the model for SAS: Succinct Atomic Swap will be

Modeling with FSharp

Then let’s model these relations using F# type. We are going to use NBitcoin to represent Bitcoin specific classes, and my own DotNetLightning off-chain specific types.

Let’s say we call each node in the above graph TxEntry,

open NBitcoin;
open DotNetLightning;

/// The object for describing the “Lock condition” of the on-chain payment.
/// It is not ready since the Miniscript is still in a research phase.
type Miniscript = NotYetReady
  with
  static member Parse(str: string): Miniscript =
    failwith “TODO”

type OnChainLock = Miniscript * Money

type EntryRelation =
  | AtomicallySetup
  | AtomicallyExecuted

type EntryId = int

// each node in the graph.
type TxEntry = {
  Id: EntryId
  Prev: TxEntry option
  Lock: OnChainLock
  Next: EntryId seq option
  RelatedEntry: (EntryId * EntryRelation) seq
}

type Entry = OnChain of TxEntry

The Contract is a set of graph of the TxEntry, so let’s define it as

type TxTree = TxEntry seq
type Contract = TxTree list

types of off-chain contracts

Bitcoin has two ways of payments, one is on-chain payment which must be recorded on the blockchain, and another is off-chain payment which is accomplished by the Lightning Network.

The mechanism of off-chain payment is quite different from those of on-chain, but some operations are analogous. (e.g. “we must commit to payment atomically to prevent option problem” or “Iff one payment is executed, another payment must be able to execute”).

So we can use a similar terminology to describe the contract.
See below table for how the “relation” of each entry is achieved in both on-chain and off-chain.

So for example, Submarine swap in the graphical representation could be.

Modeling in FSharp

So next let’s include the concept of an “off-chain” entry into the above code. It will be like

open NBitcoin;
open DotNetLightning;


/// The object for describing the “Lock condition” of the on-chain payment.
/// It is not ready since the Miniscript is still in a research phase.
type Miniscript = NotYetReady
  with
  static member Parse(str: string): Miniscript =
    failwith “TODO”

/// Same for off-chain Lock condition. Which also is not ready yet.
type Improv = NotYetReady

type OnChainLock = Miniscript * Money
type OffChainLock = Improv * Money

type EntryRelation =
  | AtomicallySetup
  | AtomicallyExecuted

type EntryId = int

// each node in the graph.
type TxEntry = {
  Id: EntryId
  Prev: TxEntry option
  Lock: OnChainLock
  Next: EntryId seq option
  RelatedEntry: (EntryId * EntryRelation) seq
}
type L2PaymentEntry = {
  Id: EntryId
  Lock: OffChainLock
  RelatedEntry: (EntryId * EntryRelation) seq 
}

type Entry =
  | OnChain of TxEntry
  | OffChain of L2PaymentEntry

We also update Contract type to include both on-chain and off-chain concept

type TxTree = TxEntry seq
type PaymentTree = L2PaymentEntry seq
type Contract = TxTree list * PaymentTree list

Finally, let’s define an actual Contract object for Submarine Swap.

let submarine_swap (alice_key: Key) (bob_key: Key) (amount_to_bob: Money) (amount_to_alice: LNMoney): Contract =
    let funding_tx_entry = {
        Id = 0
        Prev = None
        Lock =  Miniscript.Parse($“or(and( pkh({alice_key}), pkh({bob_key}) ), and(after(144), pkh(alice_key)))”), amount_to_bob
        Next = [1] |> Seq.ofList |> Some
        RelatedEntry = [(2, AtomicallySetup)]
    }
    let swap_tx_entry = {
        Id = 1
        Prev = Some funding_tx_entry
        Lock = (Miniscript.Parse($"pkh({bob_key})"), amount_to_bob)
        Next = None
        RelatedEntry = [(2, AtomicallyExecuted);]
    }

    let off_chain_payment_entry =
        let lock_condition = bob_key.PubKey
        {
            Id = 2
            Lock = lock_condition, amount_to_alice
            RelatedEntry = [(1, AtomicallyExecuted)]
        }
    [ (seq { funding_tx_entry; swap_tx_entry }) ], [ seq { off_chain_payment_entry } ]

Conclusion

This is a very rough sketch of my recent thoughts, and it is still in an early phase.

The next step will be to define the function to compile the Contract object to an actual state machine to handle the procedure of an interaction with the another party (and the blockchain), that is, function with the type

type ContractToProcedure = Contract -> Async<unit>

That is the most difficult part, and I will probably change the definition of the Contract on the way.

What I want to point out here is that in the research phase like this, it is important for me to define an algebraic data type in a succinct way so that I can alter the definition while I go back and forward with an implementation.

In other words, F# is very nice tool for Exploratory programming