TLDR;
Use curlconverter and transform tools to turn a public API into a software library; this example covers how to do it using rust with mempool.space.
This article will go over how to take the mempool.space API and turn it into some rust code. What is mempool.space? It's a website that implements an API to access bitcoin's blockchain data. The API implements an "esplora" style backend which helps to standardize bitcoin's blockchain access which helps to decentralize the network since app devs can only implement this access style and have a myriad of choices for blockchain provider.
Mempool.space is widely accepted by many in the bitcoin community, mainly because it shows all the most essential information about the bitcoin mempool at-a-glance through a stunning interface. In many ways, this is an excellent API because it turns the low-level ugliness of the bitcoin blockchain and mempool through the command line into a high-level interface with beautiful graphics. This is useful for anyone that wishes to interact with the bitcoin protocol at a low cost.
This tutorial will apply to any other public API, so read on if you have different needs!
My primary motivation is that I have a bitcoin wallet to which I want to add fee recommendations.
Rust is the most appropriate tool for this task because;
- The http crate, hyper, is an optional backend for cURL. This speaks highly of its quality.
- Type safety and the SerDe crate; anytime you reach out to a public API you should be very paranoid and verify everything. - Enforcing invariants is easy with rust. SerDe makes "Don't trust, verify" convenient to implement.
- The cargo tool is great
- C calling convention, so FFI with typescript and python is theoretically possible
If you don't know what the bitcoin mempool is, then read the mempool.space FAQ or ask your local bitcoiner. It is a fascinating aspect of the bitcoin protocol (replete with its own controversy), its especially interesting to those who are curious about distributed systems!
A mempool (short for "memory pool") is the queue of pending and unconfirmed transactions for a cryptocurrency network node. There is no one global mempool: every node on the network maintains its own mempool, so different nodes may hold different transactions in their mempools.
For this to work, you need to know how to invoke the API and what the responses are. Without docs or access to the API, this workflow will not be effective.
For brevity, I will showcase the example for three API groupings, fees, general and mempool.
Later on, I would like to make a series about this, including;
- Adding async
- Using a builder pattern to configure the API crate
- Feature guards (for different parts of the API)
- Rust's FFI for typescript and python
- Anything else you think would be useful or interesting - please let me know below, in the gh repo, or directly via email or some other means.
We will also stick to testnet for now, this will be a very simple crate with no config options. We will add network configuration later.
So if we want the difficulty adjustment, the docs tell us to run;
$ curl -sSL "https://mempool.space/testnet/api/v1/difficulty-adjustment"
So if we run that, we would get;
$ curl -sSL "https://mempool.space/testnet/api/v1/difficulty-adjustment"
{"progressPercent":83.53174603174604,"difficultyChange":7.02698236660555,"estimatedRetargetDate":1668895091192,"remainingBlocks":332,"remainingTime":186121192,"previousRetarget":247.24200134108503,"nextRetargetHeight":2407104,"timeAvg":560606,"timeOffset":0}%
which is not so great for reading, lets run that through jq;
$ curl -sSL "https://mempool.space/testnet/api/v1/difficulty-adjustment" | jq
{
"progressPercent": 83.58134920634922,
"difficultyChange": 7.082824486908046,
"estimatedRetargetDate": 1668894501603,
"remainingBlocks": 331,
"remainingTime": 185463603,
"previousRetarget": 247.24200134108503,
"nextRetargetHeight": 2407104,
"timeAvg": 560313,
"timeOffset": 0
}
So we want to emulate this functionality in rust code, how would be do that?
Well we need to make a function that takes that API endpoint and calls it, then takes the response and deserialzes whatever comes in through the wire. Curl does these things for us, but we can't easily call curl from our programs (well you can almost always have your app run a subprocess that invokes curl and reads from a file and manually parses it but that is less than ideal for a variety of reasons, namely security and sanity).
Ok! so we need a function. How do we turn curl into a rust function?
We can use curlconverter.com
Here is an example for example.com
extern crate reqwest;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = reqwest::blocking::Client::new();
let res = client.get("https://mempool.space/testnet/api/v1/difficulty-adjustment").send()?
.text()?;
println!("{}", res);
Ok(())
}
I've converted that code into a replit
Ok so we have the function, but lets go over how to turn the json response into a rust struct.
{ progressPercent: 44.397234501112074, difficultyChange: 98.45932018381687, estimatedRetargetDate: 1627762478, remainingBlocks: 1121, remainingTime: 665977, previousRetarget: -4.807005268478962, nextRetargetHeight: 741888, timeAvg: 302328, timeOffset: 0}
To turn this into a rust struct, we can use this
transformer.tools
![[Pasted image 20221117130709.png]]
Here is the basic example, hopefully this is also clear where I am going with this.
You have to paste in the reply from the GET request, on my system I use pbcopy like so
$ curl "https://mempool.space/testnet/api/v1/difficulty-adjustment" | jq | pbcopy
You must verify that the types are properly transformed, dont trust the tool. This looks correct to me, to start with. You can always go back and swap in more semantically appropriate types to the library, primitive types will do for now. Since these are only numbers, its probably fine as is, but any cryptographic primitives shoud be verified by using the appropriate types. You could also add the unix timestamp time.
use reqwest;
use serde_derive::Deserialize;
use serde_derive::Serialize;
#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct DifficultyAdjustment {
pub progress_percent: f64,
pub difficulty_change: f64,
pub estimated_retarget_date: i64,
pub remaining_blocks: i64,
pub remaining_time: i64,
pub previous_retarget: f64,
pub next_retarget_height: i64,
pub time_avg: i64,
pub time_offset: i64,
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = reqwest::blocking::Client::new();
let res = client.get("https://mempool.space/testnet/api/v1/difficulty-adjustment")
.send()?
.text()?;
println!("{}", res);
Ok(())
}
Don't forget to add dependencies, cargo has an extension that makes this easy;
$ cargo add serde serde_derive serde_json reqwest -F blocking
By using the json serde function we can get the crate to automatically generate all the json text parsers, rather than doing it by hand. If anything fails to parse, an error bubbles up and since its not handled, our program panics.
use reqwest;
use serde_derive::Deserialize;
use serde_derive::Serialize;
#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct DifficultyAdjustment {
pub progress_percent: f64,
pub difficulty_change: f64,
pub estimated_retarget_date: i64,
pub remaining_blocks: i64,
pub remaining_time: i64,
pub previous_retarget: f64,
pub next_retarget_height: i64,
pub time_avg: i64,
pub time_offset: i64,
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = reqwest::blocking::Client::new();
let res = client
.get("https://mempool.space/testnet/api/v1/difficulty-adjustment")
.send()?
.text()?;
println!("text: {}", res);
let val = serde_json::from_str::<DifficultyAdjustment>(&res).expect("could not parse json");
println!("whole struct: {:?}", val);
println!("just the difficulty change: {:?}", val.difficulty_change);
Ok(())
}
Thats pretty much it for basic functionality, in the next posts, I will turn this into a library. Stay tuned!
Top comments (0)