DEV Community: Martin Becker

API and Provider methodology

Martin Becker — Tue, 29 May 2018 11:55:40 +0000

Introduction

So, One of the biggest issues when designing a library/framework Is when you need to make a call to an external dependency.

Specifically in the context of this post, when that dependency is no longer usable for any reason, what do you do? Most would then need to implement a new interface for a new dependency, and then as a result potentially make breaking changes to your API to accommodate which isn't good.

I bring this up because I see all to often libraries that directly call the external dependency in there API methods (I'm guilty of this too -_-), then have to completly change there API when the dependency changes that requires data to be structured differently.

This then breaks things for many people.

This is why I advocate separating the API and the external calls into separate classes, it does add complexity and increases the amount of code but the advantage is that when you're external dependency changes, your API doesn't, and nor does it for anyone using you're libraries.

When I think to do this (ahem) I like to separate the logic into:

API: The API of your library
Provider: The logic that calls your an external dependency
Formatter: If needed, the logic that handles data formatting between the API and the Provider

Basic Provider Logic

Now you can do this any way you like, so long as there is a clear separation of concerns. Using ruby as an example, I like to take advantage of the fact you can define def self.[]() to make my provider API easy to use so I can do somthing like this:

Module Providers
    class << self
        attr_writer :listing
        def [](provider)
            @listing[provider]       
        end
    end
    class Provider_Base
        def self.inherited(child, name = child.class.to_s.downcase.to_sym)
            # Normally I'd add logic to ensure that only this method can add to
            # `Providers.listing`, but I can't be arsed for this example
            Providers.listing[name] = child
        end
    end
end

The reason I do name = child.class.to_s.downcase.to_sym in the function declaration is so that if you want you can easily change the way the name is set by doing:

Module Providers
    class OtherProviderBase < Provider_Base
        def self.inherited(child)
            super(child, :some_other_naming_mechaism)
        end
    end
end

Now you can inherit from OtherProviderBase instead.

Using Your provider

Using the providers in you're api becomes super easy now:

class API

    PROVIDER_NAME = :provider_name

    def some_method
        Providers[PROVIDER_NAME].some_method
    end

end

Why do it like this? Lets say you build a new dependency provider, you can just change the provider to this:

class API

    PROVIDER_NAME = :other_provider_name

    def some_method
        Providers[PROVIDER_NAME].some_method
    end

end

End result is that the API doesn't change and won't break for you of your users.

And using the above providers example, you can add formatters like this:

class API
    PROVIDER_NAME = :provider_name

    def some_method
        Formatters[PROVIDER_NAME].some_method(
            Providers[PROVIDER_NAME].some_method)
    end

end

You can even go further and do this:

class API

    PROVIDER_NAME = :provider_name

    def some_method(*args)
        method = :some_method
        output = Providers[PROVIDER_NAME].send(method, (Formatters::Input[PROVIDER_NAME].send(method,*args))
        Formatters::Output[PROVIDER_NAME].send(method, output)
    end

end

Provider Piplines

You could even provide a mechanism (where appropriate of course) to automatically pipeline, like this maybe:

module Pipelines
    attr_reader :listing

    def pipline_for(action, *pipes)
        @listing[action] = pipes
        define_method(action, do) |*args|
            run_pipline_for(action, *args)
        end
    end

    def self.extended(child)
        child.include Instance_Methods
    end

    module Instance_Methods
       def run_pipline_for(action, *args)
           Pipelines.listing[action].inject(args) do |result, (item, custom_action)|
              item[PROVIDER_NAME].send(custom_action || action, *result)
           end
       end
    end
end


class API
    extend Pipeline

    PROVIDER_NAME = :provider_name

    pipline_for :action, Formatters::Input, [Providers, :get_remote_data], Formatters::Output
    pipline_for :other_action, Formatters::Input, Providers, [Processors, :differing_name], [Formatters::Output, :custom_output]

end

Sorry, this probably out of scope for this post but couldn't help myself when I saw somthing interesting to code. Also I'm not sure if PROVIDER_NAME will be picked up in the local context of the API class when Pipeline is included so you might need to code a mechanism for setting and retrieval that does but you get the idea.

conclusion

By separating your external dependency into it's logic that is then called by your API, you can then keep your API constant between versions and then the only difference between versions becomes the dependencies, without any breaking changes between them.

p.s. I completly admit to not having checked or tested any of this code as it was just examples to get the ball rolling.

Explain RPC like I'm Five

Martin Becker — Thu, 17 May 2018 09:14:43 +0000

I'm having a little trouble understanding RPC, especially compared to say Rest API's or using protobufs to send and receive data, so please explain RPC like I'm Five?

Building My own website

Martin Becker — Wed, 16 May 2018 21:39:54 +0000

Introduction

So as far as I've seen most developers have there own website but for me, sadly I'm yet to do this so I've decided It's time to embark on the time honoured tradition[1] of a dev building there own personal site!

But I thought I'd make it interesting by over-engineering it on purpose.

My reasoning is several:

I can put to practice the knowledge I've accumulated on systems architecture
I can build using interesting technology I've been wanting to try
I can learn how to use new things
Fun!

So, I've decided to plan out what I'm thinking of doing here, any discussion or ideas would be and are welcome!

This document is also to act as a kind of repository of useful links with context that are related to the things I've chosen.

Checklists

Selections

Basic Architecture
- [ ] Message Queue: Kafka|RabbitMQ|AWS SQS
- [√] Message format: Message Pack
- [ ] Routing: Nuster|Nginx
- [√] Deployment: Docker & Chef?
- [√] CI: Travis

Front end
- [√] View framework: Marko
- [√] State: Vuex
- [√] Visual: Materializecss

Back end
- [√] Ruby: Rack + Custom Middleware
- [√] Rust: tokio-serde-msgpack
- [√] API: Roda

Features I may add

[ ] dev.to post feed

Services

Front end (I'll Add to this as I think of more services)
- [ ] Front End app distro
- [ ] Routing/caching (Nuster/Nginx)
- [ ] Front end App distro-service (maybe Node but what ever works best)

Front Facing Back-end (I'll Add to this as I think of more services)
- API (I'll add to this as I add API's)

Rear Facing Back-end (I'll Add to this as I think of more services)
- [ ] Message Queue (maybe, maybe not)

Usefull links

Docker
- Kubernetes on Macos
- Getting started with micro-services and Kubernetes

The Plan

My idea for my plan is to list things that I think of and what I can do for that item the idea is to fill the list, some of them I'm not sure about I'll put two or more items, I would appreciate some input on them!

I don't know if I'll ever get around to completly building this, but at least if I have a plan I'll have something to work towards.

Lastly, I might put somthing in the plan but this is also for fun so If somthing becomes to annoying or beyond the scope of fun I'll probably not do it.

Basic Architecture:

Host

I've decided to go with AWS, whilst it might be fairly close to the metal it will allow me to get to the nitty gritty

Code

I've decided to go with ruby and rust on the backend, since I'm building with micro-services I can build small services in which ever language I like, or build in ruby first to get somthing running then migrate to rust or hybridise and build with ruby and use FFI for rust.

Infastructure

I decided to go with micro-services becuase it's always somthing I wanted to try doing to that end I need to some infrastructure to go with it.

if I decided I can't be arsed with this I'll just go with a standard monolith and then add micro-services on the side or start as a monolith and then migrate to to micro-services instead.

Message-queue
I was thinking about going with kafka or RabbitMQ but In thinking about it, I've decided to go with SQS. As much fun and as interesting as it would be to set up, I really don't need somthing with that much throughput as Kafka or as heavy as RabbitMQ and SQS is supported by Rust, Ruby and Javascript.
Message-format
I've decided to go with message-pack, somthing I've had my eye on and should interesting to implement and use across Ruby, Rust and JavaScript(front end).
Routing
I've decided to go with trying Nuster as it looks interesting and I've never used it before. But if I can't get it to work as I want I'll go with Nginx as I've used it in the past
Deployment
I've decided to go with Docker as I've never used it before so It will be interesting to implement, I'm not sure if I'll need somthing like Chef for deployments though or if I should combine them as i'm a little confused if I need both of them or not.
CI
I've decided to go with Travis, again becuase I've never used it and it's free.

I think that's about it for Infrastructure beyond using S3 for assets and stuff, Postgresql (or amazon aura) as my DB

Website structure:

Front end

View framework
I've decided to go with Marko becuase Marko's Syntax is just so nice and is highly performant and honestly Marko just looks more interesting.
State
Vuex, no competition, it just looks easier to use then redux (and honestly, mutations is a lot easier to understand then reducers in terms of understanding it & context)
Visual
Materializecss, honestly this is my second choice, my first is fluent design but the only thing I can find is a React component framework which whilst nice looking (I would say sexy) Is still React when I want to use Vue or Marko. Materializecss is still nice and if I wanted to I could do some acrylic effects to add some to it. As framework however I do like it, way more then bootstrap.

Back end

Ruby
Rack + Custom Rack Middleware: In most cases be using Rack to mount apps with some custom built middleware. Since I intend to Build this as a front end JS app with an API backend, if someone knows how to distribute the base HTML + assets without needing somthing like Sinatra then please point in the direction!
Rust
tokio-serde-msgpack
API
I've decided to use Roda for API stuff as it looks interesting, I'll then use a messaging library to pass messages to the message queue and then any MS will then pick up the appropriate messages, send a response and then the API service will respond. To be honest though I'm not sure, as I've said I'm doing this as learning experience as my knowledge is currently almost zero, once I get a better understanding I'll change this section.

Features

dev.to post feed
Instead of building a blog I'd much rather just post dev.to and have links to my posts appear on my site. To that end I can use the dev.to articles api dev.to/api/articles?username=thermatix, there is an RSS feed but I'd much rather just use the API.

For now This is all I have, I'll add more sections as I think of them or decide they should be added, once my plan is complete I'll create a list of things I'm going to build as a separate document, publish it and then update it with any appropriate links and info.

Authors Notes

[1]: building a dev site is probably not a time-honoured tradition but I thought it fit.

Switched to Alacritty

Martin Becker — Tue, 01 May 2018 15:16:43 +0000

I just switched to using Alacritty (as in about 40 mins ago, give or take). It's a fair bit more bare bones then the ITerm2 I was using but to be completly honest I never used any of the extra features provided by it so I'm not missing anything.

It's a lot faster than Iterm2 by a lot I'm also finding that Nvim has sped up a lot and by far the best feature is that it's solved one the biggest bugbears I've ever had about Iterm2 (and Apples Term). I use Powerline fonts but one the biggest issues is that ITerm2 never rendered them properly as a result the tips of the powerline characters stuck up and as a result made everything just a little bit ugly. That problem is non-existent on Alacritty (at least for me)

AS for negatives there are a few. To use it as an .app you have to clone the repo and then build it, not a big issue but it took a while.

I did have issues getting Alacritty to find the font I wanted. I told it to look for a font called Knack Regular Nerd Font Complete but it kept saying it didn't exist. I posted an issue and someone said they tried Knack Nerd Font so I tried that and it worked.

Finally there is a bug I get with tmux, where the last character of the status line is moved to the beginning of the next line. For some people it would be a deal breaker but for me I can live with it as it appears next to buffer tab line (airline) on neovim so I don't notice it plus I can live with that over misaligned powerline characters so long as I set the last line of the tmux status line to |. I've added an issue on Github so hopefully it gets fixed soon.

Any way, I'm really liking it and can't wait for it to get even better.

Building My first Rust Project: Wercker Build Status

Martin Becker — Wed, 04 Apr 2018 17:39:37 +0000

Introduction

warning This is my first ever techy blog post so I hope you'll forgive any errors or badly written sections.

My main language of choice is ruby, I like it's ease of use, it's simplicity and expressiveness and it's malability makes it a dream to use, It is however Slow, lacks clear typing and even subtle bugs can crop up if you aren't aware of the fact that ruby is both pass by value and reference as you can see here:

a = 'hello '
b = a
b << 'world'
puts a
=> 'hello world'

Whilst all of it's pro's make rapidly building something easy it can make it difficult to maintain over long term.

Enter Rust!

As far as I'm concerned Rust is the perfect partner to ruby, it's fast where ruby is slow, it's typed where ruby isn't and you never need to worry about undefined behavior or subtle bugs due to its nature.

To that end I decided on learning rust to act as a foil to ruby. Learning rust has been hard but worth it I feel.

The tool - Werker Build Status

This tool exists because at my job we use Oracle's Wercker (I keep calling it wrecker) to handle Continues Intergration but I have a tendency not to pay attention to when a build is failing so I thought I'd build a tool to display the status of my last build in the Tmux status line.

Before I started working on this tool I'd always felt lost when working with the language, I can safely say that this is the first time I'd worked with rust and felt like I knew what I was doing (more or less).

Originally I was going to build a bash script around the Wercker CLI tool, but then it occurred to me that it's not a large project and its scope is small, wouldn't that be perfect for my first project with Rust? So I did, and I started with planning.

Planning

I realised that my tool could be broken down into steps that I could then use to work out what crates I might use to make to make it easier (something I admit I don't normally do).

1) Loading user config - config
2) Getting the list of runs - curl
3) Deserializing the json - serde_json

I admit that normally I'd just jump into ruby and start hammering code out however due to the nature of rust and the fact I don't know it that well I thought it would be a good idea to plan it out and it ended up working in my favour.

Loading the User Config

I honestly found this to be the easiest of all the steps as the documentation was easy to follow, for me I simply added a single function to take care of loading the config:

use config_rs;

use std::collections::HashMap;
type Config = HashMap<String, String>;

fn load(config_file: String) -> Config {
    // create config object
    let mut settings = config_rs::Config::default();
    // check if config_file was supplied, if it was then load it    
    if config_file != "" {
        settings.merge(config_rs::File::with_name(config_file.as_str())).unwrap();
    }
    //check for and add env variables
    settings.merge(config_rs::Environment::with_prefix(::CONFIG_PREFIX)).unwrap();
    settings.try_into::<Config>().unwrap()
}

Initially I was checking for each key I needed with an if statement to check if a key existed but in the end I found that this to be the dry-ist way to do it but still remaining flexible:


// check if config options exists
let keys: [&str;3] = ["token", "author", "pipeline_id"];

match keys.iter().position( |key| {
    !settings.contains_key(&key.to_string())
}) {
    Some(i) => println!( "No `{}` detected in config or env ({}_{}) variables",
            keys[i], CONFIG_PREFIX, keys[i].to_uppercase()),
    None    => {
}

Getting the list of runs

This was were things started getting tricky, Setting up the Curl client was fairly straightforward:

use curl::easy::Easy;

mod urls;

pub fn set_up(token: &String) -> Easy {

    use curl::easy::List;

    let mut easy = Easy::new();
    let mut list = List::new();

    // add authorisation header to curl client
    list.append(&format!("Authorization: Bearer {}",token)).unwrap();
    easy.http_headers(list).unwrap();
    easy
}

I created a function to handle doing get requests:

fn get(curl: &mut Easy, url: String) -> String {
    use std::str;

    let mut data = Vec::new();

    curl.url(url.as_str()).unwrap();
    // add mechanism to transfer data recived data into a vec
    {
        let mut transfer = curl.transfer();
        transfer.write_function(|new_data| {
            data.extend_from_slice(new_data);
            Ok(new_data.len())
        }).unwrap();
        transfer.perform().unwrap();
    }
    // Return Data!
    data
}

However this is where I encountered my first hiccup, where I was expecting a JSON string I was getting a Vec[u8] it puzzled me for a bit but then I realised that curl was outputting raw bytes, a bit of digging and the answer was to change the last line to this:

// convert byte result into utf_8 string
str::from_utf8(&data).unwrap().to_string()

Now I was getting a JSON string!

Deserializing the json

At first I was content to just filter the build runs by author and then grab the first deserialized object:

let mut client = set_up_client(&settings["token"]);
let runs: Value = serde_json::from_str(get_runs(&mut client,&settings["author"],
       &settings["pipeline_id"]).as_str()).unwrap();
println!("{}",runs[FIRST]);

However due to what appears to be a bug on the Wercker API this wasn't possible. the bug was that no matter what I set as the author (a username, random letters) it would always send back the last builds of the same incorrect user.

This means means I have to instead grab the last 20 runs and manually find the first with a matching username, to do that I had to create structs to represent the data I wanted:

pub type Runs = Vec<Run>;

// only define the structs for the data I want to pull out
#[derive(Debug,Serialize, Deserialize)]
pub struct Run {
    pub status: String,
    pub result: String,
    pub user: User,
}

#[derive(Debug,Serialize, Deserialize)]
pub struct User {
    pub meta: Meta,
}

#[derive(Debug,Serialize, Deserialize)]
pub struct Meta {
    pub username: String
}

and then deserialize to my struct:

let runs: Runs = serde_json::from_str(client::get_runs(&mut client,&settings["pipeline_id"]).as_str()).unwrap();

I honestly didn't want to have to do this as the basic deserialisation was fine for my needs as I wasn't intending to store the results however Serde wouldn't co-operate quite so nicely, I tried doing this:

runs.as_array().iter().find()

However I found that serde only returns a Vec[String] with a single element containing the entire JSON string, so manually creating the deserialized data was the only way.

Another issue I found was that following the serde documentation gave me a puzzling result:

let runs: Value = serde_json::from_str(client::get_runs(&mut client,&settings["pipeline_id"]).as_str())?;

It didn't like me using ? operator, it turns out I'd completely missed the fact you can't use it nor try! in main because they both return and you can't return in main, changing ? to unwrap() solved the issue.

Displaying the result

Lastly I iterate through the Vec[Runs] and display the first to match the username:

match runs.iter().find( |run| {
    &run.user.meta.username.to_lowercase() == &settings["author"].to_lowercase()
}) {
    // print out status and result
    Some(run) => {
        if settings.contains_key("tmux") && &settings["tmux"] == &"true".to_string() {

            match run.result.as_ref() {
                "failed"    => println!("##[fg=red,bold]{}",&run.result),
                "passed"    => println!("#[fg=green,bold,bright]{}",&run.result),
                "aborted"    => println!("#[fg=yellow,bold,bright]{}",&run.result),
                _           => println!("#[fg=blue,bright]{}",&run.status)
            }
        } else {
            match run.result.as_ref() {
                "failed"    => println!("{}",&run.result.red()),
                "passed"    =>  println!("{}",&run.result.green()),
                "aborted"    =>  println!("{}",&run.result.yellow()),
                _           =>  println!("{}",&run.status.blue(),)
            }
        }
    },
    None => print!("{}","None Found")

}

conclusion

Learning Rust was worth it and this tool I built though basic shows me that I can build things with this language. A someone that has a tendency to forget things and not notice when things are slightly off, rust helps a lot in this regard as it just won't compile when things aren't right so it's nice that if it compile it will pretty much work (barring any bugs of course).

There are a few things I did not cover in this as the point isn't to show the program in its entirety, if you want to see it all you can look at the repo the tool is stored at which can be found here.

I'm honestly glad I took the time to learn rust and I feel it will be of help in my future endeavours.

Thank you for taking the time to read this retrospective I hope it was in any way informative to you.

Devs, why do you use terminal tabs over tmux windows?

Martin Becker — Tue, 06 Mar 2018 11:46:58 +0000

Just a quick thought, I've noticed a lot of developers seem to use terminal tabs, even when using tmux which has windows, so I'm curious, why use terminal tabs over tmux windows?

One thing to point out is that when you use tabs, it steals a bit of screen real-estate which can be used to display more of the terminal. By using tmux windows it displays the windows open in the tmux status line instead of a seperate line.

Hi, I'm Martin Becker

Martin Becker — Mon, 17 Jul 2017 20:24:15 +0000

I have been coding on and off for 4 years.

You can find me on GitHub as Thermatix

I live in Farnborough (just moved here last friday(14th of july, 2017).

I work for formulate digital.

I mostly program in these languages: Ruby, javascript.

I am currently learning more about Rust.

Nice to meet you.

DEV Community: Martin Becker

API and Provider methodology

Introduction

Basic Provider Logic

Using Your provider

Provider Piplines

conclusion

Explain RPC like I'm Five

Building My own website

Introduction

Checklists

Selections

Features I may add

Services

Usefull links

The Plan

Basic Architecture:

Host

Code

Infastructure

Message-queue

Message-format

Routing

Deployment

CI

Website structure:

Front end

View framework

State

Visual

Back end

Ruby

Rust

API

Features

dev.to post feed

Authors Notes

Switched to Alacritty

Building My first Rust Project: Wercker Build Status

Introduction

The tool - Werker Build Status

Planning

Loading the User Config

Getting the list of runs

Deserializing the json

Displaying the result

conclusion

Devs, why do you use terminal tabs over tmux windows?

Hi, I'm Martin Becker