DEV Community: Stephen Solka

Astro Command unlocks the world

Stephen Solka — Sun, 19 Sep 2021 19:04:43 +0000

Introduction

This weekend I built a new Astro component that lets callers use external commands on their machine to generate html content.

trashhalo / astro-command

run commands as astro components

In the repository there is a hello world demo that is shelling out to python.

---
import { Command } from "astro-command";
---
<Command caller={import.meta.url} command="./Component.py" message="from python!" />

#!/usr/bin/env python 

import sys, json

data = json.load(sys.stdin)
print(f'<h1> Hello {data["message"]} </h1>')

Props are passed in as a json blob to stdin. Html comes out stdout.

Taking it up a notch

Python has a very deep well of libraries that can be useful for astro page generation. One example is https://matplotlib.org/ a plotting library that can generate beautiful graphs outputting SVG.

As a quick example I rewrote the hello world py component to output a graph instead.

Its not just python

Now that you have shell access from Astro you can use any command! The first one I created a custom component for is Pandoc to get access to all the types of content Pandoc can understand. With 12 lines of astro code I unlocked 30+ file formats!

trashhalo / astro-pandoc

astro component that lets you use pandoc

Astro Pandoc

Astro component for using pandoc to convert content. This allows you to embed any format pandoc supports.

Supported formats https://pandoc.org/MANUAL.html#general-options
Demo
Requires you to have pandoc installed on your machine!

Usage

---
import { Pandoc } from "astro-pandoc";
---
<Pandoc caller={import.meta.url} file="Component.tex" extraArgs={["--webtex"]} /

Component.tex

\documentclass{article}
\usepackage{amsmath} % for the equation* environment
\begin{document}
This is a simple math expression \(\sqrt{x^2+1}\) inside text. 
And this is also the same: 
\begin{math}
\sqrt{x^2+1}
\end{math}
but by using another command.

This is a simple math expression without numbering
\[\sqrt{x^2+1}\] 
separated from text.

This is also the same:
\begin{displaymath}
\sqrt{x^2+1}
\end{displaymath

…

View on GitHub

Whats next?

There are lots of commands out there. Which one should I connect to astro next?

Reddit RSS

Stephen Solka — Mon, 09 Nov 2020 16:20:48 +0000

A tiny project I have been maintaining since April 2020 is an alternative rss feed for Reddit called reddit-rss. At the time I was getting into using Feedly to read the news and found reddit's native rss feed to be pretty lackluster.

If you are interested in using it

Go to a subreddit or meta feed you like example: https://www.reddit.com/r/Android/
Add .json onto the end: https://www.reddit.com/r/Android.json
Change the domain name to, reddit.0qz.fun like: https://reddit.0qz.fun/r/android.json
Subscribe to ^^^ that url in your favorite feed reader.

Useful features

Links directly to the content not to reddit.
Extracts a readable version of articles and embeds it into the feed.
Embeds high quality images and videos directly into the feed.
Extra query params to narrow search results to your liking.

How its built

Its golang http server that uses a forked version of cameronstanley/go-reddit to parse reddit content. Then it inspects each link and decides how to build the feed. Article text content extracted using go-shiori/go-readability. Feeds are created using gorilla/feeds

Deployments are managed through shipit which has been meh. Do not recommend.

📰 Bypass paywalls; read your news on the terminal

Stephen Solka — Thu, 29 Oct 2020 15:08:02 +0000

Last night I built a new go tool to let me read websites on the command line. Readcli

readcli https://medium.com/compass-true-north/go-unit-testing-at-compass-3a7cb85ab54a

Install (on mac)

brew install trashhalo/homebrew-brews/readcli

Website Content

Due to the way I am scraping website content not every url will work. I've tested it with medium.com and nytimes.com.

The algorithm is as follows:

Use go-readbility to download a stripped down version of the website.
Use html-to-markdown to convert the clean html to markdown.
Use glamour to render the markdown content.

Bubbletea

The cli interaction is controlled by an elm inspired go library bubbletea. Its be a absolute dream to work with.

Embrace the Pipe

Stephen Solka — Thu, 22 Oct 2020 01:01:55 +0000

The unix pipe operator is magical.

As long as you write your application to read input from standard-in and write output to standard-out you can compose your cli tool with thousands of other tools.

Scanners

To read one line at a time I like to use bufio.NewScanner. This uses bufio.ScanLines to split the reader by new lines.

in := bufio.NewScanner(os.Stdin)
// Scan returns false when its out of lines
for in.Scan() {
  // line is a string
  line := in.Text()
  // if you want bytes in.Bytes() exists too
}

Lets build a tool

As an example here is a tiny command line tool I wrote to read urls for images and display them as ascii art images on the terminal.

Full source https://gist.github.com/trashhalo/6154a690bd2612ae33c8252d8051eb9f

func main() {
    in := bufio.NewScanner(os.Stdin)
    for in.Scan() {
        line := in.Text()
        err := convertLineToArt(line)
        if err != nil
    }
}

func convertLineToArt(line string) error {
    fmt.Println(line)
    resp, err := http.Get(line)
    if err != nil {
    defer resp.Body.Close()

    img, _, err := image.Decode(resp.Body)

    convertOptions := convert.DefaultOptions
    convertOptions.FixedWidth = 50
    convertOptions.FixedHeight = 50
    converter := convert.NewImageConverter()
    for _, row := range converter.Image2ASCIIMatrix(img, &convertOptions) {
        fmt.Print(row)
    }
    return nil
}

Use Pipe

Now lets combine together a few tools.

curl -A "test" https://www.reddit.com/r/cats.json | jq -r '.data.children[] | .data.url'|go run .|less

This is saying:

curl - download the json for reddit posts on cats. The -A "test" because reddit hates curl user agent so we need to change it.
jq -r 'xyz' - extract the urls for the images from the reddit json.
go run - run your command!
less - Put the output in a little viewer that lets you scroll up and down.

Mocking http clients in Go

Stephen Solka — Wed, 14 Oct 2020 20:54:19 +0000

If you ever find yourself needing intercept a http client in go, httptest provides a facility to do just that. httptest.NewServer lets you stand up a temporary server that you can control in your test case.

We will use this function as an example of writing unit tests that use httptest.NewServer.

func topPostOnSubreddit(r string) (string, error) {
    resp, err := http.Get(fmt.Sprintf("https://reddit.com/r/%s", r))
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()

    dec := json.NewDecoder(resp.Body)
    var data redditData
    err = dec.Decode(&data)
    if err != nil {
        return "", err
    }

    top := findTopPost(data)
    return top["title"].(string), nil
}

Refactor

Let refactor this so we can use httptest.NewServer in our unit tests. The required changes are ...

An instance of http client has to be passed in

func topPostOnSubreddit(client *http.Client, r string) (string, error) {
    req, err := http.NewRequest("GET", fmt.Sprintf("https://reddit.com/r/%s", r), nil)
    if err != nil {
        return "", err
    }
    resp, err := client.Do(req)
    // ...
}

The base of the url has to be configurable by the caller

func topPostOnSubreddit(client *http.Client, fullUrl string) (string, error) {
    req, err := http.NewRequest("GET", fullUrl, nil)
    if err != nil {
        return "", err
    }
    resp, err := client.Do(req)
    // ...
}

Writing the Test Case

The returned test server has both the client and url available for you to pass to the function. When the function uses the client makes the http call the handlerFunc you defined in your test case is called. This allows you to inject fixture data into the runtime to control unit test behavior.

func TestTopPostOnSubreddit(t *testing.T) {
    server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        enc := json.NewEncoder(w)
        // this is where your test fixture goes
        testData := redditData{}
        err := enc.Encode(&testData)
        require.NoError(t, err)
    }))
    t.Cleanup(server.Close)

    url := fmt.Sprintf("%s/r/android.json", server.URL)
    top, err := topPostOnSubreddit(server.Client(), url)
    require.NoError(t, err)
    require.Equal(t, "foobar", top)
}

Happy Testing!

Testable go with function types

Stephen Solka — Fri, 09 Oct 2020 00:10:16 +0000

A helpful pattern I've picked up writing test cases for go code is called Ports and Adapters. Skipping past long blog post with theory lets get into how this can look in typical go code.

func TellEveryBobHeSmells() error {
    emailSvc := getEmailService()
    db := getDBConn()
    rows := db.Exec(`select bobs from db`)
    for row.Next() {
        var bobsEmail string
        err := rows.Scan(&bobsEmail)
        if err != nil // ...
        err = emailSvc.SendSpiceyEmail(bobsEmail)
        if err != nil // ...
    }
}

This function is going to be very annoying to test. It takes no inputs and reaches out to 2 external services. First to get a list of bobs and another to send emails. Lets refactor this code to be more testable.

type sendSpicyEmailFn = func(email string) error
type bobFinderFn = func() ([]string, error)

func tellEveryBobHeSmells(findBobs bobFinderFn, sendEmail sendSpicyEmailFn) error {
    bobs, err := findBobs()
    if err != nil // ...
    for _, email := bobs {
        err := sendEmail(email)
        if err != nil // ...
    }

    return nil
}

We created two "ports" to abstract away the side effects of the implementation. sendSpicyEmailFn and bobFinderFn. Now our test cases can control the behavior of these functions. You can imagine writing a unit test for this function without even providing a database in the test suite.

// now we write a public wrapper function that fills in
// the adaptors so our callers dont have to care about ports/adaptors
func TellEveryBobHeSmells() error {
    // note we still passing in dependencies to the adaptors
    db := getDBConn()
    findBobs := defaultBobFinderFn(db)

    emailSvc := getEmailService()
    sendEmail := defaultSendSpicyEmailFn(emailSvc)

    return tellEveryBobHeSmells(findBobs, sendEmail)
}

// adaptor implementations
func defaultBobFinderFn(db *DB) bobFinderFn {
    return func() ([]string, error) {
        var bobs []string
        rows := db.Exec(`select bobs from db`)
        for row.Next() {
            var email string
            err := rows.Scan(&email)
            if err != nil // ...
            bobs = append(bobs, email)
        }

        return bobs, nil
    }
}

func defaultSendSpicyEmailFn(svc *EmailSvc) sendSpicyEmailFn {
    return func(email string) error{
        return emailSvc.SendSpiceyEmail(email)
    }
}

Now that we have cut apart the function into its business logic and external dependencies it is much easier to test. In general I would create a unit test for tellEveryBobHeSmells and the two adaptors defaultBobFinderFn and defaultSendSpicyEmailFn but skip the test case for TellEveryBobHeSmells.

Go errors.is cool

Stephen Solka — Thu, 16 Jul 2020 23:51:35 +0000

In 1.13 go introduced a new function on the errors package. Is. This lets you ask if the error you are holding is the same as a predefined defined error or the error wraps the predefined error.

var BadThingErr = errors.New("omg something happened")

type canReturnErr = func() error

err := canReturnErr()
if errors.Is(err, BadThingErr) {
  fmt.Println("that bad thing that can happen happened")
}

Ive been using Is since it was released but there is a small detail of the the documentation that I originally missed.

An error is considered to match a target if it is equal to that target or if it implements a method Is(error) bool such that Is(target) returns true.

type myCoolErr struct{
  errCode int
}

func (m myCoolErr) Is(err error) bool {
  me, ok := err.(myCoolErr)
  if !ok {
    return false
  }
  return me.errCode == m.errCode
}

This lets an error control the implementation of errors.Is when comparing equality of the error object. Neat!

How does this work?

if x, ok := err.(interface{ Is(error) bool }); ok && x.Is(target) {
  return true
}

https://github.com/golang/go/blob/43c6ada84c6ef47e3b61646d2f2e7f6b7264929d/src/errors/wrap.go#L39-L60

In the source code of the errors package there is a cool trick. This is asking if the err object complies with an anonymous interface if it does then the go compiler allows you to call the method on the anonymous interface!

I have seen casts and casts to interfaces but never casts to an interface defined in the cast expression.

This pattern seems most useful if your target interface is a single method and only used in one place. The risk is too high for the anonymous interface definitions to diverge if its used in multiple source locations. At that point I'd recommend defining a proper type Xxx interface

Managing up through calendar stalking

Stephen Solka — Wed, 08 Jul 2020 23:17:43 +0000

Working at start-ups can be a pretty crazy experience. When everyone is running hard, one of the first things to fail is effective communication. One trick that can help plug the gap of missing information is invoking the dark arts of Calendar Stalking.

Calendar Stalking

In many organizations google calendar is configured to allow anyone to see anyone else's calendar. Calendars are filled with useful information. Calendar stalking is the art of using that data to make sense of the swirling world around you.

How to see someone's calendar?

On your computer, open Google Calendar.
On the left click Search for people.
Start typing someone’s name and choose the person you want to meet with.
At the top left, click Create Add.

If they were evil and configured their calendar only says Busy/Free see appendix at the end.

Start with your manager's calendar. Information tends to get lost or distorted when moving between levels. Your manager's manager and up the chain are all great sources of information.

Ignore all the repeating meetings. Ignore the one on ones. 80% of the meetings are boring fluff everyone has. You are looking for one off meetings talking about topics that you do not know much about. Events that have not been accepted yet are especially good because often they have recently been created.

In the calendar event there may be an agenda attached. Be sure to read that. Extra information can sometimes be found in your office wiki using words in from the subject or description of the event.

What do I do with this new information?

Is the meeting on an interesting topic and you wish you were invited to participate? Great, ask if you can come!

You not being invited was probably not an intentional slight. People are more than willing to let more come to meetings if they ask. Being present and contributing at the meetings increases your knowledge and presence at your company.

Meeting with an ambiguous agenda? Sometimes asking the organizer can help get more information.

"I noticed this meeting on your calendar. I was wondering if you might share what it is about?" There are of course limits to prodding. Don't ask people ops why they are meeting with someone suddenly on a Friday. In general people are pretty open to questions.

Your manager is a good resource of extra context for meetings they are in. Your one on one is a perfect time to talk about things you have observed around the office and on calendars. Most things going on at your job are not secret. Once you ask people open up and tell you all about what's happening.

Knowledge is power

The more you understand the world around you the more effective you will be in your job. You can get a sense for what is coming down the pipeline, not just the tickets you have been assigned. Stop waiting for people to notice you and invite you to their meeting. Find the meetings you want to go to and ask!

Appendix: Workaround for free/busy calendars

Sometimes people configure their calendars to hide the names of meetings they are going to. One workaround for this is if you can figure out who this person meets with on a regular basis then you can can look at that persons calendar and look for meetings that overlap.

How to ship more points

Stephen Solka — Thu, 02 Jul 2020 01:56:28 +0000

A lot can impact your velocity number. The kinds of tickets you have been taking, your mood, how you approach your work, etc. etc. This article is about what I do to consistently deliver a high number of points.

Why do you want this?

Establish why this ticket was prioritized. What do we hope to accomplish? Often the why is lost in a sea of technical requirements, product asks and commentary with stakeholders. To find the true why it may require you to follow a technique called Abstract Laddering.

Abstract laddering

This is a thought process that helps you move from concrete requirements to abstract asks and back down. To identify the why. Given a position on the ladder ask why to move up a level and how to move down a level.

Lets say you are assigned a ticket that has the title "Make the can opener more appealing". You could start doing research on what makes can openers appealing and prototyping options. OR you could ask why. Why do I have this ticket? If its not defined in the ticket why, ask your tech lead or product manager Why?

You might get an answer "We want to design a better can opener". Great! Now you have options. Appealing is just one way to improve the can opener. Some of these options may even cost less points to deliver to production.

When why isn't enough

Perhaps you didn't get to any good options when you learned your company really wants to "design a better can opener". Sometimes to find the easy way to deliver the ticket you have to go deeper.

Push past the initial answer and continue to ask questions until you can move up a level of abstraction. Why does your company care about designing a better can opener? This can lead to new options to deliver.

Assume you found out that your company only cares about can openers because "they want to help people get soup out of their cans". Whole new options open up for delivering your ticket! You could even add a pull tab to the top of cans!

The point of all this is to identify an implementation that gets your company what its looking for but costs significantly less than the original estimate.

Negotiation

Keep in mind this is a negotiation process with your peers. If at the end of the sprint your tickets are done but no one is happy then you have lost. Make sure you really are building the right thing but do not feel constrained by the tickets you are given.

What is preventing me from shipping today?

In the middle phase of a long project I often find myself lost. Unclear what is left and how much longer its going to take to get there. A useful mantra to adopt to find your way out is to ask yourself "Could I ship this feature today? If not why?".

Work through the set of changes in your mind that are preventing you from completing this project. Focus your energy on the work that is truly "blocking" from complete. Regularly ask yourself this question.

Cut scope

Tickets are never the exact minimum viable product change needed to accomplish a goal. There are often miscellaneous changes tacked on. Sometimes those extra changes drastically increase the cost of the ticket.

Product managers are often willing to negotiate with engineers. Often through conversations and a keen eye you can split large tickets apart and put the lower priority work back into the backlog.

Points are made up and they don't matter

The number of points you complete in a sprint mostly doesn't matter. Your primary purpose as an engineer is to deliver value to your organization. The value you bring is though building or changing software. Focusing on these techniques can help you avoid getting bogged down building low value changes over extended periods of time.