DEV Community: Jon Staab

Logaralgorithms

Jon Staab — Wed, 13 Apr 2022 16:50:25 +0000

Logarithms are an elegant mathematical function with a variety of applications from measuring earthquakes to calculating interest rates. But I think they might be under-appreciated as a tool for your average application developer.

The main popular use of logarithms is in charting exponential functions in a way that is intelligible to humans.

You can see this in the bitcoin stock-to-flow price model:

Covid cases:

The Decibel scale:

Why not use this powerful tool when relaying everyday concepts to our users?

Here's an example: you might redesign Twitter to show a post's score out of ten rather than the absolute like count, since people with lots of followers have vastly oversized reach compared with nobodies. In other words, there are relatively few high-follower accounts, meaning tweets either get a like or two, or tens of thousands, depending on who you are. What if there was a way to level the playing field a bit?

Let's just say the theoretical maximum number of likes for a tweet is 1 million — anything with 1MM+ likes would have a score of ten. On a linear scale, you would need 1/10th of 1MM likes to score a 1, which obviously won't do.

But we can plot our like numbers on a logarithmic scale using the formula log base likes of max score = exponent. Substituting our max likes and max score into the formula gives us an exponent of 1/6. So, to calculate a given user's score, we just have to raise the number of likes to the power of 1/6, or const getScore = likes => Math.min(10**6, likes) ** (1/6). Here's how that distribution looks

1 like = score of 1
10 likes = score of 1.47
100 likes = score of 2.15
1000 likes = score of 3.16
10000 likes = score of 4.64
100000 likes = score of 6.81
1000000 likes = score of 10

This still feels a little top-heavy to me — let's fine-tune it a bit. One way to do this is to pad lower values with an additional score:

const getScore = likes =>
  (Math.min(10**5, likes) ** (1/5) + Math.min(10**6, likes) ** (1/6)) / 2

This gives us a much nicer distribution:

1 like = score of 1
10 likes = score of 1.53
100 likes = score of 2.33
1000 likes = score of 3.57
10000 likes = score of 5.48
100000 likes = score of 8.41
1000000 likes = score of 10

Applying it to Twitter's UI might turn this:

Into this:

Small change (and not necessarily an improvement), but it illustrates the idea.

Other applications for logarithms in user interface design might include:

Any other instance where you want to provide an opaque score, especially when incorporating a weighted average of several different data points.
When you want to weight small values more heavily than large values, for example to give proportionally more voting power to small shareholders in a DAO to reinforce decentralization.

I think this is a very interesting technique, and will be reaching for logarithmic and exponential functions more often in my design work.

Building EmbassyOS 2.17 on a Raspberry PI

Jon Staab — Tue, 28 Dec 2021 13:32:24 +0000

Start9 Labs' EmbassyOS is an exciting project, and I couldn't wait to try it - but I wasn't about to spend over $179 on the software. For true self-sovereignty, it should be free and open source, so I went that route. Unfortunately, I ran into a number of issues, enough to justify a blog post detailing what the errors were, and how I got past them.

For guidance, I am following the official build guide, supplemented by this unofficial guide.

Note that EmbassyOS 0.3.0 is coming out real soon now, so these tips will likely be obsolete by the time I publish them.

Building Stack

The bulk of my time was spent building version 2.5.1 of stack, since only version 2.1.3 is available for Raspbian.

The guide says to use stack build --stack-yaml=stack-ghc-84.yaml --system-ghc to build stack, but it appears ghc 8.4 is no longer available for raspbian. 8.8.4 is the default ghc as of this writing. When I try to compile stack, I get the following error:

No setup information found for ghc-8.4.4 on your platform.
This probably means a GHC bindist has not yet been added for OS key 'linux-armv7', 'linux-armv7-ncurses6', 'linux-armv7-tinfo6'.
Supported versions: ghc-7.10.2, ghc-7.10.3, ghc-8.0.1, ghc-8.0.2, ghc-8.2.1, ghc-8.2.2, ghc-8.6.3, ghc-8.10.1, ghc-8.10.2, ghc-8.10.3, ghc-8.10.4, ghc-8.10.5, ghc-8.10.6, ghc-8.10.7, ghc-9.0.1

Luckily, stack has multiple configuration files! I was able to get stack to build by specifying the compiler thus: stack build --stack-yaml=stack-ghc-88.yaml --compiler ghc-8.8.4 to match my system version. Sady, the following install command failed, possibly because 8.8.4 is not listed in compatible ghcs above?

At any rate, I next attempted using 8.6.3, which is listed as compatible, using stack build --stack-yaml=stack-ghc-86.yaml --compiler ghc-8.6.3. This gave me some new errors, namely Error: selected processor does not support movt [...].

Following this issue I edited ~/.stack/programs/arm-linux/ghc-8.6.3/lib/ghc-8.6.3/settings as suggested to force compiling for armv7:

("C compiler flags", "-marm -march=armv7-a")
("target arch", "ArchARM {armISA = ARMv7, armISAExt = [VFPv2], armABI = HARD}")

This seemed to work, although I got a new error that llvm wasn't available, Warning: Couldn't figure out LLVM version! Make sure you have installed LLVM ghc: could not execute: opt. Stack wants llvm 6, but apt didn't have it so I used 9. I added llvm to my path as follows:

export PATH=/usr/lib/llvm-9/bin:$PATH
export CPLUS_INCLUDE_PATH=$(llvm-config --includedir):$CPLUS_INCLUDE_PAT
export LD_LIBRARY_PATH=$(llvm-config --libdir):$LD_LIBRARY_PATH

stack build continued to yell at me about using a version of llvm other than 6, but it appears to have worked. Invoking stack install --stack-yaml=stack-ghc-86.yaml --compiler ghc-8.6.3 did the trick as well. Of course, you'll want to add ./local/bin to your path once this completes, per the guide.

Building EmbassyOS

This step was much more straightforward, the one change that needed to be made to the build guide was to install nodejs 15 rather than the latest version due to engine compatibility. To do that, use nvm install 15 rather than nvm install node.

Status

After that, my .img was ready to flash! I did, and it's running, but unfortunately the Embassy service is crashing with an error /usr/local/bin/agent: /lib/arm-linux/gnueabihf/libm.so.6: version GLIBC_2.29 not found. I am still troubleshooting this one on Embassy's chat channel. Let me know if you find a way through that!

Update: the Embassy team told me 0.3.0 is nearly there, and much easier to build than 0.2.17. Onward!

0.3.0

This indeed was much smoother than 0.2.17, however I did still run into some small issues.

When running npm --prefix ui run build-prod, I got an error when building angular: An unhandled exception occurred: Cannot find module 'webpack'. Running npm --prefix ui install webpack fixed this. npm install -g webpack will also do the trick.

I also had trouble installing the buildx plugin, getting docker: 'buildx' is not a docker command. even after installing it to ~/.docker/cli-plugins/docker-buildx with the correct permissions. It seems this is because the version of docker installed by apt on Debian is Docker version 18.09.1, build 4c52b90, while the buildx feature arrived in v19. Switching to Ubuntu 21.10 fixed the problem.

After clearing that up, everything built beautifully. The product key is in embassy-os/product_key.txt. Don't delete the repository (or the VPS you used to compile it) until you put that and the eos.img somewhere safe.

At the time of writing, EmbassyOS 0.3.0 is not yet released, so there are a number of rough edges with getting it set up initially. It required a number of restarts before the embassy user was set up and it was able to connect to my LAN. Pro tip: check that your ethernet cable is actually an ethernet cable.

Now that I've gotten my embassy built and running, it's time for me to tear it down. Don't get me wrong — I'm very excited about the project, but it needs to be more stable before I trust it with all my data. In the meantime, I'm going to give NextCloudPi a shot, with an eye toward wrapping it for Embassy down the road if I like it.

How to Query Multiple Schemas With Postgresql

Jon Staab — Wed, 07 Apr 2021 17:36:48 +0000

At work, we have a multi-tenancy setup where for performance purposes we split tenants across multiple identically-structured schemas. While giving each tenant their own schema would help with isolation, having too many separate database objects would make migrations and using pg_dump too resource intensive. Instead, we use 100 numerically named schemas to randomly distribute tenants across the schema space.

This works great as an easy way to kick the can of application performance down the road. We have a master database which contains a record of all tenants and which schema their data is contained in. The app simply looks up that tenant and uses its corresponding schema.

Oh, you wanted ALL your data?

The difficulty when performing queries across the entire database, either for analytics purposes, or to find a particular object without knowing which tenant it belongs to. This article by Dave Clark is a great starting point, but it wasn't quite doing it for me.

I've been able to combine his "application generated union query" and "union view" approaches into something that works really well for my use case of ad-hoc database exploration of just a handful of tables at a time, and doesn't require anything except pure SQL. My approach doesn't rely on periodic view refreshing, excessive load on the database, or wasting space on materialized views.

Dynamic SQL in Postgres is weird

I didn't figure this out on my own; because postgres is generally statically typed, it's tricky to get it to do things in a generic or dynamic way. If you want to see my creative process, check out this Stack Overflow question.

Here's the code:

CREATE OR REPLACE FUNCTION app_union(tbl anyelement)
RETURNS setof anyelement AS $$
BEGIN
  return query execute string_agg(
    distinct format('select * from %I.%I', schema, pg_typeof(tbl)::text), 
    ' union all '
  ) from tenants;
END
$$ LANGUAGE plpgsql;

select * from app_union(null::my_table);

Simple, right? Breaking it down, there are two Postgres features that make this sort of thing possible: polymorphic types, and pg_typeof.

In Postgres, record types are statically typed. This means that when returning a row or set of rows from a function, the caller has to provide the full row type definition of the result set. This looks something like select my_function('table_name') t(id uuid, name text...). This is incredibly laborious to write out for ad-hoc queries.

Turns out, you can simply tell postgres to chill by using polymorphic types. However, the type still needs to be something postgres can infer, which is why we pass a null row of the table we want to the function instead of the name of the table as text.

We then build the query by using pg_typeof to get the record type of the row we passed in, and casting that type to text.

This is not fast

Yes, I did promise this wouldn't hammer your database at the beginning, didn't I? Well, it turns out with dynamic sql, adding a where clause to the function is trivial. Here's my final version:

CREATE OR REPLACE FUNCTION app_union(tbl anyelement, extra text)
RETURNS setof anyelement AS $$
BEGIN
  return query execute string_agg(
    distinct format('select * from %I.%I %s', schema, pg_typeof(tbl)::text, extra), 
    ' union all '
  ) from tenants;
END
$$ LANGUAGE plpgsql;

To solve my performance issues, I simply added an extra text parameter to my function. This lets me pass any arbitrary sql into the function to go after the from in every sub-query. So, for example select * from app_union(null:my_table, 'where n > 1') lets me apply the filter at the original table level rather than pulling all 12 million records and applying the filter to the union query.

How about a real example?

I wanted to get a list of all tenants active in the last 7 days. With app_union, it's a breeze:

select name, created, last_activity
from tenant
join (
  select tenant, max(created) as last_activity
  from app_union(null::event, 'where created > now() - interval ''7'' day')
  group by tenant
) e on tenant = tenant.id;

This query took about 10 seconds on my production server with 2.5MM event records. This is fairly hefty, but not bad for a one-off analytics query!

Thanks for reading — here's hoping this proves useful in your database wrangling adventures.

Some Postgres Performance Tips

Jon Staab — Fri, 18 Dec 2020 00:42:49 +0000

This last weekend the application I develop and support went down for a couple hours while many of my company's customers were trying to make holiday shopping sales. It was not fun, and the next three days were packed with stress as I tried to resolve the underlying problems. But it was also a fun learning experience.

Below is a quick summary of what I learned, complete with links to sources, just in case you (or I) ever find yourself in the same situation.

Locking

My first thought was that I had made some mistake in a recent update, and my application was creating deadlocks. When I looked for long-running queries that were blocking others, I found... a lot. (source)

select pid, query, query_start, now() - query_start as duration
from pg_stat_activity 
where pid in (
  select pid from pg_locks l 
  join pg_class t on l.relation = t.oid 
  and t.relkind = 'r' 
)
and pid <> pg_backend_pid()
order by duration desc;

When I first started diagnosing the issue, there were over 100 rows returned, several of which had been running for several minutes.

Updates are one of the the greediest kind of statements for holding locks, so when updating data it's good to be careful not to use them too often. Long-running transactions can cause a lock to be held for longer than necessary, so instead of opening a transaction for every request, I turned on psycopg2's autocommit feature, and modified my code to opt-in to controlling transactions using a context manager.

This article defines a lock_monitor query (at the very bottom) to find what queries are doing the blocking, although I've not seen it in action, since my problem wasn't actually locks — that was only a symptom.

Caching

Something was suspect about the long-running queries. Normally deadlocks get automatically cleaned up by postgres, but these were lingering, and not getting caught by pg_blocking_pids().

The reason behind this was a degraded cache hit rate, which was in turn causing high cache turnover, and lots of disk reads and table scans which are sloooow. You can read more about that here.

If you're using heroku-postgres with the pg-extras extension installed, the cache hit rate can be retrieved using heroku pg:cache-hit. It'll give you something like this:

      name      |         ratio
----------------+------------------------
 index hit rate | 0.98304548448697819350
 table hit rate | 0.92119613681591593477

There are also some equivalent queries in the article above if you're not using heroku. That article also states that "If you find yourself with a ratio significantly lower than 99% then you likely want to consider increasing the cache available to your database."

On our standard-0 heroku-postgres plan, our cache hit rate was somewhere around 87%. After upgrading to standard-2, our rate is closer to 92%. Better, but not as good as I'd like. Our database size was around 12GB, which was mostly audit log events and related indexes.

Indexing

How big the cache needs to be to get a good hit rate depends on the size of the database and access patterns, so shrinking it and using better indexes can be a good alternative to increasing the amount of memory available to your database. Below are some links on the subject with useful tips. To find low-hanging fruit, table size can be retrieved with this query:

select pg_size_pretty(pg_total_relation_size(tablename::text))
from pg_tables where schemaname != 'pg_catalog';

Cleaning up Indexes - the PGExtras package they're using in ruby is the same as our heroku pg-extras extension.
Indexing jsonb
This article has a helpful query to find unused indexes. You can get that from pg-extras too, but it seems to be inaccurate, at least on a newly provisioned database.

After taking a look at heroku pg:unused-indexes, I was able to remove or combine enough indexes to save an entire gigabyte of space, improving my cache's performance, and reducing the amount of work the database has to do when writing to my over-indexed tables.

Load

Finally, there's CPU load. Load wasn't crucial to the problem I was facing, but it can be a good indicator that something is wrong. High CPU load means the database is doing more work than it needs to do, which means it's either not using indexes well, or it's not hitting the cache.

I had a hard time finding CPU load for my heroku database, and since many hosts share resources between multiple tenants, the number of CPUs on the machine may not match how much compute power is available to my database.

But every 5 minutes or so, heroku emits a log line with some helpful database and server metrics, defined here. To get just those lines, run heroku logs -d heroku-postgres. Adding -t is a good idea, as heroku logs has a short memory, and might not contain one of the log lines you're looking for.

The load-avg stats, at least on dynos with burstable performance, should average out at 0.2, not close to 1. I'm not sure if that applies for higher-tier plans without burstable performance though. My database's load average was around 0.6. After fixing the cache miss issues, it's much closer to the target 0.2.

Conclusion

Crisis averted — for now. I've been meaning to learn more about postgres, and it unfortunately took a crisis for me to do my homework. I will be keeping a closer eye on this stuff from here on out. I hope these tips are useful to you!

Below are some more articles I've found since writing this one, which could come in handy:

Free up space by deleting unused indexes, bloat, and null-filled indexes

Opening Command Output in a New Buffer

Jon Staab — Mon, 05 Oct 2020 21:29:57 +0000

Last time, we tweaked our kakoune query browser to open the selected query in an info window. As I mentioned at the bottom of that post, there are a few problems with this over-simplistic approach, one of which is the limitations of the info window.

Today I'll be tweaking the plugin by moving the query output into its own scratch buffer, which we'll open for the user when a query is run. As an added bonus, I'll be generalizing the plugin to work the same for any external command, not just databases.

To start with, let's review our plugin:

def eval-query \
  -override \
  -docstring "Evaluate current selection using given connection." \
  -params 1 \
  -shell-script-candidates %{
    cat .eval-query.json \
      | jq -r '.connections|keys|@sh' \
      | sed s/\'//g \
      | tr -s " " "\n"
  } \
%{
  info -title "Query output" %sh{
    psql `cat .eval-query.json | jq .connections.$1 | cut -d\" -f 2` \
      -c "${kak_selection}" 2>&1
  }
}

The salient information here is that info command. Instead of popping up an info dialog, we're going to capture the output and put it in the | register using set-register. Below is the modified command body:

set-register | %sh{
  cmd=`cat .eval-query.json | jq .connections.$1 | cut -d\" -f 2`
  eval "$cmd "${kak_selection}" 2>&1"
}

We used the | register, since :doc registers states that it should be used by commands that spawn a subshell. I'm not sure if this is 100% kosher, so leave a comment if you know better!

Next, we're going to open a new scratch buffer called eval-query-output and stick the contents of the register in there. This is pretty straightforward:

edit -scratch *eval-query-output*
exec 'geA<ret><esc>"|p;'

The edit command is probably familiar; it just opens a new buffer. -scratch tells kakoune we don't care about linking it with a file. Finally, we use execute-keys (exec for short) to go to the end of the file, add a new line, and paste the contents of the | register.

And that's it! See it in action below:

Because the mechanism for running a query is so generic, this plugin can be used just as easily to open anything in a new buffer, not limited to query results. By simply renaming connections to commands and updating our config file (I also renamed the plugin to beval for "buffer eval"), the possibilities become endless!

Let's add a fancy ls command that lets us open a directory listing in a new buffer:

{
  "commands": {
    "ls": "ls -lah"
  }
}

Finally, we can cheat by adding an eval key to run any command we want:

{
  "commands": {
    "eval": "eval"
  }
}

What we basically have now is | on steroids (and less well-designed). Let's go ahead and refactor this to split out the "pipe to buffer" functionality as a standalone thing.

def eval-to-buffer \
  -override \
  -docstring "Evaluate current selection into scratch buffer." \
  -params 1..1000 \
%{
  set-register | %sh{ eval "$@ 2>&1" }

  # Open our output buffer
  edit -scratch *command-output*

  # Put output into the buffer
  exec '%"|pd'
}

def beval \
  -override \
  -docstring "Evaluate current selection using named command into scratch buffer." \
  -params 1 \
  -shell-script-candidates %{ cat .beval.json | jq -r '.commands|keys|@sh' | sed s/\'//g | tr -s " " "\n" } \
%{
  eval-to-buffer %sh{
    cmd=`cat .beval.json | jq .commands.$1 | cut -d\" -f 2`
    echo "$cmd \"${kak_selection}\""
  }
}

Now our project-specific configuration is independent of our mechanism for piping output to a buffer, since both are useful in different circumstances.

Further refinement is of course out there, but for now, I'll leave this here. Thanks for reading!

If in doubt, list it out

Jon Staab — Wed, 08 Jul 2020 18:19:20 +0000

Today (and for the last few weeks), I've been dealing with a bad technical decision I made about three years ago, which has suddenly come back to bite me, as the hidden requirements have suddenly come to light. I want to walk through my situation, and how I'm learning the painful lesson that it's best to enumerate domain objects rather than mush them into a lossy aggregate.

Background

I work on an inventory management platform for consignment stores — when I was hired to build the system from scratch with no prior domain experience, it sounded simple.

It's not simple. Consignment normally works with unique inventory, except when it doesn't. In addition, you have two models of paying suppliers living side-by-side: traditional retail where you pay your suppliers when you acquire the inventory (or thereabouts), and consignment, where you pay your suppliers when the inventory sells.

To make this worse, it's possible for both payout models to be applied to a single SKU, either for the same instance of an item, or for multiple instances over time.

Where I went wrong

In the name of "saving space", I compressed every item in the system into a single row in the items table.

There's nothing necessarily wrong with this, as we keep an event log that lists every update to an item, so we can get at the item's state as of a point in time (e.g., when it was taken into inventory, or sold). We also keep various special purpose logs, including balance accruals and item status changes to help us decouple supplier payout state from the current state of the item.

But the code started to smell when we had to shoehorn the concept of "status" (a concept having to do with unique inventory) into our plural-cardinality model: is this item sold? lost? in stock?

I accommodated this with a jsonb field in our database that kept track of how many items for the SKU could be attributed to a given stock, e.g., {"active": 3, "sold": 1, "lost": 1}. I knew it was a bad design, but I couldn't predict how it might cause problems, so I left it.

Discovering my mistakes

I'm now writing customer-facing inventory analytics code, which includes calculating Cost of Goods Sold. Because the state of any given enumeration of an item has been lost (consignor split, price, and cost) due to subsequent edits, I now have to add a new table called "item_sales", which lists the attributes of an item needed to calculate COGS as of the time of the sale, and fill it by replaying the relevant events.

Also, because we need to support partial refunds, I can't compress these records into one per sale, with a record of quantity; I have to mark each enumeration of an item as refunded independently.

So I'm left with the design I tried to avoid in the first place, except it's ad-hoc and very difficult to maintain and understand. If I had enumerated every item instance in the first place, I would have had all the COGS data as of the sale date. And once an item sold I could have "locked" those instances to avoid mutating the data even if its un-sold brethren were modified.

Conclusion

We left designing analytics until we already had two years worth of customer data. Analytics is pretty much your worst-case scenario for understanding your data, and contrasts strongly with a CRUD-paradigm view of what's true now with no regard for history. Even an event log (though useful for migrating to a new schema) won't save you, since events often have to be procedurally re-played rather than simply aggregated.

Learn from my mistakes — if in doubt, list it out. It's as Rich Hickey says when evangelizing for Datomic: storage space is cheap, developer time isn't. Keeping redundant copies of your data is redundant until it isn't. You have to understand what's possible in your domain, and point your design in a direction that will handle that from every angle.

Building my own Stack Traces

Jon Staab — Tue, 06 Aug 2019 21:01:46 +0000

Have you ever had an error in a javascript project?

TypeError: Cannot read property 'subdomain' of undefined
      at /Users/jstaab/my-project/src/server/db/core.js:568:35
      at Array.filter (<anonymous>)
      at /Users/jstaab/my-project/src/server/db/core.js:147:1
      at log (src/common/logger.js:75:14)
      at _callee16$ (src/server/db/core.js:768:28)
      at tryCatch (node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (node_modules/regenerator-runtime/runtime.js:271:22)
      at Generator.prototype.(anonymous function) [as next] (node_modules/regenerator-runtime/runtime.js:97:21)
      at asyncGeneratorStep (src/server/db/core.js:22:103)
      at _next (src/server/db/core.js:24:194)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:189:7)

Even with source maps, that stack would only point to where the error was thrown, but I have no idea where in the application that thing was called.

Today I decided to do something about it, by implementing my own stack traces. For example, for the above error, I got this extra information:

Intercepted in:
     server/db/core/getWhereColumns
     server/db/core/deleteRecords
     server/db/domain/MyModel.delete

This just gets appended to an error's built-in stack trace (which are just strings in the first place). This makes it orders of magnitude faster for me to isolate problems and fix them, whether I'm developing locally or diagnosing a problem in production.

This approach has drawbacks: there's some nasty boilerplate, and definitely some performance downsides, which I'll go into at the end of this post. But is it worth it? I think so.

Implementation

The basic concept is that you have a decorator that catches errors and rejected promises, annotates their stack, and re-throws them.

const IS_INSTRUMENTED = Symbol('isInstrumented')

const instrument = (name, f) => {
  if (f[IS_INSTRUMENTED]) {
    return f
  }

  function wrapper(...args) {
    let result
    try {
      result = f.call(this, ...args)
    } catch (error) {
      throw modifyStack(name, error)
    }

    if (result instanceof Promise) {
      result = result.catch(error => {
        throw modifyStack(name, error)
      })
    }

    return result
  }

  wrapper.name = name
  wrapper[IS_INSTRUMENTED] = true

  return wrapper
}

The reason I'm not using an async function here is that I want to be able to instrument synchronous functions without converting them to promise-returning functions (async functions always return promises). We're also using es5 function syntax and bind so we can pass this along.

This function takes a name and another function, and returns a function that passes the arguments along to the decorated function, catching any synchronous errors that it might throw and modifying their stack. It also checks if the return value is a promise, and modifies the stack of the return value.

Note that you should only ever throw/reject Errors, since this will add wonky stack properties to anything else.

Here's modifyStack:

const modifyStack = (name, error) => {
  if (!error.originalStack) {
    error.originalStack = error.stack
    error.stack += "\n\n    Intercepted in:"
  }

  error.stack += `\n\t ${name}`

  return error
}

A Basic Example

Here's how you'd use instrument:

const concat = instrument('utils.concat', (a, b) => a.concat(b))
const append = instrument('utils.append', (a, b) => concat(a, [b]))

Note that I don't recommend decorating utility functions, since they're likely to be leaf nodes in your call graph, and are usually called synchronously so you'll get a good stack trace for them. You also don't want to add the overhead of this decorator to something that's going to get called very often. This decorator is much better for controllers, hefty logic functions, etc, to give you a sense of what code path has gone wrong. With that in mind, what if you called the above function with weird arguments?

append(1, 2)

/*
Uncaught TypeError: a.concat is not a function
    at <anonymous>:1:55
    at wrapper (<anonymous>:7:18)
    at <anonymous>:2:53
    at wrapper (<anonymous>:7:18)
    at <anonymous>:1:1

    Intercepted in:
     utils.concat
     utils.append
*/

Not that helpful, but remember, this is just a small example. In complex application code, this sort of thing can be a lifesaver.

Class Methods

Class syntax can make this approach a pain. If you don't want to add a syntax transformer, you have another option: instrument an entire class. Here's what that looks like:

const instrumentClass = cls => {
  let obj = cls.prototype
  do {
    if (obj.constructor === Object) {
      break
    }

    Object.getOwnPropertyNames(obj).forEach(k => {
      if (typeof obj[k] === 'function' && k !== 'constructor') {
        obj[k] = instrument(`${obj.constructor.name}.${k}`, obj[k])
      }
    })
  } while (obj = Object.getPrototypeOf(obj))
}

class Libation {
  pourOn(target) {
    target.wet = true
  }
}

class Tea extends Libation {
  async pourOn(target) {
    await new Promise(() => super.pourOn(target))

    target.bland = true
  }
}

instrumentClass(Tea)

We're basically following this StackOverflow answer to traverse the chain of inheritance so we can enumerate all methods attached to our instance. We then instrument those methods, marking them with the class name and method name, skipping the constructor.

Here's how this would look in practice:

const tea = new Tea()
const ground = null

tea.pourOn(ground)

/*
Uncaught (in promise) TypeError: Cannot set property 'wet' of null
    at Tea.pourOn (<anonymous>:57:16)
    at Tea.wrapper (<anonymous>:5:18)
    at target.bland (<anonymous>:63:35)
    at new Promise (<anonymous>)
    at Tea.pourOn (<anonymous>:63:11)
    at Tea.wrapper (<anonymous>:5:18)
    at <anonymous>:74:5

    Intercepted in:
     Libation.pourOn
     Tea.pourOn
*/

Notice that the original stack trace is just plain wrong — and this is running in Chrome's devtools, with no build step. But now we have more information: the error happened in Libation.pourOn, which was called by Tea.pourOn! I don't know about you, but I find this very helpful.

Footguns and Performance

Obviously, any time you're changing the semantics of your host language, you need to be careful. This code is not perfect yet, but I've tried to address some of the possible issues:

This code won't wrap synchronous functions in a Promise.
This code will pass this down along with the arguments to the wrapped function.
instrumentClass does create a new copy of each method, including a wrapping closure. What it doesn't do is create a new copy for each instance, which means we still have the general memory-conserving characteristics of object-oriented method dispatch.
We're not unnecessarily wrapping every method of Object, and we can easily add code to whitelist which methods we wrap.

Performance is also not terrible. Here's a little bit of profiling code:

const profile = f => {
  now = new Date().valueOf()
  for (let i = 0; i < 10000000; i++) f(i, i)
  console.log(new Date().valueOf() - now)
}

profile((x, y) => x + y)
profile(instrument('hi there', (x, y) => x + y))

Without the wrapper, this comes out to ~200ms. With the wrapper, it comes out to ~450ms. This is 150% slower, but it's really a worst case scenario since the wrapper is doing more work than the inner function (which is why it's a bad idea to wrap frequently-used utility functions). If the inner function did more work, the cost of the wrapper would be negligible:

const profile = f => {
  now = new Date().valueOf()
  for (let i = 0; i < 1000; i++) f(i, i)
  console.log(new Date().valueOf() - now)
}

const f = (x, y) => {
  const r = []
  for (let i = 0; i < x; i++) {
    for (let j = 0; j < x; j++) {
      r.push(i + j)
    }
  }

  return r
}

profile(f)
profile(instrument('hi there', f))

This comes out to ~11200ms for both versions.

Conclusion

This code is available on github — please let me know in the comments what you think of this approach! Am I a generously mustached daredevil maverick, or am I going to end up like Rod Kimble, desperately shouting "Whiskey" on my way toward the side of a van?

Thanks for reading!

Running Queries in Kakoune with a Custom Command

Jon Staab — Wed, 29 May 2019 17:55:31 +0000

Last Time, I wrote about building a minimal query browser using bash and Kakoune. In it, I set up a file watcher using entr, and auto-ran my queries on file change.

An arguably better approach would have been to register a custom command to do that, since my queries might not always be safe in a production environment. Kakoune's documentation on the topic is dense to say the least, so let's put together an example to go with it!

Project Structure

For this project, we're going to need two components: Kakoune commands, and connection configuration. Commands should be reusable between projects, while connections will be project-specific.

Best practice when writing Kakoune plugins is to put them in a file on their own, somewhere in one of Kakoune's autoload directories. I'll be putting my plugin into ~/.config/kak/autoload/jstaab/eval-query.kak.

Connection configuration, on the other hand, will go in my project root directory, in .eval-query.json. For now, I'll just use a simple json object describing arguments to pass to psql:

  "connections": {
    "local": "-d my_project",
    "production": "-h 192.168.0.1 -d my_project"
  }

Adding our Command

For this plugin, we're only going to need one command, which I'll name after my plugin: eval-query. Commands are defined using the def keyword with some optional switches. We of course want a docstring and we'll add the -override switch so we can redefine this command as we work on it (we'll want to remove that when we're done).

def eval-query \
  -override \
  -docstring "Evaluate current selection using given connection." \
%{
    echo "I don't do anything yet"
}

That %{} syntax is kakoune-speak for a multi-line string. The bodies of commands abide by the rules described in the command parsing docs. When the command is invoked, the contents of this string are what will be evaluated.

Let's go ahead and invoke our no-op command, by typing :eval-query in normal mode. This will print our message, "I don't do anything yet" down on the bottom-left of the screen.

Asking for Input

The first thing we'll need to do is add a way to ask the user which connection we want to use. There are a lot of ways to do this, but we'll be using -shell-script-candidates. This switch takes a shell script that supplies a newline-delimited list of possible completions, which Kakoune will then fuzzy-match over. We'll also have to tell Kakoune how many parameters are expected, using -params. We'll also just have the command echo back at us what we provide:

def eval-query \
  -override \
  -docstring "Evaluate current selection using given connection." \
  -params 1 \
  -shell-script-candidates %{ echo something } \
%{
    echo %arg{@}
}

This gives us only one auto-complete option, "something". When we invoke this command with :eval-query, it prompts us with "something", and echoes it back to us.

Let's fill in the completion script by reading from our config file and grabbing the keys using jq.

def eval-query \
  -override \
  -docstring "Evaluate current selection using given connection." \
  -params 1 \
  -shell-script-candidates %{
    cat .eval-query.json \
      | jq -r '.connections|keys|@sh' \
      | sed s/\'//g \
      | tr -s " " "\n"
  } \
%{
    echo %arg{@}
}

This looks at the project's config file, pulls out the keys, and provides them as hints to the command's autocomplete. Now we know which connection to use!

Evaluating the Query

The last piece of the puzzle is finding the query we want and passing it to our database for execution. Kakoune makes this simple, by providing a $kak_selection variable to our shell expansions. Getting the string quoting correct on the other hand can be tricky.

def eval-query \
  -override \
  -docstring "Evaluate current selection using given connection." \
  -params 1 \
  -shell-script-candidates %{
    cat .eval-query.json \
      | jq -r '.connections|keys|@sh' \
      | sed s/\'//g \
      | tr -s " " "\n"
  } \
%{
  info -title "Query output" %sh{
    echo "${kak_selection}"
  }
}

I've changed the echo command to an info command, which opens up an info box filled with the stdout of the %sh expansion. Inside the shell expansion, we're echoing the contents of the current selection.

To pass it to postgres, we grab the options from the specified connection, and pass them, along with the current selection to postgres:

def eval-query \
  -override \
  -docstring "Evaluate current selection using given connection." \
  -params 1 \
  -shell-script-candidates %{
    cat .eval-query.json \
      | jq -r '.connections|keys|@sh' \
      | sed s/\'//g \
      | tr -s " " "\n"
  } \
%{
  info -title "Query output" %sh{
    psql `cat .eval-query.json | jq .connections.$1 | cut -d\" -f 2` \
      -c "${kak_selection}" 2>&1
  }
}

Notice that I'm redirecting stderr to stdout so we can see any errors that might arise (like a nonexistent database, for example). Here's what this looks like:

Binding to a Hotkey

You'll notice that this isn't exactly ergonomic. Do I have to type out the full command, complete with my connection every time? The answer is no. We can bind our command — along with any useful variations — to keyboard shortcuts, using map.

map global user \" \
  -docstring "Evaluate current selection in local database" \
  ':eval-query local<ret>'

map global user \' \
  -docstring "Evaluate current paragraph in local database" \
  '<a-i>p:eval-query local<ret>'

I've created two bindings here; the first one just calls our command with the local connection filled in. The second one does the same thing, but selects the current paragraph.

This has the effect of letting us run a query with two keystrokes, even if we don't have it already selected. If we need more control, for example to run just part of a query, we can use the more basic double quote version. You can see that in action below:

Note that the keys I chose to bind are completely arbitrary; change them to whatever you prefer!

Addenda

This "plugin" is far from done. Like every software project, it has the potential to be a business/all-consuming obsession, so I'm going to stop here, for now. Partly as a disclaimer, and partly as an exercise for the reader, here's a non-exhaustive list of improvements that could be made:

Large result sets will not be shown; I'm not sure what's going on internally, but Kakoune appears to have an upward limit on the amount of text that can fit in the info block. One way to handle this would be to put the results in a scratch buffer instead.
Plugins really shouldn't put keyboard shortcuts in the user scope. A better approach might be to create a custom user mode to namespace the shortcuts defined by the plugin, then declare an option that a user can override in her config file. To get that into the mappings, you'd likely have to listen for a GlobalSetOption hook to re-bind the mapping when the user sets the module option.
Errors and edge cases could be handled much more reliably. I mostly ignored it for the purposes of this post.
In order to share this code (and take on the accompanying burden of maintaining an open source project), we should wrap it up in an installable package. This might be as simple as making the file downloadable somewhere.

Maybe I'll do further posts on how to accomplish these things 😉

At any rate, my point here isn't primarily to demonstrate how to create a Kakoune plugin; it's to showcase the flexibility and ease with which you can customize Kakoune to suit your workflow. An ad-hoc architecture that only concerns one person's use case can be much less robust than a plugin meant for general use. Don't be afraid to push JetBrains out of your way and make some of your own tools!

Using Kakoune as a Query Browser

Jon Staab — Mon, 27 May 2019 20:58:25 +0000

For my day-to-day work, I use TablePlus to develop/maintain various postgres and mysql databases, which I can honestly recommend as a pretty great query browser.

Like any other piece of software though, it has its warts, among which are sub-par query formatting and odd text-editing keybindings. This isn't helped by the fact that I use Kakoune (a text editor that's built around doing vim "the unix way") for most of my text-editing activities. I also use tmux for a window manager, since Kakoune doesn't come with one.

So I asked myself: instead of wishing there were Kakoune-like editing abilities in TablePlus, what if I put sql into Kakoune? It turned out to be easier than I imagined because, well, unix.

Editing queries

To start with, I created a directory to put all my scripts and sql-related projects:

$ mkdir query-browser
$ cd query-browser
$ mkdir scripts projects

Creating a place where I can edit queries was as simple as creating a sql file, opening it in Kakoune, and copying my query-in-progress into it.

$ mkdir projects/my-project
$ kak projects/my-project/queries.sql

This, so far, looks like any other code:

Thinking ahead a bit, that looks problematic. If I'm developing that last query, I don't want to run the CREATE TABLE query every time! Easily solved:

Now I can run the setup query once on the right, close that tmux pane, and get to work on the query I'm developing.

Running queries

Now that I have some sql files, I can run them using the psql command line tool:

$ psql -d postgres -f projects/dogfish/setup.sql
INSERT 0 3
$ psql -d postgres -f projects/dogfish/queries.sql
  name   | length | edible
--------------+--------+--------
 dogfish |   22.3 | t
(1 row)

Let's generalize this by saving it to a script called run.sh.

psql -d postgres -e -f "$1" ${@:2}

After a quick chmod +x ./scripts/run.sh, we can run our queries file with ./scripts/run.sh projects/dogfish/queries.sql. Notice that I'm passing the rest of the arguments directly to psql so we still have access to all of its command line options.

Developing with a feedback loop

I love quick feedback when I'm working. I almost always have some tests running in another window, or hot reload turned on for my app. We can do the same thing here, too.

Let's make another script called ./scripts/watch.sh.

find "$1" | entr -p -r ./scripts/run.sh $@

This one uses find along with entr to watch the given query file and run it when it changes.

Note that I don't recommend this if you're going to be using this technique in production. It's much better to create a custom command that explicitly runs your query. For me, I want a fast feedback loop in development, so file watching fits the bill.

We can start our watcher (don't forget to chmod +x it) with ./scripts/watch.sh projections/dogfish/queries.sql. Now every time we make a change to the file, it runs it and shows us the results!

Auto-formatting

The possibilities with this technique are endless, but let's cap it off here with adding auto-formatting to our query browser. If you're following along, install pgFormatter for your OS (brew install pgformatter works on OSX). Now, save a new script to ./scripts/format.sh:

current="$(cat $1)"
formatted="$(pg_format $1 -s 2 ${@:2}"

# Only write to the file if it changed to avoid an infinite loop
if [[ "$current" != "$formatted" ]]; then
  echo "$formatted" > "$1"
fi

Chmod it and add it into our watch script:

find "$1" | entr -p -r bash -c "./scripts/format.sh $1; ./scripts/run.sh $@"

And now we have automatic query formatting:

Final Words

Obviously this isn't a full-featured query browser, but for me, the longer I develop software, the fewer features I want in the software that I use. Kakoune is a great foundation for building custom, minimal tools for day-to-day use. And, of course, this technique isn't limited to Kakoune — it's just a matter of building small tools that do one thing well, and composing them together.

I do encourage you to try Kakoune out if you haven't! Its learning curve is not beginner friendly, but it does pay off for the dedicated student.