DEV Community: Neeraj Kashyap

See the errors your users are experiencing. From your IDE. Live.

Neeraj Kashyap — Thu, 06 May 2021 18:47:41 +0000

If you have built a library, command line tool, or API, Bugout helps you understand:

Who your users are
How they are using your software
What errors they experience

Once you set up an integration and instrument your code, you can access your user reports at https://bugout.dev. This gives you a live view of what your users are experiencing:

This week, we released the latest version of the Bugout VSCode extension.

Now you can see the latest issues your users have had from VSCode. Just hover over an exception to see how many users were affected by it:

You can even click through to see their stack traces:

This is just a taste of things to come. If you use VSCode and would like to try this out, Bugout supports Python, Go, and Javascript.

Humbug: Usage and crash reports for Python developer tools

Neeraj Kashyap — Thu, 18 Mar 2021 23:44:55 +0000

Developer tools are unlike any other kind of software. All your users are power users. This blurs the line between your users and your code.

GitHub issues and Slack conversations are a poor guide to real usage of your software. Only a small proportion of the people who ever used your tool will be motivated enough to speak with you through these channels. The issues that these users have may not be representative of the real issues - the ones, for example, that drive people away from your "Getting started" guide.

At https://bugout.dev, we are building a huge knowledge base of bugfixes to help developers overcome issues with the tools and libraries they depend on. We are also trying to help developer tool maintainers gain a better understanding of how their users use their software.

We have taken a big step forward this week with the release of Humbug, which helps developer tool maintainers collect usage and crash reports from their users only with their full consent.

Here is a short video showing how to use Humbug in your Python developer tool:

Find out more about how to set up Humbug on the GitHub repository:

bugout-dev / humbug

Get usage metrics and crash reports for your API, library, or command line tool.

humbug

Humbug helps you understand what keeps users coming back to your developer tool as well as any friction they experience.

Humbug lets you collect basic system information and crash reports while respecting your users' privacy. In addition to getting reports, you get to be GDPR-compliant from day one.

Humbug is currently available in the following programming languages:

Python
- System information report
- Error traceback report
- Packages available in the current Python process report
- Logs report
- Environment variables report
- Custom report with full content control
Go
- System information report
- Panic report
- Custom report with full content control
Javascript
- System information report
- Error traceback report

If you would like support for another programming language, please create an issue.

Using Humbug

Setup

Follow the instructions in the Getting started with usage and crash reporting guide.

From development to production

We recommend generating one token for development and testing and using different…

View on GitHub

Go and Javascript support are coming soon!

A better command history

Neeraj Kashyap — Sat, 13 Feb 2021 00:03:20 +0000

If you spend a lot of time at the command line, you know how powerful the command history is. Especially if you use Ctrl + r, and other advanced history-related features.

As powerful as the command history is, it could be a lot better.

What's wrong with history?

The two biggest problems with the command history are:

It is very difficult to build a central command history across multiple terminal sessions and multiple computers.
Command histories only store the commands themselves. They do not store the output of the commands or even the exit code.

Command history is great when running commands that you use frequently and are already quite familiar with.

If you want to perform an operation that you are unfamiliar with and very rarely use, your command history tends to be useless. Especially when you can't remember which terminal or even machine you ran that operation on previously.

Storing command history in Bugout

For context, Bugout.dev is an automated knowledge base for developers and software teams. It is free to use for individuals and for small teams (up to 5 members).

Bugout has a very simple API that you can use to add knowledge into your knowledge base and it offers search over your knowledge base out of the box.

Bugout comes with a command line tool called bugout:

bugout-dev / bugout-go

The Bugout Go SDK

bugout-go

This repository contains the Bugout Go client library. It is also the home of the bugout command line tool.

Installation

Pre-built binaries

You can get the latest pre-built release of the bugout command line tool on the Releases page.

go get

If you are familiar with golang and have it installed, you can also get bugout using:

go get github.com/bugout-dev/bugout-go

This will install the Bugout client library in $GOPATH/src/github.com/bugout-dev/bugout-go.

It will also put the bugout command line interface in your $GOPATH/bin directory.

Using the bugout command line tool

Access tokens and the BUGOUT_ACCESS_TOKEN environment variable

Many bugout commands require you to pass a Bugout token as the -t/--token argument. You can generate an access token by logging into https://bugout.dev/account/tokens.

Once you have generated an access token, if you would like bugout to use it automatically without having to explicitly pass it using -t/--token…

View on GitHub

bugout has a command, bugout trap, which captures a command from your terminal into your knowledge base. bugout trap not only captures the command, but also its:

exit code
stdout
stderr
(optionally) the environment variables it was run with
(optionally) any additional tags you would like to associate with it

Below are some examples of how I use bugout trap to turn my knowledge base into a centralized command history:

Example: Capturing API requests, responses, and headers with curl

The link takes us to the new knowledge base entry:

Example: Image format conversions

I rarely use ImageMagick directly - and never remember that its shell command is convert!

I only had an SVG version of the cover image for this post and wanted to use ImageMagick to transform it to PNG (dev.to does not accept SVG images).

I searched my history for occurrences of svg or png: history | grep svg and history | grep png. Unfortunately, I only found a bunch of commands where I opened image files: commands like xdg-open <filename>.svg and xdg-open <filename>.png.

This time, when I figured it out, I used bugout trap:

The entry:

Best of all, we can search this history from our command line and view it in our editor:

Note how we can filter the search by exit code using "tag:exit:0" in the search query. We can also restrict our search only to invocations that caused errors by using the query "!tag:exit:0". This is very powerful when working with teammates to debug an issue with a command.

Example: Tracking production database migrations

At Bugout, we are heavy Postgres users. Our backend is written in Python. We use SQLAlchemy to interact with our database and Alembic to manage our database migrations.

We run migrations manually in production, and bugout trap is very useful to keep a record of when each database migration was run so the whole team can see. This is how we use it:

bugout trap --title "Database migration: $(alembic history | head -n1)" --tags db,migration,prod -- \
    alembic upgrade head

Trapping commands we run in production into our team's knowledge base helps us stay synchronized across 15 time zones.

Try it yourself

If this workflow sounds compelling to you, you can try it out yourself. Bugout is free for individual developers (and for small teams), so all you have to do is sign up for an account at https://bugout.dev.

Reach out to me in the comments if you need any help setting up!

Thank you, @nickmaris , for suggesting I create this kind of content.

Bugout.dev - a knowledge base that builds itself

Neeraj Kashyap — Wed, 03 Feb 2021 20:30:39 +0000

Bugout is a knowledge base for programmers. It's a great place to take notes for yourself, or to share with your team.

You can integrate Bugout with your code editor, your GitHub repositories, and your Slack workspaces. This makes it easy easy to add knowledge to your knowledge base directly - and automatically - from those sources.

You can sign up and start using it for free at https://bugout.dev.

After you sign up and verify your email, you should see a Personal journal with a document on how to get started. It will also help you set up the integrations with VSCode, GitHub, or Slack if you are interested in those.

Quick demo of the VSCode integration:

Quick demo of Slack integration:

The power of Bugout is in collaboration, and you can create small teams (of up to 5 people) for free to share knowledge with.

You can create teams on your Account screen (get there by clicking on your username in the top right), and then add these teams to journals on the journal settings screen (little gear icon next to each journal name). Really useful for shared projects, especially with our Slack integration.

I'd really appreciate any feedback that you have, and will work quickly to fix any issues. Thank you!

Entropy for developers

Neeraj Kashyap — Sun, 20 Dec 2020 17:27:29 +0000

Entropy is a mathematical concept which quantifies the amount of information in an observation.

This makes entropy useful to you, as a programmer, whenever you are working with data. When working with large datasets, entropy-based heuristics can tell you where to focus your attention to get the most bang for your buck.

At Bugout, we have a tagging system for knowledge bases very similar to the tags that DEV uses for their posts. We calculate the informative power of each tag using entropy. We use these calculations to help our users categorize their knowledge and navigate their knowledge bases more efficiently. Entropy helps us power these user experiences with simple calculations we do in our database as opposed to using complicated machine learning models.

Entropy and related information theoretic notions are very useful in machine learning, as well.

Decision tree algorithms tend to use entropy to learn effective branching structures.

When training neural networks, entropy is usually incorporated into the training process via a loss function. Cross entropy, for example, is a commonly used loss function when training classifiers. A good understand of entropy not only helps you understand the behavior of such models, it also helps you tune your models more effectively.

If you are curious to learn more about entropy and start using it to improve your data-related work, this post will get you started.

Towards the end of the post, I also describe some instances during my career where the concept of entropy has come in handy.

This post assumes a basic understanding of probability and some familiarity with mathematical notation.

Definition

Say you have a probability distribution with outcomes $\ldots, n$ , each of which can occur with probabilities $p1,p2,…,pnp_1, p_2, \ldots, p_n$ .

The entropy of this probability distribution is:

H(p_1, \ldots, p_n) := -\sum_{j=1}^n p_j log(p_j)

Properties

Much more important than the definition are the properties that it embodies.

First, entropy increases with the number of (equally likely) outcomes:

H(1n,…,1n)<H(1n+1,…,1n+1)H\left(\frac{1}{n}, \ldots, \frac{1}{n}\right) < H\left(\frac{1}{n+1},\ldots,\frac{1}{n+1}\right)

Example: Coins and dice
H(1/2,1/2)<H(1/6,1/6,1/6,1/6,1/6,1/6),H(1/2, 1/2) < H(1/6, 1/6, 1/6, 1/6, 1/6, 1/6),H(1/2,1/2)<H(1/6,1/6,1/6,1/6,1/6,1/6),
which says that the roll of a fair die contains more information than the flip of a fair coin.

Second, entropy is maximized when outcomes are equally likely:

H(p_1, \ldots, p_n) < H\left(\frac{1}{n}, \ldots, \frac{1}{n}\right)

for any probability distribution

(p1,…,pn)(p_1, \ldots, p_n)

for which some

pj≠1/np_j \neq 1/n

This feeds into our intuition that the amount of information in an event is the amount of uncertainty it resolves for us.

Example: Fair coins and biased coins
H(1/4,3/4)<H(1/2,1/2), H(1/4, 3/4) < H(1/2, 1/2), H(1/4,3/4)<H(1/2,1/2),
which says that we get much less information when flipping a coin which is biased to land on heads only 25% of the time than when flipping a coin.

Units

One of the choices you have to make when calculating the entropy of a probability distribution is the base you will choose for the logarithm.

This choice doesn't usually matter much. This is because changing the base only changes the calculation by a constant factor. Since you usually use entropy to make comparisons between probability distributions, multiplying all calculations by a constant factor doesn't affect how they are placed relative to each other.

Consider the calculation of

-\left[\frac{1}{2}log\left(\frac{1}{2}\right) + \frac{1}{2}log\left(\frac{1}{2}\right)\right] = -log\left(\frac{1}{2}\right)

If you take the logarithm to the base of 2, you get

H (1/2, 1/2) = 1

If you generalize this calculation to powers of 2, you get:

H(1/2n,…,1/2n)=nH(1/2^n, \ldots, 1/2^n) = n

For any probability distribution with equally likely outcomes, using a base of 2 on the logarithm, entropy counts the minimum number of bits required to represent the possible outcomes from the distribution.

Claude Shannon, father of information theory (pictured above), made this connection more explicit:

In information theory, Shannon's source coding theorem establishes the statistical limits to possible data compression for data whose source is an independent identically-distributed random variable, and the operational meaning of the Shannon entropy.

View on Wikipedia>

The important takeaway here is that you can think of entropy as answering the question, "What is the smallest number of bits I need to represent a sample from this probability distribution?"

Real-world applications

I list a few examples from my own career where knowing about entropy was extremely useful.

Bugout tags

Bugout is a knowledge base that accepts knowledge from a variety of sources (Slack, GitHub, IDEs, terminals), and makes it discoverable to users wherever they work.

This is what it looks like when someone adds knowledge to their knowledge base from Slack:

We place no restriction on the number of tags a user can associate with an entry in their knowledge base. Instead, we rely on intelligently suggesting tags to users when they create content and when they are searching for knowledge.

The first iteration of our tag suggestions were based on how frequently tags were used in a given knowledge base. Tags which appeared more commonly were higher up in the list of suggestions.

The engineering tag shows up on close to 80% of the entries in our team knowledge base. It certainly doesn't help too much when searching through a knowledge base. As developers repeatedly accept engineering as a suggested tag when adding content into their knowledge base, the informative power of the engineering tag just decreases as its frequency of use increases.

We have found entropy to be a much better quantification of the importance of a tag than frequency. It delivers a much better user experience especially for users navigating a knowledge base.

For each tag in a knowledge base, we think of each entry as an observation of whether or not the tag appears. With $p_{tag}$ as the proportion of entries having the given tag, the entropy of the tag is $H(p_{tag}, 1-p_{tag})$ .

The best thing about this is that it's easy to calculate in our Postgres database. Here is an example of an (unoptimized) entropy query for tags in Bugout knowledge bases (internally, knowledge bases are represented as journals):

select tag,
    journal_id,
    num_entries_with_tag,
    num_entries,
    num_entries_with_tag / num_entries as proportion_entries,
    case num_entries_with_tag/num_entries
        when 0 then 0
        when 1 then 0
        else -(num_entries_with_tag / num_entries) * ln(num_entries_with_tag/num_entries) - ((num_entries - num_entries_with_tag) / num_entries) * ln((num_entries - num_entries_with_tag)/num_entries)
    end
    as entropy
from (
        select journal_entry_tags.tag as tag,
            journals.id as journal_id,
            cast (count(journal_entries.id) as float) as num_entries_with_tag,
            cast (
                (
                    select count(*)
                    from journal_entries
                    where journal_entries.journal_id = journals.id
                ) as float
            ) as num_entries
        from journal_entries
            inner join journal_entry_tags on journal_entries.id = journal_entry_tags.journal_entry_id
            inner join journals on journals.id = journal_entries.journal_id
        group by journals.id,
            journal_entry_tags.tag
    ) as intermediate_stats;

Outperforming Amazon, Google, and Microsoft APIs on image classification tasks

I used to run the AI team at Doc.ai. At the time, we built image models that provided our users and their medical providers with valuable insights about their physical and mental health.

Being serious about user privacy, especially when dealing with sensitive health data, we built our classification models to run on our users' mobile phones. This meant we couldn't just plug into AWS Rekognition, the Google ML APIs or Microsoft Cognitive Services, as that would leak user health information to Amazon, Google, and Microsoft.

We set out to train models which were competitive with the major cloud APIs by collecting a LOT of relevant images (10s of millions) and boostrapping our model with the classifications returned by the cloud APIs.

At the end of the day, we were able to outperform the cloud APIs in the many cases because of how we used entropy to our advantage.

You can think of image classification results as a probability vector over the possible classes represented in an image or in a segment of that image. We calculated the entropy of the classification results that the cloud APIs returned for each of the images in our dataset. This calculation quantified the amount of uncertainty their models had about their results. This uncertainty decomposed into two components:

The quality of the cloud API model in question
The quality of the image for the given classification task

For a given cloud API, we could take the entropy (largely) as a measure of image quality. Thus, filtering on entropy allowed us to cheaply build very high quality training and evaluation datasets without contracting any kind of labeling service (which would have been exorbitantly expensive at the scale of data we were working with).

Entropy heuristics were also very important in helping us understand the hyperparameter spaces of the models we trained. For each model, during our hyperparameter tuning phase, we would run ~30 experiments concurrently, and it cost us thousands of dollars a week. As a tiny startup, it was important that we be efficient about hyperparameter tuning. Simple entropy and cross entropy calculations over our dataset and classifications performed by various model checkpoints helped us narrow in on high-performing hyperparameters very quickly.

My team at Doc.ai released a lot of this work as the open source Tensor/IO framework (which also allows you to train models across many user devices using federated learning):

doc-ai / tensorio

Declarative, On-Device Machine Learning for iOS, Android, and React Native. Deploy. Predict. Train.

Introduction

Tensor/IO is a lightweight, cross-platform library for on-device machine learning, bringing the power of TensorFlow and TensorFlow Lite to iOS, Android, and React Native applications. We now also support PyTorch on Android and will be bringing support for PyTorch to iOS soon. Tensor/IO does not implement any machine learning itself but works with an underlying library such as TensorFlow to simplify the process of deploying and using models on mobile phones.

Declarative

Tensor/IO is above all a declarative interface to your model. Describe the input and output layers to your model using a plain-text language and Tensor/IO takes care of the transformations needed to prepare inputs for the model and to read outputs back out of it, allowing you to focus on what you know instead of a low-level C++ interface.

On-Device

Tensor/IO runs on iOS and Android mobile phones, with bridging for React Native, and it runs the…

View on GitHub

Happy to answer any questions and continue the discussion. Just leave a comment below. :)

Bugout: Add checklists your GitHub pull requests

Neeraj Kashyap — Mon, 07 Dec 2020 20:57:13 +0000

Sometimes, you need human oversight on your pull requests before you can merge them.

This is especially true for pull requests that change database schema, introduce new environment variables, or introduce changes with security or legal implications.

Bugout adds a checklist to every pull request. To add a check to your checklist, just mention @bugout-dev on a PR comment:

@bugout-dev check require <something important>

To cross an item off the checklist:

@bugout-dev check accept <something important>

Here is a live PR where you can try it out: https://github.com/bugout-dev/github-demo/pull/2

Actively looking for feedback - missing features, similar to existing apps, anything. Thank you!

Break out of the deployment death loop

Neeraj Kashyap — Thu, 03 Dec 2020 07:07:33 +0000

Continuous integration services like Jenkins, Bitbucket Pipelines, CircleCI, GitHub Actions, Gitlab CI/CD force extreme automation. They require your continuous integration flows to run non-interactively in remote environments. Most of these tools do not allow you to interact with the continuous integration environment.

All of these tools require users to define deployment workflows in custom definition languages. It is very common in my experience that users of these services can only run their deployment workflows using the service in question.

When something goes wrong with a deployment, debugging takes the form of a deployment death loop:

Make changes locally (mostly to continuous integration configuration files).
Push to the appropriate branch on your git host (usually master).
Wait for a job to start on your continuous integration service.
Wait minutes or even tens of minutes following the logs to see if your changes fixed the issue.
Repeat this procedure until you have solved the problem.

Things are even worse when you have to debug configuration that lives in a database or on a configuration management system such as Vault or Parameter Store. If you can’t manually trigger a workflow, you are forced to push superficial changes to your git repository (like adding punctuation or white space to your README) to see if your changes worked.

How to break out of this death loop?

Two suggestions:

First, any developer should be able to deploy from their local environment (provided they have the required level of access). The continuous integration workflow definition should be secondary to this mechanism. If possible, it should make use of this mechanism.

For example, at Bugout, our services are deployed by a bash script which can be executed on our servers via SSH. Even in production, if a developer has access to the production servers, they can run a deployment by simply running a git pull and then the deployment script over SSH. This is useful not only when debugging deployments but also when our continuous integration service goes down, which has been known to happen.

Second, embrace checklists. Have a checklist of external actions that must be taken before a change can be deployed. This checklist can include things like running database migrations, setting environment variables, or modifying a load balancer.

It’s okay to check items off such a checklist manually. Manual checklists may invalidate ideas of fully automated deployments. This is alright. Deployments that require manual steps should be performed with human oversight.

The general principle is to favor interactivity over automation. Make your deployments interactive by default. You will not regret it.