DEV Community: Stanislav(Stas) Katkov

Software License management with Polar.sh

Stanislav(Stas) Katkov — Sun, 11 May 2025 17:48:12 +0000

This article would have been incredibly helpful to me just a month ago—so here it is, primarily for other developers who, like me, don't usually write software that's meant to run in non-server environments.

I just introduced licenses to one of my apps—DevTUI. There's another app that suffers from my code, PoshTUI, and it might require something similar soon. I suspect all of this will come in handy again in the future.

Requirements

Starting out, I only knew what needed to be built: a licensing solution for a one-time payment product. A solution should support the following requirements:

Ability to issue free licenses to early supporters of PoshTUI
Ability to limit each license to three machines (or less)
Allow customers to manage their licenses without a need to contact support(e.g. deactivate licenses or view receipts)
Should work offline; license validation should not run every time the app starts

Licensing Server

I'm using Polar.sh as my payment and licensing provider. I'm skipping the part about how I ended up with them—no interest in badmouthing any competitors.

Rolling your own solution on top of Stripe is also possible. But that wasn't an option worth pursuing for me. A 4% cut of future revenue seemed like a fair trade-off to avoid dealing with payment handling and license infrastructure.

Polar team has been incredibly helpful. When I first contacted them, they didn’t even have a Go SDK, but they published one within two days. They also responded with useful tips and shipped fixes—again, all within 48 hours of my initial email.

Polar.sh provides me with all the server side niceties.

Offer a way to define a product that can be paid one time
There is also Discounts for those early supporters and friends
Customer portal
Checkout links for a product, even one with a 100% discount

But the client-side implementation is up to me.

License Activation

The app can do three license related actions - active, validate and deactivate. All these action are currently implemented in a CLI interface.

Commands for activating, validating, and deactivating licenses  

Usage:  
 devtui license [command]  

Available Commands:  
 activate    Activate a license  
 deactivate  Deactivate a license  
 validate    Validate a license

Activate

But I expect that most users will probably invoke only one command - activate.

devtui license activate --key=DEVTUI-2CA57A34-E191-4290-A394-XXXXXX

I'm intentionally not using any ENV variables — this would require an additional library, and I don’t want users polluting their .zshrc or .bashrc with extra variables.

After activation, the key and related info are stored in a license.json file.

During activation, it’s possible to provide "conditions". In our case, we use the MAC address to associate the license with a specific machine. The licensing server enforces a maximum of 3 machines per license.

Validate

This action uses the license.json file created during activation. While the CLI command simply checks that the license is active, this validation logic is also embedded directly in the app.

Here's a rough outline of the validation logic:

It's checking that hash sum matches expectations
If it doesn't match -> We re-validate license with server
If it does match -> We check if it's time to check license with a server.
During a license check with server, if MAC address is not similar to one we used during activation - validation will fail. This would prevent people from just moving license.json file to another machine.

Software attempts to validate license with a server every week, this time frame is completely arbitrary.

Deactivate

This also uses the license.json file created during activation.

While it's already possible to deactivate a license via the customer portal, having it available in the CLI felt important. Two use cases come to mind:

Running the app in CI environments where machines constantly get recycled
Switching from one machine to another

Identifying a Machine

To my knowledge, most software uniquely identify a machine by its MAC address. It's important to try and find the physical, manufacturer-assigned MAC address, use it during license activation, and later rely on it during validation.

However, not all MAC addresses are created equal. There are also Locally Administered Addresses (LAAs), these are not the same because they could be:

Manually assigned by a network administrator
Often used in virtual machines
Common in network virtualization scenarios

It's possible to identify those by checking the second least significant bit of the first octet:

If the bit is 1, it's locally administered
If it's 0, it's universally administered

Examples:

02:00:00:00:00:00 → locally administered
00:1A:2B:3C:4D:5E → universally administered

License File

We use the github.com/adrg/xdg library to determine where to store the license file.

This library implements the XDG Base Directory and XDG User Directory specifications, offering a standard mechanism for storing application state, data, or configuration across multiple OSes—Windows, Linux, Plan 9, and macOS.

In the app’s case, we rely on XDG_DATA_HOME, or fallbacks as per OS:

Unix	macOS	Plan 9	Windows
~/.local/share	~/Library/Application Support	$home/lib	LocalAppData

Example 'license.json':

{  
 "hash": "7394991a704a054096f4484d8a19f9ac66e3e8c98b68603652feadc785a364f2",  
 "license_key_id": "DEVTUI-2CA57A34-E191-4290-A394-XXXXXX",  
 "activation_id": "2ee71107-2ecb-4172-aff7-ceaa6b2f7cef",  
 "next_check_time": "2025-05-12T23:18:12.7724912+02:00",  
 "last_verified_at": "2025-05-05T23:18:12.772488825+02:00"
}

The hash is generated from multiple values and is used to verify that the license data hasn't been tampered with. If the hash doesn't match, the app treats the license as invalid.

I won’t go into the full details of what goes into the hash, but in general:

MAC address
License key
Activation time
Next check timestamp
A salt value (random string)

This mechanism isn’t hacker-proof, but the effort required to crack it should outweigh the $40 one-time price tag.

Final notes

This is my first time implementing client-side license logic for an app, so I’ve probably missed some edge cases. But if anyone has some feedback or recommendations, please reach out.

POSH 0.9: Experimental support for gems

Stanislav(Stas) Katkov — Sat, 14 Dec 2024 00:37:00 +0000

One of the biggest releases in a while!

The main theme for this release is experimental support for gem docsets.

This required significant changes on all fronts - both back-end and client (TUI).

There is now a “POSH builder” project that handles all the heavy lifting of building Markdown documentation. It’s currently a semi-automatic process that requires some manual setup. However, eventually, the entire process could be automated; I’ve already managed to (web)hook into RubyGems.

Most of the work has been with the TUI client.

The TUI is able to parse the Gemfile.lock file, download all documentation it can find, and present it through a TUI view. While the flow of this entire process is not yet properly defined, the features for each step seem to be working.

Here’s a quick rundown of new features:

posh docsets to download all available docsets.
posh docsets --clear will remove all docsets from the system.
posh gemfile to view project gems with the ability to search and pick a docset for review.

Additionally:

The client now stores a local cache to avoid redownloading docsets multiple times.
There are many stability improvements to the client itself and better error handling.
Underlying libraries have been updated; the biggest win here is that the markdown renderer has received numerous improvements. This should generally help to present documentation better.

New version of poshtui.com live

Stanislav(Stas) Katkov — Mon, 09 Dec 2024 16:04:00 +0000

POSH TUI is preparing for a major release, but before announcing that, I gave the homepage a fresh look.

https://poshtui.com/

TUI’s have been around for a long time, so I went with a timeless brutalism design. This is my first attempt at expressing the type of design that I’m looking forward to bringing to the web.

The website now features a proper landing page, changelog, and feedback portal. Once the core features of this tool are completed, user feedback will be guiding the priorities for this project.

There is still some work needed on this website, in short term I’ll add pricing and FAQ sections. But this website is already better than the previous one, so I decided to hit that publish button.

Marrying Tailwind with Jekyll

Stanislav(Stas) Katkov — Mon, 30 Sep 2024 21:04:47 +0000

Ruby has more than one beloved framework - there's also Jekyll.

Jekyll is a simplistic framework for static websites that originally sparked the static website and JAMstack movements. While there are many similar frameworks with more features, Jekyll remains one of the simplest on the market. It has been somewhat forgotten and hasn't evolved much lately, to the point where some people decided to take matters into their own hands and fork this framework into something called Bridgetown.

Surprisingly, in many projects, I don't really need more than what Jekyll can offer. So, I remain a loyal user. Until recently, I didn't have much to complain about, not until I decided to rebuild poshtui.com with Tailwind CSS. The only way to add Tailwind and Heroicons was through a JavaScript bundler, and I'm no longer used to that approach.

As it sometimes happens in the Open Source world - the stars aligned in the right way. I stumbled upon work by @crbelaus on jekyll-tailwind. He demonstrated that it's possible to integrate both, but his solution was lacking. I then found work by @flavorjones who extracted the tailwindcss-ruby gem from the Rails core gem.

To keep things short, over the next couple of weeks, I worked to improve the integration of Tailwind and Heroicons with Jekyll. An example of this work can be found in the jekyll-tailwind-cli-template. The current setup experience with jekyll-heroicons and jekyll-tailwind is not exactly a walk in the park, but it's much easier than working with JavaScript bundlers.

jekyll-tailwind

tailwindcss-ruby will be used in Rails, so this gem now depends on it as well.

This work brought everything you would expect from Tailwind:

Minification
PostCSS support
All the tweaking of tailwind.config.js
Ability to provide input/output files
People can move to Tailwind v4 on their own accord

@flavorjones did a wonderful job here; it was a breeze to write the integration! @crbelaus was a great collaborator, and he offered me to step up as a maintainer.

jekyll-heroicons

Besides the obvious way to distribute SVGs without a JavaScript bundler, the gem offers small quality of life improvements. Any icon can be used just by typing a name:

{% heroicon bell %}

You can define a default variant in _config.yml, but you can also overwrite it on a per-icon basis.

heroicons:
  variant: 'solid'

It's possible to define default classes that will be applied to all icons:

heroicons:
  default_class: {
    solid: "size-6",
    outline: "size-6",
    mini: "size-5",
    micro: "size-4",
  }

But it's also possible to turn that off, provide your own classes, or any other tags for the SVG element:

# Disable default_classes
{% heroicon bell disable_default_class: true %}
# Provide additional classes
{% heroicon bell class: "text-red-500" %}
# Provide any other attribute for svg html element
{% heroicon bell class: "text-red-500" aria-hidden: true height:32 aria-label:hi %}

There's no need to copy SVGs one by one or reach out to a JavaScript bundler!

Ending notes

Jekyll might have a chance for a revival in light of recent trends in the Ruby ecosystem. Ruby developers are eagerly looking for ways to simplify and condense complexity. And now, there's a simpler way to use Tailwind CSS with this amazing framework. I hope you'll find it useful.

Munster - Webhooks processing engine for Rails

Stanislav(Stas) Katkov — Tue, 18 Jun 2024 16:34:39 +0000

By the time of writing this article, I had already written webhook processing logic at least 10 times for different companies and clients. Together with Julik we have implemented one recently at our current place of employment. And guess what? Once a second service had to be built, it needed to accept webhooks too.

Our combined experience in the ingestion of webhooks had already produced a reasonable generic solution. Should we just copy some files over to a microservice and duplicate that code? Nah, let's save the world from wasting those countless hours of re-implementing webhooks over and over again! This darn well could be a gem!

Enter "Munster"

Munster is a webhooks processing engine for Rails. And you heard that right: it's only accepting and processing, not sending.

Services that send webhooks would love it if you accepted those as fast as possible and responded with a 200 status code. Pretty much always, actually - because if your service refuses to accept the webhooks they send you they will likely stop retrying. Important business events could then get lost, you would need to examine the retry policies of every webhook sender, etc.

And if you didn't manage to receive a webhook correctly, it is usually a hassle to ask the sending service to re-deliver the missed webhooks to you in bulk. Some (like Stripe) provide facilities to replay their event feed in a "pull" manner, but most do not.

How to make that happen well? The answer is pretty simple: do not process the webhook inline. Verify it is coming from the sender (most good webhooks use some form of signature that you can verify using a shared secret or a public key), and spool the webhook for processing using your background jobs. Processing webhooks asynchronously has many other advantages:

Equally manage load on servers.
Background process is not subject to any request timeouts.
If processing fails, we will have a webhook safely stored in our database for later re-processing or analysis.

Munster is an engine which will provide you the following facilities:

A small abstraction for building webhook receiving handlers. A handler is a small object with just a handful of methods that you need to implement - normally those would be valid? and process
A Rails engine that can be mounted into your application and handles webhooks from multiple senders together - every webhook sender would get its own handler definition. So you would have a handler for Stripe, a handler for Revolut, a handler for Github - and any other services you might want to receive webhooks from
A background job class which calls process asynchronously, from your job queue.

Getting started

As with any other gem, you'd first want to add gem 'munster' into a Gemfile and run bundle install. This would add a generator task to a Rails project, so run rails g munster:install. Then run rails db:migrate so that the table used for received webhooks gets created.

It will create the required migration and an initializer file at app/initializers/munster.rb. This initializer would expect you to define at least one active_handlers hash for it. I will quickly run you through the process and give you an example.

Mounting

Munster is an engine and a Rack app, so it can be mounted in your Rails routes file. Inside config/routes.rb you can mount a webhook engine on a subdomain (like webhooks.yourdomain.com/:service_id):

scope as: "webhooks", constraints: "webhooks.yourdomain.com" do
    mount Munster::Engine, at: "/", as: ""
end

Or on the main domain (yourdomain.com/webhooks/:service_id):

mount Munster::Engine => "/webhooks"

Having a separate subdomain for receiving webhooks can be useful if you have very specific security requirements, or if you would like to have a separate load balancer fronting your webhook receiving endpoint.

Defining a Handler

The next step is to define a webhook handler. For the sake of an example, let's create a handler for Customer.io at app/webhooks/customer_io_handler.rb. The handler will take care of handling two specific metrics from Customer.io - subscribed and unsubscribed. We want to store a local value per user called "subscribed" – once a user unsubscribes using customer.io, we want to record this information in our app database. Same for when a user subscribes.

class Webhooks::CustomerIoHandler < Munster::BaseHandler
    def process(webhook)
      return if webhook.status.eql?("processing")
      webhook.processing!

      # The webhook body gets stored as bytes, so senders may deliver
      # you binary data on the endpoint - it does not have to be JSON
      json = JSON.parse(webhook.body, symbolize_names: true)

      case json[:metric]
      when "subscribed"
        ActiveRecord::Base.transaction do
          user = User.find(json.dig(:data, :customer_id))
          user.update!(subscribed: true)

          webhook.processed!
        end
      when "unsubscribed"
        ActiveRecord::Base.transaction do
          user = User.find(json.dig(:data, customer_id))
          user.update!(subscribed: false)

          webhook.processed!
        end
      else
        webhook.skipped!
      end
    rescue => error
      webhook.error!

      raise error
    end

    def extract_event_id_from_request(action_dispatch_request)
      JSON.parse(action_dispatch_request.body.read).fetch("event_id")
    end

    # Verify that request is actually comming from customer.io
    # @see https://customer.io/docs/api/webhooks/#section/Securely-Verifying-Requests
    def valid?(action_dispatch_request)
      xcio_signature = action_dispatch_request.headers["HTTP_X_CIO_SIGNATURE"]
      xcio_timestamp = action_dispatch_request.headers["HTTP_X_CIO_TIMESTAMP"]
      request_body = action_dispatch_request.body.read
      string_to_sign = "v0:#{xcio_timestamp}:#{request_body}"

      hmac = OpenSSL::HMAC.hexdigest("SHA256", 'customer_io_webhook_signing_key', string_to_sign)

      Rack::Utils.secure_compare(hmac, xcio_signature)
    end
end

We're rewriting three methods from BaseHandler:

The valid? method verifies that the webhook indeed comes from customer.io. This method runs inline, before we persist the webhook.
The process method defines how we want to process data
The extract_event_id_from_request method defines how to extract an ID from a webhook, and by default it will generate a random UUID.

The extract_event_id_from_request is a very important feature. A lot of webhook senders send you unique webhooks, but those webhooks may arrive out of order, or arrive twice. Networks being unreliable, the retries being too aggressive at the sender side, you name it. Munster will use this event ID to deduplicate your webhook - if you receive the same data more than once, just one webhook will be persisted and processed. This will also protect your infrastructure if, for some reason, the webhook sender happens to DoS you with repeated deliveries.

There are more methods that could be redefined and tweaked, but these three are most commonly used.

And lastly we need to mount our handler. We use the "service ID", which will also be the URL path component for your webhook. For our handler, we will use "customer-io" (our URL to paste into the Customer.io configuration will thus be https://your-app.example.com/webhooks/customer-io):

require_relative '../../app/webhooks/customer_id_handler.rb'

Munster.configure do |config|
    config.active_handlers = {
        "customer-io" => Webhooks::CustomerIoHandler
    }
end

Final note

We went through the basic webhook endpoint setup with Munster, but it offers a bit more than that. This little framework is especially beneficial in cases where you have more than one webhook endpoint. Once you set it up in your project , it's just a matter of defining a single handler to accept new types of webhooks.

The core of this is already battle-tested at cheddar.me, but we're still working on fleshing out the details and would love your feedback! You can find Munster at https://github.com/cheddar-me/munster

Interview with Greg Molnar - Rails developer and penetration tester

Stanislav(Stas) Katkov — Wed, 05 Jul 2023 09:19:04 +0000

One of the sections in upcoming Linting Ruby book is about automating security checks in ruby. I never specialized on security in Rails application, nor did Lucian. We didn't felt qualified to give our opinions as general advice on such an important topic. So, we decided to conduct a thorough research and reach out to people who know better.

So we present you interview with Greg Molnar, who is a Rails developer for 13 years and OSCP-certified penetration tester. You might know him as an author of Spektr a static code analysis tool that finds potential vulnerabilities in Rails and work in progress book titled "Secure Code Review for Rails developers".

Stas: How did you get interested in security?

Greg: I ended up working for a company that did penetration testing and security consultancies. They hired me as a Ruby on Rails developer to work on their internal applications.
They were always telling me about all the cool things they do and got me infected with all of that. I wanted to learn more about the world of InfoSec. Unfortunately, I had to leave that company and England, but I took a penetration testing course that they recommended to me.

This course is supposedly one of the hardest out there, it's called Offensive Security Certified Professional or OSCP for short. It's a very hands-on penetration testing course. For the exam, they give you five IP addresses and 48 hours. Your task is to find vulnerabilities, and get root access on all of them, and write a report.
And since then, I have been splitting my time between doing development and security work.

Stas: Can you tell us more, what your security work looks like... Do you help with Rails code reviews or you try to break into their systems?

Greg:I do penetration tests, not only for Ruby on Rails projects. I don't mind the underlying technology actually.
Most of my work is for companies with a requirement of being PCI compliant. Part of that is to have yearly penetration test on their application. For instance if a company works in the financial industry, their customers and partners likely require them to be PCI compliant due to the sensitive data they handle. But I also work for companies getting a test done to just cover their bases.

Stas: Based on your experience which are the most common problems? Most common security issues? Is it on application level or because infrastructure is misconfigured?

Greg: Assessing infrastructure testing is not really something that I typically do. In fact, I can't remember when I last did it because usually, these tasks are separated in companies. There's the development team which handles the web penetration test - the application's test. Then, there's the operations team, which is responsible for their own security tests and they hire their own consultants.

The most common vulnerability, I believe, is still Cross-Site Scripting(XSS). XSS is very frequent, but often, I find cases where proper configuration via X-XSS-Protection
and alike headers prevent full exploitation, even when developers make mistakes. So, vulnerability is there, but nobody can exploit it. I can run the script, but it won't execute - an exception is thrown in the browser console saying you are not allowed to do this.
Besides that, authorization issues are the most common ones. When a user can access or do things they shouldn't be permitted.

Stas: There's a gem called 'bundle-audit' that checks if any of your dependencies have known security issues. However, we're seing more and more attacks these days involving third-party libraries. Is it enough to rely only on 'bundle-audit' to secure your third-party libraries?

Greg: 'bundle-audit' works same way as other alternatives as far as I know, so relying on that should be enough I think. But if someone hijacks a ruby gem and publishes a version with a malware, nothing can really save you except your own manual due diligence.
I recommend to always cross-check the repository and rubygems for updates when you upgrade a gem. When you get an new version of a gem from rubygems, go to GitHub to search for the same tag. If you can't find that tag on GitHub, you have to ask why is that. When I also always the changelog on GitHub if there is one. I consider it a red flag, if I can't find the tag or a changelog for the version I am getting from rubygems.

Stas: Do you use dependabot?

Greg: No, to be honest, never particularly liked this tool, even before GitHub acquired it. Now, I don't favor GitHub either due to the Microsoft acquisition.
I strive to maintain my independence and avoid reliance on a tool unique to GitHub. For instance, with bundle-audit, I can take my repository anywhere and still use that tool.

Stas: So you don't stay on a bleeding edge of gems and probably have some lag time before you'll update. Since there is a need to check every update for every gem?

Greg: If there's a security update, and it is likely exploitable in the apps I work on, I update immediately. As for the regular gem updates, I batch them and do upgrade days when I upgrade as many as I can.

Stas: When you're developing Rails applications, as someone who is so security-minded, do you use any tools to help prevent security issues?

Greg: Yes, 'bundle-audit' and 'Brakeman', which are very well-known and widely used.
I actually created my own gem, similar to 'Brakeman'. Mainly because 'Brakeman' was acquired a few years back and subsequently changed their license. Under the new license, you can use it for free with your own application, but once you start running 'Brakeman' on someone else's application who is paying you, that infringes on their terms of use.
My gem, Spektr, is somewhat similar, bit it built with a different parsing library and it has a different license, so penetration testers can run it for free during assignments. But the core idea is the same as Brakeman's. It might report a few false positives, but even those might give ideas for patterns to look into during a test.

Stas: How about Rubocop? Rubocop had cops that are security related.

Greg: Yes, RuboCop does have security features, but I think they overlap with Brakeman, so I don't really use them. I use RuboCop for style enforcement, but not for the security aspects. Perhaps I should take a closer look and see how it compares to the other tools.
I believe that Brakeman and bundle-audit pretty much cover everything that can be detected automatically. However, there are other security issues, particularly those of logical nature, that cannot be flagged automatically. If you forget enforce authorization for an endpoint - no automatic tool can point it out. For that, I often recommend writing tests for every different role, to ensure they have access to only the things they're supposed to.
Use a whitelist, not a blacklist - if you add something new, it's blocked by default and can only be enabled intentionally.

Stas: It seems like we can automate some aspects of security checks, correct? But it also seems like this portion is relatively limited, implying a significant amount of manual work is still necessary.

Greg: I believe you can automate about half of security checks, but the other half still requires vigilance. You still need to write tests and make sure you're implementing all authorization and authentication rules properly, and continually check that they are functioning as they should.
Certainly, you can - or rather, should - automate some portion, primarily because it's a straightforward process. Why wouldn't you take advantage of it if it's that easy and offers a head start against potential attacks?

Stas: In your experience, you mentioned having worked with PHP and other stacks. In terms of security, how do Rails developers compare to other programmers? Does Rails significantly enhance security by default? Are Rails developers generally more knowledgeable about security than PHP developers, based on your experiences?

Greg: Rails' default security settings protect against a lot of common mistakes. It's hard to create an XSS vulnerability because everything is escaped by default, with only a few exceptions. You need to explicitly state that a string is HTML safe; otherwise, it's automatically treated as unsafe. This makes life much easier.

90% of the ActiveRecord API is immune to SQL injection, which means that you don’t need extensive security knowledge to maintain a safe environment.

I believe, however, that this could lead to people not taking security as seriously as they should, they think Rails takes over that responsibility, but that's impossible, there is no way to make a functioning framework without letting people to shoot themself in the leg if they want to.

When I'm writing code, I'm solving a problem. I don't think about security as a first class citizen. I always think about, okay, let's solve this problem in the most efficient way. And then I look at it and think about the security aspect. But it's only because I care about security, but a lot of people miss that second step.

Another aspect to consider is for instance, with PHP, if I discover a vulnerability in a PHP application, it is relatively easy to gain access to the entire server or infrastructure, because it is running on a server with a bunch of tools installed and those usually help to open a remote shell and elevate privileges. This is much easier than finding a vulnerability in, say, a Rails app running on Heroku in a container.

If you're interested in linting and ruby, you might want to follow along as we write our book on https://lintingruby.com

Thank you.

Week 6: search index and distribution

Stanislav(Stas) Katkov — Fri, 28 Oct 2022 19:53:03 +0000

This is a third progress report for a command line tool called "Posh TUI". It's an API documentation browser —like Dash, but for your console.

Software is not yeat available, but there is a limited pre-sale for a lifetime license. It includes all future updates for free, but also I'll be closely listening to feedback and will be building future roadmap based on it.

Recap

Previously I have explained, that project consists of two big parts that need to work together.

Documentation generation and hosting
TUI app that would allow to review/search documentation

Since I'm unable to render html in console, documentation have to come in form of Markdown. Two weeks ago, I embarked on a journey to build RDoc plugin to generate markdown documentation for Ruby and libraries.

Progress report

RDoc-markdown gem

Building markdown documentation and previewing it!

I'm excited to report, that I completed rdoc-markdown gem. Version 0.3 of this library produces not only markdown files, but also search index file. It probably still has some quirks that would need fixing, but this is already enough for me to move forward and build a TUI client.

Since plugin was built as open source, I made sure that it's easy to approach for collaborators. Don't be shy and take a peak! I'm always open to feedback.

While working on TUI client, I'll be showing off this plugin in different ruby communities. This may yield some great improvements and feedback or will just pass unnoticed.

Docsets delivery

Another part of puzzle is how to deliver docsets, to smplify things I decided to just distribute all docsets as one archive. Docsets includes only ruby language and Ruby on Rails frameworks that did not reach end of life. In terms of size, that's not a big package to download.

This delivery mechanism will not scale well with more docsets. But I guess that will be a good problem to have later. For now, I can just skip all of it.

I have packaged 3 latest Rails release and 3 latest Ruby releases. And currently using those for local tests.

I did apply for Free AWS credits as a startup founder and got approved within 2 days. Now I'll have credits to distribute these free of charge for myself. Thank you, Amazon.

What's next?

At this stage, most of the "back office" things are finished, and I'm ready to work on a client facing TUI app. My goal for next 2 weeks would be to deliver first prototype to existing customers.

It might not do much, but it should already be possible to play around with it and have a reliable delivery mechanism where people can validate their licenses and update their software.

Off I go to work on most exciting part of this project!

Week 4: week 4 progress update.

Stanislav(Stas) Katkov — Mon, 17 Oct 2022 09:20:22 +0000

This is a second progress report for a command line tool called "Posh TUI". It's an API documentation browser for ruby developers.

My name is Stanislav Katkov. I got really frustrated that there are no viable alternatives for Dash, so embarked on a journey to build one. If you are not using Mac/OSX or prefer command line tools to GUI tools and share a similar frustration—you might want to buy a lifetime license. This will guarantee that you'll get any updates of app for free in future and your feedback will be driving roadmap for this tool going forward.

This deal is limited to 20 licenses.

Background

This project is basically split into two major components:

CLI/TUI app to review documentation
framework to generate and host required documentation

Initially, attempted skipping documentation part and tried to build CLI/TUI app based on devdocs documentation. Ran into issues converting html to markdown on a fly and couldn't proceed with implementation.

I did research about alternatives and decided to proceed with RDoc integration instead.

Progress report

In these two weeks, I had to move my entire family from Thailand back home to the Netherlands. After two months of absence from home, we had a huge layer of dust waiting for us to clean. Most of my spare time in weekends was spent cleaning everything up.

But there is still some progress to report.

Rdoc-markdown gem

Started out, by just forking rdoc repo—effectively creating a fork with markdown generator. But that's a bad idea, easier to write a plugin!

There area only couple of things to remember while writing rdoc plugin. Rdoc will look for rdoc/discover.rb files in installed gems, this file is used to load entire plugin.

Testing a rdoc plugin is way trickier, though. How would you even test it locally? Having a robust testsuit is close to impossible to cover all possible scenarios. Which flavor of Markdown to support?

I have settled with Github Flavored Markdown as a standard. But even with this well-defined standard, it's close to impossible to validate programmatically.

Entire testing is being done manually:

Build and install new versions of a gem regularly
Store source code of ruby and Ruby on Rails project
Generate documentation with --markup=markdown --debug
Compare with HTML version

All of it is a manual tedious process, time-consuming, and fragile. I probably typed these commands a couple of hundred times at this point.

gem build rdoc-markdown.gemspec
gem uninstall rdoc-markdown
gem install /rdoc-markdown/rdoc-markdown-.gem

Gem is already open-sourced at github/skatkov/rdoc-markdown. It currently generates readable documentation in Markdown, but there are still some quirks in rendering I stumble upon.

What's next

Wrapping up rdoc-markdown:

Address all the rendering quirks that I'll find during testing
Produce a SQLite database with index of entire content
Post to various ruby related communities to get initial feedback.
Produce documentation for supported versions of Ruby and Ruby on Rails gems.

After move on to building a CLI app again (looking forward to coding some GO!). My initial goal is to integrate licenses from gumroad and to be able to analyze Gemfile (so I can derive ruby version and rails version from it).

That's all for today, folks.

Week 2: week 2 progress: building API documentation browser for command line

Stanislav(Stas) Katkov — Sat, 15 Oct 2022 11:08:27 +0000

Hello everyone,

This is my first installment of development journal for Posh TUI app. I'll do everything possible to make such notes a bi-weekly occurrence. It also helps me to be accountable and to structure my thoughts to ensure that everything is on a right track.

What do I have so far?

I already have a first prototype on my local computer, that takes HTML documentation, cleans it up and converts to markdown on a fly. Why do I go through all this trouble to have a Markdown document? Well, standard terminal can't render HTML properly, it's mostly likely to show you plain-text content of HTML file.

There is a way to switch terminal renderer, but that would make it inaccessible to a lot of people -- only to major geeks. So switching terminal renderer is not really a viable option either.

My first prototype offers a nice way to review documentation in terminal like a book. But it comes with a bunch of serious downsides, that doesn't allow me to use this approach for a developer documentation browser. Namely:
It takes a while to parse HTML and bigger pages takes noticeable time to load

I can't really build a search feature on top of that. I need to know exactly which row to scroll down to, but if we convert HTML to markdown, number of rows changes and search throws us to an entirely different section.

So, there is no way around this, I need to generate markdown and work with Markdown, no HTML. And this initial prototype will be scrapped and needs to rebuilt from ground up.

But first, we must get that documentation reliably converted into markdown.

Quest to render markdown documentation

The most important step, to get this project off the ground, is to have a way to generate markdown documentation (or plain text) so we can render it in terminal. If I do not figure this out, there is not much else I can do.

My first assumption was, that I should be able to generate markdown from the source. Same ruby and rails does now, but only tweaking a couple of parameters to generate .md files instead. YARD is being used for that and it supports any markup rdoc or yard.

Yard

While on tin can it says, that it supports markdown, unfortunately output is only html. Played around with multiple parameters and it doesn't affect the output that much.

While asking around, I've been meet with people being completely baffled by my request. "You still need to render markdown to something, and it will probably be HTML", was most common sentiment I've got.

At this stage I understood, that what seemed like an easy-peasy task, might not be straight forward as I assumed.

RDoc

RDoc doesn't really claim that it can produce anything other than HTML for documentation. But multiple gems have proved otherwise, there are libraries that could convert rdoc to PDF and if you study RDoc source code closely enough -- you'll find that it can spit out JSON blob of entire content. That json blob could easily be transformed to Markdown and can easily be used to create an index of entire content (for search and navigation section).

It seems, that most of the ruby itself, rails and all other gems I've stumbled upon work with rdoc. Even DASH clearly indicates, that all gem documentation that they store are based of RDoc.

So after a lot of studies on a matter, I've decided to write a generator for RDoc that would spit out markdown + index (as sqlite database) for me. And I'm planning to open-source that as a good netizen.

I do have a bit of paranoia, though. What if things are not as easy as they seem again? That lead to research an alternative approach.

Pandoc

Pandoc is a very popular open-source document converter.

I did a quick test trying to convert existing ruby documentation to Markdown, and it worked. But every document contained different artifacts (like issues with links, random HTML tags or whole sections that are not even needed) that required tweaking this entire process.

Pandoc offers a way to write filter to deal with that, but I need to use Lua for that! I kinda decided against it. With a wrapper like pandocomatic, it is still a workable solution. But I would rather leave it in case rdoc-Markdown idea will not work out.

What's next?

I initially focused on building an CLI app, that can render markdown documentation and present a list of existing documentation on a system. The latter, doesn't seem useful, but markdown presenter might need to wait until Markdown is actually being properly generated.

My next steps would be to:

Create rdoc-markdown generator and open source it.
Create sqlite-index-builder that works with rdoc and existing markdown (and not open source it, because it's hard to make generic implementation)
Start implementing basic CLI app -- it should validate licenses from gumroad and preferable be able to parse Gemfile (to understand ruby and gem versions in use)

That's all for now, folks!

Learning Go, was it worth it?

Stanislav(Stas) Katkov — Tue, 23 Aug 2022 14:43:07 +0000

Really wanted to build a project from scratch... so I did.

Project turned into a command line utility for daily journaling - stoic. I was mainly motivated by my desire to practice golang in a greenfield project and with an itch to scratch my own niche (meaning, finally writing software that I would use myself).

At some point, after writing in various programming languages you start to wonder... Will another language make that big of a difference to justify time spent on mastering it?

In case of golang learning curve was really low. It might be language, it might be my approach to learning new language or might be combination of two. Here are some of the things, that made it easy for me to learn it.

Open source

One of the big reasons I got interested in golang, was huge amount of awesome go projects in open source. I personally use roughly around ~10 golang based project on a computer I'm sitting at now.

I couldn't stop reading source code - it was easy to digest, even without prior experience in language! So, there have become a moment where I said "this seems easy, I should try writing my app with go". Of course, a lot of that available open source not only inspired, but show me examples for possible solutions that I was looking for.

Github Copilot

Copilot is amazingly good. In some cases it fills in 80% of a method I'm planning to write and almost always guesses assertions in tests that I about to write.

With Copilot I didn't need to disrupt my coding flow. Wrote a descriptive method name and AI filled in the "blanks", I tweaked the hell out of it, especially with auto-complete - it's easy.

It's creepy, how fast Copilot allowed me to pick up a language.

BubbleTea

BubbleTea TUI framework and charm related libraries is amazingly great looking. It also follows ELM architecture and this architecture works great with golang. I was blown away to find a framework of such a quality and with so much simplicity!

I need to write more cli tools! This BubbleTea TUI thing is just incredible, a bit hard to test though.

Documentation

I generally try to code without google/stackoverflow these days, mostly relying on official API documentation. I always stumble on something interesting there, something that is usually omitted in a stackoverflow answer!

In this regard, golang documentation was mostly great!

So to sum it up, I must say, even against my initial prejudgement (not a functional language, duhhh) - I really enjoyed diving into Go and absolutely will continue to explore this language further. I already have couple of projects I'd want to build with it too.

Golang turned out to be a valuable asset to have -- only the fact, that it can easily compile for any platform! Damn it's nice! I'm eagerly looking for new projects to build with it now.

Lessons from building another URL shortener

Stanislav(Stas) Katkov — Thu, 26 Mar 2020 17:55:08 +0000

This post was originally published on my website

I'm a software consultant working mostly with the web.

I secretly wanted to build an URL shortener for quite some time. It's one of those projects that seems relatively easy at first, but during my work as a consultant, I stumbled upon in-house solutions that caused more harm in the long term, than it was worth.

I’ve always wondered what my solution would be to the problems I came across and what would be the best solutions to overcome them. I recently managed to find the time and right idea to build one. In this post, I'd like to highlight my key takeaways and design principles that I put into the project.

Introducing "the project"

There is not much to gain from a toy-project, it will end up sitting on a hard drive collecting dust. So, how can one fully experience all the consequences of the technical choices with a toy project?

I came up with a project that can potentially generate profit or at least could be sustainable to operate. There is no shortage of URL shorteners out there, therefore I decided to build a very niche one for Amazon Affiliates called aCart.to.

It works like a regular shortener, but as input it takes a list of Product IDs instead of URL. Based on the list, the system pre-builds and stores URL that:

redirects to the amazon cart page with products pre-added
could link to multiple products

After careful consideration, I’ve highlighted the 3 most important things to consider in a project like this.

Pick a response code wisely

If the browser requests a URL from an external server, it would receive a response code with a requested page. When using a URL shortener, page content will not be returned, but a redirect request will be issued with the following response code.

If a shortener will be used by others, it's crucial to have a 301 response code. If you’re building a shortener for internal use - a 302 response code could be more appropriate. For any technically savvy user this would be the biggest criteria for a choice. So it's important to understand the difference to make the right call.

301 redirect

A 301 redirect says that the URL requested (short URL) has “permanently” moved to the long URL. Since it’s a permanent redirect, search engines finding links to the short URLs will credit all those links to the long URL.

302 redirect

302 redirect is a “temporary” one. If that’s issued, search engines assume that the short URL is the “real” URL and just temporarily being pointed elsewhere. That means link credit does not get passed on to the long URL.

Other

Some URL shorteners on a market are very creative and manage to use 200 or 303 codes while redirecting requests. I’m not completely aware of the motivations for doing so, but it seems “shady” and I would personally recommend against such a practice.

Keep it (really) short!

Since the biggest use-case for shorteners are social networks with limited character capacity per message, then it's really important to lose all the fat from a short URL.

Aim for a short domain

The first step in the right direction should be your domain name - it should be as short as possible. Keep in mind that finding short .com or .org domains will be costly, so you might want to look for tld domains of other countries. (.ai sound really cool!) These domains come with a SEO penalty and you’d be less likely to rank in google, but they are short. That’s the tradeoff most shorteners go with.

But there is even more fat we can lose here.

Lose www, but don't lose performance and security

A lot of people argue that "www.domain.com" vs "domain.com" usage in domains is merely a cosmetic difference. But unfortunately, dropping www from URL can have dire consequences because of how DNS records work.

Unfortunately, most DNS providers work only with two domain record types - A record or CNAME record.

CNAME record - requires that it be the only record for that domain
A record - can only point to a IP address

In most cases, to use domains with no ‘www’, you’re required to use A-record with a pointer to the DNS provider's load balancer. In case of DDoS attack on the DNS provider, your domain will be very likely to stop answering.

A record also comes with a performance hit, requests for CNAME record could be served by a closest server. Requests for A record always get resolved through one instance, it will probably be the most loaded instance available.

Netlify goes into great lengths to explain this problem in a blog post titled “To www or not www”.

My solution was to move the domain over to a DNS provider that could handle ALIAS records or CNAME Flattering. In my case, I had to move from namecheap to dnsimple. Support for this feature usually costs an additional monthly fee.

Bijective function for “hashing”

The shortener has to come up with a public ID that uniquely identifies shortened URLs. In a lot of cases, clients will rule against your server if you're stably generating a long hash for a public ID.

I used a bijective function that could transform a number into a hashed text and inverse the text back into a number. Thanks to the internet, it wasn’t that hard to find an example on stackoverflow.

This function significantly simplified table design in the database. I’m able to convert the primary key to a hash text and pinpoint exact records in a table based on hash text. Not having other indexes in a table is a great speed improvement on it’s own.

Don't lose referrals

HTTP Referrer is an option in the HTTP header field that stores the address of the webpage that this request originated from.

There is a very thin line between a URL shortener service and a referrer cloaking service (like nullrefer). And this thin line is called "Pass HTTP Referrer". You probably don't want to unintentionally become a cloaking service.

Passing referrer information along to a long URL in ruby is just one line of code, but a really important line of code.

headers["HTTP_REFERER"] = request.referrer if request.referrer

Unfortunately, there are a number of cases when the HTTP_REFERER parameter will be empty or could not be passed. Due to specification, HTTP_REFERER will not be passed if a long URL uses insecure protocol (http://).

If a website is accessed from a HTTP Secure (HTTPS) connection and a link points to anywhere except another secure location, then the referer field is not sent.

But nowadays we have Referrer Policy as a solution that could partially mitigate these issues by just adding meta tag to HTML pages.

<meta name="referrer" content="origin">

Summary

So, when it comes to launching a URL shortener service, keep in mind that these three issues are the biggest you need to be aware of. A lot of other articles explain how to write this service, but don’t give you a project wide overview of launching it into the public.

I hope you found this article useful. If you need a hand in thinking through some of your existing projects - I’m also available for hire!

New deadline for Product Advertising API deprecation

Stanislav(Stas) Katkov — Fri, 08 Nov 2019 09:18:19 +0000

As I previously announced, Amazon was planning to deprecate PA-API version 4 on 31 October 2019. But that didn't happen and the deadline has been pushed back.

Amazon keeps tradition of secrecy on this API transition. To this date not a single company representative officially commented in support forums and we have seen zero public announcements, besides those details provided in documentation for new API.

In a private communication, some of the biggest Amazon partners that rely on PA-API have received a heads up that deprecation will be postponed. But they made them sign NDA, justifying this by the push to get everyone off older API as fast as possible.

PA-API has a huge history of abuse by people that scrape Amazon for data. And last year changes have been mostly aimed to tame all the abusive behavior. This migration to new API is not an exception -- biggest features have been improved security and even further limits to grab big chunks of data at once.

We've found leaked photo from Amazon Associate event in Munich (hence the German language) that suggest that deprecation has been pushed back to 14 January. There seems to be a typo in the year, but our private sources have confirmed this date.

This push back, makes a lot of sense -- it would not be reasonable to do such huge change during biggest sales season. So let's breathe calmly here, we have some time to migrate our websites and apps.

Thanks to Amazon for that!