DEV Community: Juan Manuel Ramallo

How to deploy Apache Superset with Dokku?

Juan Manuel Ramallo — Sun, 02 Jul 2023 20:44:22 +0000

Why?

Because it is much easier than doing everything manually
Configuration changes are deployed via git (git push dokku main)
Reverse proxy (nginx) is configured by Dokku (no lines of nginx config files are touched, not even looked at)
SSL is configured by a Dokku plugin (I don't even know how to use certbot)
Ideal to spin up a Superset instance quickly to test things out

Why not?

Well, if the plan is to serve thousands of dashboards and charts for thousands of users, maybe look into the kubernetes installation.

Before we start, ACME is used as an example of a company name. Feel free to replace with your own.

In your virtual machine

1. Install dokku

Go to Dokku's installation page and follow the steps. There's no need to have Docker pre-installed, Dokku installer will take care of that.

Make sure to follow all steps until the end including the ones about adding your SSH key and setting the global domain. A domain is required to run Superset with SSL.

The SSH key must belong to the machine from where you intend to deploy Superset, so that git push can authenticate.

2. Create the app in Dokku

dokku apps:create acme-superset

3. Install postgres/redis

Install postgres plugin and redis plugin.

Then create the services and link them to the app.

sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres
sudo dokku plugin:install https://github.com/dokku/dokku-redis.git redis

dokku postgres:create acme-superset
dokku postgres:link acme-superset acme-superset

dokku redis:create acme-superset
dokku redis:link acme-superset acme-superset

This is optional, but I recommend doing it.

By default Superset uses SQLite, which will not allow us to create multiple Datasets pointing to different tables but with the same name. This is solved by using Postgres as the metastore (metastore is how Superset documentation refers to the database where Supersets objects are stored—dashboards, charts, etc).

Redis is also optional but it's nice to have in case you want to configure data caching.

Postgres and Redis connection strings will be present as environment variables in the acme-superset application:

dokku config:show acme-superset

After running the link command you may encounter a message like App image (dokku/acme-superset:latest) not found, just ignore them.

4. Configure the default port

dokku proxy:ports-add acme-superset http:80:8088

By default Superset uses the 8088 port, but in order to properly configure SSL it is required for us to proxy the port 80.

In your local machine

1. Create a new local repository

mkdir acme-superset
cd acme-superset
git init

Create a new folder and initialize git. We'll use this folder as the Dokku application to deploy.

2. Create a config.py

curl https://raw.githubusercontent.com/apache/superset/2.1.0/superset/config.py -o config.py

The config.py from your repo will be placed instead of the default config.py.

Grab the default config.py from github.com/apache/superset/blob/VERSION/superset/config.py.

To use the database connection string from the DATABASE_URL env var, update the config.py file as follows:

SQLALCHEMY_DATABASE_URI = os.environ.get("DATABASE_URL")

To use the redis connection string from the REDIS_URL en var, update the config.py file as follows:

# Cache for datasource metadata and query results
DATA_CACHE_CONFIG: CacheConfig = {
    "CACHE_TYPE": "RedisCache",
    "CACHE_DEFAULT_TIMEOUT": int(timedelta(hours=1).total_seconds()),
    "CACHE_KEY_PREFIX": "superset_data_cache_",
    "CACHE_REDIS_URL": os.environ.get("REDIS_URL")
}

Configure anything else as needed.

3. Create a Dockerfile

touch Dockerfile

# ./Dockerfile
FROM apache/superset:2.1.0
USER root

# Set my secret key
ENV SUPERSET_SECRET_KEY=SUPER_SECRET_KEY_PLEASE_REPLACE_ME

# Use my config
COPY config.py superset/config.py

# Add database drivers
RUN pip install psycopg2
RUN pip install sqlalchemy-bigquery

# Adds vim to be able to enter the container and read files with vim
RUN apt-get update && apt-get -y install vim

USER superset

First line allows us to select what version to use. This example uses 2.1.0, latest version as of this writing.

Add any database driver needed. This example adds the driver for Bigquery. More drivers can be found here.

The base image can be found in Docker hub.

4. Add Dokku's remote repository

git remote add dokku dokku@dokku.acme.com:acme-superset

Where dokku.acme.com is the global domain configured for Dokku and acme-superset is the name of the application in Dokku.

5. Deploy

Commit all files and push to Dokku's remote.

git add .
git commit -m 'Superset initial configuration'

git push dokku main

The deploy will start and you should have output in the terminal about the Dockerfile steps we defined previously.

Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 23.48 KiB | 5.87 MiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
-----> Cleaning up...
-----> Building jm-superset from Dockerfile
remote: build context to Docker daemon  66.56kB
Step 1/14 : FROM apache/superset:2.1.0
...

Finally configure SSL in the Dokku machine via the letsencrypt plugin.

# This is performed in the virtual machine, where Dokku is running
sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

# Set global email for letsencrypt
dokku letsencrypt:set --global email admin@acme.com

dokku letsencrypt:enable acme-superset

That's it, open acme-superset.dokku.acme.com in your browser.

Superset is running and you are able to change its configuration, install new drivers, change version, and pretty much do anything in a very easy manner.

Final considerations

The first time you deploy superset, you may need to create an admin user, migrate the database and run superset init.

Enter the container by running:

dokku enter acme-superset

Then run:

# Create an admin
superset fab create-admin \
  --username admin \
  --firstname Superset \
  --lastname Admin \
  --email admin@superset.com \
  --password admin

# Migrate DB
superset db upgrade

# Init and setup roles
superset init

Refer to Superset documentation{:target="_blank"} for more information about its configuration.

Happy dashboarding.

I have a Superset instance running smoothly in a Digital Ocean Droplet. Get $200 in credit over 60 days to try this out.

How to find the PID of a process using a specific port

Juan Manuel Ramallo — Tue, 14 Dec 2021 00:20:37 +0000

To find out what processes are using a specific port, use lsof.

lsof -i :PORT

Examples

List all processes using the 5432 port (commonly used by postgres)

$ lsof -i :5432
COMMAND  PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
postgres 777    1ma    7u  IPv6 0x0123456789abcdef 0t0  TCP localhost:postgresql (LISTEN)
postgres 777    1ma    8u  IPv4 0x1123456789abcdef 0t0  TCP localhost:postgresql (LISTEN)

List all processes using the 4000 port (this blog in development)

$ lsof -i :4000
COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
ruby    59182    1ma   10u  IPv4 0x2123456789abcdef      0t0  TCP localhost:terabase (LISTEN)

Alternatives

netstat can be used with grep to look for the port, but not possible in macOS as netstat won't output the process ID.

ps -ef can be used with grep too, but we'll need some bash sorcery to keep the first line of the ps output (the headers line).

How to see all AWS services currently in use

Juan Manuel Ramallo — Thu, 27 May 2021 13:52:02 +0000

Ever wondered what a black hole looks like? My AWS Billing—or yours—might be one of the closest things to a black hole on earth. So scary, and so hard to remove things from it, firstly because we don’t know what services to remove nor the regions for those unwanted services.

Let’s focus on how to remove things from the AWS bill, which is the main task that will definitively cleanse our bill. It’s so easy to spin up new services in AWS and forget about them, some might be cheap some others might not be that cheap; and given that AWS—at the time of this writing—provides over 175 services in several regions each, it becomes quite a task to manage all of your services in all of the regions you used.

Enter the Bills page. Go to your Billing Dashboard, and then on the left sidebar, under the Billing section, click on Bills. Below AWS Service Charges you can see all of the services you have signed up for, and inside each of them, you can see the incurred costs per region. This way you just need to figure out what services are not in use, then go to their regions and remove them.

This way I was able to spot an unused RDS instance and a very old Amplify instance, both burning money with no returns.

Why avoid too many abstractions in tests?

Juan Manuel Ramallo — Sat, 13 Feb 2021 21:20:08 +0000

Intro

There's production code, and there's test code. Production code refers to any portion of the codebase that is executed in production environments. Similarly, test code refers to any portion of the codebase that is solely executed in testing environments, where production code is exercised via the test code.

Test code needs to be explicit, easy to read and understand what's going on. Test abstractions, like shared examples, are usually against what we need out of test code.

Why not to DRY test code?

Refactoring production code tested with several abstractions in test code becomes harder. This is because one single test abstraction is used across several places in test code, potentially causing your changes to break some tests but not all of them. Finally, triggering a refactoring task on the test abstraction as well or maybe even updating your tests that failed to stop using the abstraction cause it no longer fits the test.

Test reports with failures are harder to read. If a test using an abstraction failed inside the abstraction then the report will show where it failed first, and that is inside the abstraction. It requires the developer to read over the stack trace to check what test called the abstraction in order to find out what test is actually failing.

This is the output of RSpec's Shared Examples

Finished in 3.17 seconds (files took 2.49 seconds to load)
47 examples, 5 failures

Failed examples:

rspec ./spec/requests/api/v1/users_spec.rb[1:2:2:2:1:1:1] # Api::V1::Users PATCH #update behaves like ...
rspec ./spec/requests/api/v1/users_spec.rb[1:2:4:1:1:1] # Api::V1::Users PATCH #reset_change_email behaves like ...
rspec ./spec/requests/api/v1/users_spec.rb[1:2:5:1:1:1:1] # Api::V1::Users PATCH #request_remove_user behaves like ...
rspec ./spec/requests/api/v1/users_spec.rb[1:2:1:1:1:1] # Api::V1::Users GET #show behaves like ...
rspec ./spec/requests/api/v1/users_spec.rb[1:2:3:1:1:1] # Api::V1::Users PATCH #reset_api_key behaves like ...

beer's on me if you can tell where the failure occurred

Don't get me wrong

Too many abstractions for our test code is like highlighting every single word of the book you're reading. You'd only want to highlight the most important parts right?

With test code is quite similar, we don't want to abstract everything but we'd rather keep test code as explicit and readable as possible and at the same time provide abstractions for the parts that are not really meaningful for test cases. For instance, the factory pattern to create objects for test code is perfect – re: FactoryBot. Custom matchers are also a valid way to introduce abstractions while keeping our test code explicit and declarative.

To sum up, abstractions for non meaningful parts of test code are valid and really useful. Abstractions for test code that describes actual behavior of the system are never[*] good.

* I wrote a shared example last month, out of a code review suggestion, and I can live with it

I don't use nil

Juan Manuel Ramallo — Sat, 17 Oct 2020 22:01:33 +0000

I don't use nil, or at least I try not to.

NoMethodError: undefined method 'some_method' for nil:NilClass

This is one of the most common errors seen on production systems. And it's likely to happen everywhere where a variable can reference to a null value. Here are some case studies about the topic.

Migrations

When creating/updating a table I always consider adding a null: false constraint and a default value accordingly. For string values, the default would be an empty string. For number values, the default value would be zero.

This allows us to do operations with those columns without worrying about them being nil, because we took care of that at the database level. For instance, let's consider a Post model in a blogging system:

# db/migrate/...
class CreatePosts < ActiveRecord::Migration[6.0]
  def change
    create_table :posts do |t|
      # [snip]

      # not bad
      t.string :excerpt

      # better
      t.string :excerpt, null: false, default: ''
    end
  end
end

# app/views/posts/show.html.erb
<p>
  <%= @post.excerpt.downcase %>
</p>

In this particular example, sending downcase to Post#excerpt will always work because we ensured that the value will always return a String object.

Enums

Let's consider that any post can have a certain category. Since these categories won't have any extra info nor behavior, we'll use an enum attribute in the post model.

# db/migrate/...
class CreatePosts < ActiveRecord::Migration[6.0]
  def change
    create_table :posts do |t|
      # [snip]

      # not bad
      t.integer :category

      # better
      t.integer :category, null: false, default: 0
    end
  end
end

We can easily add a default enum option to represent a post without a category. And why is it better? For the simple reason that the Post#category method will always return the same object type. This will cause the code to avoid checking for nil when working with the category, and will also make the safe navigational operator useless when working with this attribute.

# app/models/post.rb
class Post < ApplicationRecord
  enum category: { general: 0, lifestlye: 1, art: 2, misc: 3 }
end

In this example we are using the category named general as the default one.

NullObject

This approach is an elegant way to represent a null value. It is somewhat similar to the enum's case study. Here's an excellent article from thoughtbot about this pattern.

undefined method 'conclusion' for nil:NilClass

To sum up, I don't use nil because I don't want the program to potentially raise this NoMethodError. To avoid it, it's just as easy as not handling null values in the codebase.

And this is Tony Hoare, the inventor of nil:

I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years. src

When objects become super objects

Juan Manuel Ramallo — Tue, 11 Aug 2020 05:16:02 +0000

A pencil. We use a pencil to write. And that's it. (Let's forget about some pencils that come with an eraser in one end, in this world they don't exist)

We are in the "digital transformation" era, right? So I came up with the idea of including a simple scanner in the body of the pencil, to be able to digitalize everything written with the pencil. It's really handy!

I always struggled to share my technical drawings when exchanging thoughts in groups, so I think a mini projector on top of the pencil would be a great addition. It's amazing, I can show my drawings directly in the wall, no need for any external projector!

Oh, and a laser pointer on it. You know, everyone loves playing with lasers.

We call it The Pencil 3000 ™

Let's stop here. I've experienced this very same line of thinking when creating/updating classes.

Multiple responsibility classes: Real life examples

These are the places where I've encountered myself working with this approach:

CTMPS means "comment to my past self".

1. Models

Several times I wrote methods in the model that were only being used in the views. CTMPS: It'd be better to use a presenter object for those.

Numerous times I found myself writing callback methods in the model, with the purpose of sending emails or adding default values in the model. CTMPS: Sending emails seems like a side effect of a user action, why don't trigger them in the controller instead? And about setting default values, what about setting them using the ActiveRecord::Attributes API?

Don't get me started with inline validation methods. This I one I abused. I'm sorry. CTMPS: There are no taxes for new classes, create a validator class and be done with it. Peace out.

2. Form objects

Select tags with options that are not DB backed. Guilty. I more than once wrote a method in the form object to provide the options to be used in the select tag, in the view. CTMPS: It's ok no big deal, but what about using a presenter object for this? Given it's a view-only matter.

At this point it feels like I'm seeking absolution from my sins. Hopefully you haven't experienced any of these. But in any case, let me share you one more example.

3. Page objects

Testing usually requires 4 steps: setup, exercise, assertions and teardown. And when testing web applications, end users interact with HTML documents. Enter page objects. So we use page objects to wrap the HTML and provide a meaningful API to be used in test code. This includes the exercise and assertion steps of testing. But I gotta be honest, it is so easy and pleasant to just write a new method into the page object to do the setup and create instances, update stuff, set up environment variables and whatever needed for our tests to run. CTMPS: It's as easy and pleasant to extract the creational part into a support module in your specs, and keep your page objects fulfilling their true purpose.

Remember The Pencil 3000™? I can't imagine how would I sharp the pencil, with all that stuff around it.

Single-responsibility principle

"[...] each software module should have one and only one reason to change." Better read it from here

Do you use form objects in your Rails apps?

Juan Manuel Ramallo — Fri, 10 Apr 2020 02:48:22 +0000

Introducing yaaf, a gem to ease the usage of the form object pattern in rails apps.

My pain points (before yaaf)

The form object pattern is widely used across Rails apps, and yet we tend to write these form objects in a different manner from project to project.

Most of the time, they don't even have the same interface across form objects in the same app.

We also tend to write validations twice, in the model and in the form object, just so that the form object has the errors and the view can display them.

Well, I can say that most of our in-house form objects make good use of database transactions. But I bet we rarely provide the controller the ability to use a banged method, expecting it to raise an error if things go south.

Business logic is present everywhere in your app, controllers, models, helpers (some hardcore scenarios might include Rails initializers), but is that right? Many times we find ourselves with the need to add business logic around the creation of an object, such as sending an email, updating other records or even calling an external service. We know that the controller is not a good place, and the model would be terrible as well because it violates SRP. So we end up creating several service objects and throwing the business logic there as if it were a trashcan.

Instead of all of that, we can...

use yaaf, it solves all of that in an easy way, let me explain:

Same interface? yaaf will encourage you to use save and just save to, well you know, SAVE the data from your form into models.

Contextual validations only? Please put them in your form objects, and keep data integrity validations where they belong, in your service objects, haha no, of course no, in your models! yaaf will take the liberty to promote all model errors collections into the form object errors collection, so you can show them in your views

So, one of the form objects need to raise an error when not saved? As you may have guessed already, use the dangerous save! method. yaaf already defines it so that it raises an error just as any Active Record model does. Of course, it will also trigger a database rollback when the models couldn't be saved. Both save and save! methods are available.

Not sure where to put business logic related to the creation of an object? Well, the form object is the place. yaaf will allow you to use callbacks the same way as your model callbacks work. For instance, if you want to send an email after the form has been submitted and persisted, the after_commit callback is the one you're looking for. More info can be found in the readme.

The Bob Ross of form objects

Using yaaf will allow you to standardize your form object definitions across your project's production code.

Why not use this?

If you would rather have controllers deal with all these responsibilities, then it's fine to keep doing it, you may not need yaaf if that's the case.

Are you a service-objects-for-all person? Then you might feel better writing service objects rather than form objects.

If you work with Java, well you don't want to use this.

A bit of history

We have been using this very same approach on a production app for almost a year now, more than 10 form objects written this way, and it has served us well. That's why we decided to extract it to a gem. (yaaf is no more than 64 lines currently, and we need no more)

Big thanks to @santib for shaping up this gem from its very beginning.

The lovely branding from yaaf was done by Sofi Salazar

rootstrap / yaaf

Easing the form object pattern in Rails applications

Have you heard of API Blueprint? 📘

Juan Manuel Ramallo — Tue, 21 May 2019 01:38:52 +0000

Recently, I came across the need of providing some sort of documentation for an API I was building. At that moment, I had never heard of the API Blueprint specification. Though they might be several options to write a good API documentation, I found this one to be very helpful for the following reasons.

1. Easy to read

Sections are well defined and grouped by route. Each route can have several different HTTP methods with names associated to each method. The documentation shows:

Attributes listed with their descriptions (when needed)
Requests examples (headers and body)
Responses of those examples (also headers and body)

2. Mock server

This language allows you to create a mock server that anyone can use to test the response of the actual server (there are tools for this, see more.)

So the mock server is actually a server that responds to the endpoints defined in the documentation and returns the expected responses (it does not care about authentication or headers, cause it actually does not run the production code, it just returns the response to the endpoint hit). So it's very helpful to have this kind of server in the early stages of an app, where the client application is still being built and the backend is not fully deployed to a production/testing cloud based environment.

3. Lots of tools

Renderers are services that take your blueprint as input and create a nice web page for others to refer to, when using you API. Apiary is one of them, and a widely used one. Apiary can be connected with your remote repository, so whenever you blueprint changes, it will automatically reflect the changes in the documentation page.

But I don't want to write another language, and probably neither do you. So here's where docs-generator libraries take place. For those using Ruby❤️, RSpec API documentation turned out to be a great tool to generate this blueprint (though it can be used to generate other type of docs). You just need to write some RSpec tests and run a rake task to generate the blueprint.

Here's an example of a basic endpoint being documented using this gem:

resource 'Books' do
  before do
    create_list :book, 5                # Creating some books 
  end

  route '/books', 'Books collection' do # Name of the section
    get 'Index' do                      # HTTP method with name
      example 'Ok' do                   # A request example
        do_request

        expect(status).to eq 200        # Basic expectation of the request
      end
    end
  end
end

After running rails docs:generate it'll generate a file with something like this:

# Group Books

## Books collection [/books]

### Index [GET]

+ Request Ok

+ Response 200 ()

    + Body

            {
              ... // The response body would be here
            }

If you are writing an API give this language a try. Make your fellow developers happy by giving them a proper documentation of your API.

To sum up, some things I like about it

Production code does not get mixed with the doc-generating code
All documentation related code lives in one place
As soon as a change is made in routes or in the response of the controllers the documentation gets updated automatically by the generator. I don't need to write anything new.
Mock server just uses the data from the docs (it does not run the production code)
There are bunch of services to render the documentation (renderers) or I can host my own 😎
My API blueprint file lives in the same repository, so I can assure it's always synced with the current branch

In case you want to look a full set of doc-generating test examples

Does anyone else think HTML5 multiple select sucks?

Juan Manuel Ramallo — Thu, 11 Oct 2018 03:58:28 +0000

Am I the only one who thinks that html5 multiple selects are not intuitive for the web? I mean they are fine, work fine but I have to use CMD (or CTRL) to select more than one, doesn't feel as a web-ish thing 🙃

Maybe it's just me being paranoid 🤪

The thing is that I'm looking for a decent javascript module to handle multiple selects like Chosen or Select2 but without jQuery as dependency. Any suggestions?

I was billed for 14k USD on Amazon Web Services 😱

Juan Manuel Ramallo — Tue, 03 Jul 2018 01:11:38 +0000

We may agree that we lie to ourselves once in a while. I remember thinking of:

I'll never put my code on a public repository since it's a freelance project maintained only by myself.

The truth is, anything can happen in life.

Month to month I use S3 from AWS to store photos and documents from several apps I got on production and also use Route 53 to route the domain with the app itself. My monthly bill is about 1 USD. Yep, that's almost nothing, but this month (april last year) I had to pay 14,267.86 USD (well at least that's what my bill says)

On april 13th (a sleepy Monday), it was like 10:00 in the morning when I got a happy email from Amazon Web Services giving me the welcome to EC2 services. By the time I received that email I neither know the existence of that service. That made me wonder if I had received that email by mistake or if my account … had been hacked.

Minutes later I wrote a message to AWS support asking about that email and they answered me very quickly and called me like five times in less than 48 hours. They told me that my account may had been compromised, and gave me some list of things to do in order to strengthen my account security and to avoid further EC2 service usage. I deleted all access keys I was using, added multi-factor authentication and changed my password. (I'm not endorsing or promoting AWS but I have to say that the guys from AWS support were extremely kind and useful in this case)

It all started in a boring weekend (at least this is what I believe). I was on the chat with a friend telling him about a side-project I had been working on. And suddenly I decided to put my code on github to show him off all the stuff. It was up, in the web, for like 10 minutes max, after I switched it to be hosted on gitlab to make it private. Two days later I received that email from AWS I told you before.

What happened?
A file containing my AWS credentials hadn't have been ignored in git, so when I pushed my local repository it all, even my credentials, went online in github (for couple of minutes, but they were there).

Suggestion
Please, store your credentials secretly ALWAYS. You can use environment variables for storing access keys and credentials that may compromise your accounts or bills. And never think of a local repository only, when everyone knows that nowadays the internet is all over around.

How this happened?
Since you can list all public repositories on github, I imagine of a job/task/process/program running constantly and cloning each project and looking for .yml files and keywords like "KEY" or "ACCESS_KEY" or something like that. This is only my thought of how could this happen. If it happen like so, it makes me sad of how people can be malicious and with no concerns of consequences of their acts.

TL;DR
Never leave your api keys on public repositories (always check before uploading)
Take a look at your email frequently and don't take a single email as a mistake
Be a hacker so you can track and catch the guys who stole your keys (well that's only a dream)

Hope you have a good day!
(If you're still wondering, no, I didn't have to pay for what I didn't use)