DEV Community: Manu

make release: save time, gain in reliability

Manu — Wed, 27 Jun 2018 09:12:15 +0000

The best way to make a diamond-solid process is to automate it

A story, where any resemblance to reality is pure coincidence...

[fictitious user] : ... and there I have a page which says five hundred errors. Thats's a lot !

[you, not mentioning that 500 is an error code, not the number of errors] : Hmmm... should not be like this.... which version are you running ?

Ah.

Which version exactly ?

Tricky question... small numbers in the footer of the footer ? hidden comments ? versionned-urls for APIs ? and so on...

[user] :It says 0.3.1

[you] : Thanks. Let me check the Changelog

Ah.

Where is the Changelog exactly ?

Do we have one to start with ? Is it up to date ? Has it been written for marketing guys ? Well, chances are huge that if you have a Changelog, it does not really help anyway...

Objective

In order to facilitate the conversations of the kind above, I would like to present you with a bit of processing and a bit of tooling.

The final objective should be

$ make release
...

And you would be set with

a new branch on github rc-X.X.X
a new release on github based on this branch
an updated Changelog with meaningful information ...
... that actually comes from the differences between this release and the previous one
a way to check the current / next version of your code
an updated badge on your README (because badges are cool)

One command to rule them all

Pre-requisites

make
your application on github
docker

Docker will relieve you from installing other dependencies like ruby (used to generate Changelogs) or python (used to manage version)

Set up

files

This repository (release-maker) is designed as a plugin: you should copy paste the following files into the repository of your app

File	Description
├── Makefile	entrypoint for all commands
├── bin
│ ├── Dockerfile	defines a (very) simple docker image FROM python:3 with the requests library available. The image is built locally with `make build`
│ └── update_release.py	management script, to run sanity checks, update version numbers, publish release to github
├── .env.sample	to define the values of your environment variables
└── versions.py	where version numbers are written to/read/imported from

environment variables

simply copy-paste (or rename) the .env.sample file to .env and fill in your details:

$ cp .env.sample .env
$ vi .env
# GITHUB_* are used for github APIs, to automatically create release & Changelog
GITHUB_OWNER?=repo owner
GITHUB_REPO?=repo name
GITHUB_USER?=your github user
GITHUB_KEY?=your key
CHANGELOG_GITHUB_TOKEN?=${GITHUB_KEY}

docker

Once you have "imported" the repository content and defined your vars, you need to build / pull the docker images that will be used. Otherwise you will get this kind of error:

$ make vars
Unable to find image 'python-requests:latest' locally

The python image is built locally with make build. It is a straightforward Dockerfile that relies on python 3 official image and will have the library requests installed.

# bin/Dockerfile
FROM python:3

WORKDIR /usr/src/app
RUN pip install --no-cache-dir requests

ENTRYPOINT ["python"]
CMD ["--help"]

Since we are setting up docker, you can actually pull the second image that will generate the Changelog. make pull will also run make build for you

$ make pull
...

The command should now work properly (with your values for the environment variables)

$ make vars
  Version: 0.1.1-rc

  GITHUB_OWNER=ebreton
  GITHUB_REPO=release-maker
  GITHUB_USER=ebreton
  GITHUB_TOKEN=xxx
  CHANGELOG_GITHUB_TOKEN=xxx

versions

The generation of the Changelog depends on the existence of (at least) one git tag. If you have none, you should run

git tag 0.1.0 # or whatever initial version number you prefer
git push --tags

You can check the current configuration with

$ make version
CHANGELOG GENERATOR:
Version: 1.14.3

APPLICATION:
Version: 0.1.1-rc

Updating release numbers...
INFO - compute - will set _version=0.1.1-rc
INFO - compute - will set _build=5026db3b8b001c5d8991c90d0528161abec7bbeb
INFO - compute - will set _release=0.1.0-12-g5026db3

Usage

Practically, you will only use make release, everytime you want to make a release (surprised?)

and that's all.

Of course, there is a few more commands, some we have seen to setup things, some to diagnose or check what the situation is. Here is the list:

Command	Description	remarks
`make version`	display the current version of the application (not the one previously deployed), and version of github-changelog-generator	-
`make vars`	display value of environment variables	requires .env file
`make build`	build a docker image FROM python:3 with requests available	-
`make pull`	pull last docker image of github-changelog-generator	-
`make ps`	list docker containers (with less information than docker ps)	-
`make changelog`	build/update new version of Changelog according to tags and PRs	valid environment variables must be defined in .env (or environment)
`make release`	after confirmation of version number, execute all release process.	must be executed in branch master
`make push-qa`	update tag qa-release and rebuild Changelog	-
`make push-prod`	update tag q a-release and rebuild Changelog	ask confirmation of branch and version

We have not yet discussed the last two lines... there is some kind of bonus behind them, that will see in the last section.

Behind the curtains

I guess you see it coming :

1. a new branch on github rc-X.X.X

this acts as an emergency branch, in case when a quick fix is needed

2. a new release on github based on this branch

this allows to update the badge or get the release number from github API

3. an updated Changelog with meaningful information ...

this is the mandatory context that you will be looking for to diagnose any issue

4. that actually comes from the differences between this release and the previous one

Thanks ferrarimarco !

5. a way to check the current / next version of your code

thanks to versions.py which you can import and use anywhere in your code

6. an updated badge on your README (because badges are cool)

indeed

Bonus

Using git tags for continuous integration and continuous deployment

If you happen

to run your applications within docker containers,
and have setup automated builds in your registry (or docker hub),
to use an orchestrator

It is convenient to use tags for continuous integration and deployment

In my case, the orchestrator relies on the qa-release tag to upgrade the integration environment, and on the prod-release tag to upgrade the production.

It is all automated, except for the trigger, which remains on your hand. Therefore, running make release will call make push-qa, and start the upgrade process for integration. You can also upgrade without going through all the release process, by calling yourself make push-qa (no confirmation)

In order to update production, you must call make push-prod, which will ask for your confirmation.

Be aware that those two commands will update your Changelog (by calling make changelog), because we want it up to date with the commits that are pointed out by each tag.

$ make push-qa
# update tags
git tag -f qa-release
Updated tag 'qa-release' (was db42b2e)
git push --tags --force
...

$ make push-prod
You are on branch 'master'
on version '0.1.2-rc'

Type [y] to comfirm push
or any other key to abort:
...

Needless to say, if you haven't configured your orchestrator to update containers based on new builds, and if you haven't configured your registry to update builds based on git tags.... well... those two commands above will not be so useful.

Conclusion

I cannot stress enough the conveniency of automatic versioning and releasing, along with a consistent (auto-magically generated) Changelog. It just saved hours of team-work to support the final users and diagnose the issues.

Within my team, we were aware of it for a long time, and we had tried hard to structure our Pull Requests, our Changelog, our version numbers. But that just could not work as long as it was manual. Too cumbersome, with absolutely no value (until the shit hit the fan).

Firstofall release maker has allowed us to focus on development, on the content of the reviews. Secondly, it has facilitated the release process so much that we could actually increase dramatically our release frequency, from once a month to once a week (sometimes more).

Nice. Handy.

Hope you will find it useful too :)

Ghost in A shell - part IV (and final): Maintenance

Manu — Wed, 13 Jun 2018 15:28:52 +0000

One very good reason to go for Ghost(Pro) is that you do not have to maintain your blogs. They just run. For ever. Eventhough servers do crash, hard drives do break, electric power do fail, blogs do need to be upgraded... they will continue serving their pages.

Nevertheless, having one's own setup is fun (as long as you are aware of the limits above), and much easier to play with. You could for instance:

run the Ghost version you want:

$ GHOST_VERSION=1.10 PORT=3002 NAME=old make

$ NAME=old make cli-version
  docker exec -it old ghost -v
  Ghost-CLI version: 1.1.1
  Ghost Version (at /var/lib/ghost): 1.10.0
  Latest version on Docker Hub: 1.24.2

maintain many blogs without having to remember the options they were built with:

# instead of 
$ PORT=3003 NAME=new URI=my-new-super-blog DOMAIN=me.com make restart

# just use
$ p=instances/new/ make restart

upgrade your blog with no risk and no hassle

$ vi instances/old/.env
  GHOST_VERSION=1.24

$ make p=instances/old/ upgrade
  docker exec -it old ghost -v
  Ghost-CLI version: 1.8.1
  Ghost Version (at /var/lib/ghost): 1.24.2
  Latest version on Docker Hub: 1.24.2

And these three reasons are good enough for me to write this fourth (and last) blog about shells of Ghost . Did I say "fourth" ? yep!. And here are the three first ones:

Part I : localhost, that focuses on localhost
Part II : HTTPs is the norm, that shows you how to spawn multiple ghost instances on a given server
Part III: To production, with performance, that applies the recommendations from the Ghost team when it comes to DB + emailing, and puts a bit of pressure on the resulting instances

But first of all you need the basic diagnostic tools

Environment variables

We have seen before the two following commands

make ps to list the running instances
make logs to tail and follow the logs, i.e to see what is happening

Those commands depends on the environment vars, which can be overridden in the command line. E.g.:

make vars will list for you the currently configured variables

$ make vars
# common
 NAME=ghost-local
 GHOST_VERSION=1
# values used for dev
 PORT=3001
# values used by traefik (qa & prod)
 PROTOCOL=http
 DOMAIN=localhost
 URI=ghost-local

NAME=my-blog URI=me PROTOCOL=https GHOST_VERSION=1.22 make vars will show how the variable are overriden

$ make vars
# common
 NAME=my-blog
 GHOST_VERSION=1.22
# values used for dev
 PORT=3001
# values used by traefik (qa & prod)
 PROTOCOL=https
 DOMAIN=localhost
 URI=me

So far (in the previous blogs) this is the way to go with multiple instances... and it sucks, especially if you want to run different set of values for each of them (which is exactly the reason why we have this series in the first place)

All-in-one configurations

Therefore, there is a newcomer in the list of available variables: please welcome p.

p is the path of the valid instance of the blog you want to act upon. you can use it with most of the make commands we have seen so far.

Valid means that the path will contain a .env file with all the overriden variables. For the sake of your sanity, this file is automatically generated when you spawn your instances with make [dev|qa|prod]. e.g.:

$ NAME=new-blog PORT=3002 make
# Simply start a ghost container making it directly available through $PORT
docker run --rm -d --name new-blog \
        -v /Users/emb/git-repos/ghost-in-a-shell/instances/new-blog:/var/lib/ghost/content \
        -p 3002:2368 \
        -e url=http://localhost:3002 \
        ghost:1-alpine
6330f60a50a834c7acee5916751815e47bba2f68b98f150a6ba2fe59627704ff
make save-vars
creating instances/new-blog/.env

The last two lines are the new sweetie... and you can now run the following to make sure your configuration as been saved:

$ make p=instances/new-blog/ vars
# common
  p=instances/new-blog/.env
  NAME=new-blog
  GHOST_VERSION=1
# values used for dev
  PORT=3002
# values used by traefik (qa & prod)
  PROTOCOL=http
  DOMAIN=localhost
  URI=new-blog

Consequently

make p=instances/new-blog/ vars will list all vars from ./instances/new-blog/.env
make p=instances/new-blog/ logs will tail and follow the logs of the ./instances/new-blog blog instance
make p=instances/new-blog/ restart will restart the blog with the settings defined in ./instances/new-blog/.env
make p=instances/new-blog/ cli-version show the versions for the new blog
and so on...

Note that 1) the path is relative, 2) the trailing / is mandatory, but it should come with the bash autocompletion anyway. (that's why it is done like this)

Upgrade (and Downgrade)

Upgrading may be painful (particularly if you do not do it often enough). It is for sure scary. No ones want to change (break) something that runs smoothly.

You first need a bit of information, e.g with the question 'how far am I behind ?'. make cli-version will help you there:

$ make cli-version
docker exec -it ghost-local ghost -v
Ghost-CLI version: 1.8.0
Ghost Version (at /var/lib/ghost): 1.23.0
Latest version on Docker Hub: 1.24.2

The upgrade mechanism relies on DockerHub tags. Which means that the most generic tags point to the latest version, and that the same version may have multiple tags. For instance, if the latest version is 1.24.2, you will get this version with all the following tags: 1.24.2, 1.24, 1, latest.

Hence the following behavior when upgrading:

GHOST_VERSION=1 -> will automatically update to all new releases of 1st version
GHOST_VERSION=1.x -> will automatically update for patches only
GHOST_VERSION=1.x.x -> will never automatically update

By default, the GHOST_VERSION is set to 1. Therefore, upgrading is as simple as make upgrade[-qa|-prod]. This command will pull the lastest (1.x.x) docker image and restart the container for you.

You can force your desired GHOST_VERSION by overriding the env var:

$ GHOST_VERSION=1.24 make upgrade
...
Ghost-CLI version: 1.8.0
Ghost Version (at /var/lib/ghost): 1.24.2
Latest version on Docker Hub: 1.24.2

If you have specified a GHOST_VERSION when you created the blog instance, e.g: GHOST_VERSION=1.10 PORT=3002 NAME=old make, the version is stored in the .env file. Therefore the p helper variable will not allow you to upgrade automatically. It will stick to the version of the instance specified when it was created, e.g:

$ p=instances/old/ make upgrade
...
docker exec -it old ghost -v
Ghost-CLI version: 1.1.1
Ghost Version (at /var/lib/ghost): 1.10.0
Latest version on Docker Hub: 1.24.2

You will have to edit manually your ./instance/$NAME/.env file and set the new GHOST_VERSION to upgrade it.

Would you like to rollback to a previous version, e.g. version 1.23.0? Just modify againt the .env file and upgrade again. Or pass the variables through the command line.

$ vi instances/old/.env
GHOST_VERSION=1.23.0

$ make p=instances/old/ upgrade
docker exec -it old ghost -v
Ghost-CLI version: 1.8.1
Ghost Version (at /var/lib/ghost): 1.23.0
Latest version on Docker Hub: 1.24.2

the actual docker image downloaded is ghost:1.x.x-alpine: the -alpine suffix is added here because I prefer it lighter than heavier.

Reliability ?

Well, we have seen quite a lot with these series and reliability has more to do with the hosting you are going to choose. And security policies you are going to enforce too. Far too much and too wide for a simple tutorial like this one. Remember also the disclaimer from the begnining: go for Ghost pro if you are looking for reliability :)

Cheers,
Manu

Ghost in A shell - part III : to Production, with performance

Manu — Mon, 04 Jun 2018 20:35:29 +0000

I am actually very happy with the outcome of the previous blog for my production: it has a proxy, it serves on HTTPs, sqlite3 is not so bad after all... and for my 7.5 users, it is just great.

Your definition of "great" might be slightly different...

What about defining production greatness as a setup that will give satisfaction based on:

performance (covered in this blog)

ease of maintenance (covered in next blog)

reliability (not sure that this will fit into the scope of the Tutorial)

Soooo... let's start with performance.

Philosophy

I am not going to dive into the depth of the performance theme, too many things to cover. Let me just suggest this path: to focus on setting up a toolbox that will allow you

to run performance/load tests easily (one command sounds good, as usual; it will be make gatling)
to view the output metrics easily (like in a web browser, thank you gatling!)

The idea behind is to base our decisions upon relative metrics, which means we are going to compare the results of the tests before and after making some changes (idealy: any change). Changes will be validated according to the variation of the results

But enough talking, let's make sure we are ready to go...

Pre-requisites

a) from the previous episodes...

I will make the assumption that you have been through my two first blogs:

Part I : localhost, that gets you familiar with the main commands of the Makefile, and focus on localhost
Part II : HTTPs is the norm, that shows you how to spawn ghost instances fearlessly on one server

There are no new dependencies, which means you are good to go with docker, make and the two repos prod-stack and ghost-in-a-shell.

b) if you wish to send emails (probability is high)

The Ghost Team makes some recommendations to send email, one solution being to leverage MailGun. Follow the link, and the steps behind this link.

That will give you credentials to send emails through their platform. Those credentials will go into ./etc/prod.env:

$ cp etc/prod.env.sample etc/prod.env
$ vi etc/prod.env
  ...
  # see https://docs.ghost.org/docs/mail-config#section-mailgun
  # you will find an updated screencast to understand exactly where to find these details
  MAILGUN_LOGIN=(Sandbox) default SMTP Login
  MAILGUN_PASSWORD=(Sandbox) default Password

Those variables will be used when spawning an instance in "production mode", as we can see within the Makefile:

prod:
    ...
    -e mail__transport=SMTP \   
    -e mail __options__ service=Mailgun \
    -e mail __options__ auth__user=${MAILGUN_LOGIN} \
    -e mail __options__ auth__pass=${MAILGUN_PASSWORD} \
    ...

Because we are now dealing with a 'production' environment, the setup is not as easy as it was when using your SMTP server... hence a few "surprises" on the way:

MailGun will ask you for a credit card, even if you keep using their free plan
With the default login and password, you will have to declare all the recipients you wish to send emails to (and they will have to approve receiving those emails)
To go around this limitation, you will have to set up your domain within MailGun. (and you will need to modify your DNS zone for this)

Therefore, if you prefer not using MailGun, you can remove the 4 lines above from the make prod command, or adapt it to another service provider (let me know how it went if you do, please!)

Next step is setting up the DB...

Switching the DB

a) objective

We wish to make use of MySQL instead of sqlite3, because it is recommended by the Ghost team. Fair enough.

b) setup

The prod-stack already runs a MariaDB in a container, as defined in the docker-compose.yml file:

  db-shared:
    image: mariadb:latest
    container_name: db-shared
    restart: always
    env_file:
      - $PWD/etc/db.env
    volumes:
      - db_data:/var/lib/mysql

There is only one piece of customization, the db.env file which holds the MYSQL_ROOT_PASSWORD. This file has been generated for you when you installed the prod-stack by running make. Here is the reminder:

# from prod-stack root directory
$ make
cp etc/traefik.toml.sample etc/traefik.toml
cp etc/db.sample.env etc/db.env
sed -i s/password/auMr9jsMrPnX0Oatb9j4yX2AYBN2jTQV/g etc/db.env
Generated etc/db.env
...

Well, I guess you know what is coming: copy the value from prod-stack/etc/db.env and paste it in your ghost-in-a-shell/etc/prod.env file. It now should look like:

# password set for db-shared, as defined in prod-stack/etc/db.env
MYSQL_ROOT_PASSWORD=auMr9jsMrPnX0Oatb9j4yX2AYBN2jTQV

# see https://docs.ghost.org/docs/mail-config#section-mailgun
# you will find an updated screencast to understand exactly where to find these details
MAILGUN_LOGIN=postmaster@your.domain.com
MAILGUN_PASSWORD=your-key

Done? You are ready to make use of a MariaDB instead of sqlite3 thanks to the following line in the Makefile.

prod:
    ...
    -e database__client=mysql \
    -e database __connection__ host=db-shared \
    -e database __connection__ user=root \
    -e database __connection__ password=${MYSQL_ROOT_PASSWORD} \
    -e database __connection__ database=${NAME} \
    ...

As a note, you can see that the connection to the DB server is hard-coded: it is made through the container name db-shared which is defined in the prod-stack/docker-compose.yml as seen previously.

`make prod`

In the two sections above, we have created a etc/prod.env file with three environment variables. This allow us to call make prod, which creates a ghost instance:

behind the Nginx+Traefik proxy (same as with make qa)
accessible through HTTPs (same as with make qa)
using MailGun to send emails ( new )
using a MariaDB instead of sqlite3 ( new )

Because we are in a relative mindset (remember the philosophy?), we are going to compare our new set up, performance-wise, with the old one...

define a new blog NAME for the exercise: export NAME=fire-at-blog
spawn the instance with sqlite3: make qa (we will use the default content to keep it simple)
measure performance (and produce our first report): make gatling

start again same blog using MariaDB: make stop prod (you could also go for make stop prod logs target to follow the logs and see when the instance is ready)
measure performance again: make gatling

compare the two reports written in the folder ./gatling-results

We do not really care about the absolute values of any of the two reports. We are interested by the differences. Whatever the expectations were.. we now have the detailed figures with one command. And we can access directly in our browser the reports.

For illustation purpose, I have only shown the high level figures for requests and active... but you have much more (thank you gatling!)

Of course, that will not alone make a decision. Let's not forget that we also want ease of maintenance (sqlite3 might be better here), and reliability (MariaDB gets several good points there)

Most important: we have a clear view on the performance of one solution vs the other

`make gatling`

The magic in the section above come from the command make gatling, which is actually quite simple:

gatling:
    docker run -it --rm \
        -v $(shell pwd)/etc/gatling-conf.scala:/opt/gatling/user-files/simulations/ghost/GhostFrontend.scala \
        -v $(shell pwd)/gatling-results:/opt/gatling/results \
        -e JAVA_OPTS="-Dusers=${GATLING_USERS} -Dramp=${GATLING_RAMP} -DbaseUrl=${GATLING_BASE_URL}" \
        --network=proxy \
        denvazh/gatling -m -s ghost.GhostFrontend

We are using a docker container which encapsulate all the (Java) layers needed by gatling.

-v $(shell pwd)/etc/gatling-conf.scala ... and mounts a few volumes for input (scenario file)
-v $(shell pwd)/gatling-results ... mounts the volume for the output.
-e JAVA_OPTS="-Dusers=${GATLING_USERS} ... adapts the script according to the environment variables (more details below)
--network=proxy allows the container to directly load-test its beloved siblings (e.g. the blog instances)

The scenario file, ./etc/gatling-conf.scala, has 4 sequences (self explicit):

atOnceUsers(nbUsers),
constantUsersPerSec(5) during (myRamp seconds),
rampUsers(nbUsers) over (myRamp seconds),
heavisideUsers(10) over (myRamp seconds)

You can see that two variables (nbUsers & myRamp) are available to give you a bit of flexibility in the loads you are expecting. Default values are provided in the Makefile and can ve overridden as usual.

GATLING_BASE_URL?=${PROTOCOL}://${NAME}:2368/${URI}
GATLING_USERS?=3
GATLING_RAMP?=5

By default, the galing container which targets the container that runs your blog instance. (hence the --network=proxy parameter in the gatling command). Here is a more realistic example:

GATLING_BASE_URL=https://your.qa.com/blog GATLING_USERS=20 make gatling

You might wonder what will be the "full URL(s)" since we make use of a "..._BASE_URL". You will have to change the configuration file here (or use another one). They are so far coded as the following array, which is use as many time as necessary (thanks to the last bit .circular).

  val uriFeeder = Array(
    Map("URIKey" -> s"welcome"),
    Map("URIKey" -> s"the-editor"),
    Map("URIKey" -> s"using-tags"),
    Map("URIKey" -> s"managing-users"),
    Map("URIKey" -> s"private-sites"),
    Map("URIKey" -> s"advanced-markdown"),
    Map("URIKey" -> s"themes")
  ).circular

By the way, we could have chosen some other strategy than .circular (e.g. random), but I have chosen a repeatable + predictable strategy here, for the purpose of being able to compare the results with a controlled scenario.

Let's apply this knowledge to load-test our proxy

Playing a bit with Nginx

Keeping in line with this methodology, we could have a closer look at the proxy (Nginx + Traefik) and compare our configuration with another one with NGinx cache... We will use the configuration prod-stack/etc/000-proxy-with-cache.conf.sample for this purpose:

proxy_cache_path /var/cache levels=1:2 keys_zone=STATIC:10m inactive=24h max_size=1g;

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;

  location / {
    proxy_pass http://traefik/;
    proxy_set_header Host $host;
    proxy_buffering on;
    proxy_cache STATIC;
    proxy_cache_valid 200 1d;
    proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
  }
}

The tricky part is

to continue playing around locally, while
making gatling access the blog through NGinx, and not directly from his container to the other container...

For this purpose, we are going to use the GATLING_BASE_URL variable along with the internal IP of our computer. This will make the gatling container query the blog through the host (which means through Nginx and Traefik) instead of heading straight at the ghost container.

Watch out: 127.0.0.1 will not work (neither localhost) since the container will query itself in that case, not the ghost instances.

Let's say our internal IP is 192.168.178.93... here is the process:

run a first container without cache: NAME=no-cache-blog make qa logs
once launched, run a gatling container: NAME=no-cache-blog GATLING_BASE_URL=http://192.168.178.93/no-cache-blog make gatling
activate Nginx cache in the prod stack

$ cd /your/path/to/prod-stak/etc/conf.d
$ rm 000-default.conf
$ cp ../000-proxy-with-cache.conf.sample 000-proxy-with-cache.conf

# at this point you can stop and start the stack again, 
# or simply go inside the nginx container and reload the conf:
$ docker exec -it nginx-entrypoint bash
root@bef39f4ad198:/# nginx -s reload
... signal process started

run a second container with cache: NAME=with-cache-blog make qa logs
once launched, run gatling again: NAME=with-cache-blog GATLING_BASE_URL=http://192.168.178.93/with-cache make-blog gatling
look at the results (no cache on left hand side, with cache on right hand side)

Performances are better with cache (as expected), but content could now "lag" and that could impact negatively the user experience... On the other end it could also impact positively the ease of maintenance, allowing us minor downtime of the blog itself (if well cached).

Again, the final choice will depend on you. At least, you will have metrics to compare the performances of the various solutions.

Keep in mind, and avoid...

Avoid changing multiple criteria when comparing performances.

For instance, doing the following comparison is a bad idea :

`make dev`	`make qa`	`make prod`
some performance	other perf.	yet-another perf.

The results are actually useless because you have multiple changes between the different sets of configurations: you are changing at the same time the DB, the proxy, and the email settings (potentially some cache layer). Therefore you will be blind of the individual effects of each of them.

Conclusion

You now can run three modes of ghost instances, for development, testing or production:

Command	Description	Remarks
`make dev`	straightforward (local) installation if you simply wish to bridge a container port on your host (3001 by default)	also known as `make`
`make qa`	The former being not really convenient if you wish to serve on standard ports (80 or 443), and if you want anyone to access your blog easily, this command sets up a traefik router	recommended usage of prod-stack
`make prod`	Same as above, using a MariaDB instead of SQLite and an email provider (Mailgun)	requires a (free) account at Mailgun

The (more) important get-away of this blog though, is the simple command make gatling, and using it to validate (small and well identified) changes along the way.

A good practice would actually be to integrate the command in the continuous integration workflow, and to raise alerts if the performance goes down

In the next blog, I will introduce a few helpers for the maintenance... in order to keep a bit of order in all the instances we can now spawn here and there

Ghost in A shell - part II : HTTPs is the norm

Manu — Fri, 25 May 2018 20:36:11 +0000

Overview of the context in three points:

The objective of this series is to facilitate (automate) the spawning of Ghost blogs
This blog comes after part I - localhost, where I focused on one's computer
The targeted audience are developers/ops who wish to build / wipe out ghost instances in a breeze

Would you be interested, please take a seat and make yourself at home, I will try 1) to make your reading comfortable, 2) to lead you to the magic world where Ghosts appear from thin air... and 3) to give you the magic wand that will pop out:

Appetizer: the repo(s)

In order to fulfil my promises, I will make use of two repositories:

prod-stack, which I have already introduced in the first blog. This repo will act as router and proxy for free, including the discussions with Let's Encrypt
ghost-in-a-shell, which basically is the material behind the blogs.

By cloning those two guys, and setup a few variables, you will be able to create your ghost instances with our old pal make

Quick setup for prod-stack

Just to save you a return ticket to the first blog, here are the few steps necessary to get the prod-stack up and serving:

$ git clone git@github.com:ebreton/prod-stack.git
Cloning into 'prod-stack'...
...
$ cd prod-stack
$ make
cp etc/traefik.toml.sample etc/traefik.toml
cp etc/db.sample.env etc/db.env
sed -i s/password/auMr9jsMrPnX0Oatb9j4yX2AYBN2jTQV/g etc/db.env
Generated etc/db.env
docker-compose pull
Pulling nginx-entrypoint ...
...

Once the logs stabilize, you can Ctrl-C to get back on the command line, and docker ps to check that the stack is running.

The Traefik dashboard will be available on http://localhost:8081 "protected" by Basic Authentication with the user test/test

Quick setup for ghost-in-a-shell

Well, you should have make, docker and the prod-stack installed at this point.

You are ready to spawn a blog on http://localhost:3001 with make, or on http://localhost/ghost-local with make traefik.

Let's see how to do better!

from here on, running a container with make will be called the standalone mode. Running a container with make traefik will be called the traefik mode (how original!)

Starters: environment variables

The two modes make use of environment variables, with default values defined in the (well commented) .env file.

If you happen to delete this file, make will complain

$ rm .env
$ make
.env file is missing
make: *** [check-env] Error 1

If .env is found, you will be able to check its values with make vars

$ make vars
# values that will be used to create the blog URL
  NAME=ghost-local
  PROTOCOL=http
  DOMAIN=localhost
  PORT=3001
  URI=ghost-local

Changing the configuration of the instance is a matter of changing those variables:

either directly in the .env file
or in your environment, e.g. adding export NAME=blog in your terminal (or in your .bashrc when it comes to some other variables that will vary less in time, e.g PROTOCOL)
or in the command line, e.g. PORT=3002 make

$ export NAME=blog
$ PORT=3002 make vars
# values that will be used to create the blog URL
  NAME=blog
  PROTOCOL=http
  DOMAIN=localhost
  PORT=3002
  URI=ghost-local

The variables should be self explanatory, especially if you think PROTOCOL://DOMAIN:PORT/URI, but it never hurts to spell the words, so here is a quick summary of the variables, with their default values and which mode uses them

Variable	Description	default value	used by standalone / traefik mode
`NAME`	for the docker container and the data folder	ghost-local	both
`PROTOCOL`	to build the URL	http	traefik
`DOMAIN`	to build the URL	localhost	both
`PORT`	to build the URL	3001	standalone
`URI`	to build the URL	${NAME}	traefik

The Main Dish: HTTPs

So here we are, we wish to run an instance on http://my.domain.com/my-blog

I guess the following command is not going to be too much of a shock for you:

DOMAIN=my.domain.com URI=my-blog make traefik

You might actually have jumped straight to http*S*://my.domain.com/my-blog with PROTOCOL=https DOMAIN=my.domain.com URI=my-blog make traefik

That's fine, if you had also anticipated

The DNS configuration, i.e my.domain.com is already pointing to your server
The traefik configuration, and in particular if you have setup your own email address in the traefik.toml file as directed in prod-stack/README.

By the way, do not forget to change the Basic Authentication used to access Traefik dashboard (also pointed out in prod-stack/README)

In this case, you could actually export PROTOCOL=https and export DOMAIN=my.domain.com in your environment, and simply run URI=my-other-blog make traefik.

If you hadn't anticipated... well... you now know what to do :

Set DNS config at your preferred provider
Adapt Traefik config in prod-stack
Ease your process by adding some variables to your environment

As a bonus, you can also force the redirection of http to http*s* (and raw domain.com to www.domain.com as an extra bonus). This is done by Nginx, and through the configuration sample in the repo, which you just have to move to the ./etc/conf.d folder, and then restart the stack

$ cd /path/to/prod-stack
$ mv ./etc 002-redirect-to-https.conf.sample ./etc/conf.d/002-redirect-to-https.conf
$ make
...

Doing all of this will

redirect http://domain.com/blog to https://www.domain.com/blog (which go well beyond our initial objectives, but hey! that's nice and free, and optional)
get a certificate from Let's Encrypt for www.domain.com
serve the blog

But there is more...

Digestive: helpers

We have already used make vars that listed the values of the environment variables.

There is a few other helpers command, which are listed in the table below:

Command	Description	Variables used
`make vars`	Display the values of all env vars	All
`make ps`	Display the running containers	None
`make shell`	Connect to given Ghost container	NAME
`make logs`	Tail and follow the logs of given Ghost container	NAME
`make stop`	Stop given Ghost container	NAME

There is really few to say here, they are just convenient wrappers around standard docker commands.

Dessert (conclusion)

I must admit that I am quite satisfied with the outcome of this second blog.

We just have one tiny little bit of work to cope with now: the technical debt !

More seriously, the choices made so far are shortcuts for most of them.

Ghost is running in development mode,
and still uses sqlite.
Not to mention there is no monitoring neither backup of anything (by the way, do not forget that you have Ghost(Pro) to relieve you from those burdens).

It is therefore a very appropriate time for a very big red disclaimer:

do NOT use this stack in production

Next?

I will look more deeply into Ghost setup in next blog: its running mode and the DB to start with...

Ghost in A shell - Part I : localhost

Manu — Thu, 24 May 2018 20:38:16 +0000

The last blog from the Foundation Ghost, "After 5 years and $3M, here's everything we've learned from building Ghost", was an eye-opener for me.

Not only the company model: a Foundation whose owners will not be able to sell, on purpose; but especially the actual love behind the product:

when we do anything you can be absolutely certain it's in the sole interest of building the best product we can for the right reasons.

Coming from WordPress, having played with Jekyll and Hugo, I could only take the bait to try Ghost. Therefore, I happily threw myself into the doc and the installation process to get a taste of the product.

Objective

The overall objective of this (short) series of blogs is to be able to spawn easily multiple (independent) ghost blogs on one's beloved server.

It's intended for developers/ops that wish to put their hands into the mechanics of Ghost, and need to easily spawn/kill blogs here and there.

I will not go through the code of the blog itself, neither on the topics of writing blogs, nor creating a theme. I will limit my sharing to the installation-side of Ghost, and how to facilitate/automate the process.

Would you not feel like a devops at the end of the day, you might find better value in the Ghost(Pro) service, which will do a much better job of hosting your blogs, and keeping them running and up to date.

In this first (of a few) blogs, I will focus on the installation of instances on localhost. The purpose is to set up locally a sandbox, as similar as possible as what we will ultimately have on production (hopefully).

Following the path

There I went on my way, clicking on the Developer > Documentation link of the Ghost website, scrolling a bit down, and finding the expected entry point for setting up things:

After clicking on the Setup box, I had the choice between "I want to install Ghost on my own server" and "I want to try Ghost out quickly". I thought I was very lucky and I clicked on the second option: local install guide.

The TL;DR version of local installs, is you want to make use of Ghost CLI's super handy command: ghost install local

From there I stumbled on

the current configuration of my environment
its Node version
and my lack of knowledge of the JavaScript stack.

I quickly switched to one of my server running Ubuntu, where it was also quite painful to get all the versions aligned... At the end of my 10mins timebox, I could not make the ghost-cli execute itself properly.

I (quickly) switched again for a Docker image, and there you go...

Thank you Docker!

There is an official image (i.e. maintained by the DockerHub staff) on the docker hub, ghost, with a few examples of the commands available.

Back on my Mac I created a simple Makefile:

NAME?=ghost-local
DOMAIN?=localhost
PORT?=3001

dev: 
    # Simply start a ghost container making it directly available through $$PORT
    docker run --rm -d --name ${NAME} \
        -v $(shell pwd)/instances/${NAME}:/var/lib/ghost/content \
        -p ${PORT}:2368 \
        -e url=http://${DOMAIN}:${PORT} \
        ghost:1-alpine

if/when copy-pasting, make sure to use tabs for the indentation (not spaces)

Running make was the only thing left to get my blog running on http://localhost:3001

Running NAME=another PORT=3002 make runs a second blog available on http://localhost:3002

Docker parameters

A quick explanation on the arguments passed above:

--rm will remove the container when it stops, thus preventing conflicts when spawning a new container with the same ${NAME}
-d will run the container in daemon mode
--name ${NAME} will name your container. This one is optional, but it is nice to have the same name for the container and the data volume defined on the following line...
-v $(shell pwd)/instances/${NAME}:/var/lib/ghost/content will mount the Ghost files locally on your file system into subfolder ./instances. That will persist data (and allow you to sneak into the code later on).
-p ${PORT}:2368 exposes the blog on the local ${PORT}
-e url=http://${DOMAIN}:${PORT} tells ghost what URL will be used to access it.

A bit of cleaning up

Before next step, you can stop the containers with docker stop ghost-local another. However, the data will not be lost thanks to the volumes.

Moreover, they will be used again the next time you start a blog with the same name.

Thank you Traefik !

My initial objective achieved, I went for the bonus...

The bonus

The first one was to be able to serve on port 80, and to have the blogs sit on different paths. Something like

The "prod-stack" companion

I have my so humbly called "prod-stack" repo that provides me with this kind of routing functionalities (among others). It is based on Traefik (and NGinx), and allows me to quickly set up a proxy, a DB and a memcached.

It's running on my Mac, therefore making this solution free for me. I could only buy it :)

For you dear reader, I have taken advantage of this blog to open my prod-stack on github, and I also have done a bit of curating with the Readme (even though it is still more a draft than a real documentation).

Here is the fast-track setup to pull and run all docker containers:

$ git clone git@github.com:ebreton/prod-stack.git
Cloning into 'prod-stack'...
...
$ cd prod-stack
$ make
cp etc/traefik.toml.sample etc/traefik.toml
cp etc/db.sample.env etc/db.env
sed -i s/password/auMr9jsMrPnX0Oatb9j4yX2AYBN2jTQV/g etc/db.env
Generated etc/db.env
docker-compose pull
Pulling nginx-entrypoint ...
...

Once the logs stabilize, you can Ctrl-C to get back on the command line, and docker ps to check that the stack is running.

The Traefik dashboard will be available on http://localhost:8081 with the user test/test

Please, do not hesitate to feedback thanks to issues if you felt the curiosity to try it.

Adding `make traefik`

After making sure that Traefik is running, I added this command to the Makefile:

traefik:
    # Start a ghost container behind traefik on path $$NAME
    # Beware of --network used, which is the same one traefik should be using
    docker run --rm -d --name ${NAME} \
        -v $(shell pwd)/instances/${NAME}:/var/lib/ghost/content \
        -e url=http://${DOMAIN}/${NAME} \
        --network=proxy \
        --label "traefik.enable=true" \
        --label "traefik.backend=${NAME}" \
        --label "traefik.frontend.entryPoints=http" \
        --label "traefik.frontend.rule=Host:${DOMAIN};PathPrefix:/${NAME}" \
        ghost:1-alpine

The parameters are similar to the ones of the previous command with the following changes:

The ${PORT} is not exposed anymore since Traefik will handle the routing
--network=proxy launches the container in the same network as the one where Traefik is running (the values proxy is arbitrary and comes from the default configuration of the "prod-stack")
--label key=value are used by Traefik to create the appropriate route

Running make traefik will now run a blog on http://localhost/ghost-local

Running NAME=another-ghost make traefik runs a second blog available on http://localhost/another-ghost

Conclusion

First of all , I wanted to easily spawn Ghost blogs on my local machine,

The native documentation did not survive to my impatience, mainly because I knew it would be fast and smooth with Docker.

Secondly , I wanted to use the standard port (80 to start with), and route the blogs according to their path,

It was given for free with Traefik (and my "prod-stack") and I must admit I had a lot of fun playing with make. However, the result is still one for a developer, surely not ready for production (despite the name of the stack).

Nevertheless, the stack can provide HTTPs thanks to Let's Encrypt, a DB and some easy way to manage all this... I will dive into this in the next part to get something more robust... maybe

https://manu.breton.ch/blog ?

DEV Community: Manu

make release: save time, gain in reliability

A story, where any resemblance to reality is pure coincidence...

Objective

Pre-requisites

Set up

files

environment variables

docker

versions

Usage

Behind the curtains

1. a new branch on github rc-X.X.X

2. a new release on github based on this branch

3. an updated Changelog with meaningful information ...

4. that actually comes from the differences between this release and the previous one

5. a way to check the current / next version of your code

6. an updated badge on your README (because badges are cool)

Bonus

Conclusion

Ghost in A shell - part IV (and final): Maintenance

Environment variables

All-in-one configurations

Upgrade (and Downgrade)

Reliability ?

Ghost in A shell - part III : to Production, with performance

Philosophy

Pre-requisites

a) from the previous episodes...

b) if you wish to send emails (probability is high)

Switching the DB

a) objective

b) setup

make prod

make gatling

Playing a bit with Nginx

Keep in mind, and avoid...

Conclusion

Ghost in A shell - part II : HTTPs is the norm

Appetizer: the repo(s)

Quick setup for prod-stack

Quick setup for ghost-in-a-shell

Starters: environment variables

The Main Dish: HTTPs

Digestive: helpers

Dessert (conclusion)

Next?

Ghost in A shell - Part I : localhost

Objective

Following the path

Thank you Docker!

Docker parameters

A bit of cleaning up

Thank you Traefik !

The bonus

The "prod-stack" companion

Adding make traefik

Conclusion

`make prod`

`make gatling`

Adding `make traefik`