Intro - Rapid Software Development in the Modern World
(Feel free to skip to the tutorial)
This post is the first in a planned series about rapid software development in the modern world.
Rapid developer onboarding is something we've been dealing a lot with at Otomato software. But what really got us started on diving deeper into this - was my conversation with Elad Meidar a few months back.
Elad was talking to me about how non-trivial it has become to choose a stack in today's world. And let's say you've chosen your tools - only the most experienced developers really know how to set up CI, testing, deployment, instrumentation, monitoring, security, etc. correctly. A lot of Ops knowledge is involved in even getting things initially running.
Our current goal is to take that knowledge that we've accumulated and make it available as a service. And while we're doing that - we're exploring the tooling that's currently available.
Enter Buffalo
Buffalo is a framework, (or a tool) for rapid web development in Go. Cloud Native DevOps folks (and that's what we are at Otomato) have a soft spot for Golang and that's why I'm starting this series with Buffalo.
The official getting started section of Buffalo documentation is great but as I ran through it I noticed it lacks some operational details that I'm planning on exposing. Again there's Ops knowledge lurking in the dark!
Installations
Quite naturally one would need to install Go.
On a Mac:
brew install golang
On Ubuntu/Debian:
sudo apt update && sudo apt install golang
Note: on older systems (such as Ubuntu 20.04) you'll get a very old version of Go (1.13) by default when installing with apt. So instead - choose the download-n-extract option here.
For additional installation options go here
Do you do frontend?
Buffalo can generate both pure backend API services and fully-fledged webapps with frontend matter included. The frontend is in Javascript - so if you want that - you'll also need Node and either yarn or npm (the default).
On a Mac:
brew install nodejs
On Ubuntu/Debian:
sudo apt update
sudo apt install nodejs npm
Do you want containers?
Buffalo makes quite a few educated assumptions when generating your project. One of them is that you'll want to wrap your app in a container. You can, of course opt out, but why? So if you're going with the flow and enjoying the benefits of containerization - you probably already have Docker installed. If not - please do so now - we'll need it further along the tutorial.
Finally - bring in the Buffalo
On a Mac:
brew install gobuffalo/tap/buffalo
On Linux;
wget https://github.com/gobuffalo/cli/releases/download/v0.18.8/buffalo_0.18.8_Linux_x86_64.tar.gz
tar -xvzf buffalo_0.18.8_Linux_x86_64.tar.gz
sudo mv buffalo /usr/local/bin/buffalo
Create a project
Buffalo has a project scaffolding feature that allows us to generate a new app complete with:
- a local git repository
- a backend api
- a db integration
- a frontend
- a Dockerfile
- a CI pipeline.
Let's create a webapp called testr. It will be used to manage test assignments for new and existing trainees. (Did I mention we do technical training at Otomato too?)
The command to create a new project is buffalo new.
The default DB backend used by Buffalo is PostgreSQL.
We will be using Github for SCM, so we'll choose Github Actions as our CI provider.
buffalo new testr --ci-provider github
After buffalo shows us what it's bringing in and generating (quite a bunch of stuff really) it will say:
Initialized empty Git repository in /Users/username/git/testr/.git/
DEBU[2022-08-28T23:37:54+03:00] Exec: git add .
DEBU[2022-08-28T23:37:54+03:00] Exec: git commit -q -m Initial Commit
INFO[2022-08-28T23:37:54+03:00] Congratulations! Your application, testr, has been successfully generated!
INFO[2022-08-28T23:37:54+03:00] You can find your new application at: /Users/antweiss/git/testr
INFO[2022-08-28T23:37:54+03:00] Please read the README.md file in your new application for next steps on running your application.
So we 'll do just what it tells us to:
cd testr
git add .
git commit -q -m "Initial Commit"
Set up the DB
Before we can actually start developing our code there's
a need to spin up a database. We could, of course, use a managed DB but it would probably cost us a few bucks. So for local development it makes much more sense to run the DB in a container.
Let's run PostgreSQL (the default Buffalo DB backend):
docker run --name buffalo-postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres
Note: we're running PostgreSQL with a very naive password here , which is fine for local development but not fit for anything production-like.
We're also exposing it on localhost:5432 - which is where a Buffalo app is configured to look for it by default.
These configurations are defined in a buffalo-generated file database.yml which we'll use shortly.
Running the DB container isn't enough. We also need to create a database for our app.
This can be done by entering the container and running good old SQL commands. But Buffalo creators recommend the use of Soda - a small and useful CLI utility that makes managing DBs easier.
Install soda:
go install github.com/gobuffalo/pop/v6/soda@latest
And create a DB:
soda create -a
Soda creates all the databases configured in the file database.yaml that Buffalo has generated for us.
Note: By default Buffalo uses its own ORM library called pop for DB integrations. Pop provides a wrapper for soda - so we can also run soda commands through buffalo aliases: buffalo pop create -a or buffalo db create -a.
Buffalo delivers even more useful DB stuff with the help of pop - like model generation. But we'll cover that in the follow-up post.
Start development
Buffalo provides us with a buffalo dev command which allows running our app with live reloading - i.e restarting the application server each time we change the code.
Let's run!
buffalo dev
Now we can visit http://localhost:3000 in browser and see our app running!
And - we're live!
The web UI we see is generated from testr/templates/home/index.plush.html using the plush templating engine. Also to be covered in a separate post.
Let's Have Some CI
As the final step of this walkthrough - let's push our code to Github and verify the generated CI pipeline works.
Generate a new Github repo
I heartily recommend using Github's gh cli tool:
From the testr directory run:
gh repo create --public <your-user-or-org>/buffalo-testr --push --source .
This will create the repo and immediately push the code to it, which in turn starts the workflow defined in .github/workflows/test.yml:
name: test
on:
push:
pull_request:
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:9.6-alpine
env:
POSTGRES_DB: testr_test
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- uses: actions/checkout@v3
- uses: actions/setup-go@v3
with:
go-version: ~1.18
cache: true
- name: setup
run: |
go install github.com/gobuffalo/cli/cmd/buffalo@latest
- name: test
env:
TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/testr_test?sslmode=disable
run: |
buffalo test
We can see that this workflow:
- spins up a PostgreSQL service container
- installs buffalo
- runs
buffalo test- which in turn creates the DB in the container and runs some tests:
[POP] 2022/09/12 15:24:04 info - dropped database testr_test
[POP] 2022/09/12 15:24:05 info - created database testr_test
pg_dump: error: connection to server at "127.0.0.1", port 5432 failed: FATAL: database "testr_development" does not exist
[POP] 2022/09/12 15:24:05 info - Migrations already up to date, nothing to apply
[POP] 2022/09/12 15:24:05 info - 0.0102 seconds
[POP] 2022/09/12 15:24:05 warn - Migrator: unable to dump schema: open migrations/schema.sql: no such file or directory
time="2022-09-12T15:24:06Z" level=info msg="go test -p 1 -tags development testr/actions testr/cmd/app testr/grifts testr/locales testr/models testr/public testr/templates"
go: downloading github.com/gobuffalo/suite/v4 v4.0.3
go: downloading github.com/gobuffalo/httptest v1.5.1
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
ok testr/actions 0.019s
? testr/cmd/app [no test files]
? testr/grifts [no test files]
? testr/locales [no test files]
ok testr/models 0.012s
? testr/public [no test files]
? testr/templates [no test files]
Voila! The workflow works. It doesn't create a Docker image for us (so no artifacts) - but it does run some basic integration testing.
Tests for Buffalo apps is another topic for yet another post.
To Sum Things Up
Buffalo is a well thought-out rapid development framework for full-stack apps or standalone backend APIs. It does pack quite a lot to get us started, but it still leaves the developer in the playground - without any clear guidelines regarding where and how to deploy their code for production.
And what are you using for rapid bootstrapping of new services?
What other rapid development frameworks would you like us to cover?
Let us know in comments - our research is only starting!
Top comments (4)
Cover asp.net core, please!
https://learn.microsoft.com/en-us/aspnet/core/tutorials/min-web-api?view=aspnetcore-6.0&tabs=visual-studio-code
I have to admit - Microsoft technologies aren't exactly our cup of tea... Not that we're against them - it's just that none of our existing client base use them. We're exclusively Linux-based.
But thanks for the pointer - we'll look into it when and if time allows.
dotnet projects have been hosted primarily on Linux for the past 5 years. If you add them to your coverage you might get some of those customers too.
Thanks for the great intro @antweiss ! Looking forward to the rest of the series!