As the principle responsible for the design of middleware software at Zebrium, I'm writing to share some of the choices we made and how they have held up. Middleware in this context means the business-logic that sits between persistent storage and a web-based user interface.
Goals
Here are some of the Zebrium Middleware design goals:
- DRY (Don't Repeat Yourself) - Limit the amount of boiler-plate code that is copied from one API to another
- Provide quick turn around for testing and deployment
- A language that has broad support and is deployed widely in similar applications
Technologies
Early on we made several selections of key technology that has proved mostly successful:
- SQL Databases, Postgres for user/session management, Vertica for analytics
- Go for middleware software
- Containers for deployment
SQL Databases
Having worked with both no-SQL and SQL databases in the past it is clear that the promise of no-SQL scaling and reliability has not born out. The vast majority of database operations are read-only and using SQL allows us to push the cache of data needed to make complex correlations down to the database software that is optimized for the task. The reliability of SQL has also been addressed, either in clustered server implementations or hosted services with replication.
Go
Coming from C/C++, I have been repeatedly impressed with Go. The execution speed allows me to introduce complex data transformations without significant overhead, while the speed of compiling renders the test cycle close to interactive. And the scope and variety of open-source support libraries is impressive; more on that later.
Containers
Building the middleware server into a container allows us to test it in simple platform independent docker-compose configurations as well full-stack Kubernetes staging and production. For example, our UI engineers can deploy the container on their MacBooks using Docker.
Templates
We used Go templates effectively for two areas of the middleware:
- APIs to perform CRUD (create, read, update, delete) on generic database tables
- Interpolating arguments into complex SQL queries
CRUD with go-raml
I was looking for an API framework that allowed me to reduce the amount of repeated specifications. RAML and go-raml (github.com/Jumpscale/go-raml) fit the bill. Though not as mature as Swagger/OAS, RAML provides more expressions on types that can be shared between APIs. And, because go-raml is written in Go and open-source, I was able to fix bugs and write my own extensions. Here is a sample RAML type:
Vote:
properties:
id:
type: string
generated: true
createTime:
type: datetime
required: false
createUserId:
type: string
generated: true
modifyTime:
type: datetime
required: false
modifyUserId:
type: string
generated: true
name:
type: string
pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"
db: "rid"
thumb:
enum: [ up, down ]
desiredPriority:
enum: [ high, med, none ]
The framework will validate inputs and translate JSON camel-case tags into SQL snake-case, with the "db" override above for name. With this type, I can describe various HTTP methods that act on it. The following is an HTTP PUT which creates or updates a vote by name. The responses document the possible return values.
/vote:
dataSource: "odbc/t_votes"
put:
securedBy: [ Token ]
description: Create or update a vote
body:
application/json:
type: Vote
responses:
200:
body:
application/json:
type: Vote
201:
body:
application/json:
type: Vote
400:
body:
application/json:
type: Meta
500:
body:
application/json:
type: Meta
The above text is translated by go-raml and our templates into Go code that serves the request. We have templates for the following HTTP operations:
| Method | URL Pattern | Description |
| --- | --- | ---|
| GET | {object} | Get all objects |
| GET | {object}/{id} | Get object by ID |
| POST | {object} | Create object |
| PUT | {object} | Create/update object by name |
| PUT | {object}/{id} | Create/update object by ID |
| PATCH | {object}/{id} | Update fields in object by ID |
| DELETE | {object}/{id} | Delete object by ID |
| POST | {object}/{action} | Custom request/response API |
We also have callbacks embedded into the generated code to deal with pre- and post-processing, allowing us to adapt this model to a wide variety of object types. About half of our APIs are handled with the above CRUD operations and callbacks, while the other half are handled with custom actions, the last HTTP pattern above.
SQL Templates
The other use of Go templates in our framework involves adding parameters to complex queries. The following is an example an SQL template:
{{ define "prefix" }}
select m.abfmt_str from {{.schema}}.t_metadata_namespaces as m
inner join {{.schema}}.t_etable as e on e.namespace = m.namespace
where e.etname in ({{ csv .etypeList }})
order by pid desc
{{ end }}
The formatting of a Go string array into a SQL list is accomplished with a template function (https://golang.org/pkg/text/template/#Template.Funcs):
"csv": func(in []string) string {
out := ""
for _, v := range in {
if out != "" {
out += ","
}
out += "'" + v + "'"
}
return out
},
This design provides a separation of concerns between the Go code that provides parameters to the often complex SQL that extracts the data.
Conclusions
To review our goals from the top:
- RAML and go-raml, together with pre- and post-processing filter functions generate about half of our API code
- Container deployment and quick compile times provide fast and flexible dev/test cycles
- Go and it's collection of open-source libraries cover relevant functionality and enjoy broad support
But most of all, this has helped the company I work for (Zebrium) to build and deploy versions of software rapidly and with high quality.
Posted on behalf of the author: Alan Jones @ Zebrium
Latest comments (0)