"Why we" articles are meant share about our engineering considerations and decisions, but it does not mean our decision would be the best for your use case, as your unique context matters a lot!
Our CLI was originally written NodeJS, 2 years ago.
It was something we quickly hacked together at the early beginnings of UI-licious when our focus was to move fast and iterate the product quickly. We wanted to roll out the CLI ASAP, so that users with a CI/CD can hook up their tests to their front-end deployment pipeline. The ever useful commander package was helpful in quickly setting up the CLI.
What's the matter with the original CLI?
This version served most users pretty well, especially in the startup beta-release days. And while we do dogfood our own CLI in our own CI/CD pipeline and felt that it could be better, it wasn't til feedback from mature software teams that were using the CLI heavily in their CI/CD pipeline that made it obvious that we need a better solution.
The issues mostly had to do with installation of the CLI. You see, the original CLI works pretty well for developers and testers. But it wasn't so friendly for DevOps, because npm can be a pretty big pain. - I'll come to that in a bit.
So we decided to rewrite the CLI from scratch and set out what would be the goals for the CLI.
Goals for the new CLI
1. Zero deployment dependencies
While node.js/npm has conquered the front-end development landscape.
Its easy to forget, that a very large segment of current web development still use good old tools. And most CI environments for non node.js based project, will not have them preinstalled.
As a result, in order to use our CLI toolchain within a CI for such projects, several users would need to at best wait an additional 15 minutes to install the whole node.js/npm stack.
Or at worse find it outright impossible due to networking policies, or dependency incompatibility with their existing projects.
So the less we can depend on - the better.
Realistically absolute zero dependency is impossible, for example you are always dependent on the OS. But it is a goal to strive towards.
2. Single file distribution
Having worked with many CLI tools, the ability to download a single file and execute commands - without an installer, or even setup process - does wonders to a user.
This has an additional benefit of making it easily backwards compatible with our NPM distribution channel. By quick single file glue code to link the NPM commands to the new file.
Evaluating our options
Node.js + NPM
good
- Works well for >75% of our use case
- Easy for company to maintain. JS is a required knowledge for all our developers
- Easy to code
- Cross platform
bad
- Not a single file
- Node.js or NPM dependency and compatibility issues for a small % of users who must use outdated builds (for other engineering reasons)
- Many enterprise network policies are not very NPM friendly
overall
This would be an obvious choice for a JS exclusive project, where node.js and NPM is a safe assumption. Or when we want to get things done asap.
Unfortunately that is not us. And compatibility hell is a huge pain when it includes "other peoples code".
Java
good
- Extremely Cross platform
- Single JAR file
- Easy for company to maintain. Java is our main backend language
neutral
- [Subjective] CLI library syntax : feels like a chore
bad
- Probably way freaking overkill in resource usage
- JVM dependency : We probably have more users without java installed vs NPM
overall
Java is notoriously known for their obsession with backwards compatibility. If we built our CLI in java 6, we can be extremely confident that we would not face any compatibility issues with other projects. Running with the same code base on anything from IOT devices to supercomputers.
However, it is still a giant dependency. While relatively easier for anyone to install then node.js / npm, the fact that 25%+ users will need to install a JVM just to support our tool doesn't suite well with us.
And seriously, other then java based tools themselves. Those who uses java for their online SaaS product is a rarity. So Β―\_(γ)_/Β―
Shell scripting + Windows shell?
good
- Smallest single file deployment (by byte count)
- Very Easy to get something to work
neutral
- Heavily dependent on several OS modules, while most would be safe assumptions for 90% of the use cases. It is something that needs to be aware and careful of. Mitigation can be done using auto installation steps for the remaining 9% of use cases.
bad
- What CLI libraries?
- Writing good, easy to read bash scripts isn't easy, nor easy to teach.
- Hard for company to maintain : Only 2 developers in the company would be qualified enough to pull this off : and they have other priorities
- Windows? Do we need to do double work for a dedicated batchfile equivalent
- Remember that 1%?, that tend to happen for what would probably be a VIP linux corporate environment configured for XYZ. This forces the script writer to build complex detection and switching logic according to installed modules. Which will form an extremely convolute the code base easily by a factor of 10 or more (an extreme would be : no curl/wget/netcat? write raw http request sockets)
overall
Despite all its downsides, its final package would be crazy small file size of <100KB - uncompressed and un-minified. (meaning it can go lower)
For comparison our go binary file is 10MB
Especially in situations with specific constraints, such as a guarantee on certain dependencies, or projects where that last 1% does not matter : This would be my preferred choice.
An example would be my recent dev.to PR for a docker run script.
Feature : docker-run.sh script + docker container build
#1844
What type of PR is this? (check all applicable)
- [ ] Refactor
- [x] Feature
- [ ] Bug Fix
- [ ] Documentation Update
Description
A single bash script that helps quickly setup either a DEV or DEMO environment
bash-3.2$ ./docker-run.sh #---
#
# This script will perform the following steps ...
#
# 1) Stop and remove any docker container with the name 'dev-to-postgres' and 'dev-to'
# 2) Reset any storage directories if RUN_MODE starts with 'RESET-'
# 3) Build the dev.to docker image, with the name of 'dev-to:dev' or 'dev-to:demo'
# 4) Deploy the postgres container, mounting '_docker-storage/postgres' with the name 'dev-to-postgres'
# 5) Deploy the dev-to container, with the name of 'dev-to-app', and sets up its port to 3000
#
# To run this script properly, execute with the following (inside the dev.to repository folder)...
# './docker-run.sh [RUN_MODE] [Additional docker envrionment arguments]'
#
# Alternatively to run this script in 'interactive mode' simply run
# './docker-run.sh INTERACTIVE-DEMO'
#
#---
#---
#
# RUN_MODE can either be the following
#
# - 'DEV' : Start up the container into bash, with a quick start guide
# - 'DEMO' : Start up the container, and run dev.to (requries ALGOLIA environment variables)
# - 'RESET-DEV' : Resets postgresql and upload data directory for a clean deployment, before running as DEV mode
# - 'RESET-DEMO' : Resets postgresql and upload data directory for a clean deployment, before running as DEMO mode
# - 'INTERACTIVE-DEMO' : Runs this script in 'interactive' mode to setup the 'DEMO'
#
# So for example to run a development container in bash its simply
# './docker-run.sh DEV'
#
# To run a simple demo, with some dummy data (replace <?> with the actual keys)
# './docker-run.sh DEMO -e ALGOLIASEARCH_APPLICATION_ID=<?> -e ALGOLIASEARCH_SEARCH_ONLY_KEY=<?> -e ALGOLIASEARCH_API_KEY=<?>'
#
# Finally to run a working demo, you will need to provide either...
# './docker-run.sh .... -e GITHUB_KEY=<?> -e GITHUB_SECRET=<?> -e GITHUB_TOKEN=<?>
#
# And / Or ...
# './docker-run.sh .... -e TWITTER_ACCESS_TOKEN=<?> -e TWITTER_ACCESS_TOKEN_SECRET=<?> -e TWITTER_KEY=<?> -e TWITTER_SECRET=<?>
#
# Note that all of this can also be configured via ENVIRONMENT variables prior to running the script
#
#---And does the deployment using docker. Includes option to do a reset prior to deployment.
Optional contextual information provided here : https://dev.to/uilicious/adopt-your-own-devto----with-a-single-command-almost-1c04
Need advice on ...
if someone can guide me on how to run dev.to in "Production" mode, it would be great in improving the overall docker container performance
Added to documentation?
- sample readme for docker hub : https://github.com/uilicious/dev.to-docker/blob/feature/docker-run-script/dockerhub-readme.md
- https://cloud.docker.com/u/uilicious/repository/docker/uilicious/dev.to
- I can modify readme if this looks good?
What gif best describes this PR
What gif best describes how it makes you feel?
Go lang
good
- Single binary executable file
- Reasonably good libraries available
- Language basics are relatively easy to learn (jumping from java)
- Has a cute mascot
neutral
- Steep usage learning curve, on following its opinionated coding practises.
bad
- No one on the team can claim to have "deep experience" with go
- Due to extreme type safety : Processing JSON data is really a pain in the ***
overall
One of the biggest draw is the ability to compile to any platform with the same code base, even ancient IBM systems.
While the language itself is easy to learn. Its strict adherence to a rather opinionated standard is a pain. For example, compiler will refuse to compile if you have unused dependencies in your code - among many many other things. This works both to frustrate the developer, and force better quality code.
Personally I both hate and respect this part of the compiler, as I have looser standard when experimenting in "dev mode", while at the same time have deep respect for it, as I follow a much stricter standard on "production mode".
So why GO?
Node.js, Java, C, C++, etc - are clearly out of the picture based on our goals.
The final showdown boiled down to either shell script or go.lang
Internally, as we used docker and linux extensively in our infrastructure, most of our engineering team do have shell script experience.
This allow us to be confident that we would be able to make shell work on ubuntu, and macosx.
What we are not confident however, is making it work well on windows, alpine, debian, arcOS, etc ...
The general plan at that point of time was to keep go.lang (which we were sceptical of) as a backup plan, and take a plunge into shell scripting - fixing any issue as it comes up with specific customer (the 9%).
However things changed when we were "forced" to jump into a small hackaton project (to fix a major customer issue) : inboxkitten.com
uilicious
/
inboxkitten
Disposable email inbox powered by serverless mailgun kittens
Open-Source Disposable Email - Served by Serverless Kittens
Inboxkitten is an open-source disposable email service that you can freely deploy adopt on your own!
Visit our site to give a spin, or ...
Docker Deployment Guide
Its one simple line - to use our prebuilt docker container.
Note you will need to setup your mailgun account first
# PS: you should modify this for your use case
docker run \
-e MAILGUN_EMAIL_DOMAIN="<email-domain>" \
-e MAILGUN_API_KEY="<api-key>" \
-e WEBSITE_DOMAIN="localhost:8000" \
-p 8000:8000 \
uilicious/inboxkitten
And head over to port 8000 - for your inboxkitten
Other Deployment Options
- Serverless deployment guide (for cloudflare/firebase)
- localhost/custom deployment/configuration guide
Support us on product hunt π
Somewhat related blog / articles
- The Stack : Making a free open-source disposable email service prototype (inboxkitten.com) in 14 hours
- What I have learnt from a 14 hours project
- Development timeline
Other References
Looking for sponsor
Noteβ¦
In that 14 hour project, we decided to use the opportunity, to give go.lang CLI a try along the way in a small isolated project.
Turns out, it can be done relatively easy (after the learning curve). And with that - a decision was made... go lang it will be...
And from the looks of it, it turned out well for us after much testing! (fingers crossed as it hits production usage among our users)
Digression, Personally I would have went this route. Hid myself in a programming cave for a week. And bash out utility scripts around all the limitations across every platform.
However until the team grows much bigger in size and experience, this would be on hold. So maybe next year? (i dunno)
Sounds good, What does uilicious do with a CLI anyway?
We run test scripts like these ...
// Lets go to dev.to
I.goTo("https://dev.to")
// Fill up search
I.fill("Search", "uilicious")
I.pressEnter()
// I should see myself or my co-founder
I.see("Shi Ling")
I.see("Eugene Cheah")
And churn out sharable test result like these ...
Which are now executable via the command line
One more thing, the go lang gophers are cute
Top comments (39)
You could use
nccfrom Zeit to compile NodeJS to a single fileHmm, ur right should have gave it a try did honestly miss this one π
There's even Zeit's pkg which I believe builds on the work of ncc. It generates a binary so you don't even need node installed.
Package your Node.js project into an executable
Disclaimer:
pkgwas created for use within containers and is not intended for use in serverless environments. For those using ZEIT Now, this means that there is no requirement to usepkgin your projects as the benefits it provides are not applicable to the platform.This command line interface enables you to package your Node.js project into an executable that can be run even on devices without Node.js installed.
Use Cases
npm installto deploy your application. Deploy it as a single fileRust seems interesting from an engineering and programming language design point of view, but the syntax is so ugly. I tried, but I can't program in it for longer than a snippet.
Rust clicked with me as well, much easier than go. Before that I would say python clicked better with me than Ruby. Just to give some context.
I'm happy to have all of them s an option overall.
I share greatly your ending sentiment. While I dislike python and ruby also Iβm glad they have enabled so many others.
I love Rust, and write about it here frequently.
But I would never insinuate its learning curve is anywhere near as forgiving as go's. =)
Came here to say this.
Structured parsing of JSON in Rust is wonderful.
Foiled by golang once again.
Gosh, thats a good idea : worth giving it a spin next time π
It would be interesting to see a comparison :D
The ooooooone point I might argue is zero-dep. I build go for scratch base images all the time. Scratch is literally zero bytes, and only provides
/.As for learning curve... Yes. It's a bit of a beast lol.
gofmthelps a lot - intellij, with its autogofmthelps more.Dealing with json - especially uncertain json - is... I don't talk about that.
For me, the draw was true cross-compile, and it's sheer perf. If you need quicker than Go, you need C.
I don't need C π
π and I have met folks who thinks my alpine approach is "hardcore" haha.
And yes I so want go to have a better way to handle uncertain json data.
BTW of course you can create single binaries with node. Just use
pkgornexe!Didn't try pkg (should have honestly), nexe did sadly create a large package for us - which is a known issue on their part : they are working on it π
Yeah ... Anyways, Golang is very awesome, although I recently switched back to Node simply for it's simplicity.
needs to be saved. btw you did not consider awesome python ...
We actually did, python, ruby, and a few others... they been removed in the article due to the repetition of the reason - which is the end user needing to install an external language library as a dependency before using the CLI (which is mentioned in java and js). Someone in the team even joked about considering PHP (its possible btw)
On another topic : we even considered c and c++, but that might be a bit too difficult for the team to do well without bugs (or overflow vulnerabilities)
One of our benchmarks for example is the ability to run in vanilla alpine.
na na you need to study py engineering particularly distribution. XD for distribution you just install your program. companies using py for production, include libs needed etc so that you install it like any other software.
e.g. if flask is needed, they choose a version for in-office use and that lib is shipped across products, they don't pip install or anything online in target env or even in dev
Why is PHP always a joke
PHP is by definition a joke :)
python is (unless you're doing ML, which you shouldn't be) not awesome
why? like give more details
I personally really dislike the syntax, and the code is inconsistent to what I think it should do. Also, going off your points, does not create a single distributable, has tons of dependencies, version conflicts, and the syntax is compiler enforced.
version conflicts and dependencies is solved by py companies, not an issue if you use your own version. single exec, pyinstaller has been killing lately but even without single exec, py companies have no problem.
for syntax dislike and enforcement, that is a matter of choice.
industrial-level python is different from casual coding.
What about Java with graalvm native image? Or even node with graalvm native image..
Never heard of it, but just took a look into.... and holy batman!
While not so comfortable with the way it does things internally. I am impressed, in the crazy sense that they are pushing the idea of compiling programs, not just to exe's but to run within Oracle / My SQL.
Shameless self plug here : its like GPU.JS style of crazy rewrites (of JS to WebGL).... Dun ask why we compile from one language to another which makes no sense... just know it has been done.
The interesting part is that the node execution performance is almost 85% for V8 and GraalVM is still in beta. Hopefully they will manage to reach the same performance and then you could compile you node app into a native one and have no drawbacks. Currently a JAX-RS rest endpoint compiled into native boots in ~5ms and uses 20mb of ram.
I like that GraalR runs an order of magnitude faster than the reference R runtime :D
I guess R is a scripting language so it makes sense, I expect python to run faster too
There's MicroPython :v
public knowledge of what or how to use graal is very low unfortunately..
The icons at the end could be a winning argument for just about anything.
Java has made major improvements in both language constructs and deployment models in the last few years (9 and 11 in particular). Although go is fun for small projects, and wins the too cool for school award, single file lightweight deployment of java is now an off-the-shelf supported option. See steveperkins.com/using-java-9-modu... for more information.
Note that all single file deployment solutions are still os dependent, go, java, etc. So you still have to maintain multiple build chains.
Interestingly, with the embedded jvm approach, you also enable development in multiple languages, such as kotlin , scala and groovy and still have a single file deployment.
There may be other reasons to reject java, just not so sure it is such an easy call.
Thanks for a nice article, but I will need to report you to The Society Against Inhumane or Cruel Abuse of Colons
Gosh you are right : Im guilty as charged : gonna try make some edits to improve that : π
How was this sold to your PM?
Nice reference to physics! My post-read nerd stokedness level: maximally high.