DEV Community: Jør∂¡

My dream come true: Launching GUI Docker sessions with DX11

Jør∂¡ — Tue, 07 Jan 2020 22:48:50 +0000

Get ready, because today we will get you to launch complete Linux Desktop Environments on demand. You’ll create your ideal graphical environment in a few minutes and launch it in a few seconds. Launch any desktop in the same way as you would launch a program.

Sounds good, doesn’t it?

If you are a Linux user, at some point you may have used Virtual Machines to try new stuff and experiment before you choose one. Virtualization is cool, but you know the downsides:

You need to boot a separate OS with its own Kernel and added overhead
The guest OS preallocates a fixed amount of RAM memory, even if only 1% of it is used
Your guest OS will never run as smoothly as your native desktop
The guest OS also needs a virtual data partition, networking, guest tools, etc
You need to manually install it and tweak it to your needs
If an upgrade breaks your system, there is no clean way to go back. With snapshots you would lose the latest changes on your personal data, whereas uninstalling or downgrading is not guaranteed to work

A new hope: Docker

Something instantly “clicked” in my head when I first heard of Docker:

Run any Linux command from any OS image, reusing the same Kernel and system resources.
Run on disposable stateless containers that you could break and relaunch right after.

That’s so cool, why don’t we all live on Docker containers right now?

Docker is mainly intended for backend projects. It’s designed for the command line and targeted to run servers, daemons, compilers and so on. Docker empowers great pieces of software, but Desktop Environments are not one of them yet.

Enter x11docker

X11Docker is a feature rich script that addresses this problem. It does the heavy lifting to start an X11 server on the host system and makes Dockerized apps talk to it. So with it, you can run GUI apps from the Docker image that you like! 😃🎉

I’ve been using it for a month now. While I applaud the brilliant work that it has behind, my conclusion is that X11Docker is not a VM replacement.

Instead, I see it as a replacement for docker run .
A minimal VM-like scenario looks much more like:

Unless you make a script of it, I bet you’ll never use x11docker for it. And yet this does not even include home folder encryption, direct connectivity with the host, etc.

The missing piece: DX11

My yearn for multiple independent Desktop systems launched on demand kept growing until I decided to ditch the script above and program a full fledged tool that made my dream come true.

Photo by Adam Whitlock on Unsplash

Without further ado, let me introduce DX11! 😃🎉
DX11 / CLI manager for X11 Docker Linux sessions

DX11 is a Linux program that allows you to run X11 Linux sessions with Docker. A “session” consists of a Docker image, a user home folder and settings like user data encryption or Tor networking.

DX11 combines the great performance of Docker on Linux with many of the features you’d get when running VM’s:

As fluid and fast as your native system
Reuse the same kernel, system memory and disk storage
Scriptable, 100% reproducible desktop environments
Immutable desktop systems (similar in concept to Silverblue)
Disposable and throwaway sessions
Independent home folder persistence for each session
Optional home folder encryption
Optional Tor networking
Unprivileged execution, with the option to attach separately as root
Isolated sandboxed environments

Use cases

The list grows pretty much as you imagination allows. But a few examples include:

Realistic distro hopping. Try a new distribution or desktop environment before you make it your daily driver. Evaluate the real performance instead of hoping that a native installation would be smoother.
Full portability. Copy your session’s data to a brand new computer, rebuild the Docker image and continue to work as if nothing ever happened.
Give your developers a reference workstation Dockerfile that they can still customize. Avoid the “works on my machine” problem.
Speed up the landing of new developers with an already functional toolkit.
Post your session’s Dockerfile on your StackOverflow question or GitHub issue. Let others reproduce the exact same environment you have when you encounter that nasty error.
Deprecation proof development environments. Your legacy React Native or Cordova setup is carved in stone. If an SDK upgrade breaks a project, nothing else will be affected. Rerun your stable image and npm install will keep working as usual.
Post your session’s Dockerfile on a public repo. Let others clone it, fork it and make it even cooler.
Evaluate bloated/unsafe software on a clone of your workstation and drop the changes when you are done experimenting.
Upgrade and downgrade at will, without the risk of losing any local data. Docker build will never interact with any session, even if it’s running.
Replace your all-in-one bloated computer with minimal isolated systems that you can use on demand. One for your sysadmin work, one for your side projects, one for gaming, one for anonymous browsing, etc.
Forget about multiple SSH key management by having a different session for each project, with its corresponding key files and credentials.
Cybercafes, where a Linux computer spins up a fresh disposable desktop when you pay for it and terminates it when you are done.
Record a screencast of that upcoming Gnome/KDE release and transfer the video file to your main system right away. No VirtualBox Guest Additions needed.
If you write extensions for KDE, Gnome, etc., you can publish a Dockerfile to let people run sessions with them preinstalled.

You name it. The more you use it, the more you realize the enormous potential it has.

Well, show me the money!

Create a session

Let’s create a simple Dockerfile that customizes a Deepin Linux base image with a few extra packages:

The hard work is already done in x11docker/deepin. All we need to do is adding the commands that we would manually run after installing the system ourselves. In this case, a few system tools and a browser.

Now, let’s create our first session with it:

This creates or updates a session named my-workstation
A Docker image is built from Dockerfile and my-workstation is linked to it

Run the new session

That’s even simpler:

This will switch to a new Virtual Terminal and start Deepin in a fraction of the time it would take to boot the entire OS.

Deepin Desktop Environment started by DX11

Do some work, create some files and log out. Run dx11 run my-workstation again. Your prior work is there, your desktop boots in seconds, your session has the exact software that you need and you can be confident that if something bad happens, you’ll be back to normal when you log back in.

How about a brand new Gnome session? No problem, let’s extend a Dockerfile from ubuntu:eoan and install a few packages that we need to use. Two commands (dx11 new and dx11 run) and two minutes later, our DX11 session looks like this:

Gnome Desktop running on Ubuntu 19.10. Started by DX11.

Et voilà! Would you like to do the same with Ubuntu Mate? No problem. We haven’t touched a single installer yet, but here is our third session in less than 10 minutes:

Mate Desktop started by DX11

That’s insane!

The last two screenshots are the result of tweaking the user settings a bit and adding a global menu. But in general, starting a session with an already built image will take less than 10 seconds.

Attach as root

While my-workstation is running, you can attach to it from your normal session:

This will give you a root console, which allows you to try and change whatever you like on the system.

If you log out of the system, any changes besides /home/user will be lost
You can consolidate them into your Docker image using docker commit if you like

More base images

You can see a few examples provided on the DX11 repo and you can also checkout the Docker Hub images contributed by the X11Docker team.

Contribute

DX11 is far from a finished tool. There are still bits and pieces that need attention, but so far I’m excited to be writing an article like this 😃

If you like it, please give it a try today and provide valuable feedback so we all can benefit from it.
Also, feel free to explore these Docker Hub images, experiment with your ideal setup and submit Pull Requests with your examples

Future work

The next thing I’d like to see is a “Docker Hub” for desktop sessions. Docker Hub itself would do the trick for the image registry, but most DX11 users would expect screenshots, screencasts, comments, etc.

That would be a nice challenge to undertake.

More conclusions: NIM

In my last article I briefly explained that I was going to check out the Nim programming language and see what I could do with it.

Nim: from scripting to C code. The all-in-one language.

After the experience of building DX11, I’d like to share a few thoughts about it:

Nim is a nice and clean language
I’ve felt that the language is well thought out and it leads to writing nice code quickly
The compiler is fast and produces small, dependency-free binaries. Sometimes, error reporting could be better, but as a newcomer to Nim I’ve been able to quickly detect the root issues and get them fixed
However, the main drawback is the lack of a graphical debugger for the most common IDE’s. You can use CLI tools, but things are likely to be slower when working on complex projects
Although not as popular as NodeJS’s, Go or Rust, Nim has many libraries available for use on your project, covering the typical dependencies you may find in systems programming

In my opinion it’s worth giving it a try. I hope more people get involved so we can tackle bigger and more complex projects with it.

That’s it for today! I hope you enjoy the Christmas toy 🎁 and if you like it, please clap out loud 👏, show some love ❤️, subscribe and give DX11 a try!

Until the next time, take care!

Photo by Simon Matzinger on Unsplash

Cover by Mike Meeks on Unsplash

Nim: from scripting to C code. The all-in-one language

Jør∂¡ — Sat, 04 Jan 2020 21:54:53 +0000

Today I’d like to talk a bit about NIM, a programming language that I have recently discovered.

You may be used to two different types of languages:

Native compiled languages (C, C++, Go, Rust, …)
Scripting or interpreted languages (Javascript, Python, Bash, …)

Each of those have their pro’s and con’s, but today we are here to discover the best of both worlds with Nim.

What makes it different?

C language as a target

Unlike the typical systems programming languages, Nim allows to compile dependency-free static binaries by precompiling your project to plain old C code.

However, Nim also allows you to compile your projects to C++ or Javascript. This enables to program both backend and frontend projects using the same knowledge.

Templates and macros are defined using Nim itself

If you are familiar with the typical #define, #ifdef, etc constructs in C, you’ll feel at home with Nim, because the language is flexible enough to support them by itself.

Nimscript

When you are targeting internal scripts and automation tasks, you can rely on Nim to do the work for you. Nimscript is a subset of the language that allows to run running custom operations on a more comfortable language. Unlike the traditional shell interpreters, Nim is fully cross-platform and you can opt for a binary or script approach without changing any mental context.

Personally, I’m excited to play a bit with it and see what it can do 🍭
To me, it looks concise, clean and well thought out. I’ve seen it being used for game development, crypto apps, scripts and more.

I’m currently working on a first use-case tool that makes use of it on a really cool topic. If all goes smoothly I’ll write a full post about both the language and the “thing” that it will provide.

Stay tuned for more.
Until then, take care!

Photo by Conor Luddy on Unsplash

Modern makefiles used the right way in 2020

Jør∂¡ — Fri, 27 Dec 2019 12:41:00 +0000

If you are an experienced developer, you may well know about makefiles. Plain text files defining rules to compile software, back from the old days. Right?

Today we will:

See the Top 3 myths I’ve encountered on my experience and prove them wrong
We will see how make shines when used as expected.

Myth #1

Only useful for C, C++ and native software

Although it’s true that the C/C++ ecosystem was heavily influenced by the presence of make within the ecosystem, there’s much more that you can do with it. make can handle any kind of file, as long as it has a path and a timestamp.

The typical example:

Creates a dependency tree of the commands that need to be run on each execution
If you run make edit, then main.o, kbd.o and command.o are compiled first, and then edit is built upon them

However, you could also use it to transform something as simple as plain text files:

In this case, our (default) target is my-content.txt and it is built by simply concatenating the output of two dependent files (created on the fly).

I am successfully using it in other scenarios like web development and mobile app development. But there’s no restriction on how it can be used.

Myth #2

It’s just another task runner, NPM scripts do the same job

That’s indeed not true. Yes, it runs tasks (the commands of a rule) but not necessarily. Let’s put the example above with text files.

When we run make the first time, it will trigger the dependencies and then the main target. So yes, we run a bunch of tasks. But what happens if we run make again?

Nothing happens, but why?

It turns out that make is designed to keep track of the modification date of files. In this case, it detects that the modification time of dependency-1.txt and dependency-2.txt has not change since my-content.txt was last built. Hence, my-content.txt does not need to be rebuilt.

What happens if we change the contents of a dependency?

Then, make is smart enough to figure out that only the first rule needs to be executed at this point.

This is not the same as what an npm script would do
Achieving the same using a shell script would need much more code than a simple makefile
If each of these 3 rules took 30 seconds to run, you would be saving one minute for yourself on every execution

Myth #3

For web development that’s an overkill tool

If all you ever do is invoking webpack then, it is. In the rest of cases, it might not be at all. Put for example, a simple web site with styles, scripts and a static media gallery like this:

We may want to:

Instal the NPM dependencies
Minify the HTML code
Transpile Typescript, bundle and minify it
Fetch a remote JSON file with data to be imported by Typescript
Compile sass code into CSS and bundle it
Generate the sitemap
Optimize the images and videos
Etc…

You may be thinking of a simple script that would do the trick, run a few commands and the job is done, right? Well, you may get the site built, but at the expense of building everything every time.

Even if you just changed one character, the videos of your web site are going to be transcoded once and again. Even if you have the same styles, sass is going to launch every time. Even if you have a static site generator and the list of products hasn’t changed, your entire application will be rebuilt from scratch.

If you care about speed and efficiency, then make is definitely your friend. But if you only need to launch a few scripts, then make is not the tool you’re looking for.

Top mistakes found when using make

They may be hard to understand if you don’t take the time to carefully read the docs.
It is quite common to see a makefile like this:

The typical approach is to see the makefile as a task/subtask tree. When you run make all then, all the dependencies are build.
While this example might eventually work, what are the main issues?

Using rules as if they were a simple task

This is more of a conceptual concern, but rules are meant to be evaluated, in order to decide whether the target needs to be built or not.

However, in the example above markdown: is being used as an “alias” instead of a rule that prevents useless computation.

A rule’s dependency files are not declared

To take advantage of make, the markdown rule should (at the very least) be written like:

Rule names should be bound to actual output files

Using abstractions like all: markup scripts styles media to make things clean and flexible is fine. However, indirect targets should always link to the specific target file that will fulfill the dependency.

When defined like this, the modification date of the dependencies and the target file tell make wether the rule needs to run again or not.

These are seconds that you can save!

Variables are there to help

If the list of source files is known beforehand, wouldn’t it be great to use a variable instead of hardcoding the dependencies each time?

Note that here, the $(MARKUP_FILES) variable is used to define the dependencies. But it could also be placed on the commands to execute:

Looks good, but we can still do better. Let’s also factorize the sass executable path as well:

Confusion with make and shell variables

In the example above, note that variables like $(STYLE_FILES) are make variables. Not shell variables.

Make variables are evaluated to generate the exact shell command and then, the shell command is executed.

When writing a command like echo $(PWD):

make will replace $(PWD) by the current value (i.e.) /home/user
bash will then execute echo /home/user

This is not the same as if you run echo $$HOME. In this case:

make will replace $$ by $
bash will execute echo $HOME

Use the builtin variables

Still on the same example, we can improve the rule.

Imagine that index.sass internally imports other sass files. How do we declare them as dependencies, too?

Ok, this change needs a bit of explanation:

The wildcard keyword evaluates the glob and puts any matching file path on the variable. So our variable contains a dynamic list of source files.
$@ is evaluated to the name of the target. In this case it is an alias for build/index.css. Instead of rewriting the own name, we can use this shortcut.
$< is evaluated to the first dependency of the rule. We use it because sass takes the entry point, instead of the whole list.
In this case, $< evaluates to $(STYLE_FILES) which equals $(wildcard src/index.sass src/styles/*.sass). This is the same as passing src/index.sass
If sass took the whole list of files, then we would write $(SASS) $^ $@.

So the command $(SASS) $< $@ would translate into something like:
./node_modules/.bin/sass src/index.sass build/index.css

Ensure that target folders exist too

If we run the main target as it was, commands would probably complain about the build folder not being present.

A clean way to ensure its existence would be to create a target for the folder and make targets depend on in before running.

markup will trigger build first and build/index.html after.

We could also use it for our NPM packages. A typical way is to define a make init static action, but hey… what if this could be automatic?

Look at this:

When node_modules does not exist (target), the ruler will be triggered.
When package.json changes (timestamp is newer than node_modules), the rule will also trigger.

Setting static actions as Phony

On actions that do not depend on any previous state, a special rule should be used. Typically on actions like make clean you want the command to be triggered, regardless of the current artifacts.

Setting .PHONY ensures that if the clean rule is matched, it will always execute.

Why do we need this? Well, imagine that a file named clean is accidentally created on the project. What would happen if we ran make clean? Well, we would get something like: make:clean' is up to date` and you would think “fine, it’s clean”.

But this message would actually mean: The target file clean already exists and it has no newer dependencies. So, no need to do anything.

If you set .PHONY: clean you ensure that clean will always run rm -Rf ./build/*

How would the end makefile of the example look like?

As final remarks:

Think of a makefile in a declarative way, not in an imperative way (a bit like a ReactJS component)
Think of rules as statements that transform some input into some output and run only if the source content has changed
Approach your makefile by looking from the end (the target files, even if they don’t exist yet) and bind any abstract rules to specific output files

And this wraps it up for today 🎉🎊
I hope you found the article cool and refreshing ❄️🍦 enough to scroll down a bit more and hit the clap 👏👏 button 😃.

There’s more to come. If you want to stay tuned, don’t hesitate to follow Stack Me Up and new articles like this will be waiting for you next time.

Until then, take care!

Photo by Sorasak on Unsplash

Time to self-upgrade

Jør∂¡ — Wed, 18 Dec 2019 11:12:52 +0000

My Twitter feed says it all:

Jordi

@crunchyspace

Putting some order into my life 💆‍♂️

Time for a dev-centric account. Time to put technology in front and talk about the great things it can do 🖥📱📡

#firstTweet

00:58 AM - 18 Dec 2019

0 2

Jordi

@crunchyspace

Some things in life can't be systematically postponed, and the future is now.

In the coming weeks/months I'm going to spend time on them: #Travel, #Reading, #Writing and even posting some #Videos.

01:03 AM - 18 Dec 2019

0 1

A journey is on the roadmap

As a rather introvert person, I've seldom taken the time to broadcast the highs and lows of my life. However, focusing too much on projects won't make you a better person if you don't devote some time to yourself and to let others know where you are.

The only constant in life is change and below are the things I'm about to change on the journey I'm about to undertake.

Move!

First off, my life partner and I will leave the comfort of our home town and live as remote workers, at least within a reasonable timeframe. The first stop of the journey brings us to Silicon Valley. We want to see how things move, how things breathe and what can we learn from the people that have shaped today's digital society.

After touring by the West Coast, we'd like to deep dive on the Eastern New York City. Next, Australia might be on the horizon, but the coming weeks will tell where we move next.

Talk!

Instead of keeping things for myself and ourselves, I won't hesitate to share thoughts, experiences and findings.

Share!

This one is for tech. I feel so grateful for the insane amount of knowledge and wisdom that I've been given thanks to the internet community. Not only it has made me a better engineer, but also a better and free person.

I believe that my Karmic debt deserves the time and effort to try to give out a bit of what I've been given myself.
This means that this account will turn into a publication. And it also means, that it will be the basis of all the topics that I'm eager to talk about and tell everyone 😃

If you spend your life waiting for something to happen, the only thing you'll do is letting your life pass.

The future is now. Go make it happen.

Until very soon!