DEV Community: Ian Eyberg

Running Linux Programs as Unikernels on macOS

Ian Eyberg — Mon, 10 Feb 2020 21:17:34 +0000

Someone emailed me the other day and was having some trouble getting
their program to work under OPS.

He had downloaded a binary release from github and knowing that OPS only runs linux binaries chose the linux version. Further, there are numerous
examples of people running linux programs under ops on osx. Hell, just do a

ops pkg list

for a short list.

However, for some reason it just wasn't working for him.

user@users-MacBook-Pro Desktop % ops run solana-validator    
*errors.errorString libstdc++.so.6: file does not exist
/Users/eyberg/go/src/github.com/nanovms/ops/lepton/ldd_darwin.go:100 (0x4c55779)
/Users/eyberg/go/src/github.com/nanovms/ops/lepton/ldd_darwin.go:129
(0x4c5607d)
/Users/eyberg/go/src/github.com/nanovms/ops/lepton/image.go:261
(0x4c52d0e)
/Users/eyberg/go/src/github.com/nanovms/ops/lepton/image.go:24
(0x4c50f5f)
/Users/eyberg/go/src/github.com/nanovms/ops/cmd/run.go:15 (0x4ca4189)
/Users/eyberg/go/src/github.com/nanovms/ops/cmd/run.go:118 (0x4ca4f75)
/Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
(0x4c8bf3e)
/Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
(0x4c8cb5b)
/Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
(0x4ca8d88)
/Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
(0x4ca8d83)
/usr/local/go/src/runtime/proc.go:200 (0x402f57c)
  main: return
/usr/local/go/src/runtime/asm_amd64.s:1337 (0x405a671)
  goexit: GLOBL shifts<>(SB),RODATA,$256

panic: libstdc++.so.6: file does not exist

After I saw the word mac and then libstdc++ I knew immediately what was wrong but I also realized that it might not be readily apparent to many people and so this is what this blogpost is about.

The binary in question is dynamically linked. What this means is that
there are several libraries that the dynamic loader (ld) will try and
load at runtime versus statically linked where all the libraries are
packaged inside the binary. This leads to a fatter binary but you can be
assured that everything is present when needed. I won't jump into a
static/dynamic flamewar as they each have their own usecases.

I should also point out that normally you can't run linux binaries directly on a mac - this is actually a uniquely interesting unikernel aspect as we don't need to boot up vagrant or a linux vm. Traditional virtualization software virtualizes the operating system. You can see using a tool like OPS as virtualizing the application instead.

One of the reasons I decided to write this post was not just the email but I've noticed in the past few years I've seen quite a lot of people that use containers and go to refer to their programs as statically linked when it is in fact absolutely not.

Let's look at this example go webserver:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "Welcome to my website!")
    })

    fs := http.FileServer(http.Dir("static/"))
    http.Handle("/static/", http.StripPrefix("/static/", fs))

    http.ListenAndServe(":8080", nil)
}

Fairly simple - all it does is listen on 8080 and serve up requests. If we compile it:

go build

We see that by default it will be dynamically linked against a few libs:

eyberg@box:~/z$ ldd z
        linux-vdso.so.1 (0x00007ffe8b1db000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2e091f4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2e08e03000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2e09413000)
eyberg@box:~/z$ file z
z: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, not stripped

VDSO gives us a fast clock. Libpthread gives us threads. Libc contains everything you'd find in section 3 of the man pages and finally LD is our loader.

You can link your go programs statically via something like this:

eyberg@box:~/z$ go build  -buildmode=pie -ldflags "-linkmode external -extldflags -static"
# _/home/eyberg/z
/tmp/go-link-057347370/000004.o: In function `_cgo_7e1b3c2abc8d_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:57: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
eyberg@box:~/z$ ldd z
        not a dynamic executable
eyberg@box:~/z$ file z
z: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=0671d22e52baf812ed9f500a8d9ee7848b8c809e, not stripped

Both ldd && file now report that it is statically linked. However, you'll notice (at least in go 1.12) it now outputs a message stating that getaddrinfo is still required! That's cause even though ldd output and file is telling us that it is indeed static we still rely on libnss for dns. You can force go to use its built in resolver instead.

Getaddrinfo is found here:

eyberg@box:~/s/solana-release/bin$ readelf --dyn-syms /lib/x86_64-linux-gnu/libc-2.27.so  | grep getaddr
   873: 0000000000107bc0  3261 FUNC    GLOBAL DEFAULT   13 getaddrinfo@@GLIBC_2.2.5
  1601: 00000000001413c0    43 FUNC    GLOBAL DEFAULT   13 inet6_rth_getaddr@@GLIBC_2.5

You'll note that your libraries are probably stripped which means you can't use something like nm to find the symbols but readelf surfaces them up in the .dynsym section.

Also, even that is not enough - you'll probably make use of various functions found in these files as well:

eyberg@box:~/z$ ls /lib/x86_64-linux-gnu/libnss_files.so.2
/lib/x86_64-linux-gnu/libnss_files.so.2
eyberg@box:~/z$ ls /lib/x86_64-linux-gnu/libnss_dns.so.2
/lib/x86_64-linux-gnu/libnss_dns.so.2

The point here is that just sticking a go program in a container does not inherently make your binary 'statically linked' and the problem is that, that is not what a lot of people have been stating and this source of
confusion is probably what leads to misunderstandings such as this.

So let's go back to our example at the beginning of the article.

The program in question is the solana blockchain.

You can download and unzip like so:

wget https://github.com/solana-labs/solana/releases/download/v0.23.2/solana-release-x86_64-unknown-linux-gnu.tar.bz2
bunzip2 solana-release-x86_64-unknown-linux-gnu.tar.bz2
tar xf solana-release-x86_64-unknown-linux-gnu.tar

You can see from the ldd output that it is trying to load a library that is dynamically linked (libstdc++) and that's what our error in the trace was.

eyberg@box:~/s/solana-release/bin$ ldd solana
        linux-vdso.so.1 (0x00007ffe338df000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f06976a8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f06984a2000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f06974a4000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f069729c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f069707d000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0696e65000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0696ac7000)

As you can see this program works out of the box on linux.

eyberg@box:~/s/solana-release/bin$ ops run -c config.json solana-validator
[solana-validator --ledger /]
booting /home/eyberg/.ops/images/solana-validator.img ...
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
assigned: 10.0.2.15
solana-validator 0.23.2
log file: solana-validator-7wgZy3ozTRyvLtRN5o3Xh6m5aEvEo2XBxP5yPfaCny1t-20200210-204507.log

eyberg@box:~/s/solana-release/bin$ cat config.json
{
  "Args": ["solana-validator", "--ledger", "/"]
}

This is all fine but if you want to run it on osx you'll need to do one
of two things:

1) Either build from source and statically link it (so ldd doesn't show
output).

2) Download the libraries it's linked to and manually add them to the
filesystem.

The reason this works out of the box on linux is that OPS looks at and
try to load the libraries as we build the disk image. This isn't
possible on a mac cause mac uses mach-o which is a different format that Nanos does not and probably won't ever support as nanos is explicitly designed to run linux server-side programs.

For example when we build common packages like nginx or node, you'll see from the ops output that all required libraries are placed on the filesystem in a known location.

➜  ~    ops pkg contents node_v13.6.0
File :/node
File :/package.manifest
Dir :/sysroot
Dir :/sysroot/lib
Dir :/sysroot/lib/x86_64-linux-gnu
File :/sysroot/lib/x86_64-linux-gnu/libc.so.6
File :/sysroot/lib/x86_64-linux-gnu/libdl.so.2
File :/sysroot/lib/x86_64-linux-gnu/libgcc_s.so.1
File :/sysroot/lib/x86_64-linux-gnu/libm.so.6
File :/sysroot/lib/x86_64-linux-gnu/libnss_dns.so.2
File :/sysroot/lib/x86_64-linux-gnu/libnss_files.so.2
File :/sysroot/lib/x86_64-linux-gnu/libpthread.so.0
Dir :/sysroot/lib64
File :/sysroot/lib64/ld-linux-x86-64.so.2
Dir :/sysroot/proc
File :/sysroot/proc/meminfo
Dir :/sysroot/usr
Dir :/sysroot/usr/lib
Dir :/sysroot/usr/lib/x86_64-linux-gnu
File :/sysroot/usr/lib/x86_64-linux-gnu/libstdc++.so.6

Without getting into the semantics of building your own package which you can find here the easiest way to do this with your own app is in your project directory you can create a directory:

mkdir -p usr/lib

then include it in your config.json that you pass to ops when it builds your image.

{
"Dirs":["usr"]
}

ops run -c config.json myprogram

After that you can always grab the image that was created in ~/.ops/images.

Hope this clears up any confusion.

Serverless Security with Unikernels

Ian Eyberg — Thu, 14 Nov 2019 20:29:31 +0000

Security is one of those topics where on one hand you see a lot of passionate developers that get upset whenever there is a new data breach (and those seem to be happening on the daily), yet on the other hand there is a very large skills gap on understanding how hackers (the bad kind) think, what makes them tick and most importantly - how they operate.

I think it's important developers start thinking about security in a
more holistic manner.

Let me give you an example. I was talking to a vp of eng the other day that said they are rather good on security cause all of their instances are inside a VPC. I agreed that was a good approach versus exposing everything on the internet but then I brought up the Capital One hack and the Door Dash hack and many others that occurred just this year. You can bet that if someone was only alerted 4-5 months after an attack such as in the case of DoorDash the miscreants have been all up and down those servers. Now exploiting a SSRF (server side request forgery) vulnerability is one thing but escalating the attack into the point where you have landed a shell inside a vpc is where this thinking falls apart.

Why is that?

At the end of the day attackers don't care about what exploit or what vulnerability they are using to get onto your server. The only care about getting onto your server to run their programs. For example cryptojacking attacks like the one that afflicted Tesla are very popular nowadays cause unlike ransomware you don't have to wait to get paid - it just starts making money immediately! At the end of the day it doesn't matter that Tesla had exposed kubernetes to the world the attackers just wanted to mine some monero - they could care less how they broke in. This is the point.

I'm going to show you real life attacks on Google Cloud here in a bit but first I want to set some expectations.

The Problem with Multiple Processes (or 'Just Use Threads')

Most attacks today rely on the capability of running other programs on a given server/instance/container/etc. If you can't do that because fork/execve and friends have been seccomp'd out the attack has gotten progressively harder cause now you have to start doing more exotic attacks using things like rop gadgets. However, at the end of the day the end desire remains the same - unfettered access to run whatever program the attacker wants so typically the end goal there is to pop a shell.

Not being a day to day js developer I did a quick search on github to see how popular forking a new process might be. This picture shows that it definitely is not unpopular.

The recent paper A fork() in the road argues very well that we should not be using fork - at all in 2020. We have had native threads since ~2000 in Linux (yes, 20 years ago).

In the past there wasn't a strong demand to get rid of it because Linux itself was designed to run on real machines - not virtual ones. This is important to point out cause how else would you run other programs on the same physical server? However, that proposition can now be re-examined at least for cloud computing use cases which are entirely built on virtual machines.

For languages such as Java and Go you get threading out of the box so you can have as much performance as you have threads/cores available. For the interpreted language class such as Javascript and Ruby it's been common to stick X application servers behind a load balancer/reverse proxy to scale up. At the end of the day you get the exact same vCPU that you buy regardless. If you've got one vcpu user-land threads, async, and such might help you out some but forking off a half dozen worker processes won't - then you are just fighting the operating system scheduler.

Serverless Security

Serverless is clearly a desire for many developers today that don't wish to manage and run infrastructure. That makes sense as we keep pumping out tremendous amounts of software and devops salaries, at least in my neck of the woods (SF), are through the roof. Unfortunately, a lot of the status quo serverless offerings are built on top of popular cloud services leading to vendor lockin.

Unikernels are a fresh set of eyes of looking at this problem space as they allow one to deploy the same set of code to any number of vendors using tried and true vms as their base artifact, albeit not the types of vms you might be used to.

Running Node the Old Way

Let's show how you might normally provision this javascript webserver. (and yes I understand that this would be automated but it's the same thing -- work with me here) First we spin up an instance. Ok, nothing abnormal here.

Then we ssh in. Wait - hold on.

Right off the bat we are explicitly allowing the concept of users to jump into an instance and run arbitrary commands. In fact every single configuration management tool out there including terraform, puppet, and chef are explicitly built on this concept which is odious from the start.

Ok, let's continue.

Once we are on the instance we install node.js:

eyberg@instance-1:~$ sudo apt-get install nodejs
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libicu57 libuv1
The following NEW packages will be installed:
  libicu57 libuv1 nodejs
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 11.2 MB of archives.
After this operation, 45.2 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian stretch/main amd64 libicu57 amd64 57.1-6+deb9u3 [7,705 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 libuv1 amd64 1.9.1-3 [84.4 kB]
Get:3 http://deb.debian.org/debian stretch/main amd64 nodejs amd64 4.8.2~dfsg-1 [3,440 kB]
Fetched 11.2 MB in 0s (42.9 MB/s)
Selecting previously unselected package libicu57:amd64.
(Reading database ... 37215 files and directories currently installed.)
Preparing to unpack .../libicu57_57.1-6+deb9u3_amd64.deb ...
Unpacking libicu57:amd64 (57.1-6+deb9u3) ...
Selecting previously unselected package libuv1:amd64.
Preparing to unpack .../libuv1_1.9.1-3_amd64.deb ...
Unpacking libuv1:amd64 (1.9.1-3) ...
Selecting previously unselected package nodejs.
Preparing to unpack .../nodejs_4.8.2~dfsg-1_amd64.deb ...
Unpacking nodejs (4.8.2~dfsg-1) ...
Setting up libuv1:amd64 (1.9.1-3) ...
Setting up libicu57:amd64 (57.1-6+deb9u3) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up nodejs (4.8.2~dfsg-1) ...
update-alternatives: using /usr/bin/nodejs to provide /usr/bin/js (js) in auto mode

Notice something strange? That's right. It didn't matter that my user is a non-root user - I could immediately 'sudo' my way to doing whatever I wanted on the instance.

That whole concept of 'least privilege' and 'user separation' that security devs like to talk about is by default on many servers not present.

Unfortunately, as soon as we do that we realize that Debian 9 (the first instance that Google offered to give us comes with node version 4.

eyberg@instance-1:~$ nodejs --version
v4.8.2

Now our options are to either trash this instance or download a tarball. Let's go for that other option (even though knowing that if someone else touches this instance it might cause problems down the road).

eyberg@instance-1:~$ wget https://nodejs.org/dist/v12.13.0/node-v12.13.0-linux-x64.tar.xz
--2019-11-14 18:08:59--  https://nodejs.org/dist/v12.13.0/node-v12.13.0-linux-x64.tar.xz
Resolving nodejs.org (nodejs.org)... 104.20.23.46, 104.20.22.46, 2606:4700:10::6814:172e, ...
Connecting to nodejs.org (nodejs.org)|104.20.23.46|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14055156 (13M) [application/x-xz]
Saving to: ‘node-v12.13.0-linux-x64.tar.xz’

node-v12.13.0-linux-x64.tar.xz                     100%[================================================================================================================>]  13.40M  --.-KB/s    in 0.1s

2019-11-14 18:08:59 (114 MB/s) - ‘node-v12.13.0-linux-x64.tar.xz’ saved [14055156/14055156]

eyberg@instance-1:~$ unxz node-v12.13.0-linux-x64.tar.xz
eyberg@instance-1:~$ tar xf node-v12.13.0-linux-x64.tar

Let's jump into the code!

What this next snippet does is pop a webserver on that offers two urls to list the contents of a directory. One is a lot safer than the other as we'll soon find out. (Again, I'm not a js dev so excuse the ugliness of the code.)

var http = require('http');
var fs = require('fs');
var url = require('url');

const { exec } = require('child_process');

var port = 80;

http.createServer(function (req, res) {

  if (req.url == '/safe') {
    var files = fs.readdirSync('/');
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end(files + '\n');

 } else {

try {
  var cmd = 'ls';
  var resbody = '';

  var query = url.parse(req.url, true).query;

  // this is *unsafe*
  if (query.cmd) {
    cmd = query.cmd;
  }

  exec(cmd, (err, stdout, stderr) => {
    if (err) {
      resbody = err
      console.error(err)
    } else {
     resbody = stdout
    }

    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end(resbody + '\n');
  });

} catch(e) {
  console.error(e)

  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end(e + '\n');
}

}

}).listen(port, "0.0.0.0");
console.log('Server running at http://127.0.0.1:' + port + '/');

Now let's run our program:

eyberg@instance-1:~/node-v12.13.0-linux-x64/bin$ ./node bob.js
Server running at http://127.0.0.1:80/
events.js:187
      throw er; // Unhandled 'error' event
      ^

Error: listen EACCES: permission denied 0.0.0.0:80
    at Server.setupListenHandle [as _listen2] (net.js:1283:19)
    at listenInCluster (net.js:1348:12)
    at doListen (net.js:1487:7)
    at processTicksAndRejections (internal/process/task_queues.js:81:21)
Emitted 'error' event on Server instance at:
    at emitErrorNT (net.js:1327:8)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  code: 'EACCES',
  errno: 'EACCES',
  syscall: 'listen',
  address: '0.0.0.0',
  port: 80
}

Oh no! We forgot ports under 1024 are 'privileged'. Well no problem here - cause sudo make me a sandwich right?

We have gone from bad to worse. A sane setup would probably have a frontend proxy sitting in front of this that can drop privileges after getting setup and forwarding on the request but now you might need to call in your devops person huh?

eyberg@instance-1:~/node-v12.13.0-linux-x64/bin$ sudo su
root@instance-1:/home/eyberg/node-v12.13.0-linux-x64/bin# ./node bob.js
Server running at http://127.0.0.1:80/

Ok, let's hit it up:

➜  ~  curl -XGET http://34.68.46.143/
bob.js
node
npm
npx

Well - that works but is it safe?

➜  ~  curl -XGET http://34.68.46.143/?cmd="touch%20tmp"

This first query passes in the command "touch tmp" which creates a new file in that directory - bad news bears. The %20 you might recognize as the url encoding for the space character.

➜  ~  curl -XGET http://34.68.46.143/
bob.js
node
npm
npx
tmp

As we can see, we can run arbitrary commands on our end server and worse it's running as root.

This is a very oftenly abused software development pattern called 'shelling out'. There is almost never any good reason to do this and if you have code linters or static analysis setup on your ci there's a good chance it'll flag it or whoever is reviewing your PRs should.

Now if we refactor the offending command injection into the '/safe'' equivalent we might get this instead:

➜  ~  curl -XGET http://34.68.46.143/safe
bin,boot,dev,etc,home,initrd.img,initrd.img.old,lib,lib64,lost+found,media,mnt,opt,proc,root,run,sbin,srv,sys,tmp,usr,var,vmlinuz,vmlinuz.old

Leaking out your root filesystem probably isn't the best thing to do but at least you aren't injecting commands anymore.

Now, this is just one 41 line program here but this is a full blown linux system. Let's see what else is on here before we retire this example.

Attack Surface

Envision Normandy 1944.

The attack surface when we talk about linux systems is the amount of utter crap that we can attack.

root@instance-1:~# find / -type f | wc -l
76369

76,000 files! Just to run a 41 line javascript program?

I wonder how many shared libraries we have on this system?

root@instance-1:~# find / -type f -regex ".*\.so.*" | wc -l
751

750?? If we check out node we can see there are only 8 explicitly linked to node - why do we want/need the rest?

root@instance-1:~# ldd /home/eyberg/node-v12.13.0-linux-x64/bin/node
        linux-vdso.so.1 (0x00007ffeaf7e9000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fef28ab4000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fef28732000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fef2842e000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fef28217000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fef27ffa000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fef27c5b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fef28cb8000)

What about executables?

root@instance-1:~# find / -type f -executable | wc -l
1339

1300?!? We can attack 1300 programs on this fresh instance? All we did was install node. Let's try that query again.

root@instance-1:~# find / -type f -executable | xargs file | grep executable | wc -l
754

Well we drilled it down close to halfway but still 750??

A heavily seccomp'd container infrastructure might prevent some of this behavior but then you are missing out on the whole serverless part of the idea and container security does not have a great track record. Also, we haven't even begun to talk about why the linux kernel is +15MLOC - half of it is just drivers for hardware that doesn't exist in a virtual machine, then there's all the support for users, and IPC and scheduling and .... anyways, that's for a different blogpost.

So we've now shown that merely setting up a node webserver can be a pain even when we aren't doing things like putting it into an init manager or dropping privileges or any other sane activity.

Securing it becomes a whole new level of batshittery.

Serverless Unikernels

Let's start fixing the problem now that we have identified it. Let's take this same node.js webserver and turn it into a unikernel using the Nanos kernel and the OPS unikernel orchestrator.

If it's the first time you've done this you might want to check out this tutorial first.

Before we build the image - want to see the entirety of the filesystem first? I didn't show you the filesystem in the previous example cause no one wants to sift through 20+ pages of a tree listing.

➜  sec-article  ops pkg contents node_v12.13.0
File :/node
File :/package.manifest
Dir :/sysroot
Dir :/sysroot/lib
Dir :/sysroot/lib/x86_64-linux-gnu
File :/sysroot/lib/x86_64-linux-gnu/libc.so.6
File :/sysroot/lib/x86_64-linux-gnu/libdl.so.2
File :/sysroot/lib/x86_64-linux-gnu/libgcc_s.so.1
File :/sysroot/lib/x86_64-linux-gnu/libm.so.6
File :/sysroot/lib/x86_64-linux-gnu/libnss_dns.so.2
File :/sysroot/lib/x86_64-linux-gnu/libpthread.so.0
File :/sysroot/lib/x86_64-linux-gnu/libresolv.so.2
Dir :/sysroot/lib64
File :/sysroot/lib64/ld-linux-x86-64.so.2
Dir :/sysroot/proc
File :/sysroot/proc/meminfo
Dir :/sysroot/usr
Dir :/sysroot/usr/lib
Dir :/sysroot/usr/lib/x86_64-linux-gnu
File :/sysroot/usr/lib/x86_64-linux-gnu/libstdc++.so.6

Yep - that's all 20 files of it. Actually 6 of those are just directory entries.

Ok, let's build the image first:

➜  sec-article  cat build.sh
#!/bin/sh

export GOOGLE_APPLICATION_CREDENTIALS=~/gcloud.json
ops image create -c config.json -p node_v12.13.0 -a main.js

➜  sec-article  ./build.sh
[node main.js]
bucket found: my-bucket
Image creation started. Monitoring operation operation-1573756133681-59752a74faae6-6944eb54-9f7ee502.
............
Operation operation-1573756133681-59752a74faae6-6944eb54-9f7ee502 completed successfully.
Image creation succeeded node-image.
gcp image 'node-image' created...

Then we can boot it:

➜  sec-article  ops instance create -z us-west1-b -i node-image
ProjectId not provided in config.CloudConfig. Using my-project from default credentials.Instance creation started using image projects/my-project/global/images/node-image. Monitoring operation operation-1573756213461-59752ac110224-644f49fc-4440b475.
.....
Operation operation-1573756213461-59752ac110224-644f49fc-4440b475 completed successfully.
Instance creation succeeded node-image-1573756213.

➜  ~  curl -XGET http://35.247.123.61/
Error: spawn ENOSYS
➜  ~  curl -XGET http://35.247.123.61/safe
dev,etc,kernel,lib,lib64,main.js,node_v12.13.0,proc,sys,usr
➜  ~  curl -XGET http://35.247.123.61/cmd\="touch%20tmp"
Error: spawn ENOSYS

So we can see we are safely not allowing any other processes to be spawned on the end machine. It isn't a matter of filtering the calls either - the system itself straight up doesn't have support for it. If you want more programs running just boot up another instance and if you want more performance out of the server look at other languages.

There are other reasons why we advocating serverless unikernels like this besides security and in upcoming blogposts we'll start showing other superpowers of this style of infrastructure.

Stateful Serverless with Unikernels

Ian Eyberg — Wed, 13 Nov 2019 21:28:27 +0000

One of the big problems with cloud native computing is the reluctance to use what is called a stateful service. Think everything from databases to even just logging a web application and actually keeping those logs. To some of us it sounds a bit crazy to even distinguish between 'stateful' and 'stateless'. Unless you are on some functional programming kick you are going to deal with state.

If you do choose to use traditional cloud native stateful applications you get to learn about everything from stateful sets to persistent volumes to everything else. For your average dev this is just too much craziness.

It gets really crazy when you start to think that most cloud native and serverless frameworks are already running on top of public cloud environments like AWS or Google Cloud. If you haven't been told before "the cloud" is simply an API to virtualization -- and there's a non-trivial amount of software that makes that work. AWS and Google Cloud have gone out of their way to provide a truly cloud native experience by writing the underlying infrastructure to do the networking and the storage and everything else. To be clear - I'm not talking about ECS and GKE here. I'm talking about when you go and boot a VM on one of those respective systems think about what all is happening underneath.

It almost makes you wonder why development teams would choose to re-write all of that knowing full well that they are going to do a crappier job than the infrastructure teams at Google/Amazon. To boot, most of the cloud native ecosystem takes serious performance hits from having to replicate networking and storage layers on top of the existing infrastructure and they suffer from serious security issues like cryptojacking.

I was re-watching this talk from Andrew Tanenbaum recently and he asks a very poignant question:

"Why are our operating systems not like our tvs? You click the tv on and you click it off and it just works."

Good question Andrew!

The cloud native ecosystem makes some developers lives easier because they don't need to mess with configuring or deploying the base operating system but isn't there something that is kinda like a container but also has persistence and security baked in? Ever wish you could have a true cloud native serverless option but with actual real persistence?

Well you can. Today we'll show you how to deploy stateful serverless with unikernels.

Stateful Serverless

For this tutorial you'll want to go grab OPS. If you'd like to build from source go check out the github repo. We'll be deploying to Google Cloud today but you can also sub it out for AWS. On Google Cloud you need a bucket to store your images in and a service key (that I stick in ~/gcloud.json) and then create this config.json:

{
    "CloudConfig" :{
        "ProjectID" :"my-project",
        "Zone": "us-west1-b",
        "BucketName":"my-bucket"
    }
}

Now let's look at some sample code. We have a very simple node.js webserver that does two things. It either appends to a file through the '/savelog' url or it shows what's in the log via the '/getlog' url.

Also - I'm not a practicing node developer so please excuse the non-idiomatic craziness of the code. Apologies in advance.

I left the port configurable here so you can try it out on a different port locally before deploying it out on gcloud.

const fs = require('fs');
var http = require('http');

var logfile = 'bob.log';
var port = 80;

http.createServer(function (req, res) {

    if (req.url == '/getlog') {

      if(!fs.existsSync(logfile)) {
        console.log("File not found");
      } else {
        var contents = fs.readFileSync(logfile).toString();
        console.log(contents);
      }

      res.writeHead(200, {'Content-Type': 'text/plain'});
      res.end(contents + '\n');

    } else if (req.url == "/savelog") {
      fs.appendFileSync(logfile, 'test' + '\n');

      res.writeHead(200, {'Content-Type': 'text/plain'});
      res.end('logged\n');

    } else {

      res.writeHead(200, {'Content-Type': 'text/plain'});
      res.end('Hello World\n');

    }

}).listen(port, "0.0.0.0");

console.log('Server running at http://127.0.0.1:' + port);

We can build our image simply by stating we want the node version 12.13.0 package and pointing it at our entryfile bob.js.

export GOOGLE_APPLICATION_CREDENTIALS=~/gcloud.json

ops image create -c config.json -p node_v12.13.0 -a bob.js

If we run that we can build our image - it's fairly quick cause all it's doing is copying the disk image up into our bucket:

➜  stateful-serverless  ./build.sh
[node bob.js]
bucket found: my-bucket
Image creation started. Monitoring operation operation-1573675163382-5973fcd1af403-c6abea13-cf898ff6.
...............
Operation operation-1573675163382-5973fcd1af403-c6abea13-cf898ff6 completed successfully.
Image creation succeeded node-image.
gcp image 'node-image' created...

One that is done we can create an instance like so:

 ops instance create -z us-west1-b -i node-image

ProjectId not provided in config.CloudConfig. Using my-project from default credentials.Instance creation started using image projects/my-project/global/images/node-image. Monitoring operation operation-1573675228678-5973fd0ff4976-a6b7132d-9aa57c18.
.....
Operation operation-1573675228678-5973fd0ff4976-a6b7132d-9aa57c18 completed successfully.
Instance creation succeeded node-image-1573675228.

Then we can hit it up a few times by querying '/savelog'. After that we can view '/getlog' to see that we saved our data.

eyberg@box:~/stateful-services$ curl -XGET http://34.83.134.230/savelog
logged
eyberg@box:~/stateful-services$ curl -XGET http://34.83.134.230/savelog
logged
eyberg@box:~/stateful-services$ curl -XGET http://34.83.134.230/savelog
logged
eyberg@box:~/stateful-services$ curl -XGET http://34.83.134.230/getlog
test
test
test

Now let's stop the instance - not terminate it but stop it. This is where traditional cloud native solutions and other serverless based options tend to have problems. They view the running image as ephemeral and not ever coming back. However for true cloud native applications like unikernels this is simply not the case. All we are doing is telling Google to merely stop the instance not kill it. This is good if you want say a cough cloud native database cough.

We can go ahead and stop the instance and then restart it.

eyberg@box:~/stateful-services$ ops instance stop -p my-project -z us-west1-b node-image-1573676028
Instance stopping started. Monitoring operation operation-1573676572018-5974021110877-37b0f038-95aa93d0.
.............................................................

eyberg@box:~/stateful-services$ ops instance list -p my-project -z us-west1-b
+----------------------------------+------------+-------------------------------+-------------+---------------+
|               NAME               |   STATUS   |            CREATED            | PRIVATE IPS |  PUBLIC IPS   |
+----------------------------------+------------+-------------------------------+-------------+---------------+
| node-image-1573676028            | TERMINATED | 2019-11-13T12:13:50.106-08:00 | 10.240.0.50 |               |
+----------------------------------+------------+-------------------------------+-------------+---------------+
| releases-150719-14-07-1563179925 | TERMINATED | 2019-07-15T01:38:47.354-07:00 | 10.240.0.9  | 35.233.232.59 |
+----------------------------------+------------+-------------------------------+-------------+---------------+

eyberg@box:~/stateful-services$ ops instance start -p my-project -z us-west1-b node-image-1573676028
Instance started. Monitoring operation operation-1573676869968-5974032d363e0-d0a18a2c-0d56186d.
...
Operation operation-1573676869968-5974032d363e0-d0a18a2c-0d56186d completed successfully.
Instance started node-image-1573676028.

eyberg@box:~/stateful-services$ ops instance list -p my-project -z us-west1-b
+----------------------------------+------------+-------------------------------+-------------+---------------+
|               NAME               |   STATUS   |            CREATED            | PRIVATE IPS |  PUBLIC IPS   |
+----------------------------------+------------+-------------------------------+-------------+---------------+
| node-image-1573676028            | RUNNING    | 2019-11-13T12:13:50.106-08:00 | 10.240.0.50 | 34.82.108.88  |
+----------------------------------+------------+-------------------------------+-------------+---------------+
| releases-150719-14-07-1563179925 | TERMINATED | 2019-07-15T01:38:47.354-07:00 | 10.240.0.9  | 35.233.232.59 |
+----------------------------------+------------+-------------------------------+-------------+---------------+
eyberg@box:~/stateful-services$ curl -XGET http://34.82.108.88/getlog
test
test
test
test
test

Hooray - your data is still there. This is precisely what you would see in a traditional linux environment yet now you get the benefit of the container/serverless like atmosphere as well.

You'll notice that the ip might change as we don't have an elastic/static ip attached to it - that's ok and expected. If you were actually deploying to prod you might assign one or stick this behind a load balancer instead.

If you're new to this style of infrastructure it's worth pointing out a few things. These instances are not linux instances - as in they don't even have a linux kernel inside - at all. They don't have the concept of usernames/passwords. You can't login to them and they can not run more than one program per instance. This is all by design to make them faster, more secure and easier to manage then existing options.

This was just a quick and dirty example to show you the basic concepts behind the idea of stateful serverless. No servers to deal with but you can keep your state cake and eat it too.

What will you deploy?

OPS - Rethinking Cloud Infrastructure

Ian Eyberg — Tue, 22 Jan 2019 21:46:39 +0000

Unikernels have been a technology that have been predicted for the past few years to be the future of software infrastructure.

Wait - hold up - what's a unikernel!? One way to think of a unikernel is what would happen if you converted your application into it's own operating system. It is a single purpose operating system versus a general purpose one like Linux. Another really large difference is that a unikernel is meant to only run one application per server. Since practically everything that is deployed to day is deployed onto virtual machines (eg: all of public cloud) unikernels propose that you don't need a full blown operating system for each application you want to deploy. This makes them run faster, more secure and they become way tinier.

However, unikernels have not been without their problems. One of these problems that unikernels have traditionally suffered from is that to truly utilize them in the past you had to have been a systems developer or a low level c programmer.

Not anymore.

OPS is a tool to help anyone build and run unikernels in whatever language or as whatever application they want. OPS simply loads ELF binaries just like Linux does but with unikernel benefits. Before 'unikernels' we had what were known as 'library operating systems'. OPS and the kernel that it uses 'Nanos' embraces that concept. They are both under heavy active development and have a full time team of paid kernel engineers working on it. (This has also been historically a big problem for the ecosystem.)

For local development OPS works on linux or mac currently.

To get started you'll want to install the cli tool - if you aren't a Go user you can just download the binary from the site but if you are a Go user than you can build it yourself as well once you obtain the source .

Install

$ curl https://ops.city/get.sh -sSfL | sh

Let's try out a quick Node.js hello world.

Put this into a file:

  console.log("Hello World!");

Then you can run it.

  $ ops load node_v11.15.0 -a ex.js

Easy huh? Want to try something a bit more complicated? Let's try a Go webserver.

Put this into main.go:

  package main

  import (
      "log"
      "net/http"
  )

  func main() {
      fs := http.FileServer(http.Dir("static"))
      http.Handle("/", fs)

      log.Println("Listening...on 8080")
      http.ListenAndServe(":8080", nil)
  }

If you are on a mac specify the os as 'linux'. It's important to note that it's not running linux underneath - at all. Unikernels aren't containers. The only reason we specify GOOS as linux is that Nanos implements the POSIX standard (to some degree) and loads our programs as ELFs.

  $ GOOS=linux go build main.go

You can verify this by looking at the filetype:

➜  twe  file main
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

Note that I built this on a mac whose native file type is a Mach-O. We mainly support ELF since that is what is deployed to production.

➜  twe  file main
main: Mach-O 64-bit executable x86_64

Now let's create a folder to put our static assets in:

  $ mkdir static
  $ cd static

  <!doctype html>
  <html>
  <head>
  <meta charset="utf-8">
      <title>A static page</title>
  </head>
  <body>
      <h1>Hello from a static page</h1>
  </body>
  </html>

Then let's specify a config.json for this:

 {
      "Dirs" : ["static"]
  }

This simply states that we wish to put the directory 'static' into /static as a folder in our filesystem. There are many advanced options that you can explore with OPS config.json but please be aware that it is under active development and a lot of it can and will change.

Now we are going to run it. OPS implements a very small wrapper around QEMU. Qemu allows us to run the resulting built image as a virtual machine. The virtual machine is the output of ops building your app as it's own unique little operating system. Have you ever wondered to build your own operating system? Well you just did.

By default OPS will run your unikernel in what's known as 'user-mode' networking without KVM acceleration. Usermode networking is a bit slower than a proper bridged network (what you'd get on AWS or GCE) but we are playing around on your laptop so this is easy to get started with. Also, keep in mind that we run this as a non-root user by default and by default KVM will want root privileges. As a more advanced lesson later we'll show you how to configure that properly. Also keep in mind that on mac we don't have access to KVM, however, there is an equivalent - Intel HAX which we can use if we want.

  $ ops run -p 8080 -c config.json server

Now that it is running you should be able to run curl against your server and wah-lah!

  curl http://localhost:8080/hello.html

You've just built and ran your first Go webserver as a unikernel! How cool is that?

Check out the github repo, star it and let's see what else can be created!

nanovms / ops

ops - build and run nanos unikernels

OPS

Ops is a tool for creating and running a Nanos unikernel. It is used to package, create, and run your application as a nanos unikernel instance.

Check out the DOCS.

Installation

Most users should just download the binary from the website:

Binary install

curl https://ops.city/get.sh -sSfL | sh

If you don't like this option you can also download pre-made packages for various systems here and you can also build from source.

Desktop applications

Operating System	Download
macOS
Windows

MacOS via Homebrew

Add the repo & install:

brew tap nanovms/homebrew-ops

brew install nanovms/ops/ops

See the formula file for details.

Debian / Redhat:

Add a deb src:

sudo vi /etc/apt/sources.list.d/fury.list

deb [trusted=yes] https://apt.fury.io/nanovms/ /

Update your sources && install:

sudo apt-get update && sudo apt-get install ops

Build and Install from source

Building from source is easy if you have used Go before.

This program…

View on GitHub