DEV Community: Andreas Offenhaeuser

Five major changes when moving to vuetify 2.1

Andreas Offenhaeuser — Sun, 06 Oct 2019 20:48:33 +0000

This is a crosspost from my personal blog blog.anoff.io

I just migrated the code for the devradar editor to the latest major version of vuetify.
There is an officiel migration guide that helped me solve 70% of the issues but here is a quick overview of the biggest issues I encountered and what actually changed.

🚨 This was written for a migration from vuetify 1.5.14 to vuetify 2.1.1
Also all code samples are typescript, but should also work for Javascript projects

Bootstrapping

Looking at the new Quick start section of vuetify 2.x you will notice that the way vuetify is added to Vue.js has changed.
Previously vuetify was just included via Vue.use() now it also needs to be instantiated.

//main.ts (vuetify 1.5)

import Vue from 'vue'
import App from './App.vue'
import '~vuetify/src/stylus/main'
import Vuetify from 'vuetify'

Vue.use(Vuetify, {
  theme: appConfig.theme
})

/* eslint-disable no-new */
new Vue({
  el: '#app',
  components: { App },
  render: h => h(App),
  template: '<App/>'
})

The new setup would look like this

// main.ts (vuetify 2)

import Vue from 'vue'
import Vuetify from 'vuetify/lib' <1>
import App from './App.vue'
import 'vuetify/dist/vuetify.min.css' <2>

Vue.use(Vuetify) <3>

const vuetifyOpts = { <4>
  theme: appConfig.theme
}

/* eslint-disable no-new */
new Vue({
  el: '#app',
  router,
  store,
  components: { App },
  render: h => h(App),
  template: '<App/>',
  vuetify: new Vuetify(vuetifyOpts) <4>
})

<1> Load vuetify
<2> include the css instead of stylus sheet
<3> Register vuetify with Vue.js
<4> Configure and instantiate vuetify in the new vue instance

Loading styles

This was already implicitly shown in the previous section but might be worth another mention.

Whereas previously the vuetify styles were provided via a styl(us) file they are now precompiled css.

// vuetify 1.5
import 'vuetify/src/stylus/main.styl'

// vuetify 2.x
import 'vuetify/dist/vuetify.min.css'

Also note that the npm module sass is required and node-sass no longer works.
You may need to swap by running

npm uninstall node-sass
npm install --save-dev sass

Adding vuetify types to typescript config

If you get a compile error from typescript stating that Argument of type '{..}' is not assignable to parameter of type 'ComponentOptions<..>' or Object literal may only specify known properties, and 'vuetify' does not exist in type you need to register the vuetify types with typescript.
This was something I had not done before and may only be necessary with the change to vuetify 2.

// tsconfig.json
{
  "compilerOptions": {
    "types": [
      "webpack-env",
      "node",
      "vuetify" <1>
    ]
  }
}

<1> add vuetify to the types property

Theme options

If you are using a custom theme you might need to adapt to the new object structure that supports light and dark mode for your app.
It is mostly moving the color specification of the theme into another nested layer.

Add MDI font

Vuetify now uses the material design icons for default icons like the hamburger navigation menu.
Install it as a dev dependency if you have not done so yet.
Alternatively you could configure Vuetify to use another icon font, see the official getting started docs for infos on that.

npm install --save-dev @mdi/font

Then add it to your main.ts

import '@mdi/font/css/materialdesignicons.css'

Component changes

With the above changes your app should build correctly, however there will still be a lot of errors in the browser as many components have breaking changes.
Below are the main changes I had to fix in my devradar editor application.

Application Toolbar

There is a new component v-app-bar that should be used for application wide navigation toolbars.

// vuetify 1.5
<v-toolbar
    app dense scroll-off-screen
    color="accent"
    >

// vuetify 2
<v-app-bar
  scroll-off-screen
  dense
  color="accent"
  >

List view

All components in the list category have been renamed from list-tile-xyz to list-item-xyz.
Best just run a replace all operation and see if it broke anything 😉

Done

These changes made my application compile and render the home app component without issues.
Various components changed and you may need to consult the migration docs for specific cases -- or just look at the new API docs directly as they are way more detailed.

If you stumbled upon this post, I hope it helped you. If it did not I would love to hear what you are missing in the comments or via Twitter DM 👋

Preview AsciiDoc with embedded PlantUML in VS Code

Andreas Offenhaeuser — Sat, 11 May 2019 16:55:18 +0000

Crosspost from blog.anoff.io/2019-05-08-asciidoc-plantuml-vscode

This post is for everyone that likes to write AsciiDoc in VS Code but also wants to inline PlantUML diagrams within their docs. In a previous post about diagrams with PlantUML I gave an intro into PlantUML and how to preview images in VS Code. With the latest release of the asciidoctor plugin for VS Code it is possible to easily preview embedded PlantUML images within AsciiDocs.

Follow me on Twitter at @an0xff for future blog updates, not all of them make it to dev.to. Currently I am writing mostly about docs-as-code.

Prerequisites

You should already have Visual Studio Code installed on your machine. At the time of writing this post I am using v1.33.1 on MacOS and also verified the setup on a Windows 10 machine.

For the AsciiDoc preview to work we will use the AsciiDoc extension that you can get by executing



code --install-extension joaompinto.asciidoctor-vscode

💡 The feature we are going to use here is rather new and shipped with 2.6.0 of the AsciiDoc plugin.

The third thing you need is a PlantUML server. There are multiple options:

use the public plantuml.com/plantuml server
deploy your own java plantuml-server
run plantuml/plantuml-server docker container on your local machine

For test cases option 1 works fine; even if the server claims it does not store any data I would advise you to host your own server if you are working on anything professionally that is not open source. Setting up a PlantUML server is rather easy if you are familiar with Docker, you can see an example setup in my blog post from march 2019. Finally the third option of running it locally within docker is great if you are on the road or sitting somewhere without WiFi.

This post will use option 1 as it just works out of the box while following these instructions.

Configuring the extension

The option we will use for this feature is asciidoc.preview.attributes that allows you to set arbitrary AsciiDoc attributes. These attributes will be injected into the preview. You could also set the attribute manually on each file but that is really something you do not want to do for generic configs like a server URL. Build systems in the AsciiDoc ecosystem like Antora allow you to set attributes during the build process (see this example), so having a local editor that also injects these attributes is super handy.

Under the hood the AsciiDoc VS Code extension relies on the javascript port of asciidoctor and the asciidoctor-plantuml.js extension. This extension needs the :plantuml-server-url: attribute to be set in the AsciiDoc document to become active and parse PlantUML blocks.

So all you need to do in VS Code is to hop into your user settings and add the following entry



"asciidoc.preview.attributes": {
  "plantuml-server-url": "http://plantuml.com/plantuml"
}

⚠️ The downside of using the public server is that it does not offer SSL encrypted endpoints and you must weaken your VS Code security settings to preview correctly.

The PlantUML images are served over http:// and you must allow your preview to include data from unsafe sources. To do this open your command palette (⌘+P, ctrl+P) and enter asciidoc preview security and choose Allow insecure content. In case you are running a local PlantUML server you may choose Allow insecure local content.

Figure 1. opening asciidoc preview security settings

Figure 2. allow insecure content

Live Preview AsciiDoc with embedded PlantUML

To test it out just create an example file with some PlantUML content.
puml

Figure 3. This image is rendered on the fly

With the attribute set correctly the above code block renders as an image

..without the attribute set or issues with the security settings you just see a code block

Hope this post helped you. If you have any questions or know of better/alternative ways leave a comment 👋

What I learned from: How Google Works

Andreas Offenhaeuser — Thu, 31 Jan 2019 08:00:00 +0000

🤞 Cross posted from https://blog.anoff.io/2019-01-31-book-google-schmidt/

As promised my book series will start with two books out of the FAANG league. The book How Google works covers many aspects of the companies culture. The book has lots of management insights that - at the time of reading - were of little interest to me. Therefore I want to briefly highlight a few points about innovation that I find fascinating and then move to the human factors like recruiting and competence management that are also covered extensively in the book.

Innovation

Photo by Sharon Pittaway on Unsplash

The book makes some clear statements about what innovation really means for Google and might help you differentiate between bullshit bingo and real innovation:

Giving customers what they want is not innovative
- Relates to the Henry Ford (or not) quote “If I had asked people what they wanted, they would have said faster horses.”
- Instead surprise customers and provide something fundamentally useful they did not have before
Multiple small non-innovative steps combined can lead to innovative products
- One example given here is Google Mail that was not an innovation per se but the combination of cloud storage, web applications, artificial intelligence created a product that entered an existing market and disrupted it
Innovators do not need to be told that they should be innovative
- Innovation is not learned it is a personality trait
- You can foster innovation but you can not learn or command it
- To achieve true innovation give innovators freedom to operate (time, resources, trust)
Large corporations attract risk-avoiding employees that often tend to hinder innovation
- Employees that seek a safe place hardly embrace change and even less actively bring it into an organization

Management

Photo by rawpixel on Unsplash

As mentioned before there are a lot of management aspects covered in this book that did not make it into my personal notes. I might have to read those up again at some point. One deduction that stuck with me is about processes in an innovative environment:

"If everything seems under control, you're just not going fast enough." (Mario Andretti, Race car driver)
- In this context: If you do not experience any problems you are not innovating as much as you could
Processes are slower than reality
- YES, YES, YES
Chaos is the normal not an exception
- In an innovative environment what you consider realistic might change any day
Order and processes can not be enforced
- Even in innovative environments you need to establish some rules
- These rules need to be decided and implemented by everyone involved from top management to junior employee
- To have this open discussion and decision making you need to have a culture of openness, transparency, meritocracy and honesty

Recruiting

Photo by Ryoji Iwata on Unsplash

A term that keeps reappearing in the book is the smart creative that is described a person that belongs to

"[..] the ambitious ones of all ages who are eager (and able) to use the tools of technology to do a lot more. Their common characteristic is that they work hard and are willing to question the status quo and attack things differently. This is why they can have such an impact.
It is also why they are uniquely difficult to manage, especially under old school models, because no matter how hard you try, you can’t tell people like that how to think. If you can’t tell someone how to think, then you have to learn to manage the environment where they think and to make it a place where they want to come every day."

This definition resonates so well with me because after reading it I realized that all the people I look up to within my personal network are not willing to accept the status quo and keep asking Why do we need to do it this way?. I fought many battles myself, often I was close to giving up but in the end being persistent in your will to change something you will most likely succeed. Too often things are done in ways that are based on assumptions that are no longer valid or might not be applicable for you. Keep in mind to stay constructive when going up against established rules though - try to understand their reasoning and explain why you think it may not apply to your cause. Playing naive also helps 😉

The main points that the book makes about hiring such smart creatives are:

Hire people that are life long learners
- People you do not have to train for the new upcoming technology but those that want to bring it in and learn it out of curiosity
- One point I take away from this is to read CVs more carefully regarding things like off-work trainings, pet projects etc.
Generalists over experts
- The reasoning is that IT changes so quickly it is likely the field you are an expert in now may no longer be relevant to the industry, your product or your company in the future. It is more important to excel at learning new concepts than to already master them.
Hiring (only) people you want to have as friends makes a bad team
- This aspect might be controversial but given the reasoning I think it makes sense - and is too often ignored
- People you like as friends share your interests and your views on things
- This results in little diversity in opinions and paths of reasoning
- Without diversity you will never have all possible options on the table when you need to make a decision
- This might lead to a sub-optimal decision because the best solution was not considered
- It matters more that you are able to have an interesting (objective) discussion with someone than to like them (LAX rule)
Hiring is every employees responsibility
- To get the best you need to cast a wide net
- Every employee should be looking for new talent
- Make it part of your yearly employee reviews and bonuses for everyone, not just HR
Always hire the best
- Train your employees to hire people they look up to
- Once you start hiring someone you consider only good and they themselves start hiring the collective skill will spiral downwards
- Never settle for good

Competence Management

Photo by Hello I'm Nik on Unsplash

One point from the recruiting category also has a big impact on the competence management of employees. Life long learning is something that needs to be enabled, supported and fostered by the organization. If you hire smart creatives they will leave as soon as they realize there is nothing new to learn or - even worse - that learning new things might be frowned upon as wasting time.

Another point made in the book that I find extremely tempting to implement in my own environment is the concept of snippets:

Every employee writes a short weekly summary of what they were working on
- Project managers might write about customers they talked about
- Developers write about tools they learned or bugs they found
0.5 - 1 page per week, bullet points
Share it openly with everyone in your company
- The organization needs to provide a central infrastructure where people can store their snippets
- Do not mail it to your manager
- Snippets need to be searchable by fulltext and person
If you are interested in a persons work, just look up their past snippets to see what they are working on exactly
If you are looking for someone that might help you with XY just search the snippets for it

I think the act of writing these snippets is very little overhead to ones daily work if you get used to it but it can be such a powerful tool to boost network opportunities and knowledge exchange.

Let me know if you liked this article or which parts you find especially fascinating by leaving a comment below, on Twitter or message me on LinkedIn

Stay tuned for the best parts of Powerful

👋 Andreas

Get the most out of hackathons

Andreas Offenhaeuser — Sun, 09 Dec 2018 20:32:25 +0000

Cross posted from https://anoff.io/blog/2018-12-09-stuttgart-hackathon/

TL;DR; Network and play with expensive toys

Intro

End of October I joined the Stuttgart 🇩🇪 Hackathon for the second time. It was my overall 4th public Hackathon. In this blog post I want to tell you why I enjoy doing Hackathons and why you should join one too if you have the chance!

How I used to approach Hackathons

The first Hackathon I ever participated in was hosted by Zeiss April 2017. Together with a Tim Großmann I formed a team to compete in Munich for ideas and projects around Digitalization. We did not join to win but mostly to have fun and write a bit of code - so on our way to Munich we were discussing what would be fun to do. Not knowing the exact scope of the Hackathon we thought doing something with augmented reality on web apps would be fun. We also did not manage to get our hands any of the gadgets we wanted to include in our solution therefore we had to go with a pure code solution.

So we built microllaborators - an awful play of words with microscope and collaborators. We discussed the idea with the coaches and also came up with a pretty solid pitch with a working live demo including the audience. I still like the solution we built but sadly we only made 5th place 😢

Last years Stuttgart Hackathon started off completely different. I did not join with a team but participated in the on site team building event to find new people to code with. Having found a group of four something simliar to the Zeisshack happened;

we did not get the gadgets we needed to implement the idea we came up with

So we fiddled around for 3 hours with the stuff we got + what we brought from our personal stashes and then decided we would not try to compete at all. Instead we spent another 10 hours teaching each other about languages, tools and programs that each of us used and left the Hackathon after 2/3.

What changed

In the past I mostly joined Hackathons to gather experience with new languages, frameworks or gadgets like Alexa, Hue etc.

By now I go there for two main reasons:

networking
play with things I would never get my hands on otherwise

As I also do a lot of side projects at home I want Hackathons to give me something that my personal projects do not. Therefore I do not want to build a fancy blockchain based, AWS hosted IoT solution. I can do that at home for very little money 💸.
What I can not do at home is meet awesome people that hang out at such events or get my hands on some prototypes or industrial equipment worth several thousand Euros 💰.

Stuttgart Hackathon 2018

At this years Stuttgart Hackathon my team did exactly this. We all joined with the clear intention to not compete to win. If we end up with a solution that would be pitch worthy we agreed we should pitch it. But we did not make it our primary goal. We wanted to have fun - and we got our hands on one of these badasses: The Festo Bionic Cobot a humanoid robot arm powered by air pressure instead of electrical motors.

After a few hours of fun with the robot we eventually thought about an actual project that would give the robot a purpose. Thus R.I.C the robot interaction companion was born. Using this prototype robot was really different from any side project I have done so far because it meant not only using a single gadget but understanding the complex system. The following diagram shows the system setup where the gray Client Code is the part that we coded ourself to control the robot. The great thing was that several experts were on site to help us with understanding and customizing the individual components of the Cobot.

We even managed to integrate a bunch of gadgets like LiDAR scanners and Echo dots with Alexa integration. We came up with a funny story to pitch and our presenter did an awesome job at the final presentations.

My future Hackathon strategy

In the future I will keep attending hackathons with the goal of getting my hands on non consumer grade hardware and networking. If you are a student you might be tempted by the prizes available at Hackathons but I seriously encourage you to use the time to meet new people and exchange ideas and experiences with others - this is the most valuable thing you can get out of such events.

Drop me a message on Twitter or leave a comment. Would love to hear your thoughts on Hackathons! 💬

Image sources

Stuttgart Hackathon - my own
Gadgets - Wikipedia
Cobot - Festo found on their website

RBAC with Google Firestore

Andreas Offenhaeuser — Sun, 12 Aug 2018 07:00:00 +0000

This post will explain how to implement role based access control (RBAC) using the Google Firestore serverless database. Firebase and Firestore in particular with the concept presented in this post offers the most seamless integration of serverless infrastructure with a mobile client at this point. It has become my go to backend for all web apps I build.

Firestore basics

Firestore is database that is part of Googles Firebase suite for mobile app development. It is currently in beta and has the potential to replace the current Firebase Realtime Database due to its superior API and features.

Cloud Firestore is a flexible, scalable database for mobile, web, and server development from Firebase and Google Cloud Platform

For those that never used a Firebase database; it is a NoSQL document oriented database. Firestore allows you to nest documents by creating multiple collections inside a document.

The Firebase suite is built for mobile development and provides SDKs for all major languages. JavaScript/Node.js, Swift, Objective C, Android, Java, Python, Ruby, Go. The SDKs allow add, query or delete data as well as other operations required when interacting with a database as a client. One feature I really like is the possiblity to register your client to receive updates automatically. This allows you to build three way data binding in realtime applications easily. This is a feature I used in my first project with Firebase.

In combination with the Firebase authentication provider you can limit access the database to people that are logged in. The auth provider also provides an SDK and requires only a few lines of code to implement in a web app.

Firestore Security Rules

The ability to create a detailed rule set make Firestore enable use cases for a serverless database without any backend code and still keeping data secure. It is also the foundation for building a role based access control.

All roles and authorization rules will be enforced by the Firestore server

Security rules are written in a JavaScript-like syntax but have their own methods. First you nest match operators to specify the document level you want to be affected by this rule. Use {wildcards} that can later on be referenced in the rule definition. Granting/denying access is done via an allow <method> if statement that grants access if it returns true or otherwise blocks the transaction.

service cloud.firestore {
  match /databases/{database}/documents {
    allow read, write: if <some_condition>;
    allow delete: if false;
  }
}

There are five methods that can be specified:

get: retrieve a single document
list: read an entire collection or perform queries
create: write to non existing documents
update: write to existing documents
delete: remove a document

The modifying operations 3-5 can be addressed using the write method instead of specifying them individually, read applies both get and list. If multiple rules match for a request only one needs to resolve to true for the request to be successful.

// allow anyone to read but only signed in users to create/update; only a specific user can delete
service cloud.firestore {
  match /databases/{database}/documents {
    match /posts/{document=**} {
      allow read: if true;
      allow create, update: if request.auth != null;
      allow delete: if request.auth.uid == 'an0xff';
    }
  }
}

RBAC example scenario

We will setup RBAC for a simple content site with posts that can be commented with the following roles:

admin: can assign roles
writer: can create new posts, can modify its own posts
editor: can edit any post, delete comments
user: can create and modify its own comments, can modify his user settings

On the root level of Firestore we add three different collections. One holds the content, one for user details and one that implements the roles per user. The reason to separate role assignments from the user document is to easily allow users to modify their own details without giving them the possibility to grant themselves new roles.

Required security rules for RBAC

Given the collection setup and the above role definition we can define the rules we need to implement for each collection.

	User	Roles	Posts	Comments
`get`	anyone	user (own), admin	anyone	anyone
`list`	noone	admin	anyone	anyone
`create`	noone	noone	writer	user
`update`	user (own profile)	admin	writer (own), editor	user (own)
`delete`	user (own profile)	noone	writer (own)	user (own), editor

Implementing security rules for RBAC

We start with a default DENY ALL policy setup for the collections in our project.

service cloud.firestore {
  match /databases/{database}/documents {
    // this addresses any entry in the user collection
    match /users/{user} {
        allow read, write: if false;
    }
    // rules for the roles setup
    match /roles/{user} {
      allow read, write: if false;
    }

    match /posts/{post} {
      allow read, write: if false;
    }

    match /posts/{post}/comments/{comment} {
      allow read, write: if false;
    }
  }
}

RBAC helper functions

We start by implementing a few custom functions that help us define role based rules.

service cloud.firestore {
  match /databases/{database}/documents {
    ...
    // the request object contains info about the authentication status of the requesting user
    // if the .auth property is not set, the user is not signed in
    function isSignedIn() {
      return request.auth != null;
    }
    // return the current users entry in the roles collection
    function getRoles() {
      return get(/databases/$(database)/documents/roles/$(request.auth.uid)).data
    }
    // check if the current user has a specific role
    function hasRole(role) {
      return isSignedIn() && getRoles()[role] == true;
    }
    // check if the user has any of the given roles (list)
    function hasAnyRole(roles) {
      return isSignedIn() && getRoles().keys().hasAny(roles);
    }
  }
}

With these functions in the security rules you can now easily implement security roles based on the users roles.

Users collection

First we make sure only the user itself can modify its data:

match /users/{user} {
  // anyone can see a specific users profile data (name, email etc), in a real scenario you might want to make this more granular
  allow get: if true;
  // noone can query for users
  allow list, create: if false;
  // users can modify their own data
  allow update, delete: if request.auth.uid == user;
}

Roles collection

Next we enforce the rules for the role collection

match /roles/{user} {
  allow get: if request.auth.uid == user || hasRole('admin');
  allow list: if hasRole('admin');
  allow update: if hasRole('admin');
  allow create, delete: if false;
}

Posts collection

This one is a little trickier because we first need to figure out who actually created the post if we want to enforce the update rule.

match /posts/{post} {
  allow get, list: if true;
  allow create: if hasRole('writer');
  // check if the post author is identical to requesting user
  allow update: if (hasRole('writer') && resource.data.author == request.auth.uid) || hasRole('editor');
  allow delete: if hasRole('writer');
}

Comments (Sub)collection

Make sure that users can modify or delete their comments and editors can moderate anyones comments.

match /posts/{post}/comments/{comment} {
  allow get, list: if true;
  allow create: if hasRole('user');
  // check if the comment author is identical to requesting user
  allow update: if resource.data.author == request.auth.uid);
  allow delete: if hasRole('editor') || (hasRole('user') && resource.data.author == request.auth.uid);
}

Summary

You created a Firestore ruleset that enforces data privacy and integrity by limiting access to resources based according to a role design.
To bring RBAC to your application you need to consider a few things:

Deploy the security rules into your Firebase project, highly recommend using the CLI or Travis
Implement a Firebase function that creates an entry in the users and roles collection for each new user using the authentication trigger
Log in to your app and then manually grant yourself admin rights via the Firebase console

To see a similar design implemented you can check out my techradar project which also showcases how to implement RBAC on the frontend and setting up Travis CI to deploy Firebase rules. There are additional features available to secure Firestore data for example checking which parts of a resource are being changed in an update process to allow users to update only specific properties.

Drop me a message on Twitter if you have any feedback about this post. Also check out my previous post on creating PlantUML diagrams.

Markdown native diagrams with PlantUML

Andreas Offenhaeuser — Tue, 31 Jul 2018 07:00:00 +0000

This post was originally published on my blog under https://anoff.io/blog/2018-07-31-diagrams-with-plantuml/ and crossposted for 💖

This post will cover PlantUML basics and how it can be used in GitLab or GitHub projects as well as a seamless local development environment using Visual Studio Code.

This is a post I have been wanting to write for months. Lately I have been using PlantUML extensively at work but also in my private projects. You can see it being used in my plantbuddy and techradar projects on GitHub. Using it in different places and for various purposes I came across a bunch of issues that I want to share in this post.

PlantUML Basics 👨‍🎨

For those that do not know PlatUML: It is an open source tool that allows you to define UML diagrams with plain text. There are different diagram types available being described with custom syntax but following a common scheme. This post will not go into the details of each of those diagram types because the PlantUML website does a pretty good job at describing sequence, component, activity and the other diagram types.

A basic component diagram showing data flow can be built using the following markup:

@startuml component
actor client
node app
database db

db -> app
app -> client
@enduml

Reasons to love PlantUML 🤗

Versioning 📦

A very important aspect for developing software and writing documentation is to keep it in sync. One part is to update documentation if the code itself is updated. Another important part is versioning - usually software is versioned using git or similar systems. By putting the documentation into the same repository as the code you make sure to always look at the correct state of documentation for a respective point in time.

For that reason I love putting all my documentation either within the sourcecode as comments or as Markdown files next to the sourcecode. One thing I was always lacking with this approach is visualizing things. Putting PowerPoint/Keynote/Visio/Enterprise Architect.. files into a repository does make sure your diagrams are always versioned with the code - but they are not browsable in Web UIs. Come PlantUML and GitLab rendering to the rescue: GitLab allows you to inline PlantUML diagrams directly into your Markdown files and they will be rendered on the fly when viewing the files in the browser.

One other benefit PlantUML has over the mentioned tools is that by defining your diagrams in plain text you make them diff-able in pull requests. Reviewers can always see what changes have been made and easily compare changes to the diagram with changes made inside the code.

Syntax 🐟

The basic syntax of PlantUML is very concise and builds a good foundation for the different diagram types. It is also well very smart in the way that it allows diagrams to be written with different flavors e.g. you can declare/instantiate all nodes at the top, but if you do not declare them they will be inferred automatically. Same goes for macros and definitions that allow you to compose larger diagrams or a common library for your team.

I recently created a PlantUML Cheatsheet for a lot of useful tricks - it does however not cover the very basics of PlantUML syntax. You can browse the latest version or the LaTeX sourcecode on GitHub.

Layouting 🏗

Compared with WYSIWYG editors PlantUML diagrams only define components and their relationship but not the actual layout of the diagram. Instead the diagram is inferred by a deterministic algorithm in the rendering process. This is beneficial when specifying the diagram because you only focus on the content - comparable to writing a LaTeX document.
Sadly the layouting engine is not as good as you sometimes wish it to be and especially in component diagrams with 10+ nodes you might end up spending a lot of time enforcing specific layouts manually.

For sequence and activity diagrams the automatic layouting works great even for very large diagrams. After you built a few diagrams and notice how easy it is to just move lines of code up and down and have changes in the code immediately reflect in your documentation you will love the automatic layouting.

Share anywhere 📱

If you want to freeze a diagram version and send it to someone outside your organization you can simply send them an insanely long url (e.g. http://www.plantuml.com/plantuml/png/5Son3G8n34RXtbFyb1GiG9MM6H25XGsnH9p8SGgslrpxzFILcHovse-yYw8QdlJl2v--N93rJ2Bg4EDlSBlG0pn6wDiu5NiDcAU6piJzTgKN5NNPu040) that encodes the entire diagram definition. You can also just embed this URL inside an HTML <img> tag. If anyone ever needs to work with the image all you have to do is swap /plantuml/png to /plantuml/uml and you will see the definition of the diagram.

This gives the entire PlantUML toolstack an extremely versatile way of passing information as well as viewable images.

Local development 💻

The fastest, platform agnostic and easiest way to start creating PlantUML diagrams is using their online editor (btw. you can easily host it on prem using the plantuml-server Docker image). This is fine for creating simple diagrams with a few nodes but larger diagrams require a lot of previewing which is annoying in the online editor.

If you have any other local setups please let me know via Twitter

Visual Studio Code

In case you already use VS Code this is a no brainer to set up. Otherwise you might seriously want to consider using it for the purpose of editing PlantUML diagrams (in Markdown) only because it is a super smooth experience.

All you need to do is to get the PlantUML extension to enable codes native Markdown preview feature to also parse inline diagrams.

By default the plugin requires a local PlantUML process to be running and accepting the rendering requests. I recommend switching it to use a server for rendering; this could be the official plantuml.com server, an on premise instance or a locally running container. After installing the plugin go to the VS Code options (ctrl/⌘ + ,) and change the plantuml.render property.

// PlantUMLServer: Render diagrams by server which is specified with "plantuml.server". It's much faster, but requires a server.
// Local is the default configuration.
"plantuml.render": "PlantUMLServer",

// Plantuml server to generate UML diagrams on-the-fly.
"plantuml.server": "http://www.plantuml.com/plantuml",

If you ever go off the grid and still want to work remember to docker run -d -p 8080:8080 plantuml/plantuml-server:jetty while you still have an internet connection. The image is ~250MB to download. Afterwards set plantuml.server to http://localhost:8080/ and you're set for an offline adventure.

On my MacBook I sometimes experience a lot of CPU consumption from the running container - even when not actively rendering. Restarting the container helps 🤷‍

Render to SVG/PDF

This method only works if diagrams are defined explicitly in files and not inlined into Markdown.

To write this blog post and build the Cheatsheet I played around with non-realtime ways of rendering PlantUML diagrams into images. You can use the Makefile and Shell script to convert an entire folder of PlantUML diagrams with .puml extension into .svg and .pdf files.

The script essentially runs the diagram definition through a dockerized PlantUML process which outputs an .svg and then uses Inkscape to create a .pdf file for importing it into LaTeX documents for example.

#!/bin/sh
# converts all puml files to svg

BASEDIR=$(dirname "$0")
mkdir -p $BASEDIR/dist
rm $BASEDIR/dist/*
for FILE in $BASEDIR/*.puml; do
  echo Converting $FILE..
  FILE_SVG=${FILE//puml/svg}
  FILE_PDF=${FILE//puml/pdf}
  cat $FILE | docker run --rm -i think/plantuml > $FILE_SVG
  docker run --rm -v $PWD:/diagrams productionwentdown/ubuntu-inkscape inkscape /diagrams/$FILE_SVG --export-area-page --without-gui --export-pdf=/diagrams/$FILE_PDF &> /dev/null
done
mv $BASEDIR/*.svg $BASEDIR/dist/
mv $BASEDIR/*.pdf $BASEDIR/dist/
echo Done

GitLab integration

This feature is currently only available with on-prem installations of GitLab, enabling it on gitlab.com is an open issue. See the GitHub integration for a workaround.

Using PlantUML within GitLab is super fun. All you have to do is set up a render server to use and you can just commit Markdown files with inlined PlantUML diagrams and they will render for everyone visiting the GitLab web UI.

What's great is that this does not only works in Markdown files committed into a git repository but in all other fields within GitLab that render markdown - virtually everything. You can have small diagrams helping illustrate things in issues as well.

GitHub integration

There is no native PlantUML integration for GitHub and gitlab.com available. To maintain the advantages listed above it is obviously not a valid workaround to just render the files locally and commit them into git.

Instead make use of the PlantUML proxy service as described in this stackoverflow discussion. The way this works is that instead of passing the PlantUML server the diagram content within the URL we define a remote URL where the content can be fetched from e.g. http://www.plantuml.com/plantuml/proxy?src=https://raw.github.com/plantuml/plantuml-server/master/src/main/webapp/resource/test2diagrams.txt. This URL can be embedded in an HTML <img> tag or within Markdown image syntax ![](). To leverage this feature when using GitHub, simply point the remote URL to a raw link of the PlantUML diagram in your repository.

You can see this method in action in the plantbuddy and techradar projects of my GitHub account.

The following diagram shows what will happen when you open a Markdown page hosted on GitHub that contains such a link:

This example shows that adding a ?cache=no might be a good idea because of GitHubs Camo caching strategy which will prevent your images from updating if you change the sourcecode.

The downside of this approach is that it will always render the latest commmit in your repository even if you browse old versions. If browsing old versions is a strong requirement for you when using an integration with GitHub then you might need to build your own plugin/renderer or optimize the local development environment because after all the correct diagram version will always be with the sourcecode you checked out.

To use the proxy service integration simply use:

![cached image](http://www.plantuml.com/plantuml/proxy?src=https://raw.github.com/plantuml/plantuml-server/master/src/main/webapp/resource/test2diagrams.txt)

![uncached image](http://www.plantuml.com/plantuml/proxy?cache=no&src=https://raw.github.com/plantuml/plantuml-server/master/src/main/webapp/resource/test2diagrams.txt)

Summary

There are two fundamental ways of keeping PlantUML diagrams

inline into Markdown
keep as individual .puml files

Depending on your toolstack one of those should be your preferred option to work with diagrams in your repository. It is highly recommend to keep diagrams as close to the code as possible and not create artificial documentation repositories.

This post covered how to write and render files locally in VS Code, using Docker containers and how to integrate into GitLab on prem as well as publich GitHub and GitLab instances.

There is a lot more to tell about PlantUML but I hope this article gave you enough infos to get started on whatever platform you are using. I recommend this PlantUML Cheatsheet which will help you to cover an even wider range of use cases.

Tell me about your experiences with PlantUML or alternative integrations on Twitter 🐦

Building realtime apps with Vue and nodeJS

Andreas Offenhaeuser — Wed, 18 Apr 2018 07:00:00 +0000

This is a crosspost of my blog at anoff.io/blog

Set up a basic Node.js server
Initialize Vue.js project
Adding Material Design library
Bring in websockets
- frontend: vue-socket.io
- backend: socket-io / server.js
Showtime 🍿

Wanting to build a simple SPA as a side project I struggled with things that might annoy a lot of newcomers that do not want to go full vanilla. Which web framework, which style library, which server framework - and more importantly how does it all work together?

In this post we will put together a bunch of great tools out there to build a realtime web app with a few single lines of code. A quick introduction into the tools that will be used:

Node.js: Javascript runtime to build server applications
Vue.js: A webapp framework
Material Design: Set of styled web components by Google using the vue-material library
socket.io: Client & server library for websockets
servers.js: An opinionated server framework for Node.js based on express

Set up a basic Node.js server

The first thing we'll do is set up a node server to provide a backend. Using the servers.js library a basic API service can be build with a few lines of code.

# in any empty directory
npm init # initialize npm project
npm install server

Creating a Hello World example:

// server.js
// load the server resource and route GET method
const server = require('server')
const { get } = require('server/router')

// get server port from environment or default to 3000
const port = process.env.PORT || 3000

server({ port }, [
  get('/', ctx => '<h1>Hello you!</h1>')
])
  .then(() => console.log(`Server running at http://localhost:${port}`))

Running the code with node server gives the following output and the website showing Hello World! will be reachable at localhost:3000

Server running at http://localhost:3000

For easier development install npm install nodemon in the project and change the start command to:

// package.json
"scripts": {
  "start": "nodemon -i myapp/ server.js"
},

💡 If you're struggling take a look at this code for reference

Initialize Vue.js project

The easiest way to set up a vue project is to use the vue-CLI which is available via npm install -g vue-cli. To initialize a project using webpack as a bundler run

vue init webpack myapp

Answer the questionnaire with the default question or disable tests you do not want to implement. I chose not to install any test frameworks for this tutorial.

Webpack comes with it's own development server with hotreloading functionality so you see changes in the browser right away. Try it out by starting the server with npm run dev (in the myapp/ directory) and opening the Vue.js template at localhost:8080

output of the webpack dev server npm run dev

Template Vue.js project at http://localhost:8080

When modifying the Vue.js components the web page will automatically reload

// myapp/src/components/HelloWorld.vue

// chnage the script content to
...
<script>
export default {
  name: 'HelloWorld',
  data () {
    return {
      msg: 'Welcome to MY first Vue.js App'
    }
  }
}
</script>
...

By simply saving the file the development server will propagate the changes to any open browser window which will automatically reload to

Modified template with custom message

💡 If you're struggling take a look at this code for reference

Adding Material Design library

To install vue-material run the following command in the Vue.js directory myapp/

npm install vue-material@beta --save

Add the following lines to myapp/src/main.js to load the vue-material components into the app.

import VueMaterial from 'vue-material'
import 'vue-material/dist/vue-material.css'
import 'vue-material/dist/theme/black-green-light.css'

Vue.use(VueMaterial)

ℹ️ You might have to restart the dev server for this new plugin to take effect

Create a new Vue.js component making use of several vue-bootstrap components like the app container.

<!-- myapp/src/components/Chat.vue-->
<template>
<div class="page-container">
    <md-app>
      <md-app-toolbar class="md-primary">
        <div class="md-toolbar-row">
          <span class="md-title">My Chat App</span>
        </div>
      </md-app-toolbar>
      <md-app-content>
        <md-field :class="messageClass">
          <label>Messages</label>
          <md-textarea v-model="textarea" disabled></md-textarea>
        </md-field>
        <md-field>
          <label>Your message</label>
          <md-input v-model="message"></md-input>
          <md-button class="md-primary md-raised">Submit</md-button>
        </md-field>
      </md-app-content>
    </md-app>
  </div>
</template>

<script>
export default {
  name: 'HelloWorld',
  data () {
    return {
      textarea: "dummy text\nblup\ndummy text"
    }
  }
}
</script>

<!-- Add "scoped" attribute to limit CSS to this component only -->
<style scoped>
.md-app {
  height: 800px;
  border: 1px solid rgba(#000, .12);
}
.md-textarea {
  height: 300px;
}
</style>

To load the new component modify the router at myApp/src/router/index.js

// change HelloWorld -> Chat
import Vue from 'vue'
import Router from 'vue-router'
import Chat from '@/components/Chat'

Vue.use(Router)

export default new Router({
  routes: [
    {
      path: '/',
      name: 'Chat',
      component: Chat
    }
  ]
})

💡 If you're struggling take a look at this code for reference

Bring in websockets

For the following development the web application will consume from two different endpoints. The webpack-dev-server sends the web app sources (HTML, CSS, Javascript) and the node server will supply the socket-io endpoint. This is typically not something you want to do in production but since we want both the node server and Vue frontend to be hot reloaded we need two systems - webpack and nodemon.

frontend: vue-socket.io

For the Vue app to communicate with the websocket backend the socket.io library needs to be installed into cd myApp/

npm install vue-socket.io

With the node backend running on port 3000 modify your vue application in myApp/src/main.js to connect to the backend

import VueSocketIO from 'vue-socket.io'

Vue.use(VueSocketIO, 'http://localhost:3000')

To bring some very basic functionality into the app we will show messages that were send from other instances in a list and add the ability to send messages.
For sending messages we need to give the Submit button an action once it is triggered by adding a v-on:click method

<md-button class="md-primary md-raised" v-on:click="sendMessage()">Submit</md-button>

The sendMessage() function and the socket interactions are specified in the <script> tag

<script>
export default {
  name: 'Chat',
  data () {
    return {
      textarea: '',
      message: '',
      count: 0
    }
  }, sockets:{
    connect () {
      console.log('connected to chat server')
    },
    count (val) {
      this.count = val.count
    },
    message (data) { // this function gets triggered once a socket event of `message` is received
      this.textarea += data + '\n' // append each new message to the textarea and add a line break
    }
  }, methods: {
    sendMessage () {
      // this will emit a socket event of type `function`
      this.$socket.emit('message', this.message) // send the content of the message bar to the server
      this.message = '' // empty the message bar
    }
  }
}
</script>

backend: socket-io / server.js

Server.js already comems with socket-io bundled into it. The only thing to do in the backend to enable a basic chat operation is to react to a message event sent from the UI and propagate this to all connected sockets.

// modify server.js to include the socket methods
const { get, socket } = require('server/router')

...

server({ port }, [
  get('/', ctx => '<h1>Hello you!</h1>'),
  socket('message', ctx => {
    // Send the message to every socket
    ctx.io.emit('message', ctx.data)
  }),
  socket('connect', ctx => {
    console.log('client connected', Object.keys(ctx.io.sockets.sockets))
    ctx.io.emit('count', {msg: 'HI U', count: Object.keys(ctx.io.sockets.sockets).length})
  })
])
  .then(() => console.log(`Server running at http://localhost:${port}`))

After running npm start in the server directory the server will now create logs for every web page that gets opened. It logs the list of currently open sockets.

Note there is no disconnect event registered yet in this tutorial so the count event will only be emitted when a new website connects.

Showtime 🍿

Running the demo in two browsers/separate devices in the same network will look like this. It is a very, very, very basic but totally anonymous chat system.

You can find a repository on github containing this demo code.

I hope this blog helped you:

set up an easy node server
bootstrap a Vue project with vue-cli
get fancy UI elements in Vue using material design
integrate websockets to provide realtime communication

What to do next:

add tests to backend/frontend
store state/sessions in frontend
possibly add authentication
improve UI (e.g. register enter button on message bar)

Feel free to leave a comment or reach out on twitter 🐦

GitLab CI/CD for GitHub — How and Why?

Andreas Offenhaeuser — Fri, 30 Mar 2018 20:08:23 +0000

GitLab CI/CD for GitHub — How and Why?

When creating a git project that you want to share with others you traditionally had the choice between GitHub with its huge community and tons of integrations, GitLab with a great overall dev experience from issues to one of the best CI/CD solutions out there and BitBucket being one of the friends you have since kindergarden. My personal decision was to host all my personal projects on 🦑 GitHub. For projects that need CI/CD I tinkered around with 👷‍♂️ Travis CI and 🅾️ Circle CI on top of GitHub.

It is no longer GitHub OR GitLab

But recently GitLab announced that their amazing CI/CD solution can now be combined with git projects hosted on Github. Having used a fullstack GitLab solution at work for the last 1.5 years I was really psyched to test it for my personal projects. Now you can finally have the best of both worlds — GitHubs reach in the community and GitLabs CI tooling 💃

How to enable a GitLab CI/CD pipeline for a GitHub project

At first glance GitLab integrates like any other CI/CD service into your project. Using webhooks that the service either adds for you automatically or you define manually. GitLab did a pretty good job on describing how to set those up so I wont repeat the steps again in detail.

CI/CD integration into GitHub via webhook

To set it up you first need to have your project on GitHub. I set up a small node.JS web app to play around with the integration. Feel free to fork it or create your own. Next you head over to GitLab sign in — I suggest using GitHub OAUTH to keep your code credentials in one place. To set up your GitLab CI/CD project simply create a new project and choose the option on the far right CI/CD for external repo and click GitHub.

You will see a list of your GitHub repos and hit Connect on the one that you want to use for this integration. In my case it is the repo anoff/node-ci-dummy

After doing this you will see that GitLab actually cloned your entire project. There are two settings that make sure this magic 🧙‍♀️ works. One is the previously mentioned webhook on the GitHub side that will trigger as soon as you git push onto your GitHub repository. The other is the repository settings on your newly created GitLab project that is set to PULL from a remote repository — namely the GitHub project you selected.

GitHub webhook (left) and GitLab repo mirror options (right)

How to configure a CI pipeline

The final thing to do is to set up an actual CI pipeline in your repository. GitLab will automatically recognize the pipeline definition and run the defined jobs. If you are unfamiliar with GitLab CI you might want to start with a minimalist pipeline definition. Just create a .gitlab-ci.yml in your root directory. Example content to test a node.JS app:

image: node:9
lint:
  script:
  - npm install
  - npm run lint

unit test:
  script:
  - npm install
  - npm test

In my dummy repository this will run standard linting and ava unit tests in parallel. Below you can see the commit that introduced this file and the respective pipeline that was triggered.

GitHub commit view with a pending GitLab pipeline

GitLab view of the pipeline in process

Success marker on GitHub after the CI pipeline finished

Check out the docs for more info writing custom pipelines. For example introducing stages might help you get a handle on more complex pipeline workflows. If you are running a public GitHub project and you want guests to see your pipeline results in detail you should make sure your GitLab project is set to Public.

Final thoughts

This post covered how to set up a basic integration workflow between GitHub and GitLab: not a lot of work 🏋️‍♀️ for you actually.

Things to do next:

introduce multiple stages to your pipeline
test against various runtimes
make use of the service tag in GitLab pipelines to spawn a database to run smaller integration tests

In my opinion the only downside of the GitLab CI/CD compared to other solutions like Travis, Circle, Drone is that the fact that GitLab creates a fully featured project for you might create some confusion for people. One thing you should definitely do is follow the permission settings above and disable issues, wiki, pull requests on your GitLab repositiry in this case. That way your permissions will prevent people from interacting with you via GitLab. In addition your project automatically shows up as Mirrored from which links back to your GitHub master project. If that is not enough for you, you can always add a link in your README and your contribution guidelines.

GitLab project showing link to the original project

Oh btw if you are still looking for the Why that the title promised: GitLabs CI/CD solution is just one of the most comprehensive out there imo.

Feel free to discuss on twitter or comments 👍

Automated dev workflow for using Data Science VM on Azure

Andreas Offenhaeuser — Wed, 21 Mar 2018 22:37:16 +0000

This is crosspost from Medium

tl;dr; I put together a bunch of scripts on Github that let you deploy a VM from your command line as well as sync code from your local directory to the VM easily to be able to use local IDE and git but execute on the powerful remote machine. Perfect for Data Science applications based around jupyter notebook.

In my previous blog post I explained how to do Terraform deployment of an Azure Data Science Virtual Machine.

Overview of available commands

Motivation 😓

Recently I started to do some #deeplearning 🔮 as part of my Udacity Artificial Intelligence Nanodegree. When I was working on the #deeplearning Nanodegree last year I started to script starting/stopping an AWS GPU VM and rsyncing code around. This time I felt like giving the Azure Cloud a try. Mostly because my daytime job lets me look at a lot of their services I wanted to venture deeper into the Azure Data Science offerings. Being more of a software developer by trait and less of a data scientist 👨‍🔬 I often feel like my standards for versioning, testing and ease of development are beyond those that the ML ecosystem offers by default (hope that doesn’t offend the data wizards out there). My development machine is a small MacBook without GPU support. So to train neural networks I wanted to get a Virtual Machine with a GPU on board. Azure offers VMs with a prebaked Ubuntu image containing all of today's Data Science tools: Python, Conda, Jupyter, GPU Neuralnet libs etc.

Top features, see this list for full stack available on DSVM (source)

Having the perfect target for running my code I was wondering how to actually keep my local machine my main development machine — meaning I don’t want to setup git on the VM to version my code. This is where our friend rsync comes into the picture 🖼. It lets you sync two directories over SSH.

rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST

The Goal 🏁

Being prone to over-engineering 🙄 my side projects I started my journey of automating my VM workflow with the following goals:

Deploy (and delete) an entire VM by a template that I can version on Github
Start/Stop the VM from my command line so I don’t pay for it if I don’t need it (GPU VMs are 💰💰💰)
Get code changes I make on the VM using jupyter notebook synchronized to my local machine so I can git commit

Deploy infrastructure 📦

Again I opted for Terraform to deploy the VM. As mentioned in my previous blog post you could also use Azure Resource Manager Templates for that, but my personal favorite is Terraform 😍. So I continued from my previous findings to build the following Terraform recipe. The suggested setup is to place the script into an infra folder into your projects working directory.

anoff/vm-automation

It creates several resources:

resource group: logical grouping on Azure that contains all the resources below
virtual network: well..a virtual private network that your resources use to communicate*
network subnet: the subnet that your VPN will use*
network interface: a network interface so your VM can bind against the virtual network
virtual machine: the actual compute resource (will spawn disk resources for file system)
public IP address: a static IP that will make your VM reachable from the internet
local executor (null resource): used to write some results of the VM creation process onto your disk

* sorry if I did not explain those correctly tbh I am not 💯% sure I understand correctly what they do either 😊

This are the created resources

Variables in the Terraform recipe

Leveraging Terraform variables some of the properties of this recipe can be customized.

The easiest way to change the variable values is to config.auto.tfvars file that contains all the variable names and their description as well.

You can find it in the Github repo right next to the Terraform recipe itself. As you can see all variables have a default value even if you not specify the .tfvars properties. The ones you most likely want to modify are admin_public_key and admin_private_key

They are a SSH key pair that you will use to authenticate when connecting to the virtual machine later. During the Terraform process the public key will be stored on the VM so it will later recognize it as a valid key. The private key will be used to SSH into the machine near the end of the process to actually prepare the local file system for later file transfers — namely create a ~/work directory. You might also want to modify the admin username or the resource location.

config.auto.tfvars

Signing the license terms for Data Science VM ⚖️

You might see the following error when trying to run the Terraform script without having read this far.

Terraform error due to missing license agreement

The problem is that the DSVM is published via the Azure market place and even though it does not incur additional charges on top of the Azure VM resources you need to read and agree to the license terms. You can do this as described in the error message via Powershell. The complete process of opening a Powershell is explained in this Readme. The short version if you already have Powershell open is to run:

# Use this command to view the current license agreement

$ Get-AzureRmMarketplaceTerms -Publisher "microsoft-ads" -Product "linux-data-science-vm-ubuntu" -Name "linuxdsvmubuntu" # If you feel confident to agree to the agreement use the following command to enable the offering for your subscription

$ Get-AzureRmMarketplaceTerms -Publisher "microsoft-ads" -Product "linux-data-science-vm-ubuntu" -Name "linuxdsvmubuntu" | Set-AzureRmMarketplaceTerms -Accept

After successfully signing the license terms you should see the following output in your shell

Run Terraform 🏃‍♂️

Once the license terms are signed you can initialize Terraform using terraform init and then can run terraform apply to bring up the infrastructure resources on azure. It may take up 5~10 minutes to fully provision the virtual machine.

After running it you may notice two new files being created. Both contain a link to the created virtual machine. .vm-ip contains the public IP address that was created and will be used to SSH into the machine. .vm-id is the Azure Resource ID of your virtual machine and is a unique identifier that we will use to start/stop the machine later. Both are plain text files and only contain one line, feel free to check them out. The machine is now up and running and you can work with it.

Bring code onto the VM 💁‍♀️

Before doing any work you might want to upload some bootstrap code onto the virtual machine — or you just want to run an existing jupyter notebook there. Again the Github repository holds a small script that will help you do this (works out of the box on Mac/Unix machines otherwise you need to install make and rsync first).

anoff/vm-automation

Place the Makefile into the working directory of your code and make sure to update the PATH definitions to the two files mentioned at the end of the last chapter containing the details of the newly created virtual machine. If you had the Terraform script in a subfolder named infra there is nothing to do. Otherwise you should either copy over the two files into such a directory or modify the PATH definition in the Makefile.

Use make syncup in your working directory (where you placed the Makefile) to sync your local directory content onto the remote machine. You can see the command that is being executed and what the remote directory will be named. In my case it is ~/work/AIND-RNN which is one of my Nanodegree projects. You can also see that the command automatically ignores all files that are defined in your .gitignore which means anything you do not want to version will also not be copied around. This is especially useful for artifacts created during neural net training processes.

Output of make syncup

Run Jupyter Notebook 📒

Let’s assume that your project also holds a Jupyter notebook you want to execute on the remote machine and access from your local browser. You could also use a similar process to execute ANY kind of script on the remote machine.

First you need to SSH into the machine using make ssh which will also do port forwarding for the Jupyter Port 8888 onto your local machine so you can open http://localhost:8888 in your local browser (my MacBook) and connect to the webserver that listens on this port on the virtual machine (Jupyter Notebook). Now that you have a shell running on the DSVM manipulate the file system, install missing packages via pip/conda or just start a process.

Starting jupyter notebook on the VM

The Jupyter notebook process started above is linked to the lifecycle of the interactive shell that we opened with the SSH connection. Closing the SSH connection will kill the Jupyter server as well. All your code should still be there as Jupyter regularly saves to disk but your python kernel will be gone and all the memory objects (state of notebook execution) will be lost. You will need to execute the notebook again from beginning after you SSH again into the machine and start Jupyter up.

Commit your changes 💾

After you did some changes and you want to git commit like a good developer you need to get those changes you did on the virtual machine to your local development environment. You can do this using make syncdown which will copy all changed remote files onto your local working directory — again only those under git version control.

_🚨_Make sure you exit the SSH connection first

Copy remote changes to local filesystem

The remote file LOOK_MOMMY_I_CHANGED_A_FILE has now been copied to my local working directory and I can use git commit -am "everyone uses meaningful commit messages right?" to commit my changes or use my local tooling to execute unit tests, check codestyle, add some comments…

Start and Stop the Virtual Machine 🎬 🛑

If you have not checked already, you should look up how much the Virtual Machine that you provisioned actually costs you. The Standard_NC6 (which is the cheapest GPU instance) will cost you a small holiday if you keep it running for a month. That is the reason why I wanted an easy way to stop it when I don’t need it and get it back up quickly if I want to continue working.

The Makefile comes with three commands for managing the state of the virtual machine itself. They all require the unique Azure Resource ID located in the .vm-id to select the correct VM in your Azure subscription:

make stop will stop stop the virtual machine AND deallocate the resources which will significantly reduce the costs as you only pay for the disks that hold your data.

make start tells Azure to allocate new resources and spawn up the virtual machine again

make status will tell you if the virtual machine is up or not

Virtual Machine start/status/stop

The screenshot shows you how long stopping and starting the VM might take. However as soon as you see the CLI saying Running \ you can shut down your local machine as Azure started deallocating your resources.

Reduce risk of bankruptcy 💸

If you are afraid of the bill that might come flying in if you miss stopping the virtual machine you should take a closer look at the auto shutdown features that Azure offers you. It lets you specify a time at which the VM will automatically shut down every day.

Virtual Machine Auto-Shutdown

But let me tell you from experience — if you accidentally keep it up for a weekend and see the bill the next week you will *always* shutdown from then on. That was actually one of the reasons why I wanted to make this workflow as easy as possible.

Summary 📚

Well I hope you liked this article and found some helpful tips. The general workflow can also be done with AWS machines but the Terraform template will look different. Feel free to submit a PR to my Repo and I will add the option to also use AWS resources.

anoff/vm-automation

I would love to hear feedback via Issues, Twitter 🐦 or comments. One thought I have is to bundle all the commands into a binary CLI so it works cross platform and can just be installed by copying around a single file. If you’re interested please let me know😻

Here is another look at all the commands you can use🧙‍♀️

Available commands

/andreas

How to create a streaming HTTP interface in Python?

Andreas Offenhaeuser — Fri, 16 Mar 2018 14:51:52 +0000

Hey folks,

lately I'm playing around a lot with artificial intelligence and Python. What I want to build now is a web interface for a python based ML model (keras etc). For full flexibility I want the app to a) take a stream of input data b) return a stream of output data. Each chunk in the input stream would be run through the ML model and the corresonding output streamed back as a chunk.

However I could not find any resources online that tell me how to SEND a stream response in a production grade python web framework. Anyone got experience how to send a HTTP stream response in python?

Currently I'm focusing on the combination of falcon + gunicorn to create a web app as Flask doesn't seem to be a production grade framework. Although my requirements are not production-grade I would love to figure out how to do this at such a level.

/andreas