DEV Community: Edoardo Tenani

Remove yourself from the critical path

Edoardo Tenani — Thu, 26 Jan 2023 21:14:33 +0000

During one of my recent mentoring sessions, an interesting discussion started with my mentee. How do you manage being a leader with being hands-on too?

If you have some experience in leading a group I suspect you faced the same issue at some point, maybe not with coding but for sure around the "I can't be one of the hands-on people anymore". I don't think this is necessarily true, but being hands-on and a great leader is a tough challenge and its practicality is heavily influenced by the environment around it.

During our session the main issue were interruptions and meetings driving away time from focused work.

I there a way out of this? I don't know, but in my experience you can help yourself. The goal in this situation is to create space for deep work, chunks of time in which you expect no interruptions and no meetings. In which you can focus and do deep work.

There are calendar blocking techniques that can be tried out, but that's like curing the symptoms and not the causes: if everybody needs you, you will never be able to get that focus time, because there will always be something requiring you attention.

We can now agree that improving on this situation is hard, requires dedication and a lot of learning. Especially if you are moving from an individual contributor role to a team leading role, as you need to adjust your thinking and approach; you are now the one everybody else count on.

My suggestion here would be: remove yourself from the critical path. Stop being the one everybody needs to go to.

What's the critical path?

Doing something, in a generic way, involves going from a Point A to a Point B. To move from A to B a series of steps are required. The critical path is that path from A to B that you cannot avoid taking; is composed of those steps that are mandatory to move from A to B.

Let's visualise this with some example. Let's say we are travelling from City A to City B. It doesn't matter which transport are we using, we will need to go through some physical places. And we hope there will be no accident, delay or impediment along our route, because that would slow us down. Or maybe we are taking a taxi, we hope there will be no slow downs as that will make the run more costly. Or maybe you need to study for an exam, so you will want to avoid getting sick, as that would reduce the time you have available for studying, making you risk to fail the exam.

The critical path is the series of steps that needs to be performed to achieve an objective. Any impediment in the critical path will make achieving the objective be more time consuming, more costly and/or not be ready for the deadline.

We can say that this critical path is the most important element you want to watch.

Am I in the critical path?

This is a worthy question! How do you identify if you are in the critical path?

Does any of this happen to you:

other come to you for answers that they need to do their work
only you have the knowledge needed by others
you get interrupted frequently (by your reports, your peers or your manager)
you need to take decisions and people depends on them, so they are blocked waiting for your decision

If yes you are in the critical path. And now what?

There are some strategies you may try to move away from it. Those strategies require time and trust as their main ingredients, as removing yourself from the critical path is all about delegation. You need to learn how to delegate (a skill that you generally don't need as individual contributor), and the more effective you will be at it, the more effective your work as a manager.

The hard part of delegation is that is not about doing something, is about not doing something. But that something has to happen anyway. Delegation is hard. Delegation involves trust. Trust is even harder.

So how do you build trust? Here are some techniques. In the next examples I'll be using teacher and student as "the one that knows how to perform a task right now" and "the one that should learn how to perform a task in the future".

Shadowing

Is there a task you do that someone else could do with some training? How do you deliver that training? Shadowing may be a solution. The student follows (or "shadows") the teacher while they perform the task. The goal is for the student to be independent in executing the task after a set period.

Other literature is available, but I want to give some hints:

define a deadline after which the shadowing period ends
apply to task that may benefit from observation (es interviewing, meetings) but do not apply to heavy concept base tasks where a lot of prior knowledge is required

Pairing

Do you have a knowledge-heavy task at hand that you would like to delegate but don't know how as it requires a lot of prior knowledge? In this case shadowing is probably not enough. Do pairing sessions, as in "code pairing" sessions, but for other topics.

During those sessions select a driver and a writer. The driver direct how to do things, the writer executes. Exchange roles after some time. This is a great way to get to know the other person and build trust, while sharing knowledge.

Proposal documents

Do you have a task that needs some thinking, where multiple solutions needs to be explored and there is no clear solution? Delegate writing a proposal document and review it together. Going through a document has many benefits: it allows asynchronous communications, will last for future people ("verba volant, scripta manent" Latins used to say) and will provide for time and space to have an insightful conversation.

Do remember to appoint a driver for the document, who will be in charge of moving from draft to the final decision. The driver does not have to be the one taking the decision too, on this subject there is a whole lot of literature, so go surfing!

Pitfalls

Delegation does not come without its risks. Watch out for things that you don't want to happen, some of which I summarise below.

Not all tasks are the right candidate for delegation

That task the CEO is really looking for and keeps you accountable for? Probably that's not the best task to delegate.

Not all tasks can be delegated, but fortunately not all tasks have the same importance (as impact * complexity * urgency). Knowing which task to delegate is one of the hidden skills of great leaders. I think that for being great at it you need to remember that half of it is your judgement but the other half is the environment around you.

Some tasks that may be good candidates are: any decision that can be reverted (from less expensive to most expensive to revert), any task outside of the critical path, any "nice to have".

Another judgement method you can apply here is what you are aiming for: are you aiming for more personal focus time? Are you aiming to making sure the critical path does not have any impediment? Are you trying to upskill someone? Keep in mind your goal when delegating, as it affects your decisions.

One framework to help here is the Eisenhower Matrix.

Don't setup people for failure

If you remove yourself from critical paths but is not safe to do so or is not appropriate given your team health and competence, you are setting up people for failure.

Failure brings pressure and frustration. Frustration leads to fear or anger. And we know how this ends, fear leads to the dark side.

Over time people will get disengaged, they may even get mentally or physically sick due to the excessive burden and in the end leave the group (or company). Don't be that leader.

Be okay with people failing!

Have you ever done something for the first time? Like riding a bike, singing, studying a subject? Where you good at it? I bet not that much.

When someone else will start doing something, in particular something you are used to do frequently, they will probably not be good at it. They are building their way of doing it, and the only way for this to happen and to make them grow is to allow them to fail in a safe space. Your goal is to build the safe space, not to prevent them from failing.

Be okay with having hard talks

People you delegate to will not do everything right. As you will be responsible for the outcome, you need to learn to have hard conversations.

You need to learn how to talk about issues, missteps and errors. My advise is to learn how to be emphatetic, respectful and transparent. We tend to associate negative feedback with something bad, but learning how to give negative feedback in a human and constructive way is a golden skill, it build trust.

If you think this is hard, consider the alternative: everything will be great until it won't be anymore, the number of wrong things will probably be too much and you will have way worst conversations to have.

Stop thinking about the effort, start thinking about the outcome

When people in your team feel safe and empowered they will do awesome things together. A team is more than the sum of its components. Your team will be more resilient, more effective. You will be more resilient and more effective.

As leaders, leading other people is a challenge but will give back many rewards if you're committed to improve and clearly show that you care for people.

This post first appeared on my blog, at https://endorama.dev/2023/remove-yourself-from-the-critical-path/

Golang divisions

Edoardo Tenani — Mon, 19 Sep 2022 19:27:07 +0000

Photo by Chris Liverani on Unsplash

At work today I read a piece of code in Golang that got me curious:

maxQuerySize := 500
// metricDataQueries is user provided
sliceLength := len(metricDataQueries)/maxQuerySize + 1
// what's the value of sliceLength?
for i := 0; i < sliceLength; i++ { 
    whatever
}

This is a simple integer division, but the result is then used as a parameter to the for loop.
Integer division has a drawback, that can create a bug in this case: the remainder, as float in for loop declarations are supported only if no truncation happens when converting to int (i.e. 10.0 works, 10.1 don't).

How does Go handle this? The answer is as simple as it gets: remainder is ignore and the result is the quotient. The reason why this happens is explained in Go specs, under Constant expressions:

Any other operation on untyped constants results in an untyped constant of the same kind; that is, a boolean, integer, floating-point, complex, or string constant. If the untyped operands of a binary operation (other than a shift) are of different kinds, the result is of the operand's kind that appears later in this list: integer, rune, floating-point, complex.

Which means that to have "true" division you need to convert at least one argument to float:

a := 10.0 / 2
b := 10 / 2.0
c := 10.0 / 2.0
d := 10 / 2
fmt.Println(reflect.TypeOf(a)) // float
fmt.Println(reflect.TypeOf(b)) // float
fmt.Println(reflect.TypeOf(c)) // float
fmt.Println(reflect.TypeOf(d)) // int

go.dev/play

Doing integer division effectively "floors" the result of the operation. What if we want the ceil of it?

Go math package has Ceil(x float64) float64 for this, which implies doing a "true" division and applying Ceil on the result.

All of this was not really surprising, but it's cool to see this being implemented in a very ergonomic way.

Just a note: if you want to get a float value you must convert to float division arguments before the division.

Do this: float64(2) / float64(10) // 0.2 and not this: a := float64(2 / 10) // 0.

asdf, little awesome tool

Edoardo Tenani — Wed, 17 Aug 2022 21:46:30 +0000

Some years ago I discovered a little tool with a huge potential. I started using it and since then it proved useful in so many ways it's now a mandatory tool in my toolset.

This tool is asdf. asdf is a CLI tool version manager. You may already be using multiple ones (rbenv, nvm, goenv to name a few) to manage versions of programming language tooling and accessories.

Having multiple tools for this purposes creates a series of disadvantages:

setting up your workstation is more painful (multiple tools to install, multiple versions to manage); this become more complex when you work in a team and want to provide a great onboarding experience
adopting a new tool force you to first search for a dedicated version manager, then install the tool (you may go without one, but that's going to bite your future self)
how to share the same version in all environments, in particular dev and CI? Was that file .go-version, .golang-version? Each tool uses it's own convention, sometimes they align but having a consistent experience is not in their DNA;
what's the command again? Each tool may have different CLI commands, making switching between them a pain; this may happen more easily if you're in a team;
remember the Good Old Times™ of “spaces VS tabs”? Well, why start a 🔥 war around which tool to use?

Let's summarise why asdf is ground breaking:

helps developer onboarding across a wide set of tool;
helps team collaborating;
removes useless discussion drive by preference;
has a consistent UI.

Would that be enough? NO 😜

It delivers more:

creating plugins is so easy you can just add whatever you want (I did it here, here, here and here);
creating plugins is most probably not needed because what you need is already available; asdf-community has 60 repositories (so around 60 plugins), just a asdf plugin-add away;
are we forgetting the CI? Hell no! Note that this only works if you are using GitHub Actions, but if that's the case there are actions ready to be used.

In case you don't want to use the install action provided by the asdf community because it does not leverages other optimized setup actions, I developed an action that parses the .tool-versions file extracting each tool version so they can then be reused in other actions. No asdf needed!

This action is available here.

If you reached this line, STOP right here and go install asdf.

URI, URL, URN?? Identifying resources on the web

Edoardo Tenani — Thu, 21 Jul 2022 08:14:24 +0000

Cover image by Shannon Potter on Unsplash

If you do web development you will, at some point, encounter 3 particular term: URI, URL and URN (this is not so familiar but you may have encountered ARNs in AWS).
You may also have seen URI and URL being used interchangeably, but it's important to note they are not the same thing even if they are used for very similar purposes: finding things and finding things on the internet.

Let's break down what do those acronyms mean:

URI stands for Uniform Resource Indicator
URL stands for Uniform Resource Locator
URN stands for Uniform Resource Name

URLs and URNs are specific classifications of URIs.

It happens that URIs are very different between each other (from rfc3986#section-1.1.2):

ftp://ftp.is.co.za/rfc/rfc1808.txt
http://www.ietf.org/rfc/rfc2396.txt
ldap://[2001:db8::7]/c=GB?objectClass?one
mailto:John.Doe@example.com
news:comp.infosystems.www.servers.unix
tel:+1-816-555-1212
telnet://192.0.2.16:80/
urn:oasis:names:specification:docbook:dtd:xml:4.1.2

Follow me in a deep dive into URIs, URLs and URNs and some good old RFC digging. Put on your safety 🥽, grab your ⛏ and let's go!!

What's a URI

A Uniform Resource Identifier is a generic way to uniquely identify any resource.

The complete definition is in RFC 3986, where you can hunt for all the details.

It takes the form of a string with this syntax:

URI = scheme ":" ["//" authority] path ["?" query] ["#" fragment]

There are 5 components:

scheme (required), arbitrary but there are popular ones like mailto, https, ftp or arn
authority (optional), for user information and top level namespace (usually a domain or IP address) with the syntax
authority = [ userinfo "@" ] host [ ":" port ]
- userinfo (optional); either a username or a username and password concatenated by : (username:password)
- host (required), can be a domain or an IPv4/IPV6
- port (optional)
path (required but can be empty - you know, parsers 🤷), a hierarchical structure separated by /
query (optional), it starts with ? and can contain ? and / (like path)
fragment (optional), it starts with # until the end of the URI

This is quite convoluted and the RFC is incredibly detailed. The Wikipedia page for URI helps!

It's important to understand the URI as it's the foundation on which URL and URN are based.

What's a URL

The Uniform Resource Locator is a string representation to for a resource available via the Internet.

It has it's own RFC, RFC 1738, where we find all the familiar names and strings we see as Web developers.

It defines some specific schemas we all know and love:

   ftp                     File Transfer protocol
   http                    Hypertext Transfer Protocol
   gopher                  The Gopher protocol
   mailto                  Electronic mail address
   news                    USENET news
   nntp                    USENET news using NNTP access
   telnet                  Reference to interactive sessions
   wais                    Wide Area Information Servers
   file                    Host-specific file names
   prospero                Prospero Directory Service

(wait, what is wais??? Think I'm too young for that!)

and it defines the usual "Internet" scheme syntax for all URLs schemes that involve usage of an IP-based protocol:

//<user>:<password>@<host>:<port>/<url-path>

I'm young enough that I basically only used ftp, http and mailto! (And well.. watching Star Wars over telnet 😆) Did you use some of the others? Let me know in the comment, I want to read your story!!

What's a URN

Quite simply, it's a URI with the urn scheme. URNs are location indipendent and persistent identifiers.

This means there is only 1 unique URN for a given resource in a given namespace forever (or until that resource doesn't exist any more).

URNs are defined by RFC 8141.

Their properties of being location indipendent and persistent makes them useful for some very interesting use cases, especially.

Their syntax definition (rfc8141#section-2) is quite more complex, here a simplified version:

URN = "urn" ":" NID ":" NSS [ "?+" r-component ] [ "?=" q-component ] [ "#" f-component ]

This is more easily akin to a URI with multiple components:

urn is the scheme
NID (required), the namespace identifier
NSS (required), the namespace specific string
r-component (optional), query parameters to pass to URL resolution services, note that it's used is discouraged: "Thus, r-components SHOULD NOT be used for URNs before their semantics have been standardized."
q-component (optional), query parameters for the named resource or the service supplying the named resource
f-component (optional), a fragment representing the location or region for the named resource, ignored during URN equivalence operations.

It should be noted that a public registry for URNs namespaces exists, and is maintained at IANA.

Does it mean you need to register a namespace before using it? No if you plan to use it internally, yes if you want it to be internet-global (like xmpp or uuid

So cool but where to use them?

AWS

If you have experience with Amazon Web Services you will have encountered ARNs: Amazon Resource Names. By their definition:

Amazon Resource Names (ARNs) uniquely identify AWS resources. We require an ARN when you need to specify a resource unambiguously across all of AWS, such as in IAM policies, Amazon Relational Database Service (Amazon RDS) tags, and API calls.

Sounds familiar? The format too is very URN-like (there are different formats, look at the docs!):

arn:partition:service:region:account-id:resource-type/resource-id

From the look of it it does not seem to be a RFC-compliant URN, but it's extremely similar.

GCP

Google Cloud Platform relies on URIs to identify resources on the platform.

(Resources names](https://cloud.google.com/apis/design/resource_names) are schema-less URIs similar to:

logging.googleapis.com/projects/myproject123/locations/global/buckets/my-bucket

logging.googleapis.com is the authority, the path the resource. Being the path hierarchical is possible to represent GCP resource structure this way (project -> collection -> resource).

Another at-scale example is LinkedIn:

URNs are used to represent foreign associations to an entity (persons, organizations, and so on) in an API. A URN is a string-based identifier with the format:
urn:{namespace}:{entityType}:{id}

Express foreign keys

Simple relational database design generally rely on (autoincrementing) int for rows IDs in tables. This system is effective and works in a single database scenario.

When scaling to multiple DBs or distributed applications (es microservices) using integers is not enough anymore. Some common problems are:

conflicting autoincrementing numbers: being auto incremental they are exposed to possible race conditions when creating records
too generic: the system (or it's operators) is not able to know only by looking at the ID what kind of resource that ID refers to. If you think is not that important, Atlassian recently blew up 883 customer's websites due to a similar confusion: a script included IDs for websites and not apps in the Atlassian backend ecosystem. Those IDs were then used for deletion, but the thing deleted wasn't, as expected, the customer app instance but their entire website.

Do you have any other examples of URNs being used in systems? I'm curious to know about them so please let me know in the comments!

Coordination does not scale

Edoardo Tenani — Tue, 02 Nov 2021 11:38:29 +0000

It's monday. You wake out and reach the office (which may very well be your sofa given the current times), turn on your laptop and open your email software.
One of them catches your attention; the title reads "New company wide Project Management tool".

A chill goes down your spine. "Make it be not Jira" is the first thought as you open the email.

Feels familiar? I guess everyone had a similar moment in their professional career.

I've had the pleasure to follow the rollout and administrate an Atlassian Jira Cloud instance for a couple of years for a small company, and I don't think I have fully recovered yet. The complexity hidden behind that subscription is massive.

After self-reflection, my question is: was it Jira's fault? Is the tool to blame for the complexity of the resulting process and the effort it required just to use it for day to day work? Would another tool change the outcome?

It seems it's the tool that matters, but it's not. It's the context around the tool that matters.

This is a perspective from an ex-DevOps working for a product company.
This post is not about Jira. Far from being perfect but you'll never be happy using a hammer when you need a screwdriver.

This post is about the fundamental wrong assumption we make when searching for a project management tool.

The first interesting question is: why do we need a project management tool?
The answer I've seen in the wild is that coordinating the work in progress becomes too difficult, and this complexity is blamed on the tooling being missing or not adequate (i.e. different or fragmented usage, missing items, lack of visibility or adoption).

The supposed benefit of a Project Management tool are:

coordination, defined as being able to make plans on top of other groups activities/plannings
consistency, defined as being able to move easily across groups/teams with a small overhead

But you are not going to achieve consistency, and once that promise is broken, good luck obtaining coordination. More interesting, coordination may not be what you need at all.

First, let's examine consistency. I've yet to see two teams (or people) use the same tool in the same way. It doesn't matter how far the teams are, even in the same company.
Most of the time, the usage is so different that they could be two different, not interoperable tools, and the complexity would be the same.
Consistency requires a massive effort in understanding, coordination, enforcement. Such effort does not scale and is so rigid that you risk being paralysed by it.

Do we really need consistency? No.

We strive for consistency because we expect it to drive better collaboration. And we struggle so much trying to be consistent that our work becomes keeping the consistency.

I would argue that once you reach proper consistency, the one-size-fits-all approach resulting from it would be a terrible product management take. (Remember it is a product company, not a consultancy).
The one-size-fits-all does not work because software development is knowledge work. It implies learning, continuously. It implies adapting.

We know that knowledge work requires "deep work", defined as long slots of uninterrupted focus work time.

The Deep Work Hypothesis.
Deep work is becoming increasingly valuable at the same time that it’s becoming increasingly rare. Therefore, if you cultivate this skill, you’ll thrive.
Cal Newport in Deep Work: Rules for Focused Success in a Distracted World

The recent remote work explosion made clear something that remote companies already knew: focus work drives productivity (and revenue).
If we already trust people to plan work and execute through long focused time slots, why don't we trust teams to do the same?

It's time to go back to the drawing board and review the question that brought us here. The question we should start from is "how do we deliver better software?".
We know some coordination is needed and we know focus is needed.
So why don't we focus on making coordination easier instead of making it consistent?

What does consistency even mean in a fast-paced context like software development? There is an industry focus on being agile to respond to changes, but coordination makes reacting to change more difficult and expensive.

What to do instead? Incidentally, we can take inspiration from the same field we are in. Software Engineers developed ways to make it easier for different code created in isolation to work once it is put all together.

We use testing, decoupling, interfaces and APIs. Those are our tools to build incredibly complex and interconnected systems.

Then why don't we apply the same principles to coordinating work? (Spoiler: hyperscalers do).
We can derive that it is not coordination that is important, but boundaries and understanding.

Boundaries: once I start thinking about what a dependency should do to fulfil a need I have, I stop worrying about my work and start worrying about someone else’s work. Boundaries drive Expectations: if I know I cannot directly influence something, I will need to find ways to integrate with more generalist approaches.
Understanding: no system can talk to another without mutual protocols that both systems understand. No teams can work together without mutual understanding and building shared processes.

Defining boundaries and understanding others can be only fulfilled through communication. Any communication protocol (in use by humans or machines) adds overhead to the conversation (why is the Lora protocol preferred over TCP/IP in IoT? Lower overhead)

Being a generalist and understating others requires communicating, which creates overhead, like any communication protocol.

Any form of collaboration has overhead. Your Project Management tool of choice will have overhead, probably hidden behind all the marketing.

But we have a North Star to drive us, we want to build better software.
We cannot answer only considering costs (collaboration overhead) we have to consider quality.
Boundaries and understanding drive quality.

You will adopt a Project Management tool. But do not use it for coordination, use it to empower teams within themselves.

Coordination is not the forcing function you want your employee to spend time on. Instead, engage teams to define boundaries and create space to understand each other.

In the end, we want better software for better products to better serve our customers in an ever changing landscape. Your silver bullets are empowered teams working towards the same goals.

Cover photo by Leon on Unsplash
Inline photo by Paul Skorupskas on Unsplash

Hey 👋 I'm Edoardo. I've been a DevOps and started digging into processes and what does it mean to improve software delivery within a company. Reading and listening creates opinion, and I'd like to discuss them with anyone interested! Join the conversation and let me know what you think in the comments!

How Stripe is actioning the osquery API at scale [osquery@scale]

Edoardo Tenani — Fri, 16 Oct 2020 16:02:53 +0000

OSquery is an operating system instrumentation framework for Windows, OS X (macOS), Linux, and FreeBSD. The tools make low-level operating system analytics and monitoring both performant and intuitive.
From osquery website

I recently watched this talk from osquery@scale.

My interest in OSquery started some years ago, as the premises are super cool: convert your system in a SQL database you can query with usual SQL syntax.

This integrates very well with standard data analytics tooling and process, so is a powerful way to gather data at scale for review, analysis and alerting.

Want to know which processes are executing on a machine? Or how many users there are? Any intrusion detection rule you want to check?
✔ OSquery can do this, and a lot more

As such I was really interested in how OSquery can is leveraged in the real world for security monitoring at scale.

The talk present how OSquery fits into a global scale security effort both at laptop and server level. Highly rewarding!

Presenter: https://www.linkedin.com/in/matthew-kemelhar-69058723

Challenge at scale: creating a security ecosystem where OSquery is a critical component

Every environment has different tools, sensors, logs and storage
most tools have their GUI and query language
OSquery usage of SQL relates easily to other business flow that have analytics value

But that needs to play together with other elements. OSquery is a piece of Security @ Stripe.

3 goals:

eliminate need to recreate detection logic for different tools
enable other teams who need information without access to security tools
centralize methodology for detection that everyone can activate without access to all machines and with reduced skill-set (less to learn = more effective)

OSquery on laptop managed using Chef, where logs are collected.
OSquery on servers managed by puppet, queries pushed through puppet and logs are stored in AWS S3 forwarded to Splunk.

Design criteria:

generalize security for our environment
extensible framework
detection logic agnostic from tooling
skills required: Python and SQL
easy to collect metrics

Taking a practice from Data Science, they started using Jupiter Notebooks, writing libraries that codified the logic. They have a centralized notebook server (sensitive information can be protected) and use GitHub repos (collaborative peer review).

OSquery is cool but get better when you can correlate other data.

The second part of the presentation is about how to detect meaningful events and reduce alert fatigue.

They started building a baseline (average size of CLI commands) to detect anomalies. Grab data in a timeframe and decorate them.
Once you have a baseline, detecting anomalies is easier.

They created a repo with classified attack stages (loosely based on the MITRE Attack framework) and extended to include custom detections. This allow to provide README as mini-runbooks for specific detection groups. (example rule in the video)

They generalized the metadata for detections (independently from the system). Each rule is automatically deployed to a detection engine, so all rules are centralized in a single repo.

They prefer JS-based rules (with fallback to SQL when is not possible). (example engine in the video).

KubeConEU 2020 - Day 2

Edoardo Tenani — Thu, 01 Oct 2020 13:37:47 +0000

The journey continues! Day 2 to 4 where packed with talks! I tried to create briefs from each one I followed.

From Day 2 onward the talks where early and late in the afternoon, leaving space for keynotes in the middle. With that many interesting talks, I have to admit I skipped all keynotes after the first one in Day 2, but I'm watching them on-demand (it was easier to interact with speakers timely after their talks, as there were no talk dedicated channels but topic dedicated channels, so following conversations long after the talk was complex at times).

Deliver Your Cloud Native Application with Design Pattern as Code

Talk link
Who is for: Ops, Backend
by @JunMakishi, @TAR_O_RIN

As organization embrace the Cloud and it's tools, managing fragmentation becomes more and more complex. The idea of applying design patterns to create composable and reusable infrastructure components is very cool. It also allows to abstract in a reusable way different cloud provider patterns.

They integrate pipelines in the equation, which is awesome. Flexibility of pipelines is one of the key advantages of being in the Cloud, leveraging that is very powerful.

It's not a one-size-fit-all approach, but their solution is inspiring. Based on Infrastructure as a Code and CueLang for configurations.

The talk explores a powerful concept: configuration as data >> configuration as code.

I agree!

Kelsey Hightower

@kelseyhightower

During the @ArrestedDevOps podcast, hosted by @bridgetkromhout, I was able to dive into what I refer to as infrastructure as data. Its the idea that configuration should be treated as data and leverage pipelines for manipulation and policy enforcement. arresteddevops.com

23:13 PM - 04 Dec 2019

Their solution does not take into account state management, that should be separately handled (they use Pulumi, previously they used Terraform).

Kubernetes Patterns

Talk link
Who is for: Backend
by @ro14nd

Companion book: Kubernetes Patterns (Free download https://k8spatterns.io/)

Nothing particularly new to me, I've had my chance to review and study Kubernetes patterns some time ago.

Worth checking out if you're a developer using K8s for app delivery. The book is definitely to be downloaded for study and future reference.

Progressive Delivery

Talk link
Who is for: Ops, Backend
by @vfarcic, @csanchez

Progressive delivery is a term to describe deployment techniques to avoid common all-or-nothing deployment pitfalls.

The goal is to shift traffic slowly from the old to the new version, being able to roll back immediately in case of unexpected behaviours of the new deployment.

Avoid downtime, limit the blast radius, ship code faster!

I was positively impressed by the visual explanations of techniques. The concepts were clear to me before the presentation, but have been explained so well it has been a pleasurable review.

Monitoring is the new testing

Today monitoring allow to know when users are experiencing issues and react automatically.

Demo time focus on Istio and flagger.app.

How to manage databases with progressive delivery?

Use Cloud hosted solutions
Make you app always backward compatible
Only non-breaking database schema changes.

Build an Automatic Canary Release Pipeline in a Kubernetes-native Way

Talk link
Who is for: Ops
by Ying Chun Guo

I was a bit mislead by the title of this talk. The focus is on Knative (a serverless platform on top of Kubernetes) and Tekton (a Kubernetes-native CI/CD system). Truly Kubernetes-native but very "locked in".

Seeing how the two systems play together has been interesting, but unless you are planning to adopt such technologies there is little knowledge that you can reuse in other environment.

Handling Container Vulnerabilities with Open Policy Agent

Talk link
Who is for: Ops, Security
by @knqyf263

This has been one of the most interesting talk I've followed.

Scanning for vulnerabilities does not scale. In 2019 there have been little less than 50 vulnerabilities per day.

How can the team keep up with that? By reducing noise and focusing on what matters to us.

Use Open Policy Agent (OPA) to define how to handle vulnerabilities discovered by a scanner. This allows to create policies to handle vulnerabilities based on internal knowledge and requirements.

Trivy has direct OPA integration.

trivy-enforcer is a custom K8s controller that allows enforcing policies on images to be deployed on K8s. Still experimental, looks awesome!

Architectural Caching Patterns for Kubernetes

Talk link
Who is for: Backend
by @RafalLeszko

Do you happen to need caching data in your services? Generally, is quite a common need.

The overview of caching pattern was quite complete but the talk felt centred around promoting the speaker company product more than giving a solid understanding of different caching patterns.

Still worth to watch if you want an overview of caching patterns or if you're interested in the product (Hazelcast).

Tracing is For Everyone: Tracing User Events with GraphQL and OpenTelemetry

Talk link
Who is for: Frontend
by @ninastawski

This talk is presented by a UI engineer! It's so interesting to see tracing applied to frontend because can really help understand where customers are having issues using your application.

Why only Backend and Ops should have the cool tools? OpenTelemetry.js

UI is much more than a "service". Users do lots of things in UI and with complex UI apps (microfrontends, complex journeys) tracing is powerful.

Collect actions in UI then send to backend and collect (in Zipkin). If you want to follow along the demo, a GitHub repo is provided.

By carefully tagging traces is possible to correlate frontend and backend traces to look at the entire user journey. COOL!

Five Great Ways to Lose Data on Kubernetes (And How to Avoid Them)

Talk link
Who is for: Ops
by @dbcicero

Database data resiliency is always an interesting topic for me. Nobody cares if the app is up and the code works if the database is broken.

The talk had cool tips, and some very nice troubleshooting/discovery commands. Highly recommended!

It starts with a clear explanation on Kubernetes related risks for stateful applications and mitigations.

The main mentioned topics have been: replicas, affinity, distance, ephemeral storage and operator error.

Replicas

Replica is one of the ways to avoid node failure.

How does Replicas cope with blast radius (how fare away you need to be to not be affected by a failure)?

Affinity

The process of requesting different replica to be nearby or distanced. Kubernetes supports Pod affinity natively.

Distance

Protect replicas using distance (affinity/anti-affinity).

Take backup in object storage (it's easy and durable).

How does distance cope with blast radius? How to protect across regions or K8s clusters? It's complicated: it's not just a data problem, is a resource problem too (think load, DNS). (This part was out of scope for the talk, unfortunately!)

The persistent volume that doesn't

In Kubernetes ephemeral storage is a feature. Unless you want it to be persistent 😅

A pod cannot understand if storage is ephemeral or not.

Persistent Volumes are not enough if you don't use them! When you expect your Pod to use persistent storage, verify it writes data to be persisted to the proper mount point, otherwise ephemeral storage is used.

Fat fingers of fate

The saddest one. When an operator deletes applications. In Kubernetes deleting workloads is very easy. Once an application is deleted, dynamically provisioned persistent volumes (like the ones created by cloud provider integrations) are deleted by default.

But fear no more, you can avoid that changing the Reclaim Policy to Retain. You can achieve the same by creating a new StorageClass with Retain policy and use that.

Escaping the Jungle - Migration to Cloud Native CI/CD

Talk link
Who is for: Ops, Backend
by @antweiss

When a company starts it's journey, there is a blank slate. Then, after a while, you get in the jungle, where everything is in your stack and there is tech debt everywhere.

It's time to find the Golden Path.

Golden Path is not "the only" path, but it makes easier for the organization to work together. It's the supported path, out of it, you're on your own.

Meanwhile, you may want to build Cloud Native apps. And what about CI/CD? What is a Cloud Native CI/CD?

"Avoid e2e test at all, they are very brittle. Adopt Contract Testing instead."

When building the Golden Path, build the map of where we are going and embed it in the pipelines.

This one has been one of the most interesting for me, as surfing the amount of different tech stacks in a company can be incredibly daunting and I can see the Golden Path value proposition.

That's the end for Day 2! Any talk you think is missing? Let me know in the comments!

A KubeConEU 2020 story

Edoardo Tenani — Thu, 17 Sep 2020 16:34:50 +0000

KubeCon Europe 2020 has been a massive virtual event this year. The price was incredibly low (no travel, no accomodation, lower ticket cost) so it was very accessible.

18k+ attendees gathered online to follow the massive schedule available!

I attended all 4 days of the conference, here is my retrospective.

The quality of content and the number of talks was impressive. My schedule had around 60 different talks I were interested in 😵 I couldn't attend all, but I'll do my best to watch the recording.

A word of mention to the staff, the platform was good, the Slack workspace was massive and very active and the overall experience has been very nice for such a massive event. All talks were registered with a final QA live session, then the discussion continue in the thematic Slack channels, one for each content track.

This year I dedicated my time on hands-on tracks (like AppDev, Security, Observability, Networking, ...) but there was a huge spot for meeting maintainers and sponsors where you could actively engage with the community. I guess I'll dedicate more time on that the next year!

I've added Youtube embed for videos of talks I watched but you can find all videos in KubeCon + CloudNativeCon Europe 2020 - Virtual YouTube playlist. Videos don't have the questions and answers section included, but if you navigate to the talk page and look for Closed Caption Transcript you should be able to read them.

Let's dive in!

Day 1

The first day has been introductory. I skimmed around lighting talks and some tutorials.

The talk that hit home was "KubeCon + CloudNativeCon 101: A Beginner’s Guide to The Conference" by Karen Chu & Michelle Noorali. Is your first KubeCon? Fear no more, as this talk can guide you around giving you a warm welcome in the KubeCon community.

The journey will continue in the next post from this series!

Did you attend KubeCon EU 2020? If yes share your thoughts in the comments! I'd love to know.

Contributing back to Ansible — flexible secrets with some Sops

Edoardo Tenani — Wed, 11 Dec 2019 08:26:43 +0000

This blog post has been published originally on Arduino blog: https://blog.arduino.cc/2019/12/04/contributing-back-to-ansible-flexible-secrets-with-some-sops/

This post is from Edoardo Tenani, DevOps Engineer at Arduino.

In this blog, we’re going to answer: How does one store sensitive data in source code (in this case, Ansible playbooks) securely and in a way that the secrets can be easily shared with the rest of the team?

Ansible is an open source community project sponsored by Red Hat, it’s the simplest way to automate IT. Ansible is the only automation language that can be used across entire IT teams from systems and network administrators to developers and managers.

At Arduino, we started using Ansible around the beginning of 2018 and since then, most of our infrastructure has been provisioned via Ansible playbooks: from the frontend servers hosting our websites and applications (such as Create Web Editor), to the MQTT broker at the heart of Arduino IoT Cloud.

As soon as we started adopting it, we faced one of the most common security problems in software: How does one store sensitive data in source code (in this case, Ansible playbooks) securely and in a way that the secrets can be easily shared with the rest of the team?

Ansible configuration system comes to the rescue here with its built-in mechanism for handling secrets called Ansible Vault, but unfortunately it had some shortcomings for our use case.

The main disadvantage is that Vault is tied to Ansible system itself: In order to use it, you have to install the whole Ansible stack. We preferred a more self-contained solution, possibly compiled in a single binary to ease portability (i.e. inside Docker containers).

The second blocker is the “single passphrase” Ansible Vault relies on: a shared password to decrypt the entire vault. This solution is very handy and simple to use for personal projects or when the team is small, but as we are constantly growing as a company we preferred to rely on a more robust and scalable encryption strategy. Having the ability to encrypt different secrets with different keys, while being able to revoke access to specific users or machines at any time was crucial to us.

The first solution we identified has been Hashicorp Vault, a backend service purposely created for storing secrets and sensitive data with advanced encryption policies and access management capabilities. In our case, as the team was still growing, the operational cost of maintaining our Vault cluster was considered too high (deploying a High Available service that acts as a single point of failure for your operations is something we want to handle properly and with due care).

Around that same time, while reading industry’s best practices and looking for something that could help us managing secrets in source code, we came across mozilla/sops, a simple command line tool that allows strings and files to be encrypted using a combination of AWS KMS keys, GCP KMS keys or GPG keys.

Around that same time, while reading industry’s best practices and looking for something that could help us managing secrets in source code, we came across mozilla/sops, a simple command line tool that allows strings and files to be encrypted using a combination of AWS KMS keys, GCP KMS keys or GPG keys.

Sops seemed to have all the requirements we were looking for to replace Ansible Vault:

A single binary, thanks to the porting from Python to Golang that Mozilla recently did.
Able to encrypt and decrypt both entire files and single values.
Allow us to use identities coming from AWS KMS, identities that we already used for our web services and where our operations team had access credentials.
A fallback to GPG keys to mitigate the AWS lock-in, allowing us to decrypt our secrets even in the case of AWS KMS disruption.
The same low operational cost.

Sops’ adoption was a great success: The security team was happy and the implementation straightforward, with just one problem. When we tried to use Sops in Ansible configuration system, we immediately noticed what a pain it was to encrypt variables.

We tried to encrypt/decrypt single values using a helper script to properly pass them as extra variables to ansible-playbook. It almost worked, but developers and operations were not satisfied: It led to errors during development and deployments and overall felt clumsy and difficult.

Next we tried to encrypt/decrypt entire files. The helper script was still needed, but the overall complexity decreased. The main downside was that we needed to decrypt all the files prior to running ansible-playbook because Ansible system didn’t have any clue about what was going on: those were basically plain ansible var_files. It was an improvement, but still lacking the smooth developer experience we wanted.

As Ansible configuration system already supports encrypted vars and storing entire files in Ansible Vault, the obvious choice was to identify how to replicate the behaviour using Sops as the encryption/decryption engine.

Following an idea behind a feature request first opened upstream in the Ansible repository back in 2018 (Integration with Mozilla SOPS for encrypted vars), we developed a lookup plugin and a vars plugin that seamlessly integrate Ansible configuration system and Sops.

No more helper scripts needed

Just ensure Sops executable is installed, correct credentials are in place (ie. AWS credentials or GPG private key) and run ansible-playbook as you normally would.

We believe contributing to a tool we use and love is fundamental in following the Arduino philosophy of spreading the love for open source.

Our sops plugins are currently under review in the ansible/ansible GitHub repository: Add sops lookup plugin and Add sops vars plugin.

You can test it out right away by downloading the plugin files from the PRs and adding them in your local Ansible controller installation. You will then be able to use both plugins from your playbooks. Documentation is available, as for all Ansible plugins, in the code itself at the beginning of the file; search for DOCUMENTATION if you missed it.

If you can leave a comment or a GitHub reaction on the PR, that would be really helpful to expedite the review process.

What to do from now on?

If you’re a developer you can have a look at Sops’ issues list and contribute back to the project!

The Sops team is constantly adding new features (like a new command for publishing encrypted secrets in latest 3.4.0 release, or Azure Key Vault support) but surely there are interesting issues to tackle. For example, the Kubernetes Secret integration being discussed in issue 401 or the –verify command discussed in issue 437.

Made with <3 by the Arduino operations team!

Ansible® is a registered trademark of Red Hat, Inc. in the United States and other countries.

My 2019 Hacktoberfest

Edoardo Tenani — Mon, 11 Nov 2019 19:18:10 +0000

This is my third year participating in the Hacktoberfest challenge!

2017 has been terrible, I've not been able to submit any Pull Requests.

2018 was way better, mainly due to me learning Golang that created a whole lot of new opportunities for contribution.

This year I started gathering interesting issues and projects since summer, so that I would have a pool of issues to choose from during Hacktoberfest.

It all started with finding a bug in Ansible while at work. 😅

ACMEAccount.get_request check status code value lower boundary #63140

endorama commented on Oct 04, 2019

SUMMARY

Any HTTP code below 200 cannot be considered a success, should be handled like a failure instead.

This is particularly true for below zero status codes.

This change is required to avoid ACMEAccount.get_request to consider any HTTP status code below 400 to be a success. In #63139 status -1 is returned due to a .netrc file permission error, but the function does not report it as an error.

Fixes #63139

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

module_utils/acme

ADDITIONAL INFORMATION

View on GitHub

Second one has been found by share luck! Following the Arduino main repository, I saw a weird issue:

Bug in arduino-linux-setup.sh affecting non-bash users #9281

jwcollins commented on Oct 04, 2019

Hi. This script has an obvious, but easily fixed bug. Currently there are a bunch of comments before #!/bin/bash. And it assumes, very incorrectly, that pound-bang /bin/bash is a comment. In fact, "#!" is a valid Unix/Linux magic number, and "/bin/bash" is the path to the program which needs to interpret / execute the script. When run by a user who uses /bin/bash as their everyday shell, it works, but only because the user happens to have invoked bash. But when run by a user who uses another shell (e.g. /bin/tcsh) as their shell, it doesn't work, because pound-bang is not at the top of the file, and thus /bin/bash is not invoked to execute the script. And since the script is written to bash, as opposed to tcsh (or ksh or ... ) syntax, it won't work.

This bug report is against distribution 1.8.10. Screen capture of trying to run this script from a non-bash shell is pasted below. Read the below wikipedia link, which mis-pronounces "pound-bang", but is otherwise valid. This construct has worked since handed down by Real Programmer Dennis Ritchie in Jan 1980. Please learn it.

https://en.wikipedia.org/wiki/Shebang_(Unix)

View on GitHub

An easy one, but fun: the shebang mentioned has been in the wrong place probably since the file has been added to the repo, on May 2, 2018. The fun part is that the file was coming from another repository, where was dated 2017. And as the copyright suggested, it may have been created way before, in 2015.
Which actually means that script has been broken for 4 years and was working by share luck: in 4 year either no one tried to run that script on a shell different than bash or no one bothered reporting it (this would make me sad!).

This as a reminder that broken code can live a long life in any codebase. Even very simple things may have hidden edge cases no one just happened to hit before.
A project that I followed closely during the last year has been sops. I really like the approach and available functionalities and I wanted to contribute to it.

I spent some time digging in sops open issues and contributed to the filestatus command:

add filestatus command #545

endorama commented on Oct 08, 2019

Closes #460

Add filestatus command, reporting if the file is in encrypted or unencrypted state.

I reused ensureNoMetadata logic, thus the command would return 0 when the file is not encrypted and 203 when the file is encrypted (respecting the codes.FileAlreadyEncrypted error code)

It does not output the error message as returned by ensureNoMetadata, preferring a simpler output: "File is encrypted" or "File is unencrypted".

As I'm not so fond on sops internals, I'm sure there are multiple things to review, I'll gladly update the code to reflect any suggestion.

View on GitHub

Last one, is probably my favourite: a simple one indeed, but (I hope) with a future. This year I got a new laptop, and decided to move from my trusted and loved Ubuntu 16.04 installation to something newer.

I've been following Elementary OS since 0.3, tried 0.4 but were not able to use it successfully for work.

But this was the year, Elementary OS 0.5 Juno felt ready for my daily workloads, and is my stable OS since a couple of months now.

I can say I'm pretty in love with it. It has some small glitches (is OSS and donation based so ❤) but the overall experience it awesome. Is snappy, easy to use and has a polished touch that really make it shine. Kudos to the development team, they did an incredible work!

There is one issue I'm struggling with: being a not native English speaker, I switch keyboard layout a lot, between italian (my main writing language) and English (my main coding language).

It always interested me to contribute to an Operative system, but I have never been able to (due to my inexperience and web specific knowledge).

Contributing to an Operating System is daunting, I don't know where to start and don't know most of the tools (from the programming language to how a Desktop Environment works, I'm a newbie!). Thus starting small was my best choice. A simple task that required reading a lot of documentation and a bit of understanding how apps work on Elementary:

add AppData configurations #124

endorama commented on Oct 24, 2019

Hello! I would love to contribute to ElementaryOS, but I'm new to Vala, Meson, Ninja so I'm starting from simple stuff!

This PR provide the AppData configuration file as requested in #111

Thank you!

View on GitHub

This is a starting point, my final goal is:

Different keyboard layouts for individual windows #180

liminspace commented on Feb 17, 2018

Hi there! It's very important to have different layouts for each windows if you use more than one layout. But it is impossible in Elementary OS. Is there some solution how to fix it? Thanks.

ps: Elementary OS can be the first candidate to move from Ubuntu with Unity, but it must have at least base functionality like above.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

</div>
<div class="gh-btn-container"><a class="gh-btn" href="https://github.com/elementary/gala/issues/180">View on GitHub</a></div>

In a couple hours I've been able to compile and replace the Window Manager from source code (😎). A pretty happy and motivating start!

This was my 2019 Hacktoberfest. What about you? Did you contribute?

This year Hacktoberfest is over, but Open Source Software is waiting for your contribution: it may be small, it may sound simple or daunting but is everyone's job to keep that software in good shape. Everyone starts small.

Go Make the world better!