DEV Community: salaboy

KubeCon & Spring One 2020: From Monoliths to K8s

salaboy — Thu, 24 Sep 2020 09:29:09 +0000

Last month was quite a rollercoaster, I’ve presented at KubeCon EU (Virtual) and Spring One (Virtual). Both conferences were amazing and even that the title of the presentation was different the code examples and material used can be found in the following repository: https://github.com/salaboy/from-monolith-to-k8s

Both conferences were focused on Kubernetes, of course targeting different audiences, which is constant validation of where the industry is going. You can find all the recordings and links at the end of this blog post.

KubeCon Session

KubeCon is the largest conference where I’ve presented so far and it was a pleasure this time to present with Tracy Miranda (Executive Director of the CD Foundation). We focused the session on how Jenkins X and other projects likeZeebe are helping teams to accelerate while they are going to the Cloud.

The presentation heavily relied on the Accelerate book which describes all the practices that can boost your team’s productivity while building complex and distributed applications.

We got amazing feedback from the people joining the live stream and further discussions were started from there that made me believe that I need to keep working and expanding on the working examples for people to use as a reference.

Spring One Session

This session was more focused on the Spring Ecosystem and the tools that Java Developers are using to build, package, and deploy their Cloud Native Applications on top of Kubernetes.

It was really interesting to see how Pivotal/VMWare are going all in towards Kubernetes with their new iteration of Cloud Foundry for Kubernetes.

This presentation included examples inside the Demo Application on how to use Spring Cloud Gateway, Consumer-Driven Contract Testing with Spring Cloud Contracts and how to add fallbacks and circuit breakers with Resielience4J.

The goal for all these examples, is not to show how these projects work but how they were architected following Cloud Native patterns and Best Practices.

The feedback and the number of live attendees were amazing (almost 1200 online viewers while I was doing the presentation) so I am looking forward to participating in this amazing conference in the future.

Links and References

Check out the recordings and if you have any questions, comments, feedback, or if you want to help me to make the examples better, feel free to drop me a comment here or a DM in twitter @salaboy

https://springone.io/2020/sessions/from-java-monoliths-to-k8s

KubeCon 2020 EU Virtual: How we migrated our monolith to K8s from Mauricio (Salaboy) Salatino

Jenkins X on Linode LKE Webinar

salaboy — Mon, 17 Aug 2020 07:27:26 +0000

A couple of weeks ago, I did a webinar for Linode and their Managed Kubernetes Engine (as part of LearnK8s) where I run Jenkins X and demonstrated the project capabilities. Here you can find my slides and the links to watch the webinar on-demand.

Jenkins X on Linode LKE from Mauricio (Salaboy) Salatino

Here you can also find a tutorial on how to get started with Jenkins X on Linode, which were the preparation steps that I followed to get Jenkins X ready for my webinar.

https://www.linode.com/docs/kubernetes/how-to-deploy-jenkins-x-in-linode-kubernetes-engine/

Finally, here is the link to watch the webinar recording on-demand.

https://event.on24.com/eventRegistration/EventLobbyServlet?target=reg20.jsp&partnerref=linkedinsocial&utm_content=Oktopost-linkedin&utm_medium=social&utm_source=linkedin&eventid=2455137&sessionid=1&key=505A2E08A3D786778378EDFBF77C0007&regTag=1210267&sourcepage=register

Make sure you watch the KubeFlow webinar from my friend Salman which you can also choose from the previous link.

Stay tuned and join me tomorrow at @Kubecon where I will be showing more of Jenkins X and Zeebe to provide business visibility and microservices orchestration.

JNation 2020: From Monolith to K8s

salaboy — Tue, 02 Jun 2020 10:40:50 +0000

I was really happy to participate in this online edition of JNation. I would have preferred to be presenting in Portugal, but the quality of other speakers and the content that was presented at the conference was amazing.

During my presentation, I wanted to share pointers to tools and lessons learned while going from a big Java monolith to a more distributed approach. I touch quickly on two main concepts from DDD (Domain Driven Design) Bounded Contexts and Context Maps, without trying to define them too deeply but more to show some practical considerations derived from them while implementing real-life solutions. The presentation is heavily oriented to share insights on what to look at in certain situations, more than to dictate solutions.

The conference is still live here: https://2020.jnation.pt/schedule/

Slides & Video

https://www.slideshare.net/salaboy/jnation-2020-from-monolithto-k8s?ref=https://salaboy.com/

Video Link to be added when available

Links

Workshop Style Guide (work in progress, contributions are welcome): https://github.com/salaboy/from-monolith-to-k8s
Repositories: https://github.com/salaboy?tab=repositories&q=fmtok8s
Jenkins X: http://jenkins-x.io
Zeebe: http://zeebe.io
There were some questions about Saga Patterns, here are some links about what’s coming for Zeebe:
- https://blog.bernd-ruecker.com/orchestrate-aws-lambda-using-camunda-cloud-8d27dc640f69
- What to expect being supported in Zeebe, already working in Camunda BPM: https://blog.bernd-ruecker.com/saga-how-to-implement-complex-business-transactions-without-two-phase-commit-e00aa41a1b1b

Congrats to all the organisers for such an amazing experience!

I look forward to the 2021 edition of JNation.

Why isn’t Jenkins X’s future more Open?

salaboy — Tue, 19 May 2020 18:34:22 +0000

or as an alternative title: “How to kill a community project 101”

It is quite common to see companies behind Open Source projects fighting to provide a sustainable business model that builds up on top of these community projects. I know because I worked as a core engineer and also contributed to projects like jBPM, Drools, Activiti, Spring Cloud, Jenkins X, JHipster, Zeebe, etc. and I’ve seen these situations arise from the inside.

Finding the right commercial and Open Source balance isn’t always straightforward and it isn’t painless either. You might have heard of the stories of community engineers and teams arguing with “the company’s upper management“ to keep working on their open source project with their already open-source practices while the product is being built. Particularly after the team spent so much effort to nurture the right culture and collaboration with the community.

More recently I’ve noticed the Jenkins X community project going down the same path https://jenkins-x.io/blog/2020/05/15/helm3/

I decided to try to summarize the downsides of approaching building a product on top of an open-source project by killing its community.

A bit of background

I’ve been involved with the Jenkins X initiative since the early beginning. Back in 2018, I delivered a workshop at JBCNConf about deploying distributed services using Jenkins X (https://www.jbcnconf.com/2018/infoTalk.html?id=24 / https://docs.google.com/document/d/1kfSYhIuPa249poM8quyiazolOyn-feWPbVHyHmy5pD8/edit#heading=h.h38mzmjnh0xo) and since then I’ve kept spreading the word and advocating its benefits.

During this time the project owners and maintainers were always active in Slack and they kept an open and transparent way of running the project. What happened from 2018 until Jan of 2020 was remarkable, as the project grew a strong community of over 2500+ members in the Slack channel. Most of them were engaged in constructive discussions.

In 2019 I was nominated as a temporary Steering Committee Member for the Jenkins X project https://github.com/jenkins-x/jx-community/tree/master/committee-steering . I was honored to be awarded such a position. In 2019, the Continuous Delivery Foundation was created and Jenkins X joined as a founding member.

Jenkins X is unlike any other project that I worked on in the past.

Most of the project complexity comes from downstream projects that are glued together. The speed of changes in these projects is unprecedented and keeping up with all of it is a big challenge. Because of this, Jenkins X is more than a solution to a CI/CD problem, it is a shared community learning experience and a research project that pushes the boundaries of best practices for CI/CD in Kubernetes.

In 2019, there was a lot of pressure to stabilize dependencies. The Jenkins X team and the community really came together to find solutions to continuous integration and continuous delivery challenges in a cloud-native way (the actual purpose of Jenkins X). Also in 2019, the Jenkins X core team expanded thanks to CloudBees (the company behind the project) which hired a lot of people working in the community.

However, towards the end of 2019, the core engineering team began shifting its work to more private initiatives such as the SaaS offering of Jenkins X. As you would expect, the open-source project saw a slowdown.

I thought “that’s normal and there is nothing to worry about.” The company behind the project has to make a profit at the end of the day. CloudBees also announced that the project was to take a more defined route and not all decisions would be taken in the open, including the new work on the innovation side (https://jenkins-x.io/blog/2020/05/15/helm3/#why-isnt-this-all-more-open). It’s a tradeoff, of course. Cloudbees employees have more to say on the project and development might go faster in the short term, but you could lose the community engagement and contribution. What’s important is how you strike the balance.

What we lose

When a popular Open Source project like Jenkins X decides to stop innovating in the open, you can expect a less engaged community and fewer core engineers looking after the project. The project might also lose:

The strong sense of community
The trust between core and community members
The sustained pace and quality of work
The drive to work together and solve challenges
The Open Source way of working is no longer applied. Building, maintaining and growing a healthy community around it is hard, if not impossible
The project loses its competitive edge
The company’s appeal to new Open Source clients/users
A transparent roadmap

Imagine if Linus Torvalds, or even a multinational company like Red Hat would decide to roll out their own version of Linux in private: Linux++. The next thing you’ll see is thousand of upset contributors looking for alternatives and customers wondering if it is time to move on.

I always wonder, “If you ask an Open Source project to stop working in a way that made your project popular, do you actually understand how Open Source works?”

What can be done about this?

I’ve learned so many things from these last 2 years involved with the project that I think it is worth the time and effort to express my discomfort with their decision. Knowing personally the engineers behind the project, their careers, and their way of working, this is clearly a company’s decision to stop Open Source innovation in favor of product development. If you are part of the Jenkins X community, feel free to express your opinions about these changes in priorities and approaches and comments.

I’ve just created the following GitHub issue https://github.com/jenkins-x/jx/issues/7207 for you to upvote if you agree with the points covered in this blog post. As a community, we should express our concerns, so the company behind the project can see that doing things in the open matters.

If you want to get in touch, drop me a message via twitter or drop me a comment here.

Orchestrating Cloud Events with Zeebe

salaboy — Mon, 18 May 2020 07:58:55 +0000

Disclaimer: This blog post is about Cloud-Native software, containers, Cloud Events, and Workflows. It describes a concrete example that you can run yourself using Kubernetes, Helm, and Zeebe.io. You should be familiar with Kubernetes and Helm to follow along and will learn about zeebe.io and cloudevents.io on your way.

While working with Kubernetes the chances are quite high that you’ll find services written in different languages and using different technologies stacks. CloudEvents (cloudevents.io / CNCF spec) was born to enable these systems to exchange information by describing their events in a standard way, no matter which transports these services are using (HTTP, Messaging AMPQ/JMS, Protobuf, etc).

In such scenarios, where you have events being produced and consumed by different systems, there are common requirements that start to arise when the system grows larger, such as:

Visibility : being able to see and understand event interactions from a business perspective.
Coordination : sometimes you just need to define and check that a certain sequence of events happened or are produced to meet certain goals.
Error Handling : mechanisms to deal with problems if things are not going as expected, to make sure that we correct these situations and, from a business perspective, we keep track and continuously improve how we operate.
Compliance : enforcing certain rules and orders between these events to make sure that regulations are being met.

To tackle such requirements without reinventing the wheel you can use a Workflow Engine which provides very flexible solutions for these common sets of requirements. In this blog post, I cover Zeebe, a Cloud Native Workflow Engine, and how you can consume and emit Cloud Events from a Workflow.

Example

Let’s imagine that we have two services to deal with customers wanting to buy tickets for music concerts on our website. Most of these systems are required to handle high loads when a new concert goes online, as fans might want to buy all the tickets in a short period of time.

The website and customer flow look like this:

In the backend, there will be a service in charge of queuing customers that want to buy tickets and a payment service in charge of processing the payments. Due to the high demand, timeouts need to be in place to make sure that customers that are not willing to buy the tickets get removed from the queue if they spend too much time deciding.

These two services have been designed with an Event-Driven Architecture in mind, so they emit events and react to events. The following events are consumed and emitted by the TicketsService :

Tickets.CustomerQueueJoined: A customer is interested in buying a ticket so they have joined the queue to buy tickets
Tickets.Requested** :** A customer has selected one or more tickets for the concert and is ready to checkout.
Tickets.CheckedOut: A checkout action was confirmed by the customer.
Tickets.Emitted: the tickets have been emitted to the customer.
Notifications.Requested** :** A notification was sent to the customer. In our example, this can happen due to time outs while checking out the tickets or due time out from the payment service while authorizing the payment.

On the other hand, the PaymentService can react and emit the following events:

Payments.RequestReceived: A payment request has been received and it is queued to be processed.
Payments.Authorized: The payment was successfully processed and authorized.

By looking at these events you can quickly realize that the interactions between these services and the Customer on the website are asynchronous. Events are emitted and some other service will react and process them, probably emitting another event as a result.

In the following sections, an Orchestration approach is introduced without changing our service, and we can tackle the points mentioned in the introduction ( Visibility, Coordination, Error Handling, Compliance ) .

The Orchestration Approach

Workflow Engines were designed to orchestrate services and people together providing visibility and audit logs for how the work is being done. These engines work by automating (running) models that represent these service-to-service and service-to-people interactions. These models provide a graphical representation that can be understood not only by technical people but by every person inside the company. Workflow Engines help us to make sure that the services being orchestrated don’t need to know about each other, which makes our services decoupled and independent.

If we add CloudEvents to the picture, the workflow engine will emit and consume cloud events itself and can then enforce certain rules about their sequence. A workflow engine will also provide out-of-the-box visibility to all these interactions and automatic data aggregation to understand how our systems are working together from a business perspective.

A well understood and industry-wide used modeling language is BPMN by the OMG. A workflow model in BPMN will look like this:

This diagram is not imposing any runtime constraints about where the Services should be or how to interact with them, it is only defining what the sequence of these events should be and, for example, when to emit certain events. Both the Tickets Service and the Payments Service can be running in the Cloud, On-Prem, with or without containers. Zeebe can orchestrate these events to make sure that ticket purchase events are tracked correctly.

It is important to recognize the flexibility and features provided by BPMN in this very simple example. Out-of-the-box with BPMN we can use Timer Boundary events to define time outs for receiving Cloud Events as well as do Error Handling in case things go wrong. BPMN can do much more than what is shown in this example, for example, split the flow based on conditions, do tasks in parallel, more complex error handling, etc. You can find more about BPMN here.

You can find the BPMN models used in this blog post and some other examples here: https://github.com/salaboy/zeebe-cloud-events-examples

Architecture

By using a Cloud-Native Workflow Engine like Zeebe, we can define this flow of events with a simple BPMN Diagram that allows us to express quite complicated scenarios in a graphical way.

Zeebe, the workflow orchestrator, is pretty flexible in supporting different deployment scenarios. It can be installed into your local environment using Docker Compose, run on Kubernetes using Helm, or you can leverage Camunda Cloud which gives you an installation-free and managed Zeebe Cluster instance.

To enable Zeebe to consume and emit Cloud Events, you will need the Zeebe Cloud Events Router, which can map BPMN Events and Commands to Cloud Events which are understood by the existing services. This is a bidirectional router, as it is in charge of translating events that are generated outside the orchestrator scope (outside Zeebe) and emitting events that are meant for Domain-Specific Services.

Cloud Events could be transported via various protocols, e.g. Kafka, WebSockets or HTTP. This blog post focuses on HTTP, in future posts you will see how these HTTP Cloud Event endpoints can be integrated with services that use other transports.

The Zeebe Cloud Events Router exposes a set of HTTP endpoints to receive external Cloud Events:

/ POST -> receive a Cloud Event that will be mapped to a Job Completion, which will make the task to move forward.
/jobs GET -> Get a list of Pending Jobs to be completed
/message POST -> Receive a Cloud Event that will be mapped to a BPMN Message
/workflow POST -> Receive a Cloud Event that will be mapped to Start a Workflow Instance
/workflows POST -> Create a mapping between a Cloud Event Type and a Workflow Key, so we can start a new Workflow Instance with a Cloud Event
/workflows GET -> Get all available Cloud Events to Workflow Mappings

You can always access the Open API UI in the Router by pointing your browser to http://:/swagger-ui.html for a more interactive list of the REST endpoints provided by the Zeebe Cloud Events Router.

Cloud Events Mappings

There are three patterns supported in BPMN workflows::

1# EMIT_ONLY , meaning that you want to emit a Cloud Event and move forward. We can always catch/consume another event later in the workflow, for example:

1-A) Emit and catch Cloud Event with a BPMN Intermediate Catch Event

1-B) Emit and catch Cloud Event with a BPMN Receive Task. As we will see in the following example, using a Receive Task enables use to use Boundary Events.

#2 WAIT_FOR_CLOUD_EVENT emit and wait for a Cloud Event to continue inside a BPMN Service Task

#3 Start a Workflow Instance by sending a Cloud Event

The first approach ( EMIT_ONLY ) can be used when you want to use a BPMN Service Task to Emit a Cloud Event, usually to delegate the logic to a third party service and it doesn’t need to wait for an incoming event to complete. An incoming event can be mapped by using a BPMN Intermediate Catch Event or a BPMN Receive Task.

The second approach ( WAIT_FOR_CLOUD_EVENT ) allows you to map two Cloud Events to the same BPMN Service Task. The first Cloud Event will be the one that is emitted and the second one will define when to complete the Service Task. This is usually to map a service that reacts to an event and, when finished processing, emits one.

A third ( Start Workflow Instance ) pattern can be considered the mapping between a Cloud Event and the action of starting a Workflow Instance. For this pattern, you will need to map which Cloud Event can trigger the creation of new Workflow Instances.

The following example uses these three patterns to illustrate how to emit and consume Cloud Events from a BPMN Workflow. You can find the BPMN model here.

For example, if we want to emit a Cloud Event, the Queue To Request Tickets Service Task defines the following properties and headers:

As you can see there is one important property and 3 headers being defined:

ServiceTask.Type = “cloudevents”, this delegates the work to our Zeebe Cloud Events Router.
Header Mode = EMIT_ONLY defines the behavior, for this mode, a Cloud Event will be emitted
Header Type = “Tickets.CustomerQueueJoined” defines the type of Cloud Event that will be emitted
Header Host = “” this is where the Cloud Event will be emitted to.

It is important to note that the data of the Cloud Event created will contain all the workflow variables with their current value.

The example also shows how to emit and consume different Cloud Events from the same Service Task:

For this mode, the Service Task will wait for a Cloud Event of type “ Payments.RequestReceived ” to complete.

Running the Example

There is a Java Spring Boot implementation of this example available here: https://github.com/salaboy/zeebe-cloud-events-examples. There are several ways of running this example (starting all the services locally in your laptop, docker, dockercompose, Kubernetes). We will take a look at two of these options:

Running in your own Kubernetes Cluster with Helm
Running with Camunda Cloud

In order to run the example you will need the following components:

Zeebe
Zeebe Cloud Events Router
Tickets Service
Payment Service

The example, for the sake of simplicity, uses only HTTP transport, but it can be easily expanded to use other transports such as WebSockets and Kafka.

Running in your own Kubernetes Cluster with Helm

Because we are building Cloud-Native Applications, we will install these services in a Kubernetes Cluster.

We can leverage Helm to install a Zeebe Cluster including Operate into your Kubernetes cluster:

helm repo add zeebe http://helm.zeebe.io

helm install my-zeebe zeebe/zeebe-full

Then we can install the Zeebe Cloud Events Router:

helm install router –set env.ZEEBE_CLIENT_BROKER_CONTACTPOINT=my-zeebe-zeebe-gateway:26500 zeebe/zeebe-cloud-events-router

Notice that we are pointing the Zeebe Cloud Events Router to our Zeebe Cluster Gateway address.

And finally, we install the Tickets and Payment Services:

helm install –set env.ZEEBE_CLOUD_EVENTS_ROUTER=http://zeebe-cloud-events-router

tickets zeebe/tickets-service

helm install –set env.ZEEBE_CLOUD_EVENTS_ROUTER=http://zeebe-cloud-events-router payments zeebe/payments-service

Notice how both services are referencing the Zeebe Cloud Events router to be able to interact with it.

In order to interact with Operate, the Zeebe Cluster and the Cloud Events Router inside your cluster you can use kubectl port-foward:

kubectl port-forward svc/my-zeebe-zeebe-gateway 26500:26500

Now you can run for example zbctl status –insecure from your terminal to check the Zeebe Cluster Topology and Status.

kubectl port-forward svc/zeebe-cloud-events-router 8085:80

Now you can interact with the Zeebe Cloud Events Router in your localhost:8085 address. Later you will send HTTP requests to some of these endpoints using curl from your terminal.

kubectl port-forward svc/my-zeebe-operate 8081:80

Now you can point your browser to http://localhost:8081 and use Operate to see which workflows are available and how many executions do you have in your environment. User: demo, Password: demo.

Now we can deploy the Tickets Workflow, which can be found in this repository: https://github.com/salaboy/zeebe-cloud-events-examples

zbctl deploy tickets.bpmn –insecure

{
  "key": 2251799813685505,
  "workflows": [
    {
      "bpmnProcessId": "TICKETS",
      "version": 1,
      "workflowKey": 2251799813685504,
      "resourceName": "tickets.bpmn"
    }
  ]
}

In order to register a Cloud Event to start a new workflow instance, you need to register this mapping into the Zeebe Cloud Events Router, which associates a Cloud Event to a Workflow Key:

> curl -X POST http://localhost:8085/workflows -H “Content-Type: application/json” -d ‘{“cloudEventType” : “ Tickets.Purchase “, “bpmnProcessId” : “ TICKETS “}’

Now you can start a workflow by emitting a Cloud Event “ Tickets.Purchase ” to start a workflow:

curl -X POST http://localhost:8085/workflow -H “Content-Type: application/json” -H “Ce-Id: ABC-123” -H “Ce-Type: Tickets.Purchase ” -H “Ce-Source: curl” -d ‘{“sessionId”:”1″ }’

In Operate you will see the token moving forward until the workflow complete.

Running the example with Camunda Cloud

Now if you want to make your life easier, you can run Zeebe in Camunda Cloud, which is a managed service for Zeebe offered by Camunda. In this case, you only need to operate your services, and the Zeebe Cloud Events Router yourself.

After signing up for an account in Camunda Cloud(Beta) you can create a Zeebe cluster in self-service. I’ve created one called CloudEvents :

Getting my cluster up and running took me less than 5 minutes. And as you can see you also have Camunda Operate installed.

You can now interact with this new Development Cluster by using zbctl as we did in the previous section, but instead of using kubectl port-forward you can now export your credentials as environment variables to allow zbctl to connect with the remote cluster. If you scroll down in your Camunda Cloud Console, you will see the export commands that you can copy and paste into your terminal. Once this is done, you can deploy your workflow definition to the remote cluster using the same command as we did before:

Note: Remember that you can find the BPMN models in this repository: https://github.com/salaboy/zeebe-cloud-events-examples

zbctl deploy tickets.bpmn

Notice that now we are using SSL so there is no need for the –insecure flag.

For running the Zeebe Cloud Events Router against a Zeebe Cluster in Camunda Cloud you just need to make sure that the Zeebe Cloud Events Router can connect to the Zeebe Cluster, and you do that by sending the following variables when you install the Zeebe Cloud Events Router with Helm:

helm install router –set env.ZEEBE_CLIENT_BROKER_CONTACTPOINT=7ece08ec-f0df-43e4-ada6-55f46368ef64.zeebe.camunda.io:443 –set env.ZEEBE_CLIENT_ID=KChsFczamyQ604uGlCHployFmcavMo3n –set env.ZEEBE_CLIENT_SECRET= –set env.ZEEBE_AUTHORIZATION_SERVER_URL=https://login.cloud.camunda.io/oauth/token –set env.ZEEBE_CLIENT_SECURITY_PLAINTEXT=false zeebe/zeebe-cloud-events-router

Notice that you can obtain from the Camunda Cloud Console, as you did for the terminal, this secret is needed to establish the connection over SSL.

Once the Zeebe Cloud Events Router is running you can start new Workflow Instances in the same way as we did before.

Register a Cloud Event to Start new Workflow Instances, you do this mapping into the Zeebe Cloud Events Router by associating a Cloud Event to a Workflow Key:

curl -X POST http://localhost:8085/workflows -H “Content-Type: application/json” -d ‘{“cloudEventType” : “ Tickets.Purchase “, “bpmnProcessId” : “ TICKETS “}’

Now you can start a workflow by emitting a Cloud Event “ Tickets.Purchase ” to start a workflow:

curl -X POST http://localhost:8085/workflow -H “Content-Type: application/json” -H “Ce-Id: ABC-123” -H “Ce-Type: Tickets.Purchase ” -H “Ce-Source: curl” -d ‘{“sessionId”:”1″ }

Notice that now, the execution of this workflow will be happening inside Camunda Cloud. This means that you can monitor these executions in the Camunda Operate hosted right beside your cluster:

Sum up

This blog post covers a basic example of how you can orchestrate Cloud Events using Zeebe Workflow. You saw how you can run all the components in your own Kubernetes Cluster as well as using Zeebe inside Camunda Cloud to avoid installing, running, and maintaining Zeebe in your infrastructure (one less thing to worry about!).

This example shows how you can leverage a Workflow Engine (like Zeebe) and the BPMN language to orchestrate Cloud Events to:

Gain visibility and business traceability about how your services are interacting, where the bottle necks are and how to identify changes for continuous improvement.
Have an orchestration layer for your Event Based systems. Having the right balance between orchestration and choreography is key to develop a robust system that can continuously evolve.
Have a robust set of mechanisms to deal with exceptions and non-happy paths, such as time outs and business error handling.
Make sure that events follow a natural order to fulfil a business objective, this is very useful to identify if the flow can be improved and to understand new business requirements.

There are other ways to run the example, for example using Docker Compose or even running the services outside of Kubernetes and locally on your laptop. Feel free to reach out (comment here or via twitter) if you have any questions about this.

On a future blog post, a more advanced example will be covered using Knative, Cloud Events, Kafka, and WebSockets.

Stay tuned!

Lessons Learned: Offline Time / Holidays

salaboy — Tue, 24 Mar 2020 15:52:26 +0000

I just came back from holidays in Cuba, it was an amazing experience (including rushing back due to COVID-19) but I’ve learned some valuable lessons from being completely offline for at least 9 days. Internet access was limited, to say the least, usually restricted to hotel lobbies and paid by the hour. Because I really needed some holidays I was avoiding getting online as much as I could so the environment was perfect for that.

Some interesting things that I’ve learnt from this experience:

After 3 days of being offline, you don’t miss it anymore, and you start caring less about going online again.
After 3 days of being offline, creativity starts to kick in again, I found myself writing a lot of interesting ideas about tech and other stuff that I am interested in. This is something that I find hard to do after a normal day of work.
Being offline allows you to step back and reprioritize with an external point of view. This is very productive when you need to balance between things such as conferences, coding, initiatives, networking with different groups, etc. Stepping back from the day to day helps you to have a clear view of how important each of these things is and how they will impact your productivity and life/work balance.
Being offline and isolated even for a week allows you to have a fresh perspective on the really important things that will not change. It was quite fascinating to see how fast companies are changing nowadays and how if you just miss a week
I’ve learned about “Screen Time -> Downtime” in iOS to automatically block apps, which I set to From 20:00 to 7:30 every day. So now that I get back online I control myself with some time off.
I’ve reminded myself of having at least a couple pet projects, those projects that you just do for fun. In my career, pet projects have been a source of inspiration and learning.
The power of Open Source and how it affects my priorities. I ended up realizing that priorities were easy to set if the focus is made on the open-source aspect of each project/initiative.

Having loads of ideas and some inspiration from Jordan Mechner‘s books pushed me to have now a dairy (with daily entries) full of potential pet projects and reminders about what things are important.

I know.. this is not a tech blog post, but it might help someone else out there to share their experiences around their time off / holidays preferences.

Stay tuned! more tech blog post are coming!

Why is the @ljcjug awesome?

salaboy — Mon, 24 Feb 2020 15:57:48 +0000

Last week I had the pleasure to talk with my friend Ryan Dawson from Seldon.io at the London Java Community meetup, which was hosted at the Birkbeck University here in London.

The main topic of the evening was GoLang and our personal experiences as Java Developers while working on some projects with a different language, the toolchain, and how bad it feels when we are out of our comfort zone.

I was surprised by the audience interaction, as we were not alone, others in the community has experienced the same pains with other languages and our presentations quickly became more like an open debate with several members of the group sharing their experiences and their opinions.

The value of these groups is directly proportional to the group interaction and how much each member gets out of the meetup and it really felt like we all won/learned something that night. We had both students and professors from the university present in the room and even if some bits of our presentations were too advanced for them, they came up with very good questions.

LJC is awesome because the people that show up really care about what they do and they have strong opinions, they are not shy to share them and the setup allows everyone to express their opinions in a safe environment.

Slides

Java to GoLang by Ryan Dawson

Zeebe Operator, a practical use case for GoLang in Kubernetes Land

Java vs Challengers

Getting Started with Knative 2020

salaboy — Thu, 20 Feb 2020 12:09:48 +0000

The Knative project offers a set of components to build cloud-native applications on top of Kubernetes. These components offer higher-level abstractions enabling developers to focus on their actual services instead of all the boilerplate required to deploy their applications in Kubernetes.

This blog post is all about getting started with Knative as fast as possible so you can experiment with Knative features by getting your hands dirty. Unfortunately, you will realize rather soon that Knative has a steep learning curve. This is not due to Knative concepts and it is directly related to the tech stack on which it is built.

The main reason why I’ve created this blog post is to make sure that Java developers like myself can install and understand the basics of Knative with simple examples, so I can provide more advanced blog posts on how the Zeebe ( http://zeebe.io ) Knative integration will work, the main reasons behind it and how you can take advantage of it.

In this blog post you will learn:

How to get started with Knative in your own cluster
How to deploy and interact with a simple Knative Service with an HTTP endpoint,
Send/Receive CloudEvents using HTTP transport
Write a Spring Boot + Spring Cloud Streams applications that communicate sending Cloud Events using Kafka

Installing Knative with Gloo

There are several ways to install Knative and most of them depend on Istio. In this blog post, I chose to use glooctl for simplicity and leveraging the fact that it uses fewer resources. glooctl replaces Istio with Gloo which is a lightweight ingress controller and provides all the features required by Knative. Alternatively, you can install Knative from the project official documentation: https://knative.dev/docs/install/

You can find the full instructions here: https://docs.solo.io/gloo/latest/installation/knative/

If you are running on Mac OSX you can install glooctl with brew:

brew install solo-io/tap/glooctl

Once you have glooctl and a Kubernetes Cluster up and running and reachable with kubectl, you can install Knative Serving and Eventing 0.12.0 with:

glooctl install knative -g -e –install-knative-version=0.12.0

Once Knative is installed, we need to install Gloo as an API Gateway to route traffic from outside the cluster to our Knative workloads. We do this by using their Helm charts:

helm repo add gloo https://storage.googleapis.com/solo-public-helm

helm repo update

Let’s create a new namespace for Gloo and install the Helm chart in there:

kubectl create namespace gloo-system

helm install gloo gloo/gloo –namespace gloo-system -f values.yaml

Where the values file that I am using looks like this

values.yaml

gateway: 
  enabled: false 
  settings: 
    integrations: 
      knative: 
        enabled: true 
        version: 0.12.0

With this, you now have Gloo ingress controller and Knative Serving and Eventing up and running.

The files listed here and in the other sections can be found in this GitHub repository: https://github.com/salaboy/knative-getting-started-java-resources

What did I just install?

We just installed Knative Serving and Knative Eventing into our Kubernetes Cluster.

Knative Serving provides an abstraction layer on top of Kubernetes Services Deployments and Ingress.

Knative Serving is usually mentioned when people talk about Serverless on top of Kubernetes, as it provides Zero Downscale features out of the box for our applications, something that can’t be achieved with plain Kubernetes. The main concept in Knative Serving is a Knative Service. Initially, I was totally confused with the naming choice. Why call it a Service if Kubernetes already has a resource called Service? You can consider a Knative Service as a Kubernetes Service + Deployment + Ingress on steroids. A Knative Service provides us with a high-level concept to build cloud-native applications without worrying about all the low-level components required in Kubernetes to deploy and access our applications.

Knative Services, for example, uses an autoscaler which enables our services to be scaled based on demand. Enabling our services to be scaled down to zero when they are not being used – you will notice that, because of this, the serverless term is used quite a lot in the Knative site. Plus it provides an automatic point in time snapshots of our deployed code and configurations, advance traffic routing built on top of a service mesh.

With Knative Eventing we can define Event producers and Event consumers in a decoupled way – enabling our applications to be built independently. Knative Eventing allows us to push the decision on how our applications will communicate with each other (transports) to later stages.

If you are a Java Developer like me, implementing services or microservices is something that we do by writing Spring Boot applications, these applications are going to be our Knative Services. If we are building highly scalable systems, it is quite common to rely on a messaging infrastructure to communicate our services. Messaging tools such as Kafka, RabbitMQ and others provide our services guarantees about message delivery and an out of the box retry mechanism for when things go wrong. On top of Spring Boot, we have frameworks such as Spring Cloud Streams that came to make our life easier by abstracting all the low-level details that we need to know while using these messaging tools (Kafka, RabbitMQ, Google PubSub, etc). This push us to understand how frameworks such as Spring Cloud Streams will work with Knative Eventing, so let’s get our hands dirty with some examples that you can run in your cluster.

Did I hear “Hello World”?

Now we have Knative installed with Gloo, let’s give it a try by creating a simple example. As usual, for any Kubernetes deployment we will need to create our app, then a Docker Image and then some YAML files to deploy our application into a cluster. We will be pushing these Docker Images to http://hub.docker.com so you will need to have an account there.

Repository here: https://github.com/salaboy/knative-hello-world

The simple SpringBoot Application in Java, which exposes a REST Endpoint

KnativeHelloWorldApplication.java

@SpringBootApplication
@RestController
public class KnativeHelloWorldApplication {
... 
  @GetMapping("/{name}") 
  public String hello(@PathVariable("name") String name) { 
    return "Hello there Knative Traveller: " + name; 
  }
...
}

We need to compile this project with

mvn clean package

To generate the fat-jar into the target/ directory.

The most basic Dockerfile using OpenJDK:11 slim

Dockerfile

FROM openjdk:11-jre-slim
COPY target/\*.jar /app.jar
CMD ["java", "-jar", "app.jar"]

We need to build the docker image with:

docker build -t /knative-hello-world .

And then push to docker hub with

docker push /knative-hello-world

And here is where Knative starts helping us to focus on our business logic instead of writing tons of YAML files for Kubernetes. For Knative we just need a single Knative Service manifest:

kservice.yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata: 
  name: knative-hello-world
spec: 
  template: 
    spec: 
      containers: 
      - image: salaboy/knative-hello-world

kubectl apply -f kservice.yaml

When a Knative Service is created, Knative automatically creates other resources (Configuration, Route, and Revision) including a Kubernetes Service, a Deployment and an Ingress. On the Knative side the following resources are automatically created:

service.serving.knative.dev
route.serving.knative.dev
configuration.serving.knative.dev
revision.serving.knative.dev

This Knative Service will be automatically downscaled to 0 if there is no traffic going hitting the service after 90 seconds of inactivity.

Interacting with our new Knative Service

Once we have our Knative Service up and running we can get more information by using kubectl :

kubectl get ksvc

We will see that the Knative Service exposes the URL that we can use to contact our Knative Service.

But http://knative-hello-world.default.example.com is definitely not reachable as we probably don’t own the domain example.com. In order to find the external IP from our ingress controller we can run:

kubectl get svc -A

This should list the knative-external-proxy which is a LoadBalancer and has an External IP

gloo-system knative-external-proxy LoadBalancer

Then with this address, we can curl our service by setting the Host header to the request, so it gets redirected inside the cluster to our Knative Service.

curl -H “Host: knative-hello-world.default.example.com” http://<EXTERNAL IP>/salaboy -v

Notice: that we need to include the Host header just because we don’t have a publicly available domain.

Curl -H what? Why?

If you are running on a Cloud Provider which can provision a LoadBalancer with a Public Ip, something that we can do is use XIP.io and configure Gloo DNS to use a XIP.io address.

https://knative.dev/docs/install/knative-with-gloo/ -> Configuring DNS

kubectl edit cm config-domain –namespace knative-serving

And modify it to look like this:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: config-domain 
  namespace: knative-serving
data: 
# xip.io is a "magic" DNS provider, which resolves all DNS lookups for: 
# \*.{ip}.xip.io to {ip}. 
  <EXTERNAL\_IP>.xip.io: ""

You can use XIP.io to have a subdomain which is linked to your transforming the previous curl to

curl http://knative-hello-world.default.<EXTERNAL IP>.xip.io/salaboy -v

This is much nicer. With just a Knative Service deployment we have a public route that we can expose for people to consume outside the cluster. Notice, no Ingress + Service + Deployment configurations required besides our Knative Service. Knative Serving will take care of managing the lifecycle of all the Kubernetes Core resources required for our application to run.

“Hello World”s are for kids, bring it on!

If we are designing a distributed system we might want to communicate between different services. Cloud Events are becoming a popular way to do that by interchanging data in a common format/envelope. Luckily for us, Knative Eventing already supports Cloud Events and we can use that straight away. Knative Eventing gives us a set of components that enable late-binding for event sources (producers) and event consumers. This means that we can create our consumers and producers in a generic way and then link them together at runtime.

Let’s start simple, let’s modify our simple Knative Service from before to accept Cloud Events. Cloud Events and how we receive them will vary depending on which transport we want to use, but the content will always be the same. Because we want to start simple, we will expose an HTTP endpoint to accept CloudEvents.

Repository here: https://github.com/salaboy/knative-hello-world

In Spring Boot you will do something like this:

...
@PostMapping
public String recieveCloudEvent(@RequestHeader Map<String, String> headers, @RequestBody Object body) { 
  CloudEvent<AttributesImpl, String> cloudEvent = CloudEventsHelper.parseFromRequest(headers, body); 
...
}
...

Now we can send a CloudEvent with curl by adding the right headers

curl POST http://knative-hello-world.default.<EXTERNAL IP>.xip.io -H “Content-Type: application/json” -H “ce-id: 536808d3” -H “ce-type: my-first-cloud-event” -H “ce-source: curl” -d ‘{“name”:”salaboy”}’ -v

Notice: ce-type: my-first-cloud-event which represents what kind of event it is. We can use this to decide if we are interested in it or not. Both the data (send with -d) and the ce-type are user-defined. Cloud Events doesn’t impose any restrictions on the data that is sent.

Notice: Content-Type: application/json, to avoid sending a form-data content.

And there you go, now every other application producing CloudEvents can send events to our small application.

When you look at Knative you will find that there are a lot of available Event Sources already, for example, KubernetesEventSource, GitHubSource, GcpPubSubSource, AwsSqsSource, ContainerSource, CronJobSource, KafkaSource, CamelSource.

But let’s create our own producer, another spring boot application that just sends a CloudEvent by posting content via HTTP.

Repository here: https://github.com/salaboy/knative-event-producer

KnativeEventProducerApplication.java

@SpringBootApplication
@RestController
public class KnativeEventProducerApplication {

  ...

    @Value("${SINK_HOST:localhost}")
    private String HOST;


    @GetMapping("/{name}")
    public String doSomethingAndSendCloudEvent(@PathVariable("name") String name) {

        final CloudEvent<AttributesImpl, String> myCloudEvent = CloudEventBuilder.<String>builder()
                .withId("ABC-123")
                .withType("my-first-cloud-event")
                .withSource(URI.create("knative-event-producer.default.svc.cluster.local"))
                .withData("{\"name\" : \"" + name + "-" + UUID.randomUUID().toString() + "\" }")
                .withDatacontenttype("application/json")
                .build();

        WebClient webClient = WebClient.builder().baseUrl(HOST).filter(logRequest()).build();

        WebClient.ResponseSpec postCloudEvent = CloudEventsHelper.createPostCloudEvent(webClient, myCloudEvent);

        postCloudEvent.bodyToMono(String.class).doOnError(t -> t.printStackTrace())
                .doOnSuccess(s -> System.out.println("Result -> " + s)).subscribe();

        return "OK!";

    }
...

kservice.yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: knative-event-producer
spec:
  template:
    spec:
      containers:
        - image: salaboy/knative-event-producer
          env:
            - name: SINK_HOST
              value: knative-hello-world.default.<EXTERNAL IP>.xip.io

Notice: now we are configuring the SINK for the events produced in our service with an Environment Variable called SINK_HOST and we are pointing to the internal DNS name for our knative-hello-world Service.

If you build the project, the Docker image and then apply this Knative Service manifest, you can curl the event producer to emit a CloudEvent to our Knative Hello World Service.

Now you have two services exchanging data using CloudEvents, that is pretty cool, but let’s take it up a notch.

Toys for Grownups (aka Kafka)

In a real distributed application, you might want to use all the power of messaging and streaming of information. For such scenarios, Kafka is becoming a very popular option, so let’s give it a try with Knative.

In this scenario, we will have an application producing Kafka messages and we want to route the content of these messages to our HTTP CloudEvent consumer endpoint. We haven’t designed any of these applications to talk to each other, but because we are using CloudEvents (and JSON payloads) they can now interchange information.

In order to get this working we need to:

Install Kafka or have it running somewhere where it is reachable by the cluster
Install KafkaSource for Knative
Create a Kafka Source for Knative to understand where the messages are coming from and where are they going. This is referred to as ‘late binding’ because you change the source, create new ones or remove them in runtime.
Produce Kafka messages

Let’s install Kafka in our cluster using Helm:

helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator

helm install my-kafka incubator/kafka

Next we need to install KafkaSource for Knative

kubectl apply -f https://storage.googleapis.com/knative-releases/eventing-contrib/latest/kafka-source.yaml

Now we are ready to create our Kafka Source with a sink pointing to our knative-hello-world Knative Service:

kafka-knative-source.yaml

apiVersion: sources.eventing.knative.dev/v1alpha1
kind: KafkaSource
metadata:
  name: kafka-source
spec:
  consumerGroup: knative-group
  bootstrapServers: my-kafka:9092 #note the kafka namespace
  topics: knative-topic
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: knative-hello-world

Now we can send messages to the knative-topic and they will be automatically forward to our Knative Service called knative-hello-world. Notice that my-kafka is the name of the Kubernetes service that was created by installing Kafka with Helm.

A very lazy way to post messages to the knative-topic might be using a Kafka client from a Docker container, like this:

kubectl run kafka-producer -ti –image=strimzi/kafka:0.14.0-kafka-2.3.0 –rm=true –restart=Never — bin/kafka-console-producer.sh –broker-list my-kafka:9092 –topic knative-topic

Then type a message in the prompt:

{“msg”: “This is a test!”}

That will send a Kafka message, which will be transformed into a cloud event and forwarded to our HTTP Cloud Event consumer (knative-hello-world Knative Service).

How would you do it for real?

In a real project, it really doesn’t matter how you produce the Kafka message. One option would be to just grab the Kafka Java APIs and make your application write messages to the previous Kafka topic. But if you really like the late-binding approach you can leverage that from your Spring Boot applications as well by using Spring Cloud Streams, which gives you a common API for different messaging providers.

If we want to now create a Spring Boot application that can send messages to the Kafka Topic that we previously defined, we just need to use Spring Cloud Streams dependencies and the Kafka Binder. You can find the source code of this example here:

https://github.com/salaboy/knative-spring-cloud-event-producer

pom.xml

<dependencies>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream-binder-kafka</artifactId>
  </dependency>
...
</dependencies>

And then a simple application that sends messages

KafkaProducerApplication.java

@SpringBootApplication
@EnableBinding(MyChannels.class)
@EnableScheduling
public class KafkaProducerApplication {

  @Autowired
  private MessageChannel myProducer;

  ...

  @Scheduled(fixedDelay = 3000)
  public void sendMessageEveryX() {
    System.out.println("> Sending a message...");
    myProducer.send(MessageBuilder.withPayload("{\"name\":\"salaboy : " + UUID.randomUUID().toString() + " \"}").build());
  }

}

This simple app will send a message every 3 seconds, but where?

That is late binding in action. Where and which binder (which messaging implementation) can be defined later on. In this case, we can do that by changing the applications.properties file with:

application.properties

spring.cloud.stream.kafka.binder.brokers=my-kafka
spring.cloud.stream.kafka.binder.defaultBrokerPort=9092
spring.cloud.stream.bindings.myProducer.destination=knative-topic

But because we are working with Docker, we can easily do that with Environment Variables as well. Meaning that our container will not have these details hard coded and we will be able to reuse the same docker image in different environments with different Kafka installations and different topics.

To build the project you can run:

mvn clean package

You will need to build this docker image with:

docker build -t /knative-spring-cloud-event-producer .

And then you can define your Knative Service:

knative.yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: knative-spring-cloud-event-producer
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
        - image: salaboy/knative-spring-cloud-event-producer

Notice: that our application doesn’t expect any HTTP request. By default, Knative autoscaler will downscale the application based on traffic. For this particular scenario, we don’t want our application to get downscaled, for that reason we use the annotation: autoscaling.knative.dev/minScale: "1". This will keep at least one instance of our service running at all times.

You can create this Knative service by applying this file into your cluster:

kubectl apply -f kservice.yaml

If you log the output of the knative-hello-world pod you will see that every 3 seconds you will receive a Cloud Event.

kubectl logs -f knative-hello-world-

You can try removing the autoscaling.knative.dev/minScale: "1" and see how after 90 seconds your pod is scaled down to zero, as there is no traffic going to your event producer.

Sum Up

There you have it, an example that uses Knative Serving + Knative Eventing + HTTP Transport Cloud Events + Knative Kafka Source + Spring Cloud Streams.

This is quite a mouthful, but this blog post covered all the pieces that you will need to build highly distributed applications that can scale using Knative and Kafka. As we have seen in this blog post, Knative provides a programming model on top of Kubernetes concepts which allows us to concentrate on our applications instead of how Kubernetes will deploy our application, or how it will route requests to it. Knative will automatically downscale to zero our service if it is not being used, something that cannot be achieved with plain Kubernetes. We have also experimented with Cloud Events and Knative Eventing, to demonstrate a powerful and robust approach to communicate between our services while using messaging and late binding.

In future blog posts, I will be sharing some details about how Knative and Zeebe ( http://zeebe.io ) integration is being built to provide features such as:

Long-lived and highly scalable orchestration Service and functions orchestration leveraging late binding
Cloud Events orchestration, sequencing, and monitoring
A framework for application development built on top of Knative which provides business visibility and business insights to non-technical people

If you have comments, suggestions or questions feel free to reach out as I will be evolving the examples.

2019/2020

salaboy — Fri, 03 Jan 2020 10:11:59 +0000

I’ve learned a lot of valuable lessons in 2019, this blog post it is just a quick summary of the year and my experiences while working with Open Source technologies such as Kubernetes, Zeebe, Jenkins X, Spring Cloud, Camunda BPM, JHipster, KubeBuilder, etc.

Stay Focused / Reduce Stress

During 2019, I’ve faced a lot of challenges around finding the right activities to focus on. Being on the road presenting, being involved in several projects which require active development, preparing new proposals, and engaging with different communities lead me to not being able to make progress or even finding which were the key initiatives that really deserved my attention.

Lucky, in the second half of 2019, I was fortunate enough to meet new friends that helped me to stay focused and reduce stress levels. In general, this was achieved by choosing the right things to do, the right conferences to present at and the right way to scope and measure progress around my work.

Company Culture is a must

It can make such a difference when you are in a company that has the right culture, you immediately notice the change. Camunda has over-delivered in this angle. I’ve definitely learned a big lesson in 2019, technology matters but without the right culture, it is impossible to sustain or even find the right path and speed to achieve great things. As has been said before, culture is ever-shifting and based on people, so it feels to me that I am up to keep learning on how the company cultivates and improve its culture as it grows.

Keep Learning + Keep Sharing

In 2019, I’ve decided that I wanted to be out of my comfort zone and I set up some projects to learn new things. After being in the Java ecosystem for such a long time, I’ve decided to start with GoLang and this opened the door for a lot of new interactions and also set me up for learning new things in 2020.

A big part of 2019 was to go out and meet other developers and smart people to understand how they are tackling their Cloud-Native journey, I had the pleasure to present in the following conferences/meetups:

MadridJUG, Spain, Dec 19
CoimbraJUG, Portugal, Dec 19
Jenkins World Lisbon, Portugal, Dec 19
Jenkins X Webinar 19 (Online), Nov 19
KubeCon US San Diego 19 / CD Foundation Summit, USA, Nov 19
SouJava Webinar 19 (Online), Nov 19
GeeCon Prague 19, Prague, Czech Republic, Oct 19
London Java Community Meetup 19, London, UK, Sept 19
Jenkins World San Francisco, USA, Aug 19
JHipster Conf Paris, France – June 19
JBCNConf, Barcelona, Spain – May 19
La Plata University Meetup, Mar del Plata, Argentina – March 19
Healthcare in Gov Meetup, Buenos Aires, Argentina – March 19
London DevOps Meetup, London, UK – March 19
Alfresco DevCon, Dublin, Ireland, Jan 19

Where my last presentation of the year focused on the path from a Java Monolith to Kubernetes focusing on two tools that can make your life easier: Jenkins X for CI/CD on Kubernetes and Zeebe for MicroServices and Functions Orchestration in Kubernetes.

I am still blown away by speaking at KubeCon US in front of 4000+ people. So it is worth mentioning that I had a lot of fun Learning and Sharing being part of LearnK8s.io, in 2020 I expect this relationship to grow and I hope to be able to meet all the LearnK8s team face to face.

What’s next

There are several things coming my way on 2020, Kubernetes and Orchestration is still my main focus, but I want to focus on getting heavily involved with new communities where I am still a newbie. You can expect the Zeebe Kubernetes Operator to evolve iteratively along with the Zeebe Helm Charts as there is a lot of work to do on those fronts. I am really interested in making the life of Developers using these tools as simple as possible, so if you are looking into these tools get in touch and we can work together on some cool projects.

I will not spoil too much now but stay tuned for more news. I will try to keep my twitter updated with 2020 conferences, so I am looking forward to meeting you all.

Zeebe Kubernetes Operator

salaboy — Fri, 20 Dec 2019 09:34:24 +0000

The Zeebe Kubernetes Operator was born out of the need to manage more than one single Zeebe Cluster running inside Kubernetes Clusters. Zeebe Clusters have their own lifecycle and in real implementations, the need to update, monitor and manage some of these cluster components while applications are running becomes challenging. The objective of the Zeebe k8s Operator is to simplify and natively integrate Zeebe with k8s, to solve operational burden and facilitate the creation and maintenance of a set of clusters.

This operator has been built with Kubernetes Helm in mind, meaning that at the end of the day, this operator will be in charge of managing Helm Charts. If you are not familiar with Helm, Helm is a package manager for Kubernetes, which help us to package and distribute Kubernetes manifest. Helm also deals with installing, labeling and dependency management between packages (charts). Because we have Zeebe Helm packages already here: http://helm.zeebe.io which are automatically versioned and released, the Zeebe Kubernetes Operator will use these charts to create and manage new clusters and other related components.

Because we are in Kubernetes realms we need to provide a declarative way of stating that we want a new Zeebe Cluster to be provisioned. For this reason, the ZeebeCluster Custom Resource Definition ( CRD )is introduced. This resource contains all the information needed to provision a cluster and it will also reflect the current cluster status. The Zeebe Kubernetes Operator is built to monitor ZeebeCluster resources and interact with the Kubernetes APIs under the hood to make sure that the Zeebe Cluster is provisioned, upgraded or deleted correctly.

Getting Started

The Zeebe Kubernetes Operator can be installed using Helm, as it is provided as a Helm Chart as well. In contrast with zeebe-cluster , zeebe-operate and zeebe-full charts, the operator chart installation by itself doesn’t install any Zeebe Cluster, but allows you to do that by creating ZeebeCluster CRD resources.

The following steps will guide you to install the Operator with Helm3 (which is the default version now)

This will also work if you have correctly installed Helm2 in your cluster with tiller.

Add the Zeebe Helm Repository:

helm repo add zeebe https://helm.zeebe.io helm repo update

Now you are ready to install the Zeebe Kubernetes Operator:

helm install zeebe-operator zeebe/zeebe-operator

Create my-zeebe-cluster.yaml

apiVersion: zeebe.zeebe.io/v1 kind: ZeebeCluster metadata:   name: my-zeebe-cluster

Create the resource with in the Kubernetes cluster with:

kubectl apply -f my-zeebe-cluster.yaml

This will create a new namespace with the name stated in the ZeebeCluster resource ( ZeebeCluster.metadata.name ) and provision a new Zeebe Cluster plus ElasticSearch by default.

Future versions will allow you to specify in the ZeebeCluster resource which ElasticSearch instance to use.

Notice that the first time provisioning a cluster, docker images will need to be downloaded to the Kubernetes Docker Registry so the first cluster might take more time to be provisioned.

You can now query for your Zeebe Clusters using the kubectl CLI:

kubectl get zb

If you delete the ZeebeCluster resource the actual ZeebeCluster will be automatically removed from your cluster.

Now you can check that there is a new “ Namespace ” created with:

kubectl get ns

And also check that the cluster is correctly provisioned by looking at the Pods created inside the newly created namespace with

kubectl get pods -n <Zeebe Cluster Name> -w

The next video show these commands in action along with the installation of the Zeebe Kubernetes Operator:

Technical Details and Dependencies

This Kubernetes Operator was built using KubeBuilder V2.1+, Tekton 0.8.0+ and Helm 3.

The Operator Defines currently 1 CRD: ZeebeCluster, but in future versions, new types will be defined for other components such as Zeebe Operate and Workers. The ZeebeCluster resource represent a low-level resource which will instantiate a Zeebe Cluster based on predefined parameters. This low-level resource definition can be used to define the cluster topology and HA configurations.

The Zeebe Kubernetes Operator was built using the kubebuilder framework for writing the controller’s logic and scaffolding the CRD type. Internally it does interact with Tekton Pipelines in order to install and manage Zeebe Helm Charts. The project itself is being built, released and tested using Jenkins X. This lead to some changes in how KubeBuilder’s project is structured, as in it current shape the project is not organised in a way that is easy to create a Helm Chart out of it.

The main flow of the Operator works like this:

First, the Operator will be constantly looking for ZeebeCluster resources, when one is found a new Namespace is created and a Tekton Task and TaskRun are created to “upgrade” the Helm Charts defined inside the Version Stream repository (for now hosted here: https://github.com/salaboy/zeebe-base-chart ).

This repository (referred as Version Stream Repository ) contains a list of blessed versions that will be installed when a new ZeebeCluster resource is detected by the operator. Using a Version Stream Repository provide us with the flexibility to evolve the operator code and the charts that define what needs to be provisioned independently. Allowing a simple upgrade path to future versions by using a Git repository as central reference to a stable version.

In future versions, the option to choose a version stream repository will be provided. Allowing different streams

The Task created in Tekton Pipelines execute two basic operations

Clone Version Stream Repository (using simple git clone)
Run Helm Upgrade of the chart defined in the Version Stream Repository (it does actually an upgrade/install if it doesn’t exist)

The second step, running Helm upgrade/install will create a Helm Release which can be upgraded if new versions of the charts are available. These releases can be queried using the Helm cli tool: helm list --all-namespaces.

Once the Task is created an execution is triggered by the creation of a TaskRun (an actual instance of the task) and the operator will monitor for this task to be completed. Once the task is completed, the Operator watch for the Zeebe Cluster to be provisioned. In a more detail look, the Operator will look for a StatefulSet (Zeebe Broker Nodes) with a set of labels matching the ZeebeCluster name, inside the created namespace.

Once the StatefulSet is located, the Operator assign the ZeebeCluster resource as the _ Owner _ of this StatefulSet , hence it will be notified about the changes emitted by the resources associated to the StatefulSet. This allows the Operator to externalise a Health Status of the Zeebe Cluster at any given point, understanding the actual state of the cluster itself.

Sum up

The Zeebe Kubernetes Operator helps you to create and manage Zeebe Clusters. It uses Tekton Pipelines and Helm3 to actually provision these clusters. Each ZeebeCluster will correspond with a different Kubernetes Namespace, which provide resources and data isolation, as currently an ElasticSearch Cluster is provisioned as part of the Zeebe Cluster Helm Chart.

Future versions will include more advanced management features and other components related to the ZeebeCluster, such as Zeebe Operate, infrastructure tools such as Prometheus. If you want to contribute with the project and get involved early on into making this operator awesome, drop me a message as I have at least 20 issues pending in my TODO list.

The source code of the operator will probably find a new home in the Zeebe organisation early next year.

Boosting your `kubectl` productivity @ KubeCon

salaboy — Wed, 20 Nov 2019 18:23:16 +0000

Oh my.. that was a big stage and a very very large audience, but it was totally worth it! I kinda rushed in the end but I got a lot of positive feedback about the content and a lot of kudos to the @learnk8s guys who are doing an amazing job.

Here some useful links about the projects that I’ve mention during the presentation:

@LearnK8s Article: https://learnk8s.io/blog/kubectl-productivity/
KubeCTX and KubeNS: https://github.com/ahmetb/kubectx
AutoGen Aliases: https://github.com/ahmetb/kubectl-aliases
Krew: https://github.com/kubernetes-sigs/krew
KubeCtl KubeCon Plugin: https://github.com/salaboy/kubectl-kcon

Also on the cd.foundation summit a quick talk about why Jenkins X:

From Monolith to Kubernetes Webinar with CloudBees

salaboy — Mon, 18 Nov 2019 16:56:45 +0000

I am super happy to announce that I am doing a Webinar about how to go “From Monoliths to Kubernetes using Jenkins X”. This is being organised by CloudBees and it is going to happen the 26th Nov 2019. I will be showing live how to work with multiple projects and what main challenges are you going to face while going cloud native, along the tools (Zeebe, Helm, etc.) that will help you to move faster.

If you are interested in the webinar please register here: https://register.gotowebinar.com/register/4496029892230348035

I’m really looking forward to hear your questions and comments about it.

DEV Community: salaboy

KubeCon & Spring One 2020: From Monoliths to K8s

KubeCon Session

Spring One Session

Links and References

Jenkins X on Linode LKE Webinar

JNation 2020: From Monolith to K8s

Slides & Video

Links

Why isn’t Jenkins X’s future more Open?

A bit of background

What we lose

What can be done about this?

Orchestrating Cloud Events with Zeebe

Example

The Orchestration Approach

Architecture

Cloud Events Mappings

Running the Example

Running in your own Kubernetes Cluster with Helm

Running the example with Camunda Cloud

Sum up

Lessons Learned: Offline Time / Holidays

Why is the @ljcjug awesome?

Slides

Links

Getting Started with Knative 2020

Installing Knative with Gloo

What did I just install?

Did I hear “Hello World”?

Interacting with our new Knative Service

“Hello World”s are for kids, bring it on!

Toys for Grownups (aka Kafka)

How would you do it for real?

Sum Up

2019/2020

Stay Focused / Reduce Stress

Company Culture is a must

Keep Learning + Keep Sharing

What’s next

Zeebe Kubernetes Operator

Getting Started

Technical Details and Dependencies

Sum up

Boosting your `kubectl` productivity @ KubeCon

From Monolith to Kubernetes Webinar with CloudBees

Recording