DEV Community: Julian Setiawan

Developing an IoT Hub to Facilitate the Support of Your IoT Architecture

Julian Setiawan — Thu, 01 Apr 2021 13:44:26 +0000

Every quarter, Solace’s PubSub+ Cloud team participates in a two-day hackathon. It’s an opportunity to experiment with the Solace PubSub+ Platform, either by utilizing it in a novel way or extending the platform itself. It’s also a great way to work side by side with others on the team that we haven’t had a chance to work with and possibly dabble in other areas.

Our team’s goal for the hackathon was to provide an IoT hub where users only had to worry about writing code and could leave the other details to us. PubSub+ Event Broker is an extremely powerful and versatile broker with a staggering number of features across many protocols. Within two days we went from an idea to actually creating the IoT hub. This article shares how we used PubSub+ Event Broker: Cloud to develop the hub.

How the IoT Hub Works

The IoT Hub uses PubSub+ Cloud to provision event brokers. When a new device is registered on the hub, a new client username is created on the event broker.

This is as simple as adding a device from the IoT hub’s UI.

How We Developed the IoT Hub

We used the ACL substitution variables of PubSub+ to ensure that each device only has access to their own topic space. This was all done with a straightforward integration with the PubSub+ Cloud REST API and SEMP.

We then offered some useful shortcuts for developers who want to dive straight into coding. There are Java and Python snippets ready to download and run, pre-populated with your connection details.

For easier debugging, we also display recent events within your topic space and provide a way to publish events right from the hub.

./run.sh and you’re publishing and subscribing!

Now we needed devices to use the IoT hub. Our choice? Nerf guns.

We created an internet-connected target and score display. Using code samples and connection details from the hub, we were able to start iterating immediately.

When the target powers on, it sends a registration event to the hub. A micro switch push button is depressed when the target is hit. The button triggers “hit” events to be published to the hub. The score display receives the “hit” events and updates the score.

The hub gave us a fast and simple way to start publishing, subscribing, and seeing events. We were able to add the score display as a second device with the press of a button, giving it its own set of credentials. This quick validation allowed us to concentrate on the actual events instead of debugging connection parameters.

The Hackathon – Working with Students

Solace has participated in many university hackathons, exposing students to event-driven architecture and getting them to think beyond REST. However, many students don’t have much experience with event brokers.

Historically, we helped students sign up for PubSub+ Cloud and guided them through connecting their applications. One of the broker’s greatest assets of being language agnostic and multi-protocol ended up being overwhelming with so many different ways to connect.

We thought the IoT hub could resolve all of these. Choosing MQTT as the protocol, picking which libraries to use for the different languages, and setting up the connection parameters gave students a baseline functionality they could start with. As they explored more advanced use-cases, we could point them to PubSub+ Cloud, but they were armed with a working prototype.

In two days, we had a few tweaks to help us out during an upcoming university hackathon. First, we secured the dashboard with our company’s SSO. This allowed all the Solace volunteers to create new devices (i.e. credentials) and help students debug issues by viewing their connection details. Then, we added a way to generate unauthenticated, hard-to-guess links to the different devices so we could easily onboard the students. Finally, we polished up the interface, added more samples, and hosted the application online.

During the hackathon, we started seeing the benefits immediately. We were able to help students debug simple issues by checking out their connection details and using the hub to publish events to their applications. Instead of the usual questions about how to connect, we heard “How can we tell when a client disconnects?” (will messages) and “Can I publish with one protocol and subscribe with another?” (absolutely!). Without being bogged down by connection issues, students were starting to think beyond the basics and concentrating on the events!

Looking Beyond the Hackathon

The IoT Hub wasn’t the most elaborate use of what Solace has to offer, it was a few opinionated integrations with Solace products that are readily available to anyone. The only difference is that we have the valuable experience of advising countless customers on their event-driven architecture and guiding users to proven patterns.

We hope to bring these lessons to our products. Who knows, maybe sometime in the future you’ll see an “IoT Hub” icon appear in PubSub+ Cloud and you’ll be assured that you have the expertise of Solace powering your products.

The post Developing an IoT Hub to Facilitate the Support of Your IoT Architecture appeared first on Solace.

A New Architect’s Take on O’Reilly Software Architecture Conference

Julian Setiawan — Fri, 05 Jul 2019 17:56:51 +0000

As a relatively new architect, I jumped at the opportunity to attend the recent O’Reilly Software Architecture Conference in San Jose, California, and was eager to absorb as much as I could.

Before I became an architect, I thought the role was mainly about designing systems, and evaluating and selecting the technologies it takes to bring them to life. I didn’t fully appreciate the importance of effectively explaining these designs and technology selections to others. As such, I chose to focus on soft skills during this conference with the goal of improving my design and communication skills. My conference schedule featured two half-day tutorials on the first day and shorter talks and various events on the following two days.

Shaping and Communicating Architecture

In the first tutorial I attended, Seth Dobbs, vice president of engineering for Bounteous, helped crystallize some things I’ve been discovering about architecture in succinct and elegant ways. While many speakers and attendees lamented the difficulty in defining the role of an architect, I thought Seth’s description of “translating between business and technology” to be quite apt. He began by talking about the basics of communication and some common pitfalls, such as the tendency to assume there is no disconnect even if there is one. This failure in communication leads to deflections of responsibility like “I told them that” or “It’s on the wiki” . As Seth puts it, “communication is a two-way street, but we own being understood”.

The first step was recognizing that architecture is a lot like sales because ultimately, “our ideas don’t matter if we can’t get others on board”. Seth went into details about the disparate stakeholders involved in the decisions being made, such as developers and project managers. He then formed archetypes around those roles to better define what those stakeholders provide (technical or business insights), what they want to know (how a feature is being built or its cost), their goals (ease of development or communicating the business), and finally, the sorts of friction that may occur with them (skepticism or armchair solutioning).

Seth then discussed how to identify problems. A core principle of this tutorial was the business side of the technology, rooting back to Seth’s definition of an architect’s role. Therefore, it is important to distinguish between a business problem and a technical problem. For example, a database query being slow is a technical problem whereas customers waiting a long time and not returning is a business problem. There are a couple risks in attempting to solve the former instead of the latter. First, it simply may not be a problem worth solving; perhaps it is a backend query run on a schedule with no impact on the user. Secondly, you may miss more pressing issues; the database query may represent a small fraction of the slowdown and the real issue is a third-party API call. After a problem is identified, Seth eloquently described the solving of a problem as “mapping from needs/goals to solution while honoring constraints”.

The final step is presenting the architecture. Seth proposes beginning with a problem statement, then the context such as assumptions and constraints, followed by the high-level solution. The solution details can come afterwards which are more tailored for different audiences.

Seth’s tutorial was a fantastic introduction to the conference and while many of his ideas seem intuitive, his ability to formalize and present them were really helpful. The next tutorial I attended shifted from the collaborative aspects of architecture to the internal aspects.

Thinking Architecturally

Nathaniel Schutta, solutions architect for Pivotal, started by explaining that every solution has trade-offs. He then drilled down into how to identify and communicate these trade-offs through non-functional requirements or, more endearingly, the “ilities” (a play on words like scalab ility or reliab ility ). The strategy was to identify the focus of the architecture by ranking a few ilities and using that ranking to break ties when having to make a conflicting choice between them. Having too many ilities is a sign that a solution is complex and may require multiple, separate investigations.

The lessons were reinforced by architectural katas, exercises where a small group is given a theoretical project and they work together to create a rough design. We were asked to rank the ilities for a random kata and use that to inform our design. It was an engaging exercise that helped practice architecture skills (the purpose of the katas). As an aside, a question was asked that has been on my mind ever since I became an architect, “How do you continue to grow as an architect?” Nathaniel elegantly explained that developers are able to develop expertise through their responsibilities at work, whereas architects may not get as many chances to architect major solutions. This is where learning about exercises such as architectural katas are exciting for me in that they give me an outlet to evolve as an architect.

Just as the first tutorial helped formalize intuitive thoughts I had about communication, Nathaniel was able to characterize my understanding of trade-offs in decisions and how to improve those skills.

The rest of the conference consisted of various keynotes, discussions, and shorter talks.

The book Building Evolutionary Architectures was heavily referenced by many speakers. Specifically, applying fitness functions to architecture by testing the structure of the code. There’s a part of me that is interested in trying this out, but I am always hesitant in enforcing high-level rules with dubious effectiveness as it tends to lead to workarounds that are more harmful to the codebase in the end.
There were some competing ideas around different code analysis tools and methods. A common one was cyclomatic complexity and its various forms. However, during the keynote, Adam Tornhill focused on an alternative analysis with the basis that complex code isn’t inherently bad. Adam’s approach was to find areas in code that were coupled, old, or changed often using the VCS commit log and associating those changes with bugs or features. I plan on trying both and use my knowledge of the codebase to evaluate what seems more accurate and see if our teams would find this helpful.
A couple talks focused on social aspects such as architecting for different team formations (instead of restructuring the software, consider restructuring the teams) and mapping biological size limits to team size limits (the size of an animal is limited by how biological functions scale). At our current size, these observations were less applicable, but good to know since we are constantly growing.
Beyond accidental architecture with James Thompson had some basic tenants for architects that I strive to follow, such as architect playing the role of a guide (instead of authoritarian leader) and being heavily involved in the codebase to foster confidence in technical skills. I liked that he took the idea of an architect as guide a step further and said that dev teams truly own the architecture since they own the implementation whereas the architect only produces an abstract form of one.
Architecting for testing and continuous delivery with Ken Mugrage was a talk I was particularly interested in because we are in the middle of our journey towards CI/CD. I was elated to discover that this talk was similar to one I recently gave internally at Solace about the strategies we should use to achieve our goals (backwards compatible changes, use of the parallel change pattern). Sounds like we are on the right track!

The conference was a wonderful opportunity to learn about the myriad of strategies and approaches to software architecture from not just the speakers, but the other attendees as well. My immediate action is to reflect on our architecture process here at Solace and see if we can improve the structure of the meetings, documents, and other output to take advantage of the communication skills I have learned. We’ll do whatever it takes to enable our teams to continue creating awesome technology!

The post A New Architect’s Take on O’Reilly Software Architecture Conference appeared first on Solace.

MQTT Client Certificate Authentication with Solace’s PubSub+ Broker

Julian Setiawan — Thu, 16 May 2019 13:10:52 +0000

MQTT is the de-facto messaging protocol in the IoT world. When it comes to authenticating and identifying thousands of devices, you may see mention of client certificate authentication, but very little on how to actually achieve this. Although managing certificates, revocation lists, and everything that comes with being a certificate authority are complicated, getting started with client certificate authentication is not.

In this article, we will be performing all the steps necessary to begin creating a Node.js MQTT application authenticating via client certificates. There are three basic things we will be doing:

Creating a certificate authority (CA) and a client certificate
Uploading the CA certificate to the MQTT broker (Solace PubSub+ in this case)
Configuring a JS MQTT client to use the client certificate for authentication

There are a few things you need before we get started:

OpenSSL
Node.js
NPM
Solace Cloud PubSub+ Developer or Starter Service

Now let’s create the beginnings of an IoT solution!

First, we’ll create a certificate authority (CA) to sign client certificates using OpenSSL.

openssl req -nodes -new -x509 -newkey rsa:4096 -keyout root.key -out root.pem -subj "/C=/ST=/L=/O=/OU=/CN=root"

Next, we’ll create a client certificate that your MQTT client will authenticate with and a request to be signed by your CA.

How the client certificate is handled is dependent on your MQTT broker. This article will use Solace’s PubSub+ Broker which, in addition to authenticating the client, can use the common name (CN) or subject alternative name (SAN) to identify the client (see Solace’s documentation for more details). By default, the CN will map to a client username. In this article we’re using the CN “demo-client”.

openssl req -nodes -new -newkey rsa:4096 -keyout client.key -out client.csr -subj "/C=/ST=/L=/O=/OU=/CN=demo-client"

Finally, we’ll sign this certificate with our CA.

openssl x509 -req -in client.csr -CA root.pem -CAkey root.key -CAcreateserial -out client.pem -days 1825 -sha256

We should have all the files we need to continue. In particular, we’ll need the following files:

root.pem
client.pem
client.key

We now need to setup the MQTT broker. In PubSub+ Cloud, you require either a Developer or Starter service to take advantage of client certificate authentication.

The client certificate will be sent to the MQTT broker as part of the TLS-handshake. The validity of the client certificate will be determined based on the broker’s implementation. For the Solace PubSub+ Broker, this is done by uploading the CA’s certificate stored in root.pem.

In your service’s detail page, click on Manage, and then Authentication.

Under the Service Authentication tab, enable Client Certificate Authentication, then click Save.

You can also disable Basic Authentication like in the screenshot.

Back in the Manage tab, click on Certificate Authorities .

Then, click on the + Add New button. Choose a name, and paste the contents of your root.pem then click Submit .

Next, a client username must be created using the CN of the client. Under Manage in your service’s detail page, click on Access Control.

You will be brought directly to the Solace PubSub+ Broker’s management interface. Click on the Client Usernames tab and click on the + Client Username button.

Enter “demo-client” as the client username. Click on Enable then Apply.

Your Solace PubSub+ Broker is now configured to authenticate clients using certificates signed by your CA.

We can now finally setup our client and send and receive a message.

We’ll be using Node.js and MQTT.js to subscribe to a topic, send a message to that topic, and print out the message we receive via the subscription.

Create a file named demo.js in the same directory as the certificate files with the following contents:

**const** mqtt = require('mqtt');
**const** fs = require('fs');

**const** client = mqtt.connect('<HOST>', {
  key: fs.readFileSync('client.key'),
  cert: fs.readFileSync('client.pem'),
});

client.on('connect', () => {
  client.subscribe('demotopic');
  client.publish('demotopic', 'Hello mqtt');
});

client.on('message', (topic, message) => {
  console.log(`Received message: ${message.toString()}`);
  client.end();
});

Replacing with the secured MQTT host of the broker (remember that the client certificate is sent as part of the TLS handshake, so insecure connections will not work for client certificate authentication). In Solace PubSub+ Cloud, in your service’s detail page, under the Connect tab, you can use the WebSocket Secured Host under the MQTT section.

Then run npm install mqtt in this directory. Finally, publish and receive a message with node demo.js. You should see the following output:

Received message: Hello mqtt

We now have the foundations of an IoT solution. There is more to work on from here, including how to provision, manage, and revoke the client certificates. Although this example used OpenSSL, you may want to look into more sophisticated solutions like CloudFlare’s CFSSL, Vault, or something managed from your favourite cloud provider.

As a bonus, Solace PubSub+ uses the same client certificate authentication for other protocols, so you can start using the same client certificate to authenticate with SMF, AMQP, and others. See the official documentation for more details.

I hope this helps you on your journey to connecting the world, one device at a time.

Elasticsearch‘s Java QueryBuilder

Julian Setiawan — Wed, 31 Oct 2018 18:11:51 +0000

In Solace PubSub+ Cloud, we began storing metrics early on in anticipation for accounting and billing. The problem was that we weren’t quite sure which metrics would be used nor what sort of queries would be needed to support our accounting and billing needs.

We chose Elasticsearch for storage as we trusted its powerful search capabilities and scalability. However, one aspect that we grossly undervalued was its fantastic Java API. Although it is generally a facade for Elasticsearch’s REST API, a particularly clever feature has been helping us build our metrics microservice with great velocity and flexibility without compromising robustness.

When we first started using Elasticsearch, we built queries in a pretty straightforward way:

BoolQueryBuilder()
 .must(QueryBuilders.termQuery("metricName", "Host"))
 .must(QueryBuilders.termQuery("metricType", "DiskSpace"))
 .must(QueryBuilders.termQuery("organizationId", organizationId))
 .must(QueryBuilders.rangeQuery("startTime").gte(startTime))
 .must(QueryBuilders.rangeQuery("endTime").lte(endTime));

We eventually realized that we usually had to tack on an organization’s ID and some time range to the query so we abstracted that out and just required the metric-specific part of the query to be given.

This worked at first, but we didn’t want to have to edit code every time we needed to calculate a new metric or slightly change an existing one. This is when we discovered Elasticsearch’s Wrapper Query.

On the surface, this is simple functionality where you can feed the QueryBuilder object a JSON string. Something like this:

{
 "bool" : {
   "must" : [
     { "terms" : { "metricName" : ["Host"] } },
     { "terms" : { "metricType" : ["DiskSpace"] } }
   ]
 }
}

Which you feed into the QueryBuilder like this:

QueryBuilders.wrapperQuery(json);

The next question is how to start augmenting the query to search across organization IDs and time periods. A gut reaction could be to add a token somewhere in the JSON string to be replaced, but this is where the Elasticsearch API shines.

You may have noticed that the Wrapper Query is just another QueryBuilder, which means you get back a builder on which you can simply add more parameters to. This let us re-use most of our abstractions of dealing with adding organization ID and time periods to our metric queries:

BoolQueryBuilder()
 .must(QueryBuilders.wrapperQuery(json))
 .must(QueryBuilders.termQuery("organizationId", organizationId))
 .must(QueryBuilders.rangeQuery("startTime").gte(startTime))
 .must(QueryBuilders.rangeQuery("endTime").lte(endTime));

And with this, we had our solution. We were able to churn out new Elasticsearch queries easily or update existing ones without any code changes while re-using our well-tested abstractions for specifying well-known search parameters. Another awesome benefit was being able to directly use our JSON files as queries to Elasticsearch’s REST API for easier testing and validation.

What do you think of this solution? Are there any other Elasticsearch API features we should have used instead? We are still learning and love hearing about new features and use cases.