DEV Community: Qlik Branch

Ski Simulators, Qlik Core and Real-Time Analytics — a Qonnections Story

Rey Riel — Wed, 29 May 2019 17:56:44 +0000

Ski Simulators, Qlik Core and Real-Time Analytics — a Qonnections Story

Qlik Core, React and a whole bunch of open source. Read about the fun I had developing an awesome app to go with some cool hardware.

Myself on the super fun SkyTechSport Ski Simulator

Another Qonnections has come and gone, and this year I got to be part of something really fun. Our keynote speaker for the conference was Lindsey Vonn, the US alpine ski racer with 3 olympic medals and 7 world cup medals. Because of this Qlik wanted to do something really cool and I had Adam Mayer — a Senior Manager here at Qlik for Technical Product Marketing — approach me to lead the development portion of this exciting project.

Myself and Lindsey Vonn at Qonnections

To get this job done Qlik teamed up with SkyTechSport, a badass company that makes killer equipment to help athletes stay on top of their game. The plan was simple: SkyTechSport would provide the super cool Ski Simulator for our attendees to ride and the people to maintain it, do a bit of development on their end to get us access to the data points the simulator generates and we would build some awesome data visualization to go around it. Our implementation would include both a real-time in game dashboard as well as a post game leaderboard to track who was topping the list. All of this would encompass a charitable effort where Qlik would donate $1 to the Special Olympics for every gate that was passed in a successful run. I was to be in charge of the real-time app and the amazing Arturo Munoz would handle the leaderboard. Some great development ahead for sure, but challenges immediately started to present themselves.

Source Code for project: https://github.com/Qlik-Branch/qonnections-ski-simulator

The first challenge that needed to be dealt with was how the simulator was passing the data. The simulator is a fast piece of equipment and the software behind it is built for the visual and physical feedback, so all the data happens in milliseconds. 30 milliseconds to be exact. So the simulator is saving the data to one file every 30 milliseconds. Over a network. And not just saving the data, overwriting the data. This brought up two concerns.

First is that we needed to make sure the network our systems were connected on weren’t going to be bogged down by external influences. Simple enough, we just have a dedicated router with the systems hard-wired to it and problem solved.

The second concern required a little more thinking and some serious testing. We wanted to make sure we got all the data. That meant getting every write of data with this 30 millisecond timeframe with no file lock issues. After a while of trying to figure out if both writing and reading a file over a network within 30 milliseconds was even feasible I decided to come up with a solution that would simply eliminate our restriction: move the file. If we could move the file out of the way before the simulator had a chance to overwrite it we could work with the data in our own time. The result was actually a really simple script that would just constantly try to move this file to a different folder with the file being named with a timestamp:

First gate passed. YAY!. The next thing to figure out was where the data was going and how it was going to get there. The answer? The awesome Qlik Core mixed with R&D’s super cool command line tool corectl. By having Docker Desktop installed on the system we used I could write three files and have the entire back end setup done. The first file is the docker-compose.yml file that will tell docker the engine we want set up:

The above file tells docker we want to use the latest (at the time of writing) qlikcore/engine image, accept the End User License Agreement, store our Qlik apps in a /docs directory (which is mounted to a local core-docs directory) and route the standard engine port 9076 to our local port 19076 . We’re also mounting a local data directory as well for when we want to load data. Once we have this file we can run docker-compose up -d and docker will have our engine running in no time.

The second file we need is a file called corectl.yml which is leverage by corectl:

This file tells corectl everything it needs to know to create the Qlik app we want. It points to the engine, indicates the name of the app we want, a connection to the data folder we need and a path to the load script that will take in the data necessary.

The final file necessary is our load script that we reference in the file above:

The key thing to note in the load script above is the ADD keyword in the second block. This allows us to leverage the partial data load feature of the engine meaning we could load new data in quickly without losing the data already in the app, keeping our round trip from data load to front end output quick. So with the load script and the corectl file I could run corectl build and have our Qlik app up and ready to go.

Now with the app up and the data being saved from oblivion I turned to the script that would actually handle the simulators data. Using enigma.js for engine interaction we first wanted to create a generic object for the attendees badge ID as well as the race ID. That way we could subscribe to the object and keep an eye on it to know when a badge was scanned:

When a badge was scanned on the front end it would update this generic object and our script can start looking for new race files. Once the race has started it was a simple loop that loads in any existing data files, saves this data to the /unprocessed/ski-data.csv file referenced in the load script and tell the engine to do a partial reload:

finally we can look through the current data to see if a finishing status is found and if so we can clear out the generic object and stop looking for files:

Once we have our data loading script running and waiting, it’s time to get the front end in place. This front end ended up being a React app designed by Arturo, built by myself and incorporates enigma.js, d3.js, picasso.js and Qlik GeoAnalytics. There’s a bunch of parts involved in it, but the important bits are that we set the generic object when a badge is scanned and create some hypercubes that will update when the partial reload happens.

With all the pieces put together it was time to do some serious testing. The upside to the way the simulator saves data is that it was incredibly easy to simulate. I just needed to write new file every 30 milliseconds and watch all the scripts do the rest.

The one concern I had through the whole thing was the speed. This was meant to be an in-game dashboard, meaning it had to update quickly and there were a lot of moving parts. The simulator saves the data, the rename script moves the data, the data load script reads and writes the data, the engine reloads the data, recalculates the data to send down to the front and sends it, then the front end re-renders with the new data. I wasn’t expecting to be blown away by the entire round trip taking under 400 milliseconds! With metric in place to measure how long the engine was taking we had 200 millisecond partial reloads happening within that time too. It’s exciting to see Qlik’s engine be put to the test in a real-time use case and come out shining.

In the end we had a great attraction in the Expo that showed off the awesome power of Qlik and Qlik Core. We raised a significant donation for the Special Olympics and generated a ton of excitement throughout the week.

I wanted to give a big shout out to everybody I worked with both developing and staffing the booth. Katie Abbott and Mike Marolda killed it with logistics and helping day of, Adam Mayer was fantastic with all the organization and Arturo Munoz was a design wizard, thanks to all for making this such a success.

Qlik Core for Developers: Lessons Learned in Workshop Creation

Rey Riel — Tue, 12 Mar 2019 15:29:57 +0000

Another ForwardJS has come and gone in San Francisco and as usual, I had a blast while I was there. This time around, I was representing Qlikwith a sponsored workshop, so it was my first crack at getting Qlik Core into the hands of fresh-faced developers new to the engine — and here’s some things I learned.

An example of data retrieval, presented at ForwardJS San Francisco, January 2019

Don’t make assumptions about what tech your developers may know

Qlik Core is a containerized version of our engine, so we have a Docker image on Docker Hub for it.

I made the silly mistake of assuming that developers would know about Dockerand the whole concept of containerization, when half the developers at the workshop was unaware of this awesome technology.

When building out a workshop you should always have time in the beginning to explain the technologies you’re working with an assume the developer has no knowledge of what those technologies are or how they work.

Qlik Core — Dockerized Engine

Test your workshop setup on multiple operating systems

To show my workshop attendees the apps files that were generated when creating a new app in Qlik, I created a volume for the Docker image that was a directory in the main project folder.

While this setup worked perfectly on my MacOS, the same Dockerfile caused problems for my attendees who used Windows, particularly the ones who were using Docker for the first time.

There was some sort of permissions or sharing issue that left other attendees sitting around while I tried to help these developers get their Docker setup working correctly.

If I had run through the workshop on a Windows machine, I may have been able to pick up on this beforehand instead of wasting valuable time. Always make sure to test your setup in multiple environments so there aren’t any surprises.

Give your attendees examples of what they’re working towards

One of the things I did that I feel was key to a smooth running workshop is to have codebases for each of the steps in my workshop. The code for my workshop project sits in a GitHubrepository and I have a separate branch for each of the learning points that I have.

There will inevitably be points in your workshop where people will have issues getting the project to do what they need it to and you’ll have to move on. Without having codebases to get them to the next point, your attendees will be separated from the pack and not have a good experience.

Creating branches will allow users to simply update their project to the next step so they can keep up with the rest of the class.

An example of set up infrastructure

Simplicity is key, advanced topics are for the back pocket

With a powerful engine such as Qlik Core there’s so much that can be done, but the last thing developers want to delve into on day one with a piece of tech is the nitty gritty stuff. Luckily the basics of Qlik Core are super simple and easy to pick up, with a bunch of tools created in house to make developers lives easier.

When building out a workshop to introduce developers to a technology, don’t even bother bringing up the complicated stuff. Trust that the developers that want to know the really advanced stuff are going to ask you for it, and that’s a great thing to leave for the end or talk about after the workshop.

Get the basics, build the foundation, and the developer will let you know when they want to start adding the bells and whistles.

Getting Started with Qlik Core

Wuzhong — Mon, 29 Oct 2018 11:56:59 +0000

A few months back, Qlik announced Qlik Core, a Dockerized version of Qlik’s associative data engine. After playing with Core for a few weeks (the developer version is free…plus, I work for Qlik), I’ve come to realize how awesome the product is for visualizing and analyzing data. So I’m writing this blog to explain, in my humble opinion, who will benefit from Core, and how to start using it.

First of all, why Qlik Core?

Nowadays, there are several visualization libraries, D3 probably being the most widely used library of the bunch. Click here to see a fully baked example of an interactive visualization built on top of D3. For an application like this, static data is packed and loaded together with your application. But there are a few limitations with this approach, especially when the size data is large (for example, greater than 200 MB)…

The app takes a lifetime to load.
The browser occupies a large amount of memory, which can even cause the browser to crash if the size of data exceeds the limit.
It’s not easy to handle filtering and association.
It can be difficult to manage state as the data is explored and filtered.
It’s difficult to handle dynamic data.

One alternative is to query a database, requesting and receiving data as we go, but this also has limitations. The main drawback is the query latency that the user may experience. Usually, a query based system takes 0.5–3 seconds to respond to user action, which diminishes the user experience and purpose of an interactive dashboard. 😟

Qlik Core to the rescue — but what is Qlik Core?

Irrelevant Paw Patrol picture (I’m a new parent), from here.

Qlik Core gives you direct access to the first data engine of its kind. And “data engine” is neither a data warehouse nor a database. It is a mechanism to handle data association, analytics and state, especially in large volumes of data.

And when you have this data engine at the 🧡 of an application, you don’t need to send a query to receive the data. Even pulling from different sources, Qlik Core can load the data into memory, process and index it, then power web applications reactively. The data becomes an in-memory instance and your web app now connects to Qlik Core via a WebSocket— all calculations and associations are handled remotely.

So, as mentioned, there are many advantages to using Qlik Core, especially if the data volume is large…

The application data resides in a server instead of being passed to the frontend. This not only increases usability from the viewer’s perspective, but it also reduces the load of the server, causing both frontend and backend performance to improve significantly.
The developer doesn’t need to worry about wiring up filtering and association through their code. All of this, as well as state management, is handled by Qlik Core now and the app’s visualizations and interactivity can simply hook into the engine.
On top of all this, Qlik Core is great at loading data from different data sources and meshing them into one.

So, we’re killing several birds with one stone as they say. And now you know why I think this product is amazing (😍 → that’s me building with Qlik Core and I swear it’s not just because I work at Qlik).

Ready to get started?

Hopefully, you're nodding like this.

Qlik Core is pretty new so getting started can be somewhat tricky (we’re working on improving this but in the meantime, please give us feedback). The official documentation for Qlik Core is at https://core.qlik.com/. There you’ll find this tutorial which I find pretty straightforward. Following the instructions will get your Qlik Core instance running quickly, especially if you’re familiar with Docker. There is also a tutorial video created by Qlik Branch (more to come but this our home for all that is developer and Qlik) which goes through the basics.

But what is after this?

Let me show you with some simple examples…

I have built some examples to illustrate how Qlik Core is used. Basically, to use Qlik Core, you need enigma.js (one of Qlik’s open source libraries) to easily communicate with the engine using JavaScript. The only way to talk to Qlik Core is via our Engine API which is a language-agnostic, WebSocket-based API. Enigma.js is simply a wrapper library for this.

So, here are some examples so you can better understand how Qlik Core works…

*️⃣ Hello engine example

*️⃣ Hello data example

*️⃣ Check if a Core instance is healthy

*️⃣ View data model in an app

*️⃣ Upload a pre-built app to Qlik Core

Hopefully, this is enough to get you started learning about and using Qlik Core. Over time, we’ll be adding more tutorials, tips and tricks to help developers not only get started with Qlik Core but also with the APIs themselves.

So keep an eye for that on Knowledge section of Qlik Branch. In the meantime, check out the tutorials there explaining more about our associative data engine, how it works and how to work with it. And if you are really interested in Qlik Core or need help, let me know. Just leave a comment below or join our Qlik Branch Slack (see the #qlik-core channel or DM me [at]wu_zh).

Cheers, happy coding and happy Halloween everyone.

— Wu

QlikHacks 2018 — An Ottawa Hackathon

Rey Riel — Tue, 02 Oct 2018 20:11:20 +0000

QlikHacks 2018 — An Ottawa Hackathon

Me at the deCODE Hackathon in Ottawa

“Hey, I saw you do a talk at Ottawa JS. You work for that Queue-Lick company, right?”

This is pretty standard for me in Ottawa and I’m definitely not the only one. Qlik has a ton of great employees working in Kanata but very few people in Ottawa know who Qlik is, what Qlik does or even that it’s pronounced “click.”

We wanted to do something to fix that while connecting the Ottawa developer and designer community AND making the world a better place.

Our solution?

A HACKATHON at our Qlik Ottawa office!!! From Friday, October 19th to Sunday, October 21st we are hosting developers and designers of Ottawa for a hackathon that’ll be a ton of fun. 🎉 And rest assured when we say developers and designers, we mean ALL developers and designers are invited.

Regardless of your skill set, whether you’re a student or a professional, we’d like you to get involved. You don’t need to know Qlik or Qlik Branch, you don’t need to know JavaScript, you can come in with any skills and have a great time.

I won’t go into too much detail right now, but at Qlik we take our Corporate Social Responsibility (CSR) very seriously. Because of this, we’re partnering with the United Nations to get the most out of this hackathon and make the 🌍 a better place.

Are you in? We hope so. Here’s the schedule of events…

Friday, October 19

We’ll kick off the hackathon in the evening with a fun meet and greet. We’ll have food, give a couple of talks on who we are and what to expect during the hackathon, then music and mingling so you can meet your fellow hackathon participants and potential teammates.

Saturday, October 20

At 9 AM, we’ll begin with a workshop detailing some necessities and then the hard work begins. Breakfast, lunch, dinner, snacks, refreshments and some entertainment will all be provided by us. We’ll give you the whole day and will have mentors on hand to help you with any challenges you may come across. We’ll close up shop around 11 PM to give you time to head home and rest up for the next day.

Sunday, October 21

Starting at 9 AM, you’ll have the opportunity to finish up your work before you’ll present your results and let the judges decide which team will take home the prize. Yup, that’s right, we even have a prize. 🏆

We’re super excited to be doing this and hope you’re excited to join. To sign up for this awesome event, please visit http://bit.ly/QlikHack18. If you have any questions at all you can contact me at rie@qlik.com. See you there!

Qlik ❤ D3

Alexander Karlsson — Mon, 17 Sep 2018 14:35:32 +0000

Chances are that if you have seen a piece of data visualization on the web you have come across D3. It’s heavily used by organizations like The New York Times to produce wonderful graphics that engage their readers and enhances their stories.

TL;DR Together with Qlik you can create very lean data visualizations using D3, letting D3 bind the data to the DOM and draw visualizations. Letting Qlik perform the heavy lifting in terms of data manipulation and a reactive data flow, abstraction away data state management from D3.

Sample project: https://github.com/mindspank/d3-blogpost-example

Ironically, while you can create wonderful data visualizations using D3 it is not a data visualization library. It has no bar chart or line chart methods. Instead it focuses on binding data to the DOM and performing manipulations of the document based on data. The data binding paired with additional components such as layout algorithms and components such as scales and axis makes it a perfect library for general purpose data visualization.

Where D3 suffers is when you have many visualizations on the page that you want to keep in sync, manipulating larger data sets and having to fetch the full data sets to calculate a proper scale.

// Looks familiar?
d3.scaleLinear()
    .range([height, 0])
    .domain([0, d3.max(data, function(d) { return d.value; })]);

Qlik to the rescue!

Luckily the Qlik Associative Engine is a blazingly fast in-memory computation engine that provides tons of useful metadata around your data geared towards building visualization components.

By letting Qlik take care of the data computation you can omit heavy data wrangling on the client and push those calculations onto the server, leaving you with very lean front-end code that focuses on what D3 does best; binding data to the DOM and outputting a visualization representation of that data.

To cherry pick a few nuggets that Qlik gives you out of the box:

Minimum and maximum values for calculations for scale inputs
Glyph counts for the longest dimensional value for label widths
Data reduction for “glanceable” visualizations
Server-side configurable sorting and formatting

Most of the points above D3 handles quite fine, however for larger datasets it’s quite taxing on the browser to crunch all those numbers.

However, for me personally, the biggest win is that I can write code that is readable, neat and doesn’t mutate my data.

System Integration with Qlik’s APIs — Why and How

Wuzhong — Thu, 30 Aug 2018 03:35:57 +0000

First of all, this article is not a “how to” tutorial but rather a white paper. It is NOT MEANT FOR TECHNICAL USERS. If you understand the value of APIs and how to use Qlik’s APIs, this is not for you.

____________________Qlik Pro’s Stop Here______________________

“What are Qlik’s APIs?” 🤔

“Why would someone want to use an API?” 🤔

“What can you build using Qlik’s APIs.” 🤔

If you have these questions, you may find this article helpful. This blog is about various types of system integration using Qlik Sense APIs.

So let’s dive in. With APIs, you can do amazing things. But wait, what is an API?

Behind every click, various events are triggered on the back-end.

The APIs are the bridge between the front and the back.

Usually, user interactions are restricted to front-end elements: buttons, charts, filters, etc.

So when a piece of software implements APIs, it means interactions with the back-end are no longer limited to the software’s native front-end, which opens many possibilities.

Now that you know what an API is…what can you build with Qlik Sense APIs?

APIs have always been an important and unique part of Qlik. They are the building blocks of mashups and extensions. If you don’t know about those, below is a brief explanation:

But since this article is about system integration, we will focus on mashups.

Here is a random fact: you can build a mashup with extensions. That way, technically you put the visualization you put into Qlik system into your own system.

Mashup Lv.1 — build a webpage of iframes

This is probably the simplest kind of system integration. Every chart or table in Qlik Sense has an ID and an URL, which can be found in Qlik Sense DevHub -> single configurator portal.

If you copy the URL into your browser, you can see the object on the page. A single line of code extracted from the DevHub can, therefore, be added into an existing website to embed a Qlik chart to a web page. Put multiple iframes onto an html file and you have a dashboard.

So in less than 1 minute, you can build a web page with four lines of code copied from DevHub. 👍

Required skill level: ★☆☆☆☆

Mashup Lv.2 — mashup with APIs and libraries

See how it is implemented in the [Walkers Dashboard](http://playground.qlik.com/liveshowcase/playground-zombie/main.html)

The Walkers Dashboard is an example on Qlik Playground created by Francis Kabinoff which uses Qlik charts within a web page showing stats on the tv show The Walking Dead. It is a typical mashup built with the Qlik Sense capability APIs. Different from an iframe, charts are rendered using a Qlik javascript library to control the user interaction and object retrieval. This means faster and cleaner execution, customizable elements and other features.

There is an easy way to create and host your own mashup too. In Qlik Sense DevHub, you can find a tool called Mashup Editor. It allows users to design their own web application by dragging and dropping Qlik objects onto predefined templates. But if you want to build something different, like The Walker Dashboard, you need to learn javascript and html.

Required skill level: ★★☆☆☆

Mashup Lv.3 — building a Qlik dashboard which looks completely non-related to Qlik

See how it is implemented in theMaple Leafs Dashboard

Qlik Sense is business intelligence software, but not only business intelligence software. By using APIs, users may create their own visualizations, on top of a Qlik application, without using charts from Qlik. In this use case, Qlik is no longer a visualization server. It is a data warehouse plus data engine (Qlik’s Associative Engine) behind your dashboard. It handles the data processing, storage and association part but lets third-party technologies take care of the presentation part.

If you are doing this, you are entitled to call yourself a Qlik Pro. 😁 The Qlik Sense DevHub Mashup Editor won’t help you at this level. You have to write every line of front-end code by yourself.

Required skill level: ★★★☆☆

Mashup Lv.4 — using enigma.js and engine API

Most Qlik developers do not use enigma.js and the engine API. For many, the capability API is enough for most use cases.

Enigma.js is a wrapper library on top of the engine API. While the capability APIs are run from a user’s browser, enigma.js can be run on an application server to interact with Qlik before serving a web page to a user. In fact, the capability API also implements enigma.js inside of an Angular application packed using require.js.

The engine API is the communication protocol between the Qlik Sense server and the Qlik Sense client. The Engine API Explorer is a tool in DevHub where you can learn how the front-end talks to the back-end. Unless you are building your own support library, Enigma.js should cover most of your integration needs.

Required skill level: ★★★★☆

Mashup(?) Lv.MAX — leveraging on the engine itself

When you know the technology inside out, you may want to use the data engine (aka Qlik’s Associative Engine) outside of the business intelligence space. Examples speak for themselves, check them out...

See how it is implemented in theBoston Building Emissions By Axis Group

C40 cities building energy by Slalom

Qlik Core

Then there are some use cases that are difficult to define — they are certainly not mashups, some of them are not even web applications.

For example, a business intelligence chatbot…

Qlik BotNet

And a mirror…
// Detect dark theme var iframe = document.getElementById('tweet-981633788244647937-571'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=981633788244647937&theme=dark" }

And many people don’t know this —there is a Qlik application behind branch.qlik.com. The application handles search and content management specifically. Qlik Branch is an open source project, you can find the code here.

Qlik Branch

Required skill level: ★★★★★

So, how should you get started?

Just start experimenting! There are resources and examples on Qlik Playground, open source examples on Qlik Branch and plenty of community members on our Qlik Branch Slack. You can also find blogs and Youtube videos about Qlik’s APIs almost effortlessly online.
Let me know your thoughts.
Happy coding. And shout out to all of the amazing developers out there.
Cheers.
— Wu

RHoK: Hacking Towards a Better Ottawa

Rey Riel — Fri, 13 Jul 2018 12:15:07 +0000

Photo Credit: https://rhok.ca/events/rhok-8/

Hardly a week goes by where I get to think of how awesome my job is.

The second weekend of April had me back at the Adobe conference room in Ottawa where just a week before, ForwardJS Ottawa bought twenty awesome talks to the Nation’s capital.

This time, it was for Random Hacks of Kindness (RHoK), a twice a year hackathon where organizations submit projects in need and participants apply their skills to hack out a solution.

Project one involved Growing Futures, an organization committed to bettering the next generation by using hydroponics to teach our youth about good food and good business. They were seeking a way to remotely manage the growing system of hydroponic stacks around the city. The resulting solution was a fantastic mix of hardware, software and data visualization.

Project two was submitted by the Gifts in Kind program, which takes on the massive and highly important task of connecting donors with gift in-kind donations to non-profit organizations in Ottawa. With a limited operating budget, the program needs help capturing usage data of its donations and a smarter way to let donors and recipients connect to the program. The RHoK team for this project did a great job laying the groundwork to an intuitive system that will connect donors and recipients to the program with greater ease.

Project three saw codemyrobot.ca seeking help. An awesome program that provides free robots to school libraries while giving students access to some robot coding fun, the current submission process for their robot challenges is highly manual and server intensive. The team tasked with fixing the problem came up with a great solution to not only automate the submission process, but also keep the safety and privacy of students as top priority while ensuring the system could scale with the barrage of videos coming their way.

Project four was submitted to help iSisters educate women in need. The iSisters organization was created to mentor women in need at no cost to them. Unfortunately the technology its currently built on is outdated and needs major modernization. Their team did a great job of revamping the site and giving it a crisp facelift and more intuitive UI.

Project five was about an undertaking by Fair Vote Canada to try and make every Canadians vote more equal. The organization was looking for software to allow its canvassers an easier time tracking and managing canvassing areas. The resulting project will surely help Fair Vote Canada take great leaps towards achieving its goal.

The final project (which I was a part of) was submitted by the Carlington Community Health Centre to breathe life into an Ottawa Bad Date List that would help the sex-trade workers in the city have a safer working environment. The organization was looking for an app that could be used by phone or desktop that would allow workers to not only see vital information to clients that were posing harm to the trade, but submit their own information quickly to ensure the safety of their fellow workers. The end product in the hackathon was a great step towards achieving that goal.

The final presentations of the hackathon were live-streamed here (audio was missing for the first 13 minutes but they caught all the presentations), but the amazing thing that comes out of this is that these projects never seem to end when the weekend does.

Participants of the RHoK hackathons seem to have a habit of continuing on to make sure these projects reach a conclusion and it’ll be exciting to see the results of all the hard work in the real world.

Photo Credit: https://rhok.ca/events/rhok-8/

The ForwardJS Battle — Part 2: Ottawa

Rey Riel — Fri, 01 Jun 2018 12:56:32 +0000

With ForwardJS San Francisco behind me, I turned my sights on bringing Forward home with me to Ottawa for the second year in a row.

Here I am taking ForwardJS Ottawa attendees through Halyard.js.

ForwardJS Ottawa in 2017 was a cool, calm introduction to the Canadian capital. But this year, we wanted to go bigger…and better.

The only way to do that was to pack the conference with great workshops and two days of stellar speakers, and boy did we deliver.

Day 1 saw ForwardJS jump out the gate in full force. With Andy Mockler and Kristin Spencer starting the show schooling us on soft skills, the day was filled with awesome talks like Adam Daw touting the power of transpilation, Jan C. Liz-Fonts showing how to harness Blockchain in Javascript and Mary Snow decoding the NodeJS Event Loop.

That first day brought so much great knowledge — so the speakers on day 2 really had their work cut out for them. And they were more than up to the task. The day saw Jenn Voss bring us Tales from the QA Crypt, Eric Adamski’s amazing energy with Rx and Async, Ksenia Coulter teaching us to Get the Most out of Code Reviews and Brian Tavares closing out with a fantastic React Native talk.

ForwardJS donuts, anyone?

I’m more than willing to admit bias and won’t deny it could be in part to my involvement in organizing or the fact it was in my hometown, but in the battle of San Francisco vs. Ottawa I’d say the “north of the border” contender took the ForwardJS crown this year.

The ForwardJS Battle — Part 1: San Fran

Rey Riel — Fri, 11 May 2018 12:54:45 +0000

February of 2018 saw me leaving the cold winter of Ottawa to bathe in the mild warmth of San Francisco. This wasn’t a vacation. I was tasked with donning the Qlik cap and representing our developer relations team with a talk. With five days full of workshops — I couldn’t just fly in, speak and fly out. I had to take advantage.

Donning my Qlik cap while wear a ForwardJS hat

Day 1 of ForwardJS had me digging into WebAPIs with Aysegul Yonet. A tiny class with fewer than ten attendees meant we could get a lot done quickly and have some hacking fun of our own. While Aysegul covered the basics of a bunch of the different options now available to us devs in the browser, the funnest part was coding a script to see exactly how much of my hard drive IndexedDB would allow a site to eat up. Not very often a workshop has me routing for a script to win against my machine.

Day 2 for me was the conference itself, which meant a less technical talk on the downfall of this years Ottawa Senators entitled “My Hockey Team Sucks” and a day of chilling at the Qlik table with Ana Nennig pimping our awesome Branch swag and talking about the awesome power of the QIX engine.

// Detect dark theme var iframe = document.getElementById('tweet-963582133112520704-383'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=963582133112520704&theme=dark" }

Day 3 brought me Brian Holt and his React + Redux workshop. I was really impressed with how Brian started with the basics, making code each element using React.createElement to see how the process works. A little while later (after getting over that icky feeling in my stomach that came from putting HTML in Javascript) we were introduced to the wonderful world of Redux, reducers and Redux middleware. This was a great workshop that Brian has brought around the world (which he tracks in the repo), so if it’s coming to a conference near you I suggest grabbing a seat.

Last day of the conference had me sitting in on Webpack 101 with Freddy Rangel. Starting at the basics of initialization and running all the way through tree-shaking, Freddy did a great job of helping me de-mystify one of those things I usually leave to the Angular CLI to figure out. It’s also always nice when a teacher introduces a new teaching technique and Freddy’s approach of having a separate GitHub branch for each module made keep pace a breeze for everyone in the workshop.

Overall I’d say the San Fran ForwardJS was a pretty great success and I knew I’d have to work extra hard with Dave Nugent to make sure its Canadian counterpart lived up to the Forward reputation.

deCODE Hackathon + Qlik = Awesome Results

Rey Riel — Fri, 10 Nov 2017 14:39:59 +0000

The second weekend of October myself and a band of Qlik developers teamed up with some of the top students in Ottawa for this year’s deCODE Hackathon.

The Challenge

But what challenge should our team tackle? After some discussion with Qlik’s corporate social responsibility (CSR) team, we decided the coolest thing for our team to work on would be the Data Competition that the United Nations had at The Oceans Conference in June. As part of this competition, “participants were encouraged to use their imagination to produce visualizations that highlight insights from all the 1,380 voluntary commitments available in The Ocean Conference Registry of Voluntary Commitments.” The overall goal? #SAVEOUROCEAN.

The Work

With our challenge selected, it was time to get to work. We gave the students access to the data with Qlik Playground, taught them Qlik’s APIs and let them dive in…

And boy did they impress us. The first thing I was impressed by was how quickly the students not only learned the Qlik APIs, but how quickly they realized the power of it.

When they learned that they didn’t have to bother with keeping the state of their selections, but simply make a selection call and let the Qlik Associative Engine handle all the data updating — they were blown away. They definitely put that power to use…and quickly in true hackathon style.

The Awesome Results

Over the span of just 22 hours the team brought together an awesome page that not only showed off the data, but allowed the individual user to find the answers themselves by filtering based on country, company, target —pretty much whatever they wanted.

This project even recently made a United Nation’s Ocean Action Newsletter (under Updates from Voluntary Commitments). We’re so incredibly proud of this team, especially the hardworking and eager students. This is well-deserved.

So where is this awesome visualization? Check out the Qlik Playground Showcase and view the “UN Our Oceans Challenge” project.

And how can you help us with this important initiative? If you want to get involved or play with the code yourself, the project is open sourced on GitHub as well.

Enjoy! And let us know what you think in the comments.

From Custom Blogs to Medium: Our Transition (Part 2)

Rey Riel — Wed, 08 Mar 2017 14:39:12 +0000

It didn’t take long to realize that it wasn’t just the APIs, but the RSS feed itself was limited. After publishing a bunch of stories to our publication, we realized that stories were being removed from the Sense search. Would you like to guess why? The RSS feed ONLY GIVES THE LATEST 10 STORIES!!! There’s a lesson in load testing for you.

I was now stuck with two problems:

Problem 1: How to properly keep the app up to date

I decided to take another look at possible solutions. My brain wouldn’t stop iterating “there’s got to be a better way than this.”

After doing some more digging and searching, I learned about PubSubHubbub. Yes, that’s really the name. And you would assume that is some sort of company or product name, but it’s actually a protocol that’s pretty similar to WebHooks.

Basically you subscribe to a PubSubHubbub feed giving a callback url. The PubSubHubbub provider then sends a POST request to the callback url to confirm the subscription.

Once this process is done, the provider will send a GET request to the callback URL any time new content is published to it. Sounds perfect!

This time I was skeptical however. APIs didn’t help. RSS was limiting. Surely there was going to be something wrong. My assumption proved correct after doing a test run of the process and learning that the GET request from Superfeedr (Medium’s PubSubHubbub provider) didn’t actually have the content of the article, making this useless.

Having learned this, I decided the best thing to do was to stick with the RSS feed at this point and simply remove the code that deleted stories from the db.

Problem 2: How to get the missing stories back

Since the RSS only shows the last 10 stories and the PubSubHubbub method is for future stories, I was left with two possible options. The first was to manually enter the information into the database, which as a coder would go against every impulse in my brain.

So the only option left is writing a script to download the data through HTTP requests. That involved doing the following:

Find out where Medium lists stories for a publication (https://medium.com///)
Analyze the HTML to determine how to pull out the link for each story
Analyze the HTML of the story page to determine how to best get the data from it

1 and 2 were pretty easy by themselves and Medium was very helpful with the third by providing most of the data we needed in meta tags. As for the content itself, it was simply a matter of finding the right div. With all this information, I just needed to find an npm package that would help me parse through the HTML (thank you cheerio) and write the script.

So now we’re done, right?

We could be, or we could probably make things better. There still isn’t any code to remove stories if they have been deleted on Medium and there’s probably a way to use PubSubHubbub to use a subscription process instead of having a script running every once in a while.

If you’d like to have some fun, I encourage you to fork the repo, try to solve these problems and submit a pull request if you do.

From Custom Blogs to Medium: Our Transition (Part 1)

Rey Riel — Thu, 23 Feb 2017 20:20:11 +0000

When I first signed on to the Dev Relations team at Qlik and began working with our main site Qlik Branch we were working with our own custom built blogs section. While this seemed easy enough for us to start with, there were a few problems with this.

Exposure: Our blogs were only available in our site and required existing users or posting links in order for the world to know about them.
Third Party Tools: In order to allow the ability to use formatting in our blogs, we needed to use a third party content editor. When I came in we were using Summernote, then we tried moving to markdown with SimpleMDE. While these options were both fairly robust, any found bugs became an issue and customization was a pain.
Maintenance: We were in charge of the storage of the blogs as well as the process for moderating them.

After much back and forth within the team and discussion with others, we decided that Medium would be the way to go. While this would solve the problems above we had one major hurdle that we needed to figure out.

While we were going use Medium to host our blogs (hereafter referred to as Mediums term “stories”) we still wanted to use the Search capabilities available in the Qlik Engine API. This involved any time a story was published that we pull all the information (including the story’s content) so that the Engine could load it into our Sense App.

The first solution I searched for was using the Medium API. As a developer this seemed like the obvious solution, but I quickly ran into a big problem here. The Medium API itself is fairly limited and doesn’t allow you to either retrieve a list of stories for a publication or retrieve the information for an individual story. Without these abilities the API was pretty much a no go for me.

Now I needed to find any type of source that could list a publications stories. After tons of searching, the only solution I could find was good ole RSS. Each publication has an RSS feed of its stories, and when you set the “RSS Feed” setting to Full on a given publication you get all the content of the story as well. So I wrote a quick script that will ping the RSS feed every X minutes, compare each story in the feed with a quick checksum to see if there are any changes to the stories and clear out the ones that aren’t in the feed (in case stories were removed for some reason). Problem solved!!!!!

Actually, it wasn’t. More to come on that.