DEV Community: Mike Chung

Functional Listening and its Generalization: Towards Having Control Over Your Productivity

Mike Chung — Thu, 10 Aug 2023 23:10:42 +0000

Originally published at https://mjyc.github.io on November 24, 2021.

At some point in my life, I realized that I wasn't listening to music ~~to join the revolution and change the world!~~to enjoy myself but to put my kids to sleep or to focus while I'm working. This sad realization made me write this post.

Sounds

Like many other new parents, I was introduced to white noise sounds when I was struggling to put my son to bed. One day after a sleepless night, I found myself listening to the white noise sounds even when I was working. I felt good and focused.

I started exploring more noise sounds to see if any of them can boost my productivity because, at some point, I felt white noise was too harsh for my ears. Then I found brown noise, which was much warmer and easier on my ears than white noise, and I enjoyed listening to it for awhile. However, after many days of listening to it for a longer period of time, it became too soothing to the point where it made me feel sleepy, especially on Winter days with long nights. So I moved on to listening to pink noise, which felt like a great compromise between white noise and brown noise.

Around the same time, my friend, who is also a dad, suggested I check out fan noise. I didn't initially like it, but it grew on me over time. There was something about listening to slightly more familiar sounds that kept me listening to it despite the initial dislike. This observation led me to explore non-synthetic noise sounds like nature sounds, more specifically, ocean sounds and rain sounds.What I really liked about listening to the nature sounds was that not only were they more comfortable to listen to than synthetic noise sounds like white noise, but they also put me into a certain mental state that had a specific effect on me. For example, listening to the wave noise tricked my brain into putting me into vacation mode, helping me relax, and the heavy rain sounds took me back to the hot and humid monsoon season in South Korea I weirdly enjoyed when I was a kid.

Understanding that listening to certain sounds (or music) can have certain effects on me made me want to leverage such effects. For example, I intentionally listened to my favorite finals week music from my college years, like Nujabes-like lo-fi hip-hop music, Bon Iver-like indie folk music, and Sigur Ros-like post-rock music, when I needed to learn new techniques or tools. When I needed to pump out code, I listened to my favorite coding music from my bachelor years, like Daft Punk's French house music or Chemical Brothers' big beats music. I turned to nature sounds like ocean sounds to help me calm down and detach from problems, especially when I'm responding to an incident under time pressure.

Lights

During the pandemic—specifically in the winter of 2020—I learned how sensitive I was to the lighting conditions in my room. At the time, I was using a single not-so-bright yellow light bulb in my room, and I felt like I couldn't focus because my room wasn't bright enough. So I ordered a 3-in-1 light socket splitter and three smart light bulbs to get my focus back by making my room brighter.

After trying out my new smart light bulbs without any customizations for a few days, I realized that while I really enjoyed having their default bright white lights during the daytime--especially on cloudy days, I didn't like the same bright white lights in the evening hours. They didn't mix well with the other yellow light bulbs we already had. I felt like the new white lights were taking away the cozy warm feeling created by the existing yellow lights. I also felt too awake if I worked under bright white lights in the evening and night hours, it was useful if I needed to work long hours, and it made me tired and difficult to concentrate during the day after.

So I used the IFTTT-like feature in the smart light app to adjust the light bulbs' brightness and color temperature throughout the day. Here is a set of rules I created:

6 am: set brightness to 80%, set color temperature to 50%
7 am: set brightness to 80%, set color temperature to 50%
10 am: set brightness to 100%, set color temperature to 100%
11 am: set brightness to 100%, set color temperature to 100%
1 pm: set brightness to 100%, set color temperature to 100%
2 pm: set brightness to 100%, set color temperature to 100%
4 pm: set brightness to 100%, set color temperature to 75%
5 pm: set brightness to 100%, set color temperature to 75%
8 pm: set brightness to 100%, set color temperature to 50%
9 pm: set brightness to 100%, set color temperature to 50%
12 am: set brightness to 10%, set color temperature to 1%

My smart light bulbs' brightness ranges from 0 (0%) to 800 lumens (100%), and the color temperature ranges from 2700k (0%) to 6500k (100%)[^1].There are redundant rules (in the non-bold font) because there was no easy way to keep a certain condition (e.g., set brightness to 100%) throughout a certain time period (e.g., between 10 am and 4 pm) using the IFTTT that came with the bulb[^2].For example, if I power off my lights at 10 pm and power them back on at 9am the next day, the brightness and color temperature will stay at what they were at 10 pm last night and they won't change until the next rule kicks in, e.g., at 10 am.One of the ways to get around this issue was using the "sun match" feature that continuously adjusts the brightness and color temperatures to match that of the sun throughout the day.However, I liked having a set of rules that changes brightness and color temperature in a bit more discrete manner.

A funny thing about having these rules was that they had a reminder-like effect on me.For example, adjusting the color temperature slightly at 4 pm was like a reminder that I should wrap up and get ready for picking up my son at 5:30 pm, whereas the 5 pm change was like a "final warning." The dramatic reduction of brightness at 12 am was like a reminder that I should go to bed even though there is this one thing I really want to finish.It was really effective, especially because I like to hide the menubar, where the clock is located, to maximize screen space and minimize distractions when I'm working.

Screen Color Temperature, Dynamic Wallpaper

Once I understood my room lights' reminder-like effect on me, I looked for other things that I can control, e.g., to better control myself.The first one was the screen display color temperature.

I'm not a fan of dark mode for my workstation desktop. I know I'm a minority among fellow developers, but I just find using a white background (e.g., on IDE and terminal) easier to read. However, one problem with using the white background is it hurts my eyes towards the end of the day, i.e., in the evening hours. I became a big fan of the "night shift/light" feature that adjusts the screen display color to a warmer color after sunset because it made using a white background in the evening much more comfortable.

To go beyond the default "start night shift at sunset and stop it at sunrise", I set up the following cron jobs on my work desktop running Ubuntu:

# Run `sudo crontab -e` to open this file

0 0 * * * /usr/bin/gsettings set org.gnome.settings-daemon.plugins.color night-light-temperature 1000
0 6 * * * /usr/bin/gsettings set org.gnome.settings-daemon.plugins.color night-light-temperature 5000
0 10 * * * /usr/bin/gsettings set org.gnome.settings-daemon.plugins.color night-light-temperature 6000
0 4 * * * /usr/bin/gsettings set org.gnome.settings-daemon.plugins.color night-light-temperature 5000
0 5 * * * /usr/bin/gsettings set org.gnome.settings-daemon.plugins.color night-light-temperature 4000
0 8 * * * /usr/bin/gsettings set org.gnome.settings-daemon.plugins.color night-light-temperature 3000

By the way, here I didn't need to have redundant rules because my laptop was always on. I even applied the same idea to control dynamic wallpaper, noticing dynamic wallpaper is another thing that I implicitly interact with.

For some time, I played with synchronizing such changes across the room lights, screen display, and wallpaper but screen color temperature and wallpaper rules didn't stick. I ended up just using the default nightshift mode (i.e., on/off on sunset/sunrise) because changing the screen display color temperature was too subtle to give me the kick. As for the dynamic background, I often missed the changing background because I usually maximize my IDE or terminal window.

Productivity

My exploration of trying to understand the effects of various kinds of lights on myself reminded me of human-computer interaction (HCI) research papers investigating the possibility of behavior change via technologies (disclosure: I only read their abstracts):

Then I thought, can we take a computational approach to figure out the optimal environment (e.g., sounds/lights sequence/placements) for productivity? For example, if we have an accurate model of how certain environments like sounds or lights affect our productivity, we could find an optimal sequence and placement of sounds and lights that maximizes our productivity! Building such an accurate model would be difficult, but maybe we can take a quantified-self approach to collect data and augment it with data from smart home devices?

But then I asked myself: Do I really want to go this route? Would it truly boost my productivity?

Self-Control via Environment Control

The goal of this entire journey was to improve my productivity, and what I discovered was not just about coffee but rather about self-control. This realization stemmed from the following observations:

Certain things put me into certain moods ("Sound").
Environmental cues act as triggers ("Lights").

And thinking about how to leverage these observations. Expanding to different modalities ("Screen Color Temperature, Dynamic Wallpaper") taught me how to avoid distractions and stay focused on my goals. One thing I've learned from daydreaming about the computational approach to boosting my productivity ("Idea") is that, as a first step, I could try to internally learn my productivity model by paying more attention to how my mind responds to various environments (let's meditate people!). Only then can I leverage such a model (understanding) by changing environments.

Footnotes

[^1] For more information on the color temperature unit Kelvin (K), see this article.
[^2] My gradschool labmate published a research paper about this.

Robo-Observability

Mike Chung — Sat, 15 Jul 2023 00:29:37 +0000

Originally published at https://mjyc.github.io on April 21, 2023.

I care about observability in the context of debugging and monitoring robotics systems.

Logs

Debugging (e.g., ROS-based) robotics systems by digging through logs hasn't been fun for me. To investigate a bug report, I first SSH into a robot, look through multiple log files from multiple teams' software, and once I find relevant logs and data (e.g., camera images), I start downloading them to my local dev machine and wait for 10~20 minutes (at best). Only after that, I could dive deep into them to understand the reported issue. I'm describing my ~worst experience, but debugging (and monitoring, too) robotics systems has always been painful for one or two reasons I just mentioned.

When I started using log management tools from the distributed systems community, I was pleasantly surprised by their amazing developer experience (DX). Three aspects in particular stood out to me:

Structured logs made logs machine-parsable and version controllable, eliminating the need for regex gymnastics. This greatly facilitated the development of user-facing tools like interactive data visualizers.
Centralized logs helped bring together all relevant information for users.
Log visualization tools allowed users to effortlessly navigate and process large amounts of log data.

During the adoption of these aspects, I observed the following challenges faced by organizations:

Large and complex codebases made it difficult and laborious to structure logs consistently across diverse subsystems.
Large data volumes posed challenges in centralizing data. Even in robotics companies that deal with non-autonomous vehicles, data generation reaches petabyte scales, making it incredibly challenging to work with.
Non-textual logs made the utilization of existing log management and visualization tools more difficult.

Here are my suggestions:

Start structuring logs by adding metadata such as robot ID and customer ID (i.e., robot and customer information) to the logs of multiple teams. Doing so should spark discussions about standardizing the data structure and tooling for logs. Nudge stakeholders to think in terms of logs generated from fleets instead of individual robots, and manage the lifecycle of logs independently, for example, from the software that generated them.
While aiming to standardize the metadata structure and tooling to simplify the consumption process, log data types and data channels carrying logs should be treated differently and separately to optimize performance in terms of transportation, visualization, and so on.
Invest in adopting or even building data visualization tools. "A picture is worth a thousand words." Non-textual data is essential when it comes to debugging, and each organization may have bespoke needs.

Metrics

Typical metric categories I've seen are:

Customer and robot-specific metrics, such as the total number of completed deliveries and the total distance traveled for an indoor delivery robot company.
Resource utilization and health-related metrics, such as CPU, memory, and disk usage, network traffic of onboard and cloud machines.
Service health and availability-related metrics, such as request rate/error/duration, service uptime/response time of onboard and cloud services.

These metrics aren't specific to robotics companies and are standardized (e.g., across services) for ease of consumption and operational scalability. However, I have found that specializing metrics for core robotics engineers (e.g., who also engage in operations work in smaller organizations) is helpful for monitoring purposes. Here are examples of such specialized metrics for motion control and planning:

Motion control: control frequency, number of staged controllers, response time of dependent hardware devices.
Motion planning: planning request rate, errors, and duration, distance and duration of planned motion (trajectory).

Notice that these metrics are still high-level, i.e., general across different kinds of motion planning or control algorithms. There are also robotics algorithm-specific metrics (e.g., the number of nodes explored for a sampling-based planner) that can be computed and tracked. While I do like to collect such robotics-specific metrics to gain a deeper understanding of algorithm performance, doing so requires caution, e.g., I like to ask questions such as: What's the overhead of computing algorithm-specific metrics? How can we extract meaningful information from these metrics and avoid adding noise to the dashboard? How much maintenance work do we anticipate?

Final Thoughts

Observability is crucial for debugging and monitoring robotics systems at every stage of an organization. In small organizations, observability is a must for quickly detecting and resolving issues. For larger organizations, the ability to collect and process large log data from a fleet of robots effectively or monitor the health and utilization of such a fleet is a must. Ensuring observability of a robotics system at scale requires not only careful design and nontrivial implementation work on tooling but also the establishment of conventions and practices that are agreed upon and adhered to by multiple teams.

Acknowledgements

I thank Chris Palmer and Rastislav Komara for sharing their experiences and insights.

Getting started with programming, kind of

Mike Chung — Sat, 11 Sep 2021 05:06:48 +0000

https://leetcode.com/problemset/all/ for coding interviews
- https://youtu.be/xAxgzrj8zgU this guy shows how to meta-study for coding interviews
https://refactoring.guru/ for refactoring & reviewing design patterns
- https://www.martinfowler.com/architecture/ if you want more
https://github.com/donnemartin/system-design-primer for system design
https://dev.to/ explained stuff to me like I'm 4 years old
- https://dev.to/mjyc I used to post some too!
https://homes.cs.washington.edu/~jstn/#/teaching my grad school buddy who has amazing Git & C++ beginner’s tutorials
- https://girliemac.com/blog/2017/12/26/git-purr/ git purr
https://github.com/alebcay/awesome-shell shell is awesome
https://github.com/Mayccoll/Gogh I like dracula, nord, github, night owl, dissonance/brogrammer, and broadcast
- https://www.reddit.com/r/terminal_porn/ oh yeah
https://carbon.now.sh/ if you need to share your amazing command line script
https://roadmap.sh/ cause developers need a roadmap too

Testing robotics systems in fast-paced startups

Mike Chung — Sun, 03 Jan 2021 00:06:03 +0000

Originally published at https://mjyc.github.io on December 16, 2020.

Starcraft II, Photo by Zach Tirrell on flickr

Testing robotics systems is hard. Based on my experience working at startups with fewer than 200 employees and fewer than 100 robots providing RaaS using fleets of indoor mobile robots or lines of robot manipulators, the main reasons for the difficulty were as follows:

Edge cases and corner cases in production environments.
The difficulty of using simulation.
Challenges with adopting automation.

Why is Testing Robotics Systems Hard?

In production, I've encountered various edge cases and corner cases:

Edge cases for robotics algorithms. Input spaces for robotics algorithms, such as perception, control, and motion planning, are vast and challenging to effectively cover for edge cases; there is always a specific layout that causes navigation failures or a particular scene with specific objects that leads to grasping failures. Characterizing such instances is difficult and algorithm-dependent, which complicates the testing setup.
Rare hardware issues. Rare hardware issues that are not (directly) detectable are the worst, such as a small damage in the robot cell structure that requires adjusting the collision map. Anticipating such issues requires input from domain experts (e.g., mechanical or firmware engineers), who may not be easily accessible and speak different jargons and reproducing them often requires changing interfaces, which can be expensive (e.g., it becomes yet another layer to maintain).
Subtle regression. The complexity of robotics systems makes it challenging to establish a robust regression testing pipeline. For example, handling low-frequency flaky tests, implementing robust test selection and prioritization[^2] is difficult and hence elusive bugs slip back into the production code. Performance regressions are particularly challenging--especially ones caused by low-level concurrency issues[^3], as they are subtle and require expensive measures such as repeated end-to-end tests and delicate statistical methods to detect.
Unexpected peak load condition/usage pattern. It is common for multiple (custom) software, such as core robotics, monitoring, and infra-related software, to run in parallel. Unexpected high demands can adversarially impact your program, e.g., by consuming all of the available resources. Anticipating and recreating such situations is challenging, especially when dealing with (custom) software at all levels, including firmware and system software.

Using simulation for testing robotics systems effectively is not as easy as it seems.

Inadequate usage. I find that simulation testing is most useful for end-to-end testing of robot applications. However, I often encounter test cases that would benefit from using other tools and techniques (e.g., for efficiency). All too frequently, I come across test cases for robot behaviors (e.g., implemented in finite state machine or behavior tree) that use simulation, making the test cases much more expensive than they need to be. In such cases, using alternatives like fake or model-based testing[^4] would be much more efficient, as they can be used to only "simulate" the directly relevant modules[^5]. This often stems from organizational issues, such as unclear boundaries between teams that result in poorly defined interfaces and testing strategies[^6] or more typical insufficient allocation of time for testing/addressing technical debts (e.g., in favor of prioritizing other deliverables).
Generating test cases. Even with simulation libraries that provide high-level interfaces for building scenarios, creating effective test scenarios is challenging. Creating a single simulated environment for end-to-end testing alone is laborious enough, so diversifying the test scenarios (e.g., to cover extreme cases) becomes a nice-to-have[^7]. There are commercial products that address this issue (e.g., AWS RoboMaker WorldForge), but they are not easy for smaller organizations (i.e., startups) to integrate due to reasons such as integration cost, vendor lock-in, etc.
Expressing specifications. Specifications for many robotics programs, e.g., those involving perception, motion planning, and behaviors, are difficult to express due to their spatiotemporal nature. This leads to verbose and unorganized (e.g., containing duplicates) test code, which makes it difficult to maintain and scale.
Managing infrastructure. I haven't met a single person who loves managing simulation testing infrastructure, e.g., for continuous integration. Simulation test code is expensive to run, requires special hardware such as GPUs, and is difficult to optimize and move around (e.g., in cloud environments). This leads to a poor developer experience and can even result in the disabling of simulation testing.

Automating robotics software testing is still hard.

Challenges with automating build and deployment. Here are tech talks and a blog post that shed light on this topic:
- Packaging ROS with GitHub Actions from PICKNIK Blog, 2023.
- Building Self Driving Cars with Bazel from Cruise, BazelCon 2019 - shares Cruise's experiences with building and testing robotics software at scale
- The landscape of software deployment in robotics from Airbotics - summarizes the typical challenges with deploying robotics software
- Physical continuous integration on real robots from Fetch, ROSCon 2016 - shares Fetch's experience with setting up and using a physical continuous integration pipeline
No standard. Automating the testing of software requires agreements among engineering teams on build, deployment, and test models. Given how robotics brings multiple communities together, such as research (e.g., computer vision, robotics), web development (e.g., frontend, backend), DevOps, embedded, etc., reaching such an agreement, or even discussing ideas (e.g., due to different backgrounds), is difficult. While the Robot-Operating System (ROS) and the communities around it have made significant progress in this regard, the lack of standards still seems to be a significant problem in organizations.

Where to Start with Testing: General Techniques

To figure out where to start with testing a robotics system, I use the followings:

Prioritization framework for creating tests.
Systematic procedure for identifying what to test.

The techniques discussed in this section are general/not-so-technical and are mostly aimed at addressing the abovementioned challenges regarding "edge and corner cases in production environments".

Eisenhower Matrix for Test Prioritization

In robotics startups that build complex systems, creating comprehensive test suites is impossible. To produce high-impact tests within the time budget, I use my adapted Eisenhower Matrix to prioritize a list of failure scenarios by first categorizing (potential) failure scenarios according to their (expected) frequency and risk.

First Quadrant (upper left): frequent and high-risk. In Quadrant 1 (Q1), I place failure scenarios that need to be covered immediately, e.g., that I hear all the time from internal communication channels, such as introducing breaking changes to APIs and dependencies.
Second Quadrant (upper right): frequent and medium-risk. In Quadrant 2 (Q2), I place failure scenarios that occur frequently but allow continued operations with short downtime like unreliable hardware or unresponsive user interface issues with well-established alerts and recovery procedures.
Third Quadrant (lower left): infrequent and high-risk. In Quadrant 3 (Q3), I place failure scenarios that occur rarely but causes significant disruption in operations such as core robotics component failure scenario or unexpected peak usage pattern.
Fourth Quadrant (lower right): infrequent and medium-risk. In Quadrant 4 (Q4), I place failure scenarios that occur relatively infrequently and allow continued operations.

The placement of example failure scenarios in quadrants will differ across companies. For instance, depending on the maturity of the robot product/prototype or the amount of time invested by the engineering team in designing the system, an unreliable hardware failure scenario may belong in Q1 (e.g., if it is causing multiple issues) or a robotics algorithms failure scenario may belong in Q2 (e.g., if the failure is not catastrophic or easily recoverable). In general, I create or improve tests for one quadrant at a time, in increasing order. After working on tests for Q1, I move on to tests for Q2 before addressing those for Q3. This is because creating tests for Q3 requires a significant time investment, for example, to ensure the reproducibility of the failure. Usually, there is no time available to work on tests for Q4. But adjustments should be made to meet organization-specific requirements and constraints.

Test Scenario Identification Procedure

So far, I have assumed that the failure scenarios to test are known; however, this is usually not the case.
To determine what to test, I follow these steps:

Gain access to internal alerts, dashboards, and logs._ Investigating recently reported problems or analyzing the latest trends using monitoring tools[^8] is the easiest way to identify high-risk failure scenarios. If monitoring tools are not set up (e.g., in smaller companies), I get involved in operations work, which is another way to uncover potential high-value tests to create.
Identify interface and service boundaries. Understanding how software components interact with each other provides insights into potential integration failures and their impact.I start by looking for internal documentation with system diagrams (or examining the codebase and creating them if such diagrams don't exist) and ask questions such as: which interactions must not fail? which interactions are changing frequently?Such exercises reveal missing must-have contract tests or high-impact opportunities to improve integration tests.
Identify implicit dependencies. I consider edge cases such as low resources, unexpected hardware states, or unseen inputs to robotics algorithms (e.g., those that crash applications) as unmet runtime dependencies. Taking this view nudges me to specify these not-well-understood requirements for keeping the system (or "implicit dependencies") well-behaving as explicitly and clearly as possible. Once defined, such requirements can be used to create extreme failure scenarios to test.

Ensuring Testability as a Startup Grows

Below, I share my insights on key practices to employ at each growth stage/funding round of robotics startups. The real motivation behind testing is the reliability (e.g., of the provided service), and so the shared key practices below cover related areas such as debugging and observability[^9].

If you have experience with SaaS/web service startups, you might notice that the RaaS/robotics company size (i.e., the number of employees) at each growth stage is larger, and some key practices occur later in robotics startups. This is because RaaS/robotics products not only consist of a more diverse set of software components but also require additional teams like electrical/firmware engineering, mechanical engineering, and manufacturing operations.

Series A: 5-20 Employees

At this stage, startups are likely to have fewer than 5 customers/design partners and a handful of developers who are relentlessly building (and fixing) major components of the company's first product. The goal of the startups is to prove the value of their product to their (rather forgiving) customers by succeeding in basic tasks performed by robots as much as possible. For example, a delivery robot should navigate without colliding with obstacles, and a robot manipulator should pick and place objects without dropping them in customers' (production) environments.

Key Practices:

Setting up continuous integration or nightly tests (e.g., using Jenkins, GitLab CI/CD, GitHub Actions, or Debian build farm) with end-to-end tests involving a high-fidelity simulator (e.g., Gazebo) to quickly smoke test rapidly changing codebases.
Collaborating on internal communication channels (e.g., Slack) and utilizing a (custom) teleop solution (e.g., built on MQTT, WebRTC) to quickly respond to critical incidents.
Creating metrics and dashboards to track business-critical measures (e.g., using Grafana/Prometheus) such as the number of deliveries/distance traveled, throughput/object knitted to guide all developers/employees at a high level.

Series B: 21-200 Employees

Startups at this stage start expanding their customer base and aim to scale their operations to deploy and handle, for example, more than 100 robots. The companies now have (small) teams of developers working on enhancing the robustness of core robotics software components to handle diverse environments of new (and unforgiving) customers, providing a non-beta-user-acceptable user experience (e.g., by building proper onboard and/or desktop UIs), and/or infrastructure to scale operations. The robotics system powering the product becomes much more complex, and to manage such complexity, its architecture becomes more modular, composable, and distributed, and boundaries/ownerships occur.

Key Practices:

Employing narrow integration tests and contract tests using test doubles like fakes and spies—sometimes going as far as implementing low-fidelity simulators with them (e.g., using pytest, GoogleTest) to test each team's evolving software component in isolation and complex interactions between such components, efficiently.
Establishing a deployment strategy with rollback support (e.g., using tools that enable infrastructure-as-code/gitops like Ansible, Terraform and support over-the-air updates, etc.) to avoid manually applying untrackable hotfixes in fear of losing customers. By the way, this practice is nontrivial to achieve for technical and cultural reasons; for more information, see "The landscape of software deployment in robotics" from Airbotics. Typically, adopting a deployment tooling sparks discussions like adopting cloud-native tooling or not, which then sparks discussions on system launching mechanisms, and so on.
Implementing nightly tests on real robots in a mirror warehouse or manufacturing line or [physical continuous integration](https://youtu.be/JNV9CkARhg)** in a production-like environment to prevent edge and corner cases and performance regressions in production environments as much as possible.
Utilizing (custom) fleet operation solution to scale teleop-based incidence recovery (e.g., more robots to monitor/rescue per person), open up such operations to non-developers, etc.
Structuring and centralizing logs (e.g., structlog, spdlog, and Loki, ELK, Splunk) to track performance across growing fleets and enable dynamically querying information for fast debugging or monitoring, e.g., with robotics-specific metrics. In addition, centralizing data would be ideal; however, given the large size of robotics data, doing so requires more care and resources—which usually becomes possible to afford in the next stage.
Implementing a data record and replayer and visualizer (e.g., rosbag and RVIZ, Foxglove Studio, or custom-built ones) to enable debugging or optimizing robotics algorithms running in distributed systems. By the way, building a performant data record and replayer, e.g., that works well with both real robots and simulators, is no joke; there are issues like not being able to record or replay data fast enough, doesn't work well with simulators that run slower or faster than real-time, etc.

Series C: 201-2000 Employees

Robotics companies at this stage have experienced significant growth and have a large customer base. Their goal is to further scale their operations and efficiently handle a substantial number of robots, potentially exceeding 1000 units, without compromising performance. The company now consists of multiple teams working on various aspects of the robotics system, including core software development, user experience enhancement, infrastructure scalability, and customer support. Additionally, the focus expands beyond purely technical challenges to encompass broader organizational considerations, such as team structure, talent management, strategic partnerships, and market expansion.

Key Practices:

Implementing an advanced test automation framework that encompasses a wide range of tests, including unit, integration, performance regression, and security tests. To improve testing efficiency, the company utilizes techniques such as test impact analysis and test selection and prioritization, which identify the most critical tests and prioritize their execution. Furthermore, they leverage cloud resources to run a large number of simulation-based end-to-end tests in parallel, accelerating the testing process.
Establishing a formalized release management process that includes thorough testing, staging environments, and controlled deployments. This involves leveraging continuous integration and continuous deployment (CI/CD) practices, utilizing version control systems (VCS), maintaining release documentation, and employing release dashboards. The company also invests in advanced deployment and orchestration tools such as Terraform, Docker, and Kubernetes to facilitate large-scale, cross-team deployment and operation of a diverse fleet, encompassing robots, peripheral devices (e.g., elevator controller, manufacturing line controller), and (web) servers.
Enhancing developer productivity through faster iteration, improved debugging and testing tooling, and ultimately aiming to enhance system reliability. At this stage, companies have the resources to invest in infrastructure and pipeline improvements. This includes setting up cloud development environments utilizing platforms like GitPod, Coder, and DevZero. Additionally, efforts are made to speed up build times through techniques such as remote caching. Streamlining code review and merge progress is achieved using tools like Merge Queue. Furthermore, specialized tooling is developed to cater to the specific needs of the company. For instance, enhancing the data record and replayer to support streaming robotics data enables faster debugging by bypassing the need to download large datasets onto local development machines. The data record and replayer are also optimized to handle high throughput and enable navigation between critical timestamps. Integration with high-fidelity simulators is improved to facilitate seamless interaction. Another example of specialized tooling is the creation of a test scenario generator, which automates the generation of realistic test scenarios, enhancing test coverage and enabling the identification of edge cases and potential failure scenarios.
Implementing incident response and resolution workflows to minimize downtime and efficiently handle critical incidents. The fleet operation tool should leverage automation, e.g., incident detection, require minimal human input for recovery, etc. Also the workflow or procedure should allow each team to be responsible for the sfotware they write.
Establishing a security and compliance framework to ensure the protection of sensitive data, intellectual property, and customer privacy. This includes implementing secure coding practices, conducting regular security audits, and complying with relevant industry standards and regulations.

Beyond 2000 Employees

I don't have any firsthand experience with companies at this size, and in fact, I only know a handful of robotics companies at this stage, such as large autonomous vehicle companies like Waymo, Cruise, Aurora, and Zoox. I will update here after gaining more experience with companies at this stage, one day.

Closing notes

In this post, I have listed the challenges of testing robotics systems in fast-paced startups, shared my techniques for getting started with testing work, and provided insights on key practices for ensuring testability as an organization grows.

Let me know what you think by leaving comments below or messaging me on LinkedIn or Twitter!

Do my experiences/approaches resonate/align (or not resonate/align) with yours?
Do you have any test-related war stories or effective testing strategies you'd like to share?

I'd love to hear your thoughts.

Acknowledgements

I thank all my colleagues who have engaged in conversations with me on related topics.

Significant Revisions

2023/06/29: Added the "Ensuring reliability as an organization grows" section
2023/05/28: Rewrote the whole post

Footnotes

[^1] The identified challenges and my approaches may not generalize to other settings, such as testing in robotics companies that are much smaller (i.e., < 10 employees) or much bigger (i.e., > 1000 employees), or involving a different product, such as an autonomous vehicle-based ride-hailing service or an autonomous-based inspection service. For example, I don't have much experience with testing robotics systems that make heavy use of machine learning or real-time programming.
[^2] See also The Rise of Test Impact Analysis by Martin Fowler.
[^3] E.g., see ROSCon 2017 Vancouver Day 2 Determinism in ROS and ROSCON 2019 MACAU: CONCURRENCY IN ROS 1 AND 2: FROM ASYNCSPINNER TO MULTITHREADEDEXECUTOR.
[^4] For code examples, see Stateful testing (Python) or Detecting the unexpected in (Web) UI (JavaScript).
[^5] Check out IntegrationTest by Martin Fowler for related discussion, e.g., about narrow and broad integration tests.
[^6] Check out On the Diverse And Fantastical Shapes of Testing (at least the last paragarph starting with "If you're paying my careful prose ...") by Martin Fowler for related discussion.
[^7] I enjoy following research papers in this space, such as those taking a grammar-based approach like Scenic: a language for scenario specification and scene generation.
[^8] See also this twit thread from Gergely Orosz.
[^9] See also my another post Robo-Observability.

Getting started with robotics: Just do it!

Mike Chung — Sun, 24 May 2020 21:27:43 +0000

Getting started with robotics is confusing.
Robotics is an interdisciplinary field and people think of many different things when they are trying to learn about it. For example, google searching "getting started with robotics" gives me the following top three results:

They talk about learning skills related to the fields of mechanical engineering, electrical engineering, and computer science. At first, it just felt overwhelming. Reading each of them slowly again, they were great tutorials especially because they all shared one great message--"learn by doing projects" (there was even a book named with a similar spirit).

I 100% agree with the message, I think people should learn robotics by doing projects. In fact, I recently shared my curated list of opensource (and other) robotics projects for those who are interested in building robots. Because I'm a programmer by training, one additional suggestion I like to add is "start by working with a simulator". Working with hardware is fun but it can be extremely time-consuming so by working with a simulator first you can feel out the robot and identify potential problems early. Projects like MuSHR and bobble-bot are great because they provide robot simulators as well as detailed instructions for building robots. PythonRobotics is another great entry point for learning about robotics algorithms. The repository contains provide tiny, simple environments for testing the algorithms which are great for learning purposes. Here is a list of ROS-based simulators that I've curated in ROS Development studio, a cloud service that allows you to work on ROS projects in a browser. In a similar spirit, I encourage using a single board computer such as Raspberry Pi or NVIDIA Jetson products instead of using a microcontroller like Arduino. Programming a microcontroller can be fun and it can allow you to develop a solution that is highly tailored to your use case, but for learning purposes, it can become a rabbit hole that prevents you from completing the project you started. However, if your goal is learning mechanical or electrical engineering my advice (rather opinions) is not for you.

Finally, I believe getting involved with robotics communities is effective for learning. The below list could be good entry points for learning about software-focused robotics

github repos with #robotics tag
ROS discourse
Foxglove blog
PICKNIK blog
Isaac Sim Technical Blog
The AI Driving Olympics (AI-DO)
Balena blog - they provide less robotics and more IoT-centric contents
Cruise news
Awesome Robotics System Design - where I keep interesting software-focused robotics stuff
Awesome Robotics Problems - where I keep interesting robotics problems, datasets, and algorithm implementations

the list below for learning about electronics-focused robotics

and the list below for learning about hardware-focused robotics

This may be a bit off topic, but since people relate "robotics" with AI/ML computer science research, it might be fun to skim robotics-related papers in open paper review and curated paper list websites:

https://arxiv.org/
https://openreview.net/
https://paperswithcode.com/ - one note: not all researchers are great coders/documenters.
http://bohg.cs.stanford.edu/list/

Talking about skimming, it might be inspiring to skim the class materials from CSE 478: Autonomous Robotics. Unlike many other class materials, their class slides provide application examples of introduced concepts with an open-source autonomous mobile robot platform MUSHR.

WARNING Reading papers and learning class materials can become yet another rabbit hole. There are endless interesting papers (on surface) or concepts (from class slides) and they can distract you from finishing your project. What happens is that because you feel achievement/growth and you get temped to keep learning. Being able to focus on the track and learn only necessary skills (and taking the project to the finish line--and defining the finish line) is a huge challenge/probably the most important skill to learn.

With that said, go explore project ideas, check out robotics communities and start your project! I believe now is the time to learn about robotics and I hope this blurb can be helpful to aspiring roboticists.

Installed ROS Lunar on macOS Mojave! But...

Mike Chung — Mon, 18 May 2020 04:17:10 +0000

Motivation

I've been playing with Comzo lately and tried to integrate its sdk with ROS. Initially, I used a Ubuntu desktop for running ROS but I wanted to use my Macbook Pro laptop instead so I tried installing ROS on macOS. I found the following instructions:

and the first one seems like an official one, so I tried it. There was a big yellow warning

This is a work in progress! At present, the instructions cover only the > installation of ROS-Comm (Bare Bones) variant and tested on the following > configuration:
- macOS Mojave + native (Apple) Python 2.7.10 + XCode 11.2.1

but my setup met the described requirement and I only needed ROS-Comm (Bare Bones) packages so just went ahead with it.

Initial Installation

First, I ran all commands in Setup section they went through smoothly. I felt great, so additionally ran ominous brew upgrade since it's been a while I have done that.

In Installation section, I chose ROS-Comm option in 2.1 as I planned earlier. Everything went through until running catkin_make_isolated in 2.5.2 Building the catkin Workspace subsubsection. It outputted:

CMake Error at test/CMakeLists.txt:108 (set_target_properties):
  set_target_properties Can not find target to add properties to: test_socket

Pfft! who needs test? I added -DCATKIN_ENABLE_TESTING=0 flag and ran the below install command again:

./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release -DCMAKE_MACOSX_RPATH=ON -DCMAKE_INSTALL_RPATH=$HOME/ros_catkin_ws/install_isolated/lib -DCATKIN_ENABLE_TESTING=0

then it outputted:

...
[ 81%] Building CXX object CMakeFiles/roscpp.dir/src/libros/transport_subscriber_link.cp
p.o
/Users/mjyc/ros_catkin_ws/src/ros_comm/roscpp/src/libros/transport/transport_udp.cpp:68:
18: error:
      expected '('
, server_address_{}
                 ^
...
/Users/mjyc/ros_catkin_ws/src/ros_comm/roscpp/src/libros/transport/transport_udp.cpp:69:
1: error:
      expected unqualified-id
, local_address_{}
^
3 warnings and 2 errors generated.
make[2]: *** [CMakeFiles/roscpp.dir/src/libros/transport/transport_udp.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
...

This output didn't look too good. I google searched the error messages and it sounded like I was trying to compile src code that is not compatible the C++ compiler I have on my mac. Arr... I didn't want to up/downgrading my system/brew libraries, so I searched for a different solution on internet but didn't like them at all. Then I thought about giving a shot at installing an older version of ROS and see if that works so I decided to install ROS Lunar. I ran this command to download ROS lunar packages

rosinstall_generator ros_comm --rosdistro lunar --deps --tar > lunar-ros_comm.rosinstall

and ran the same catkin_make_isolated command above to start building. It now complained about missing boost_signals:

CMake Error at /usr/local/lib/cmake/Boost-1.72.0/BoostConfig.cmake:119 (find_package):
  Could not find a package configuration file provided by "boost_signals"
  (requested version 1.72.0) with any of the following names:

    boost_signalsConfig.cmake
    boost_signals-config.cmake

  Add the installation prefix of "boost_signals" to CMAKE_PREFIX_PATH or set
  "boost_signals_DIR" to a directory containing one of the above files.  If
  "boost_signals" provides a separate development package or SDK, be sure it
  has been installed.
Call Stack (most recent call first):
  /usr/local/lib/cmake/Boost-1.72.0/BoostConfig.cmake:184 (boost_find_component)
  /usr/local/Cellar/cmake/3.17.2/share/cmake/Modules/FindBoost.cmake:444 (find_package)
  CMakeLists.txt:25 (find_package)

It turned out the solution was mentioned in ROS melodic installations instructions. After modifying some CMake config files as suggested, the built everything successfully!

Testing and More Installation

After sourcing the setup.bash file, i.e., ran source ~/ros_catkin_ws/install_isolated/setup.bash, and I ran roscore and it worked!!

$ roscore
... logging to /Users/mjyc/.ros/log/a15e85b8-98a1-11ea-8b7b-f45c898e9ff7/roslaunch-mjmbp
r.local-92159.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt

started roslaunch server http://mjmbpr.local:63709/
ros_comm version 1.13.7


SUMMARY
========

PARAMETERS
 * /rosdistro: lunar
 * /rosversion: 1.13.7

NODES

auto-starting new master
process[master]: started with pid [92166]
ROS_MASTER_URI=http://mjmbpr.local:11311/

setting /run_id to a15e85b8-98a1-11ea-8b7b-f45c898e9ff7
process[rosout-1]: started with pid [92169]
started core service [/rosout]
[ERROR] [1589763055.896770000]: [registerService] Failed to contact master at [mjmbpr.lo
cal:11311].  Retrying...
[ERROR] [1589763055.960174000]: [registerPublisher] Failed to contact master at [mjmbpr.
local:11311].  Retrying...
[ERROR] [1589763056.025336000]: [registerSubscriber] Failed to contact master at [mjmbpr
.local:11311].  Retrying...

There were some error messages but that didn't matter. I tried rostopic list and I got:

Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6

It seemed like I used the brew installed python when I should have used the system installed one. For example, the command below worked.

/usr/bin/python /Users/mjyc/ros_catkin_ws/install_isolated/bin/rostopic list

Urgh, I didn't change my python settings because doing so could mess up my other python projects (I don't use virtualenv, conda, pyenv, etc). I also noticed adding source ~/ros_catkin_ws/devel/setup.bash to ~/.bash_profile slowed down opening every new terminal window, which was super annoying. To make these problems go away, I added the following bash script to ~/.bash_profile, which makes the sourcing setup.bash on-demand and creates aliases for basic ROS commands:

if [ ! "$(type -t __init_ros)" = function ]; then
  declare -a __ros_commands=('roscd' 'roscore' 'rosrun' 'catkin_make')
  declare -a __ros_commandspy=('rostopic' 'rosmsg')
  function __init_ros() {
    for i in "${__ros_commands[@]}"; do unalias $i; done
    for i in "${__ros_commandspy[@]}"; do alias $i='/usr/bin/python /Users/mjyc/ros_catkin_ws/install_isolated/bin/'$i; done
    source $HOME/ros_catkin_ws/install_isolated/setup.bash
    [ -f $HOME/catkin_ws/devel/setup.bash ] && source $HOME/catkin_ws/devel/setup.bash
    unset __ros_commands
    unset __ros_commandspy
    unset -f __init_ros
  }
  for i in "${__ros_commands[@]}"; do alias $i='__init_ros && '$i; done
  for i in "${__ros_commandspy[@]}"; do alias $i='__init_ros && /usr/bin/python /Users/mjyc/ros_catkin_ws/install_isolated/bin/'$i; done
fi

Now the commands below runs without errors and there were no more delay in opening up a new terminal!

rostopic pub -r 1 /chatter std_msgs/String hello
rostopic echo /chatter
rostopic type /chatter | rosmsg show

Adding More ROS Packages

My goal of installing ROS on macOS was to use ros_cozmo which is dependent on actionlib and dependent packages. So I added them by running

mv lunar-ros_comm_plus.rosinstall lunar-ros_comm_plus.rosinstall.old
rosinstall_generator ros_comm actionlib --rosdistro lunar --deps --tar > lunar-ros_comm_plus.rosinstall
rm src/.rosinstall
wstool merge -t src melodic-desktop-full.rosinstall
wstool update -t src

and rebuilt the workspace. It built without any errors.

Then I became interested in teleoperating Cozmo via a web app with roslib. I updated .rosinstall file via

rosinstall_generator ros_comm actionlib --rosdistro lunar --deps --tar > lunar-ros_comm_plus.rosinstall

and tried rebuilding the workspace. This time I faced error messages like

...
/Users/mjyc/ros_catkin_ws/src/rosauth/src/ros_mac_authentication.cpp:15:10: fatal error:

      'openssl/sha.h' file not found
#include <openssl/sha.h>
         ^~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/ros_mac_authentication.dir/src/ros_mac_authentication.cpp.o] Er
ror 1
make[1]: *** [CMakeFiles/ros_mac_authentication.dir/all] Error 2
make: *** [all] Error 2
<== Failed to process package 'rosauth':
  Command '['/Users/mjyc/ros_catkin_ws/install_isolated/env.sh', 'make', '-j8', '-l8']'
returned non-zero exit status 2

...

There were some suggestions on internet to solve this problem but at this point, I felt like installing packages like this is not a scalable approach.

Afterthoughts

I felt like what I tried wasn't the right approach for using ROS on mac in 2020. I guess there are

ROS docker
ROS development studio
ROS on cloud like AWS robomaker, Rapyuta Robotics, and ROS development studio

and I think ROS docker and ROS development studio are reasonable but they weren't amazing. Maybe I just need to wait for ROS2.

git-issue: an offline-friendly project management tool with potential

Mike Chung — Sat, 04 Apr 2020 19:38:31 +0000

Background

Before COVID-19, I've been reviewing issues on a bus to work. My team used gitlab, so without a stable internet connection on a bus, I wasn't able to concentrate. I started look for an offline friendly issue management tool and narrowed the contenders to (i) git-issue and (ii) git-bug.

I decided to give a shot at git-issue because I liked its simplicity. My first impression of git-issue was basically an issue manager powered by git. It provided command-line tools for basic issue management like creating, editing, removing, listing issues and comments and also had additional features like logging time estimates/spent and setting milestones.
It was written in shell script.

The Pros and Cons List

After 2 weeks of using it here is my pros and cons list.

Pros:

lightweight
based on git
supports interacting with github/gitlab

Cons:

github/gitlab integration is buggy
written in shell script so hard to read & debug the src code

My Honest Opinion

My initial motivation was doing project management when I have don't have access to the internet, e.g., on a bus. I couldn't replace my existing workflow--I used to use GitLab's issue board--because there were tons of project management features GitLab provided that I didn't realize I was dependent on. For example, I wanted

customizable board view
markdown preview
issue number auto-completion the editing view

Also, I realized (i) connecting to the internet on a bus wasn't as hard via tethering and (ii) there are no severe consequences of editing issues concurrently (as in using a cloud tool). For coding (ii) is not true since doing so will likely crash your program.

I ended up mostly using git-issue when I know when exactly what to do, e.g., comment on a specific issue. While I still think it has a lot of potentials, for now, it has not replaced my goto tool, the gitlab issues/board pages on the browser.

Misc.

git-issue helper scripts for fuzzy searching issues (and editing)

My failed attempt to set up an old MacbookPro as a deep learning workstation

Mike Chung — Sat, 21 Mar 2020 05:36:42 +0000

I recently got a new laptop but didn't want to say goodbye to my 2014 MacBookPro (10,1) yet. Then I remembered it has an NVIDIA graphics card, so I thought maybe I'll use it for training toy deep learning models. I never train a deep learning model before.

My goal was to training a tacotron2 model. Tacotron is Google's deep learning-based speech synthesis system. I was not able to set up my old laptop to train a tacotron2 model but found a different way to achieve my goal, i.e., using Google Colab. Still, I'm sharing my journey in case someone could learn from the mistakes I made.

I installed Ubuntu 18.04. I searched for related documentation from help.ubuntu.com or wiki.ubuntu.com but they seem outdated so I just googled "install ubuntu 18.04 MacBookPro and found a medium post. Following the instructions on the post just worked. I also installed an NVIDIA driver via Software Settings.
Training tacotron2 requires Pytorch 1.0. Although it seemed like I could install Pytorch via pip, I wanted to try installing it from source to gain better control of it. So I decided to install cuda & cudnn myself. It seemed like cuda 10.1 and cudnn 7.6.x are the latest pytorch compatible version of cuda and cudnn in March 2020.
NVIDIA's instructions for installing cuda and cudnn weren't as straightforward as I hoped them to be. After running the main cuda install command sudo apt-get install cuda-10-1 (not sudo apt-get install cuda-10 because I want to control the cuda version) I was seeing errors like
```
...
  found 'diversion of /usr/lib/x86_64-linux-gnu/libGL.so.1 to /usr/lib/x86_64-linux-gnu/libGL.so.1.distrib by nvidia-340'
...
```
I found a solution at stackoverflow and shamelessly applied it without understanding the commands fixed the problem. I suspected the error was due to installing an NVIDIA driver at 1. but never confirmed it. I also did not understand why the solution worked, but in the interest of time, I marched forward. cudnn installation was smooth.
Now it was time to build Pytorch from source. Although I should be using conda, I just used pip3 in the interest of time and started following the instructions. After running the grand python setup.py install, I got stuck:
```
...
    /home/mjyc/.local/lib/python3.6/site-packages/torch/cuda/__init__.py:134: UserWarning:
        Found GPU0 GeForce GT 750M which is of cuda capability 3.0.
        PyTorch no longer supports this GPU because it is too old.
        The minimum cuda capability that we support is 3.5.
...
```
It turns out MacBookPro 10,1's GPU GeForce GT 750M was too old for PyTorch 1.4 (the latest compatible PyTorch version seems to be 0.3.1). My first reaction to this error message was just buying an external GPU (eGPU). However, quick google search results showed that the eGPU case alone costs ~$300.00! and learned that just choosing which GPU to buy requires some research work.
At this point, I realized I spent much more time and efforts and I originally budgeted so I gave up turning my old MacBookPro to a workstation and started looking for an alternative solution. I read a Reddit thread suggesting a cloud solution and looked into which service is a good starting point. Seems like Google Colab is a good place to start. So I stopped my exploration here.

To me, the lessons learned from this journey was always focus on the end-goal instead of the mean. Given my goal was to train a deep learning model, the mean to achieve the goal--using a laptop workstation or cloud--should not have mattered.

Understanding challenges with large robotics system development

Mike Chung — Sat, 07 Mar 2020 23:41:21 +0000

Originally posted on GitLab

Robotics system development is hard. To understand causes for the robotics system development challenges, I interviewed a few robotics engineers who have been involved in large robotics projects and identified the following themes.

There aren't many performant off-the-shelve tools

As the field of robotics is not matured, it is not easy to find performance libraries for perception, manipulation, human-robot interaction that fits your needs. Many existing off-the-shelve code is research code and hence requires expert knowledge, e.g., a user needs to see through undocumented assumptions and limitations. Essentially, identifying whether they will be useful for your problem is an art of itself.

There aren't many generalist robotics systems engineer

Although more robotics educational materials are becoming available, there are not many engineers who can design and implement large robotics systems. Many robotics engineers often focuses on one subfield of robotics engineering such as computer vision or control but does not have much experience with working with the whole system. On the other hands, good systems engineers are often lacks the robotics knowledge and treats robotics libraries as black boxes.

Gathering system requirements or software specifications is not trivial

A robotic system that interact with physical world is complicated and consequences of using such system in real world is hard to predict. This makes the gathering of system requirements or software specifications challenging. Therefore the system specifications are often underspecified which yields brittle or over-prepared systems.

Maintenance and testing are challenging

Often existing dev-ops tools are unfit for the robotics system development purposes. For example, robotics data collection, analysis, and visualization are different from those of web services. Testing is especially challenging since setting up a real-world testing environment is not trivial, e.g., a clean "reset" of the real robot testing environment is near impossible or time-consuming. Also, the simulators that are supposed to help with testing do not serve their purpose because of the gap between simulation and reality.

Although the list above is based on a small number of interviews and my personal experience, I hope it to be used as a starting point for brainstorming for solutions. Please let me know if you see missing themes or any comments!

Miscellaneous

Sep, 2021. "How to Build Good Software" - "Why Bad Software Happens to Good People" section felt relevant.
Apr, 2021. Found more related papers!
- "State of the Practice and Guidelines for ROS-based System"
- "A Study on the Challenges of Using Robotics Simulators for Testing"
While I was writing this post, I learned about this excellent paper "State of the Practice and Guidelines for ROS-based System" and discussions about the paper in the ROS Discourse. The paper is focused on ROS yet the high-level goals of it seem similar.
The study notes this article is based on are available in github and gitlab repos
Thank you! to all those who participated in my interview studies

Please help me build a cloud visual SLAM system for cellphones

Mike Chung — Mon, 10 Jun 2019 00:25:16 +0000

Hello hackers, tinkers, webdevs, sysdevs, roboticists, and all coders! I've been excited about cloud robotics, a field of robotics that utilizes the power of cloud computing, and want to share the excitement with you and suggest a project we can potentially work together. The project that I'm thinking of is "cellphone visual SLAMing". The idea is to run a visual SLAM system on cloud so mobile devices like a cellphone can build 3D maps by simply uploading camera data to the cloud.

Here are the steps I'm thinking:

Try creating a 3D map using ORB_SLAM2 and desktop camera images. The main goal of this step is to get comfortable with a visual SLAM library and feel out the limitations.
Try creating 3D maps using ORB*SLAM2 running on a desktop and cellphone camera images. ORB_SLAM2 supports ROS. So one can easily capture device camera images using HTML5's MediaDevices.getUserMedia(), turn them into ROS image messages, and publish them using roslibjs so ORB_SLAM2 can use the images collected from a remote device.
Run the ORB_SLAM2 to cloud. I have not tried it, but it seems like it is fairly easy to containerize a ROS package and deploy it on cloud.

That's it! Are you interested in trying this idea out? If you have experiences with visual SLAM and have suggestions? Let me know, I'd love to hear your thoughts.

Updates

2021/01/02 I have moved on as I don't get to spend time on tinkering but still think this is a fun project to try one day.
2020/11/23 Fyusion and CANVAS seem to provide products with related technolgoies.
2020/05/02 It seems like se2lam could be used instead of ORB_SLAM2.

Implementing a finite state machine in Cycle.js

Mike Chung — Fri, 09 Nov 2018 06:24:07 +0000

Originally published on Medium

Note: Check out other posts on programming a social robot using Cycle.js too:
1. Programming a social robot using Cycle.js
2. Implementing a finite state machine in Cycle.js

In this post, I'll show you how to implement a reactive social robot program as a finite state machine. We'll continue from where we left off in the previous post Programming a social robot using Cycle.js--so check it out if you haven't already! If you are in a hurry, here is the demo and complete code of what we are building in this post.

Making existing "travel personality quiz" program more complex

Previously, we programmed a tablet-face robot to test your travel personality. Concretely, we implemented a tablet-face robot program that

looks at a person when it sees one and
asks travel personality quiz questions as shown in this flowchart

as a Cycle.js application. Here are the demo at Stackbliz and complete code in GitHub from the previous post.

IMPORTANT!! The main package we use in the demo and in this post, cycle-robot-drivers/run, only works on Chrome browsers (>= 65.0.3325.181) for now.

Now, what if we want the robot to

look at a person only when the robot is waiting for a person's response,
stop asking a question if the robot cannot see a person and resume asking the question if it sees a person again, and
stop asking questions completely if a person abandons the robot, i.e., the robot does not see a person for more than 10 seconds.

How difficult would it be to update the existing program to have these additional behaviors? Try implementing the new behaviors on top of the travel personality quiz program.
What kind of challenges do you face?

From my experience, it was difficult to implement, or even just express the "stateful" behaviors in reactive programming. For example, to implement 1., I needed to know whether the robot is in the "waiting for a person's response" state but it wasn't clear how to represent such state in a scalable manner; I tried keeping all states in drivers (e.g., SpeechRecognitionAction emitting status events), as proxies (e.g., $lastQuestion in the previous code), or in higher-order streams, but none of them felt simple nor scalable. This was very concerning since many robot behaviors are expressed and implemented as stateful behaviors.

To address this problem, I propose using finite state machines to clearly express the desired robot behaviors. In the following, I first present a pattern for implementing a finite state machine in a reactive programming framework (Cycle.js) without scarifying maintainability. Then I demonstrate a use case of the FSM pattern via implementing the first additional behavior.

What is a finite state machine?

A finite state machine (FSM) is a computational model that can be used to represent and control execution flow. Due to their simplicity, FSMs have been frequently used by roboticists, UI developers and many others for a long time. An FSM we are using in this post is comprised of five parts:

A set of states, e.g., 'SAY_SENTENCE', 'WAIT_FOR_RESPONSE', etc.
A set of variables, e.g., currentSentence = 'Can you see yourself working online?'
A set of inputs: e.g., VALID_RESPONSE, INVALID_RESPONSE, etc.
A set of outputs: e.g., speechSynthesisAction = 'Can you see yourself working online?'
A transition function that takes a state, variable, and input and returns a state, variable, and output.

If you are familiar with FSMs, the FSM we are using is a mealy machine extended with "variables".
Like a mealy machine, it has the following constraints:

the state set is a finite set
the FSM can only be in one state at a time in the state set
the transition function is deterministic; given a state, variable, and input the function always returns the same new state, new variable, and new output.

Representing the "travel personality quiz" program as an FSM

We'll start from representing the "travel personality test" program we implemented in the previous post as an FSM:

Here we have three states, PEND, SAY, LISTEN, and five input types, START, SAY_DONE, VALID_RESPONSE, INVALID_RESPONSE, and DETECTED_FACE. We omitted variables associated with each state and outputs associated with each transition for visual clarity.

Notice that we use verbs as state names (as a popular robotics FSM library SMACH does). This is because we define the states based on distinct actions each state is performing, where the distinct actions are triggered by outputs emitted from transitions. You may have wondered why we did not create each state in the travel quiz flowchart as an individual state, e.g., ASK_CAREER_QUESTION, ASK_WORKING_ABROAD_QUESTION, ASK_FAMILY_QUESTION, etc. This is because representing the states that behave the same except the sentence the robot says with a single SAY state with a variable currentSentence (not shown in the diagram) yields the simpler, more maintainable FSM.

The inputs can be considered as the events that could occur in each state and are originated from actions, e.g., SAY_DONE, sensors, e.g., DETECTED_FACE, or external systems, e.g. START. We represent an input as a type-value pair. For example, the VALID_RESPONSE type input is paired with a value "yes" or "no", which is used to determine the transition between LISTEN to SAY (input values are not shown in the graph).

Now, let's update the FSM to express the first additional behavior mentioned above: looking at a person only when the robot is waiting for a person's response.

All we did here is remove the two self-loop transitions from the PEND and SAY states to stop the robot from looking at a person while the FSM is in those states.

Implementing the "travel personality test" FSM using Cycle.js

Let's now implement the "travel personality test" FSM we defined above using Cycle.js.

First, we'll try to define the FSM in javascript as follows:

const State = {
  PEND: 'PEND',
  SAY: 'SAY',  //_SENTENCE
  LISTEN: 'LISTEN',  //_FOR_RESPONSE
};

const InputType = {
  START: `START`,
  SAY_DONE: `SAY_DONE`,
  // QUIZ_DONE: is not an input type but a transition
  VALID_RESPONSE: `VALID_RESPONSE`,
  INVALID_RESPONSE: `INVALID_RESPONSE`,
  DETECTED_FACE: `DETECTED_FACE`,
};

function transition(state, variables, input) {  // a dummy transition function
  const newState = state;
  const newVariables = variables;
  const newOutputs = null;
  return {
    state: newState,
    variables: newVariables,
    outputs: newOutputs,
  };
}

/**
 * // Example state, variables, input, and outputs
 * const state = State.PEND;
 * const variables = {
 *   sentence: 'You are a vacationer!',
 * };
 * const input = {
 *   type: InputType.START,
 *   value: null,
 * };
 * const outputs = {
 *   SpeechSynthesisAction: {
 *     goal: 'You are a vacationer!'
 *   },
 *   SpeechRecognitionAction: {
 *     goal: {}
 *   },
 *   TabletFace: {
 *     goal: {
 *       type: 'SET_STATE',
 *       value: {
 *         leftEye: {x: 0.5, y: 0.5},
 *         rightEye: {x: 0.5, y: 0.5},
 *       },
 *     }},
 *   },
 * }
 */

Here we define the set of states State, the set of input types InputType, and the transition function transition. The sets for the variables and outputs of the FSM are not explicitly defined, but I provided example values that the variables and outputs can take in the comment.

Setting up FSM in Cycle.js

We'll now setup the FSM as a Cycle.js application. You can fork the Stackblitz demo code and start coding or set up a Cycle.js application.
For the latter, create a folder:

mkdir my-second-robot-program
cd my-second-robot-program

Download package.json, .babelrc, index.html, create an empty index.js file in the folder, and run npm install to install the required npm packages. After installing, you can run npm start to build and start the web application--that does nothing at this point.

Now add the following code in index.js:

import xs from 'xstream';
import {runRobotProgram} from '@cycle-robot-drivers/run';

const State = {
// ...
const InputType = {
// ...
function transition(state, variables, input) {  // a dummy transition function
// ...

function input(  // a dummy input function
  start$,
  speechRecognitionActionResult$,
  speechSynthesisActionResult$,
  poses$,
) {
  return xs.never();
}

function output(machine$) {  // a dummy output function
  return {
    SpeechSynthesisAction: xs.never(),
    SpeechRecognitionAction: xs.never(),
    TabletFace: xs.never(),
  };
}

function main(sources) {
  const input$ = input(
    sources.TabletFace.load,
    sources.SpeechSynthesisAction.result,
    sources.SpeechRecognitionAction.result,
    sources.PoseDetection.poses,
  );

  const defaultMachine = {
    state: State.PEND,
    variables: {
      sentence: null,
    },
    outputs: null,
  };
  const machine$ = input$.fold((machine, input) => transition(
    machine.state, machine.variables, input
  ), defaultMachine);

  const sinks = output(machine$);
  return sinks;
}

runRobotProgram(main);

If you run the application, it should load a robot face that still does nothing on your browser.

The most important thing to notice here is that we divide the main function into three functions; input, transition, and output. The input function takes incoming streams in sources and returns a stream that emits the FSM's input values. We then use the fold xstream operator on the returned stream ($input) to trigger the FSM's transition function. Note that the fold operator is like Array.prototype.reduce for streams; it takes

an accumulator function that takes an emitted value (e.g., an FSM input value, input) and a previous output of the accumulator function (e.g., the latest FSM status, machine) or a seed value and
an initial output of the accumulator function (e.g., the initial FSM status, defaultMachine).

Finally, the output function takes the stream that emits FSM status ($machine) and returns outgoing streams.

Input, transition, and output

Let's implement the three functions.
First, update the dummy input function to:

// ...
const Response = {
  YES: 'yes',
  NO: 'no',
}

function input(
  start$,
  speechRecognitionActionResult$,
  speechSynthesisActionResult$,
  poses$,
) {
  return xs.merge(
    start$.mapTo({type: InputType.START}),
    speechRecognitionActionResult$
      .filter(result =>
        result.status.status === 'SUCCEEDED'
        && (result.result === Response.YES || result.result === Response.NO)
      ).map(result => ({
        type: InputType.VALID_RESPONSE,
        value: result.result,
      })),
    speechSynthesisActionResult$
      .filter(result => result.status.status === 'SUCCEEDED')
      .mapTo({type: InputType.SAY_DONE}),
    speechRecognitionActionResult$
      .filter(result =>
        result.status.status !== 'SUCCEEDED'
        || (result.result !== Response.YES && result.result !== Response.NO)
      ).mapTo({type: InputType.INVALID_RESPONSE}),
    poses$
      .filter(poses =>
        poses.length === 1
        && poses[0].keypoints.filter(kpt => kpt.part === 'nose').length === 1
      ).map(poses => {
        const nose = poses[0].keypoints.filter(kpt => kpt.part === 'nose')[0];
        return {
          type: InputType.DETECTED_FACE,
          value: {
            x: nose.position.x / 640,  // max value of position.x is 640
            y: nose.position.y / 480,  // max value of position.y is 480
          },
        };
      }),
  );
}
// ...

Try testing whether the input function is behaving properly. For example, you can attach the addListener xstream operator to the returned $input stream and return some outgoing streams from the output function.
Like this:

// ...
import delay from 'xstream/extra/delay'
function output(machine$) {
  return {
    SpeechSynthesisAction: xs.of('Hello world!').compose(delay(1000)),
    SpeechRecognitionAction: xs.of({}).compose(delay(1000)),
    TabletFace: xs.never(),
  };
}

function main(sources) {
  const input$ = input(
    sources.TabletFace.load,
    sources.SpeechSynthesisAction.result,
    sources.SpeechRecognitionAction.result,
    sources.PoseDetection.poses,
  );
  input$.addListener({next: value => console.log('input', value)})
// ...

Do you see the expected outputs on your browser console? You should see many inputs with the DETECTED_FACE type if the robot is detecting a person.

Let's now remove the dummy transition function and create a new one:

// ...
const State = {
// ...
const InputType = {
// ...
// // Remove the dummy transition function
// function transition(state, variables, input) {  // a dummy transition function
// ...
const Response = {
// ...
function input(
// ...

function createTransition() {
  const Sentence = {
    CAREER: 'Is it important that you reach your full career potential?',
    ONLINE: 'Can you see yourself working online?',
    FAMILY: 'Do you have to be near my family/friends/pets?',
    TRIPS: 'Do you think short trips are awesome?',
    HOME: 'Do you want to have a home and nice things?',
    ROUTINE: 'Do you think a routine gives your life structure?',
    JOB: 'Do you need a secure job and a stable income?',
    VACATIONER: 'You are a vacationer!',
    EXPAT: 'You are an expat!',
    NOMAD: 'You are a nomad!',
  };

  const flowchart = {
    [Sentence.CAREER]: {
      [Response.YES]: Sentence.ONLINE,
      [Response.NO]: Sentence.FAMILY,
    },
    [Sentence.ONLINE]: {
      [Response.YES]: Sentence.NOMAD,
      [Response.NO]: Sentence.VACATIONER,
    },
    [Sentence.FAMILY]: {
      [Response.YES]: Sentence.VACATIONER,
      [Response.NO]: Sentence.TRIPS,
    },
    [Sentence.TRIPS]: {
      [Response.YES]: Sentence.VACATIONER,
      [Response.NO]: Sentence.HOME,
    },
    [Sentence.HOME]: {
      [Response.YES]: Sentence.EXPAT,
      [Response.NO]: Sentence.ROUTINE,
    },
    [Sentence.ROUTINE]: {
      [Response.YES]: Sentence.EXPAT,
      [Response.NO]: Sentence.JOB,
    },
    [Sentence.JOB]: {
      [Response.YES]: Sentence.ONLINE,
      [Response.NO]: Sentence.NOMAD,
    },
  };

  // this transitionTable is a dictionary of dictionaries and returns a function
  //   that takes previous "variables" and "inputValue" and returns a current
  //   FSM status; {state, variable, outputs}
  // this transitionTable is a dictionary of dictionaries and returns a function
  //   that takes previous "variables" and "inputValue" and returns a current
  //   FSM status; {state, variable, outputs}
  const transitionTable = {
    [State.PEND]: {
      [InputType.START]: (prevVariables, prevInputValue) => ({
        state: State.SAY,
        variables: {sentence: Sentence.CAREER},
        outputs: {SpeechSynthesisAction: {goal: Sentence.CAREER}},
      }),
    },
    [State.SAY]: {
      [InputType.SAY_DONE]: (prevVariables, prevInputValue) => (
          prevVariables.sentence !== Sentence.VACATIONER
          && prevVariables.sentence !== Sentence.EXPAT
          && prevVariables.sentence !== Sentence.NOMAD
        ) ? {  // SAY_DONE
          state: State.LISTEN,
          variables: prevVariables,
          outputs: {SpeechRecognitionAction: {goal: {}}},
        } : {  // QUIZ_DONE
          state: State.PEND,
          variables: prevVariables,
          outputs: {done: true},
        },
    },
    [State.LISTEN]: {
      [InputType.VALID_RESPONSE]: (prevVariables, prevInputValue) => ({
        state: State.SAY,
        variables: {sentence: flowchart[prevVariables.sentence][prevInputValue]},
        outputs: {
          SpeechSynthesisAction: {
            goal: flowchart[prevVariables.sentence][prevInputValue],
          },
          TabletFace: {goal: {
            type: 'SET_STATE',
            value: {
              leftEye: {x: 0.5, y: 0.5},
              rightEye: {x: 0.5, y: 0.5},
            },
          }},
        },
      }),
      [InputType.INVALID_RESPONSE]: (prevVariables, prevInputValue) => ({
        state: State.LISTEN,
        variables: prevVariables,
        outputs: {SpeechRecognitionAction: {goal: {}}},
      }),
      [InputType.DETECTED_FACE]: (prevVariables, prevInputValue) => ({
        state: State.LISTEN,
        variables: prevVariables,
        outputs: {
          TabletFace: {goal: {
            type: 'SET_STATE',
            value: {
              leftEye: prevInputValue,
              rightEye: prevInputValue,
            },
          }},
        }
      }),
    },
  };

  return function(prevState, prevVariables, prevInput) {
    console.log(prevState, prevVariables, prevInput);
    // excuse me for abusing ternary
    return !transitionTable[prevState]
      ? {state: prevState, variables: prevVariables, outputs: null}
      : !transitionTable[prevState][prevInput.type]
        ? {state: prevState, variables: prevVariables, outputs: null}
        : transitionTable[prevState][prevInput.type](prevVariables, prevInput.value);
  }
}

const transition = createTransition();

function output(machine$) {  // a dummy output function
// ...

Here we define and return the FSM's transition function inside the createTransition function.

Finally update the dummy output function to:

// ...
const transition = createTransition();

function output(machine$) {
  const outputs$ = machine$
    .filter(machine => !!machine.outputs)
    .map(machine => machine.outputs);
  return {
    SpeechSynthesisAction: outputs$
      .filter(outputs => !!outputs.SpeechSynthesisAction)
      .map(output => output.SpeechSynthesisAction.goal),
    SpeechRecognitionAction: outputs$
      .filter(outputs => !!outputs.SpeechRecognitionAction)
      .map(output => output.SpeechRecognitionAction.goal),
    TabletFace: outputs$
      .filter(outputs => !!outputs.TabletFace)
      .map(output => output.TabletFace.goal),
  };
}

function main(sources) {
// ...

Try running the application and test whether it behaves as we defined in the FSM.

You just implemented a social robot program as an FSM!

Relation to the Model-View-Intent pattern

The FSM pattern is an application of the Model-View-Intent (MVI) pattern, an adaptation of Model-View-Controller in reactive programming, where "intent" is input, "model" is FSM status, and "view" is output. In addition to the MVI pattern, the FSM pattern additionally requires a specific structure for the "model"/FSM status and the "update"/transition.

Updating the "travel personality quiz" FSM

The true power of the FSM pattern is its maintainability. The crux of the FSM pattern is dividing the main function into the three functions that have separate concerns:

the input function that focuses on turning incoming streams into "input" that the FSM can work with and
the transition function implements the FSM's transition function.
the output function that maps the outputs returned from transition into the outgoing streams (sinks in Cycle.js) to make side effects, e.g., trigger actions.

This separation allows programmers to only update the portion of code in the two functions when they need to make the program more complex.

For example, if we were to implement the rest of additional behaviors mentioned in the Making "travel personality quiz" program more complex section, we'll need to first update the FSM to reflect the new desired behavior, e.g.:

and update the input and transition functions accordingly. Checkout the complete code to see how I updated the input and transition functions to implement the remaining additional behaviors.

The biggest challenge for using FSM is defining FSM. If you are using the FSM pattern and having problems with it, double check the current definition of your state machine. For example, look for the redundant states or input types that make updating the transition function cumbersome (merge them into one state with variables), or look for state or input type that is not being used as intended for (add new necessary states or input types). Another point to check is, making sure your FSM is taking reactive programming approach, e.g., make sure the three functions (input, transition, output) are as pure as possible. Defining effective FSM is art, but I believe using FSMs in reactive programming greatly helps the programmers to better organize their programs.

Thank you for reading! I hope I got you interested in using FSMs in Cycle.js. Let me know if something isn’t clear, and I’d be happy to chat.

My name is Mike Chung. I'm a graduate student interested in the field of human-robot interaction and machine learning. You can reach me on Twitter and on GitHub.

Programming a social robot using Cycle.js

Mike Chung — Tue, 06 Nov 2018 20:23:59 +0000

Originally published on Medium

Note: Check out other posts on programming a social robot using Cycle.js too:
1. Programming a social robot using Cycle.js
2. Implementing a finite state machine in Cycle.js

In this post, I'll show you how to program a social robot using Cycle.js. I assume you are familiar with reactive programming. If you are not, check out The introduction to Reactive Programming you've been missing. If you are eager to get your hands dirty, jump to the Implementing "travel personality test" section.

What is a social robot?

Wikipedia introduces it as:

A social robot is an autonomous robot that interacts and communicates with humans or other autonomous physical agents by following social behaviors and rules attached to its role.

Cynthia Breazel, the mother of social robots, once said:

In short, a socialable robot is socially intelligent in a human-like way, and interacting with it is like interacting with another person. At the pinnacle of achievement, they could befriend us, as we could them.

I see social robots as embodied agents whose main task is to communicate with humans to help humans. So, interactive robots for education or eldercare fit my definition the best.

Programming social robots is similar to programming web applications. In both cases, programmers write code for handling inputs, e.g., a button click or sensor reading, and outputting data accordingly, e.g., displaying information on screen or sending control signals to motors. The major difference is programming social robots involves working with multi-modal inputs and outputs, e.g., speech and motion, to interact with humans instead of solely using a screen interface.

In this post, I'll use a tablet-face robot for demonstration purposes. The tablet-face robot is just a web application running on a tablet, but we'll make it speak, listen, and see you to make it more like a "social robot".

What is Cycle.js?

Cycle.js is a functional and reactive JavaScript framework. It is an abstraction that separates all side effect producing code into drivers so the core application logic code remains pure in one "main" function. The author of Cycle.js describes a web application as a dialogue between a human and a computer. If we assume both are functions, the human as y = driver(x) and the computer as x = main(y) where x and y are streams in the context of reactive programming, then the dialogue is simply two functions that react to each other via their input stream, which is an output of the another function.

Why Cycle.js for social robots?

To me, Cycle.js essentially enforces functional reactive programming, e.g., using streams, and ports and adapters architecture, e.g., separating side effects, to make it easy to create and understand complex and concurrent interactive programs--beyond web applications. This is why I chose Cycle.js for programming a social robot. I believe the patterns enforced by Cycle.js will help programmers to battle the concurrency problems originated from supporting multi-modal interactions and stay in control when complexity of the desired robot behavior grows. In fact, you don't need to use Cycle.js if you can enforce the patterns yourself. For example, you could use Yampa with reactimate, Flapjax, or one of ReactiveX stream libraries to do this in a language in which your robot's API is available.

Implementing "travel personality test"

Enough backgrounds, we'll now create a robot program that tests your travel personality. Specifically, we'll make the robot

look at you while you are interacting with the robot and
ask questions as shown in this flowchart.

If you are curious, check out the complete code and the demo at Stackblitz.

IMPORTANT!! For now, the cycle-robot-drivers/run package we use in this post and in the Stackblitz demo only work on Chrome browsers (>= 65.0.3325.181).

The code examples in this post assume you are familiar with JavaScript ES6. To build code, I use browserify and Babel here, but feel free to use a build tool and a transpiler you prefer. If you are not familiar with them, just fork the Stackblitz demo code and start coding!

Let's set up a Cycle.js application. Create a folder:

mkdir my-robot-program
cd my-robot-program

Then download package.json, .babelrc, index.html and create an empty index.js file in the folder. Run npm install to install the required npm packages. After installing, you can run npm start to build and start the web application that does nothing.

Now add the following code in index.js:

import xs from 'xstream';
import {runRobotProgram} from '@cycle-robot-drivers/run';

function main(sources) { }

runRobotProgram(main);

Then run this application, e.g., by running npm start. It should load a robot face on your browser.

We just successfully set up and ran a Cycle.js application!

Robot, look at a face!

We'll now focus on implementing the first feature--looking at a face.

Let's make the robot just move its eyes by adding the following code in main:

// ...

// "sources" is a Cycle.js term for the input of "main" / the output of "drivers"
function main(sources) {
  // "const" (and "let") is a javascript ES6 feature
  const sinks = {
    TabletFace: xs.periodic(1000).map(i => ({
        x: i % 2 === 0 ? 0 : 1,  // horizontal left or right
        y: 0.5  // vertical center
      })).map(position => ({
        type: 'SET_STATE',
        value: {
          leftEye: position,
          rightEye: position
        }
      }))
  };
  // "sinks" is a Cycle.js term for the output of "main" / the input of "drivers"
  return sinks;
}

// ...

Here we are sending commands to the TabletFace driver by returning the sink.TabletFace stream from main. The periodic xstream factory creates a stream emitting an incremental number every second and the map xstream operator create a new stream that turns the emitted numbers into positions and another new stream that turns the emitted positions into control commands. If you run the updated application, the robot should look left and right repeatedly.

Let's now work on detecting a face by adding more code in main:

// ...

function main(sources) {
  sources.PoseDetection.poses.addListener({
    next: (poses) => console.log('poses', poses)
  });

  // ...
}

// ...

Here we use the addListener xstream operator to add a callback function that prints the detected pose data to the poses stream, the stream returned from the PoseDetection driver.

When you run the application you should see arrays of objects printed to your browser's console. If you don't see them, make sure you are visible to the camera and being detected via the pose visualizer located below the robot face (try scroll down). Each array represents detected poses at current moment, which has the following format:

const poses = [
  // the first detected person
  {
    "score": 0.32371445304906,
    "keypoints": [
      {
        "part": "nose",
        "position": {
          "x": 253.36747741699,
          "y": 76.291801452637
        },
        "score": 0.99539834260941
      },
      {
        "part": "leftEye",
        "position": {
          "x": 253.54365539551,
          "y": 71.10383605957
        },
        "score": 0.98781454563141
      },
      // ...
  },
  // the second detected person if there is one
  {
    "score": 0.22838506316132706,
    "keypoints": [
      {
        "part": "nose",
        "position": {
          "x": 236.58547523373466,
          "y": 360.03672892252604
        },
        "score": 0.9979155659675598
      },
      // ...
    ]
  },
  // ...
]

While the application is running, try disappearing from the camera.
You should see one less object in the poses array. Also try hiding one of your ears by turning your head left or right. You should not see an object that has a string nose for its part field in the keypoints array.

Now that we know how to move the robot's eyes and retrieve detected face data, let's put them together to make the robot look at a face. Concretely, we'll make the robot's eyes follow a detected person's nose. Update main as follows:

// ...

function main(sources) {
  const sinks = {
    TabletFace: sources.PoseDetection.poses
      .filter(poses =>
        // must see one person
        poses.length === 1
        // must see the nose
        && poses[0].keypoints.filter(kpt => kpt.part === 'nose').length === 1
      ).map(poses => {
        const nose = poses[0].keypoints.filter(kpt => kpt.part === 'nose')[0];
        return {
          x: nose.position.x / 640,  // max value of position.x is 640
          y: nose.position.y / 480  // max value of position.y is 480
        };
      }).map(position => ({
        type: 'SET_STATE',
        value: {
          leftEye: position,
          rightEye: position
        }
      }))
  };
  return sinks;
}

// ...

Here we are sending commands to the TabletDriver by using the stream created from the output stream of the PoseDetection driver (sources.PoseDetection.poses).
To convert pose data into control commands, we use the filter xstream operator to filter pose data to the ones containing only one person whose nose is visible. Then we use the map xstream operator twice to convert the detected nose positions into eye positions and turn the eye positions into control commands.

We have made the robot look at a face!

Exercise ideas:

Make the robot look at one of your hands instead of your nose?
Make the robot smile (happy expression) when you are looking away from the camera?

Taking a closer look at `runRobotProgram`

While following code examples above, you may have wondered:

when and where is the TabletFace driver created
how and when a driver produces side effects

Here is the answer to the first question: the two drivers we used in the example code, TabletFace and PoseDetection, are created in runRobotProgram.
Normally when you program a Cycle.js app, you need to create drivers explicitly and pass them to the Cycle.js run function. We skipped this step because we used runRobotProgram that creates the required drivers for programming a tablet-face robot and calls Cycle.js run for us. The runRobotProgram function is a wrapper function for Cycle.js run that

creates five drivers, AudioPlayer, SpeechSynthesis, SpeechRecognition, TabletFace, PoseDetection
creates and sets up five action components FacialExpressionAction, AudioPlayerAction, TwoSpeechbubblesAction, SpeechSynthesisAction, SpeechRecognitionAction to allow programmers to use them as drivers, and
calls Cycle.js run with the created drivers and actions.

In fact, if you are comfortable with Cycle.js, you could use Cycle.js run instead of runRobotProgram to have more control over drivers and actions. You could also create a new runRobotProgram function that provides drivers for your own robot that is not a tablet-face robot!

Regarding the second question, check out this page from the Cycle.js website.

Robot, ask questions!

We'll now focus on implementing the second feature--asking the travel personality quiz questions.

First, we'll represent the quiz flowchart as a dictionary of dictionaries for convenience. Add the following code:

// ...
import {runRobotProgram} from '@cycle-robot-drivers/run';

const Question = {
  CAREER: 'Is reaching your full career potential important to you?',
  ONLINE: 'Can you see yourself working online?',
  FAMILY: 'Do you have to be near my family/friends/pets?',
  TRIPS: 'Do you think short trips are awesome?',
  HOME: 'Do you want to have a home and nice things?',
  ROUTINE: 'Do you think a routine gives your life structure?',
  JOB: 'Do you need a secure job and a stable income?',
  VACATIONER: 'You are a vacationer!',
  EXPAT: 'You are an expat!',
  NOMAD: 'You are a nomad!'
};

const Response = {
  YES: 'yes',
  NO: 'no'
};

const transitionTable = {
  [Question.CAREER]: {
    [Response.YES]: Question.ONLINE,
    [Response.NO]: Question.FAMILY,
  },
  [Question.ONLINE]: {
    [Response.YES]: Question.NOMAD,
    [Response.NO]: Question.VACATIONER,
  },
  [Question.FAMILY]: {
    [Response.YES]: Question.VACATIONER,
    [Response.NO]: Question.TRIPS,
  },
  [Question.TRIPS]: {
    [Response.YES]: Question.VACATIONER,
    [Response.NO]: Question.HOME,
  },
  [Question.HOME]: {
    [Response.YES]: Question.EXPAT,
    [Response.NO]: Question.ROUTINE,
  },
  [Question.ROUTINE]: {
    [Response.YES]: Question.EXPAT,
    [Response.NO]: Question.JOB,
  },
  [Question.JOB]: {
    [Response.YES]: Question.ONLINE,
    [Response.NO]: Question.NOMAD,
  }
};

function main(sources) {
// ...

Notice that I modified the quiz questions to change all response choices to "yes" and "no".

Let's now make the robot ask questions and take your verbal responses.
First, we'll make the robot to just say the first question on start, i.e., on loading the robot's face, and start listening after saying something:

// ...
function main(sources) {
  sources.SpeechRecognitionAction.result.addListener({
    next: (result) => console.log('result', result)
  });
  // ...
  const sinks = {
    TabletFace: sources.PoseDetection.poses
      .filter(poses =>
      // ...
    SpeechSynthesisAction: sources.TabletFace.load.mapTo(Question.CAREER),
    SpeechRecognitionAction: sources.SpeechSynthesisAction.result.mapTo({})
  };
  return sinks;
}
// ...

Here we are sending commands to the SpeechSynthesisAction driver and the SpeechRecognitionAction driver by returning the created streams via sink.SpeechSynthesisAction and sink.SpeechRecognitionAction from main.
The input stream for the SpeechSynthesisAction driver emits Question.Career on the tablet-face-loaded event emitted in the sources.TabletFace.load stream.
The input stream for the SpeechRecognitionAction driver emits an empty object ({}) on finishing the speech synthesis action event emitted in the sources.SpeechSynthesisAction.result stream.
Both streams are created using the mapTo xstream operator.
We also print out events emitted in the sources.SpeechRecognitionAction.result stream using the addListener xstream operator.

When you run the application, you should hear the robot saying "Is reaching your full career potential important to you?" and see the output of the SpeechRecognitionAction printed to your browser's console. The output has the following format:

const result = {
  "result": "yes",  // transcribed texts
  "status": {
    "goal_id": {  // a unique id for the executed action
      "stamp": "Mon Oct 01 2018 21:49:00 GMT-0700 (PDT)",  // "Date" object
      "id": "h0fogq2x0zo-1538455335646"
    },
    "status": "SUCCEEDED"  // "SUCCEEDED", "PREEMPTED", or "ABORTED"
  }
}

Try saying something and see how well it hears you.

Now we want to improve the program to make the robot ask more than one question. For example, we can try to send questions as commands to the SpeechSynthesisAction driver whenever the robot hears an appropriate answer, i.e., "yes" or "no". Let's try to express this by updating the code above as follows:

// ...
function main(sources) {
  // ...
  const sinks = {
    TabletFace: sources.PoseDetection.poses
      .filter(poses =>
      // ...
    SpeechSynthesisAction: xs.merge(
      sources.TabletFace.load.mapTo(Question.CAREER),
      sources.SpeechRecognitionAction.result.filter(result =>
        result.status.status === 'SUCCEEDED'  // must succeed
        && (result.result === 'yes' || result.result === 'no') // only yes or no
      ).map(result => result.result).map(result => {
        // Hmm...
      })
    ),
    SpeechRecognitionAction: sources.SpeechSynthesisAction.result.mapTo({})
  };
  return sinks;
}
// ...

Here we are merging the commands from the stream that emits the first question (sources.TabletFace.load.mapTo(Question.CAREER)) and the commands from the stream that emits a subsequent question on hearing "yes" or "no" (sources.SpeechRecognitionAction.result.filter(// ...) using the merge xstream factory.

There is one problem with this approach. We cannot figure out which question to return in the second stream since the question is dependent on the last question the robot asked, which also is dependent on the last last question and so on. In other words, we need a previous output of the current stream we are creating as a input to the current stream.

To solve this circular dependency problem, we adopt the proxy pattern by updating the main function as follows:

// ...
function main(sources) {
  // ...
  const lastQuestion$ = xs.create();
  const question$ = xs.merge(
    sources.TabletFace.load.mapTo(Question.CAREER),
    sources.SpeechRecognitionAction.result.filter(result =>
      result.status.status === 'SUCCEEDED'  // must succeed
      && (result.result === 'yes' || result.result === 'no') // only yes or no
    ).map(result => result.result)
    .startWith('')
    .compose(sampleCombine(
      lastQuestion$
    )).map(([response, question]) => {
      return transitionTable[question][response];
    })
  );
  lastQuestion$.imitate(question$);

  const sinks = {
    TabletFace: sources.PoseDetection.poses
      .filter(poses =>
      // ...
    SpeechSynthesisAction: question$,
    SpeechRecognitionAction: sources.SpeechSynthesisAction.result.mapTo({})
  };
  return sinks;
}
// ...

Here we have moved creating the code for a stream for sink.SpeechSynthesisAction outside of the sink object definition. We create an empty proxy stream lastQuestion$ using the create xstream factory and use it when creating the question$ stream.
Then use the imitate xstream operator to connect the proxy stream, lastQuestion$, to its source stream, question$. We also use the compose and sampleCombine xstream operators to combine events from the stream originated from sources.SpeechRecognitionAction.result and the lastQuestion$ stream. Note that I add $ at the end of stream variable names to distinguish them from other variables as Cycle.js authors do. Try the updated application and see if the robot asks more than one question if you respond to it with "yes" or "no".

You may have wondered when did we update the code to send the "start listening" command ({}) after all questions. We didn't update the code; the code we had before already works as desired since the sources.SpeechSynthesisAction.result stream emits data on finishing every synthesized speech.

One problem you may have faced is the robot failing to ask a next question when it hears an answer that is not "yes" or "no", e.g., by mistake. In such case, the robot should start listening again to give the person a chance to correct their answer. Let's update the code to fix the problem:

// ...
    SpeechSynthesisAction: question$,
    SpeechRecognitionAction: xs.merge(
      sources.SpeechSynthesisAction.result,
      sources.SpeechRecognitionAction.result.filter(result =>
        result.status.status !== 'SUCCEEDED'
        || (result.result !== 'yes' && result.result !== 'no')
      )
    ).mapTo({})
  };
  return sinks;
}
// ...

Run the updated application. You should see that the robot will continue to listen and print whatever it hears to the console until it hears "yes" or "no" before asking a next question.

We are done at this point. Try taking the travel personality quiz to find out your travel personality and enjoy!

Exercise ideas:

Implement one of "The 24 Most Important Flowcharts Of All Time" to make the robot answer one of the biggest questions in life?
Make your robot to read Tweets from a certain Twitter user whenever that user post a tweet, e.g., using a Twitter API?
Make your robot alert you whenever a stock's price goes below or above a certain threshold?

Please let me know if something isn’t clear, and I’d be happy to chat about your concerns. Thank you for reading!

Miscellaneous

Fun fact: many social robots today use a screen as a face.
Check out RxJS Marbles for visualizing stream operators with marble diagrams, e.g., interval (periodic in xstream), map, filter, mapTo, and merge.
If you are a ROS user, check out my experimental Cycle.js driver for communicating with ROS using roslibjs.
Help me improve cycle-robot-drivers library by participating in this brief survey!

My name is Mike Chung. I'm a graduate student interested in the field of human-robot interaction and machine learning. You can reach me on Twitter and on GitHub.

DEV Community: Mike Chung

Functional Listening and its Generalization: Towards Having Control Over Your Productivity

Sounds

Lights

Screen Color Temperature, Dynamic Wallpaper

Productivity

Self-Control via Environment Control

Footnotes

Robo-Observability

Logs

Metrics

Final Thoughts

Acknowledgements

Getting started with programming, kind of

Testing robotics systems in fast-paced startups

Why is Testing Robotics Systems Hard?

Where to Start with Testing: General Techniques

Eisenhower Matrix for Test Prioritization

Test Scenario Identification Procedure

Ensuring Testability as a Startup Grows

Series A: 5-20 Employees

Series B: 21-200 Employees

Series C: 201-2000 Employees

Beyond 2000 Employees

Closing notes

Acknowledgements

Significant Revisions

Footnotes

Getting started with robotics: Just do it!

Installed ROS Lunar on macOS Mojave! But...

Motivation

Initial Installation

Testing and More Installation

Adding More ROS Packages

Afterthoughts

git-issue: an offline-friendly project management tool with potential

Background

The Pros and Cons List

My Honest Opinion

Misc.

My failed attempt to set up an old MacbookPro as a deep learning workstation

Understanding challenges with large robotics system development

There aren't many performant off-the-shelve tools

There aren't many generalist robotics systems engineer

Gathering system requirements or software specifications is not trivial

Maintenance and testing are challenging

Miscellaneous

Please help me build a cloud visual SLAM system for cellphones

Updates

Implementing a finite state machine in Cycle.js

Making existing "travel personality quiz" program more complex

What is a finite state machine?

Representing the "travel personality quiz" program as an FSM

Implementing the "travel personality test" FSM using Cycle.js

Setting up FSM in Cycle.js

Input, transition, and output

Relation to the Model-View-Intent pattern

Updating the "travel personality quiz" FSM

Programming a social robot using Cycle.js

What is a social robot?

What is Cycle.js?

Why Cycle.js for social robots?

Implementing "travel personality test"

Robot, look at a face!

Taking a closer look at runRobotProgram

Robot, ask questions!

Miscellaneous

Taking a closer look at `runRobotProgram`