This is how I like to imagine it: one day, Filip, our Platform Manager, got anxious.
The lack of internal comfort wasn't that big and resembled something more of irritation rather than overwhelming frustration, but it was significant enough not to ignore it.
Filip was troubled by reality. To be specific, a certain characteristic of the reality of working in a product company – predictability. Even if the company could call itself innovative, actively heading toward the improvement of business communication, the innovativeness had to be put in the frames of controllable progress. After all, solutions must be delivered to customers regularly and efficiently.
Product Iterations, sprints, releases, bugs, fixes, meetings, prototypes – a solid and safe system of product development you could depend on. But after a longer period of time, even the most reliable solution seems boring and tediously predictable.
Yes, we had a perspective of innovation.
Yes, we were moving toward its realization.
Everything seemed cool on the surface, yet something wasn't exactly in it's place. That something was a tiny bit of unrestricted creativity.
Filip started to dream about a touch of unpredictability.
It's one of these moments when the history of friendly cooperation with Karolina, our Chief Product Officer, makes possible things that in normal conditions could never happen. For sure, we should take care of this piece of chaos. We cannot allow it to get out of control completely. We could benefit from it, but it's difficult to hold in your hand a piece of infinity. One day will be enough, Filip thought. One day.
I could never know for sure what the real order of events was, but this is how I like to imagine the origins of Side Project Day.
When it comes to facts, there was a day at LiveChat when the company’s life just stopped. The whole team (except for our irreplaceable Support Heroes) divided into small teams and focused on creating solutions for a specific problem, which was bringing online communication to the level of face-to-face conversations. Or higher.
Communication better than face2face
We've already been working toward making the best of online communication for quite some time. With the use of our software, companies could connect with customers with whom they could never talk in real life. But we hoped that Side Project Day would be a breath of fresh air on the subject.
Team 4 hasn't let us down. Ziemowit, Aleksandra, Krystian, Maciej, and Jakub examined available technology and created a solution allowing users with motor disabilities to communicate with businesses in an efficient way.
The ultimate inspiration
"Our inspiration for the project was Stephen Hawking", says Maciej Pankiewicz, the leader of Team 4. The famous scientist suffered from amyotrophic lateral sclerosis (ALS), a form of motor neuron disease that gradually paralyzed him over the decades. The team dug deeper into data to find out that ALS is the third most common neurodegenerative disease worldwide. Add to that, strokes, spinal cord injuries, and multiple sclerosis, which are all potential causes of paralysis. Suddenly, what seemed to be a narrow niche, turned out to be a huge group of people for whom speaking, typing, or clicking can be real barriers.
The integration
To create the LiveChat integration, Team 4 used LiveChat Moments and connected them with Teachable Machine by Google (TM). TM offers three different types of projects:
- Images
- Audio
- Poses
For this particular use case, they created a new Pose Project.
One of the biggest challenges was to teach the model recognizing head movements. Nodding corresponds with a "yes" answer, shaking your head both ways means "no". The model was also trained to recognize tilting your head to the right or left.
Team 4 decided to visualize the capabilities of their application on a specific use case. They designed a path one might take when dealing with medical issues. In the proposed scenario, left and right head tilts are used to choose from two options: treatment or prescription. Nodding (yes) and shaking (no) movements confirm or reject the selected option.
Because the model was very sensitive, they decided to use a timer to improve the experience of confirming a selected option. A user would have to nod for 3 seconds in order to successfully confirm an option.
After you train your model, TM lets you export it with some basic setup code for you to use in your integration.
The power of LiveChat Moments
The integration was created via the LiveChat Developer Console. To communicate with customers, Team 4 made use of LiveChat Moments. The communication goes as follows:
An agent sends a Moment to the customer. When the customer opens the Moment, the Pose Project loads in. (Enabling access to the camera and microphone on the customer side is required for the integration to work.)
Then, customers are presented with options to choose from. In this case, it's treatment or prescription. When the customer chooses and confirms an option, the integration composes a message that is then sent to the agent.
Some code
If you're curious to see how the integration works under the hood, the snippet of code below might give you some perspective.
/// To initialise a moment and load the Pose Project model:
window.onMomentAsyncInit = function (createMomentsSDK) {
createMomentsSDK({ title: "Side Project Day" }).then(() => {
init();
});
};
// Load the image model and setup the webcam
async function init() {
const modelURL = URL + "model.json";
const metadataURL = URL + "metadata.json";
// load the model and metadata
// Refer to tmImage.loadFromFiles() in the API to support files from a file picker
// or files from your local hard drive
// Note: the pose library adds "tmImage" object to your window (window.tmImage)
model = await tmImage.load(modelURL, metadataURL);
maxPredictions = model.getTotalClasses();
// Convenience function to setup a webcam
const flip = true; // whether to flip the webcam
webcam = new tmImage.Webcam(150, 150, flip); // width, height, flip
await webcam.setup(); // request access to the webcam
await webcam.play();
window.requestAnimationFrame(loop);
}
async function loop() {
webcam.update(); // update the webcam frame
await predict();
window.requestAnimationFrame(loop);
}
// run the webcam image through the image model
async function predict() {
// predict can take in an image, video or canvas html element
const prediction = await model.predict(webcam.canvas);
const predictions = {};
for (let i = 0; i < maxPredictions; i++) {
predictions[prediction[i].className] = prediction[i].probability;
}
...
// Your logic based on predictions
}
The outcome and the future
Team 4 ended up with well-programmed mechanism, trained to recognize Maciek's head. It actually allowed them to win Side Project Day, competing with 11 other teams of which you could say that they also have put significant effort into their solutions.
Maybe one day I will tell you about them.
Nevertheless, it's not the end of the story. Winning a Side Project Day, even if rewarding (management made sure of it 🤙), wasn't the primary goal. It was (and still is) to make communication better than face-to-face, which in this case means making digital communication available for people to whom face-to-face is often the only option.
Given that, Team 4 kept working.
They got in touch with Wrocław-based Work from bed – a marvelous marketing agency that employs people with motor disabilities. They met Krzysiek, who after a tragic water jump has his body paralyzed. As he's only able to move his head, he writes on the computer thanks to special sunglasses and a teether, which serves as an 'enter' key. Krzysiek's case is particularly interesting for them, as the integration is designed especially for this kind of person. He agreed to work with the team, helping them develop a MVP – a solution that would work for him personally.
What's more, a nurse helped the team get to the one case, a girl named Majka with the neuromuscular disorder (a combination of paralysis, increased tendon reflex activity, and hypertonia). Her father is also a developer and they’re considering involving him in this project.
First steps have been taken, but there's always more.
a word from Team 4
We're still looking for opportunities for collaborations, insights or partnerships. If you can help us or this project simply looks interesting to you, reach out to us! We'll gladly share our ideas and vision and talk about how we could work on it together.
You can contact us via DM, in the comments or send an email to l.wardega@livechat.com
PI's, sprints, releases, planning are good for everyday work. They really are. This way of work funnels creativity, skills and the knowledge of the whole LiveChat team into the structure of the company, allowing us to move forward, creating and developing products that help thousands of companies around the globe. This experiment showed us that this potential, when occasionally shaped in another way once in a while, can also produce a lasting effect. After all, we might've been able to touch the very core of creativity in a micro-scale – discovering the solution for the real problem of the real human one more time.
And maybe it was the thing that bothered Filip. This itching spark of creativity, waiting to be unleashed a bit more.
Tell me what you think!
If you'd like to develop your own app or integration and enhance the way businesses communicate, check out our Developer Platform. We can help you get accustomed to our platform, shape the idea and distribute it in our Marketplace.
Top comments (0)