DEV Community: Sai Phanindra

Console #79 -- Medusa, Firezone, and Unoriginal Rice Patties

Sai Phanindra — Sun, 14 Nov 2021 16:09:37 +0000

Sponsorship

Boost employee engagement with Axios HQ

Fortune 100s and startups alike are elevating company-wide comms with Axios HQ --- intuitive software that helps your team think more clearly, communicate more crisply, and send focused, effective updates.

What you can expect: rising transparency, engagement, and trust across the organization.

Easy to start, unlimited seats. Get your teams smarter, faster with Axios HQ.

See how it works. Get a sample newsletter built with Axios HQ.

Welcome to the * new Consolites** who joined since last week!* 👋

Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Subscribe now

Already subscribed? Why not spread the word by forwarding Console to the best engineer you know?

The Console Career Service

We've decided to re-vamp the Console job board to make it more engineer-centric! Where previously the job board was simply a curation of jobs that I felt the Console readers would be interested in, we'll now provide a white glove job finding service for our readers. The benefits of signing up include:

Automatic first-round interviews
One application, many jobs (1:N matching)
Free candidate preparation service
New opportunities updated regularly
All roles, from PM to SWE to BizOps
High potential, venture-backed, and open-source opportunities

Ready to sign up? Click below and sign up in less than 5 minutes.👇

Find My Next Job

Projects

medusa

Medusa is an open-source headless commerce engine that enables developers to create amazing digital commerce experiences.

language: JavaScript, stars: 1257, watchers: 11, forks: 98, issues: 92

last commit: October 23, 2021, first commit: January 18, 2020

social: https://twitter.com/medusajs
Github: https://github.com/medusajs/medusa

firezone

Firezone is a self-managed WireGuard-based VPN server and Linux firewall designed for simplicity and security.

language: Elixir, stars: 736, watchers: 8, forks: 18, issues: 8

last commit: November 12, 2021, first commit: April 22, 2020

Social: https://twitter.com/firezonevpn

GitHub: https://github.com/firezone/firezone

Unoriginal-Rice-Patty

Unoriginal-Rice-Patty is an example replay-based attack on Honda and Acura vehicles.

stars: 222, watchers: 11, forks: 20, issues: 1

last commit: August 04, 2021, first commit: August 03, 2021

Social: https://twitter.com/HackingInHeart

GitHub: https://github.com/HackingIntoYourHeart/Unoriginal-Rice-Patty

Patreon

*Console is powered by donations. We use your donations to grow the newsletter readership via advertisement. If you'd like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation *😊

Donate To Console

An Interview With Sebastian of Medusa

Hey Sebastian! Thanks for joining us! Let's start with your background. Where have you worked in the past and how did you learn how to program?

I have always been very interested in computers and programming, which resulted in me studying computer science; while studying I began working on different projects and eventually focused all my efforts on helping e-commerce businesses. When Oliver and I started working together we had a customer who wanted to migrate away from WooCommerce, we weren't able to find a solution that accommodated their requirements so we ended up building a custom solution, which after many iterations became Medusa.

Who or what are your biggest influences as a developer?

Interestingly my granddad happens to be one of the biggest influences in my path to becoming a developer. That is in spite of him not knowing anything about computers - he still calls me whenever he has to download a new app for his smartphone. The story is that he gave me one of his old work laptops when I was 12. It was my first computer and it didn't have an internet card so I couldn't use it to browse the internet; however, it did have internet explorer installed and together with the computer my grandad gave me HTML for Dummies which I read end to end. From this I learned to create small sites that I could browse for myself - it was really ridiculous the sites I made: things like a personal website with pictures of me and my family and buttons that changed the background color, that sort of stuff.

What is your favorite software tool?

VIM - in the age of VSCode everyone thinks I am a lunatic, but the native feel and portability of vim will keep me locked in forever.

If you could dictate that everyone in the world should read one book, what would it be?

Black Swan by Nassim Taleb has made a pretty big impression on me - am currently rereading for the 3rd time. The book stresses the fact that it is the rare, high impact events that dictate our world, not the steady and predictable. Understanding this fact and really allowing the idea to develop makes it clear that most of what so-called experts are saying on the news is based on wrong presumptions.

Have experienced this first hand while studying economics; while the mechanics of economic models are interesting at a mathematical level they are mostly useless as a foundation for planning the future.

If you had to suggest 1 person developers should follow, who would it be?

Brandur - despite not being the most active on social media he pretty much only posts high quality stuff that is worth the feed real estate.

If you could teach every 12 year old in the world one thing, what would it be and why?

Learn programming fundamentals; surely pursuing a career as a developer is not something that everyone should do, but I think everyone could benefit from just having a little understanding of programming. Why? Primarily efficiency:

Knowing a bit of programming gives you a framework for understanding at an abstract level what is possible with tech.

Most jobs involve repetitive tasks from time to time; being able to write small scripts to automate many of these don't require advanced programming knowledge; so having a bit of fundamental knowledge could really superpower you.

If I gave you $100 million to invest in one thing right now, where would you put it?

At Medusa we are trying to change what is possible for e-commerce - the first step to doing that is to make developers more powerful. We have experienced the joy that comes with good tooling and our goal is to give that joy to the users of Medusa, so that they can do more and be more creative. This is a pretty hard problem to be working on and it requires a lot of resources, so if I was given a large sum of money I'd probably throw a lot of it into Medusa, so we can double down on the product and engineering efforts - maybe keep a small chunk for some crypto :D

Why was Medusa started?

Oliver and I did agency work for a couple of e-commerce businesses and saw a pretty clear issue with too often not being forced to "hack" our way into a solution. We had a customer, Tekla Fabrics, who wanted a good long term solution that could support their growth over many years so we decided to create something that was flexible and extendable enough for us to never be in a situation where we felt that we were doing a "hacky" solution.

Where did the name for Medusa come from?

Our project actually didn't have a name for a long time until we decided to open-source it. When brainstorming names with our customers, Kristoffer from Tekla Fabrics came up with Medusa as she was decapitated, we liked the name and the subtle link so we ended up going with it.

Where does the phrase "headless commerce" come from?

The phrase headless commerce comes from the fact that the data layer and presentation layer are decoupled.

Who, or what was the biggest inspiration for Medusa?

Obviously the fact that we built the solution with Tekla makes them a pretty strong source of inspiration. That being said, I would say that the entire approach that we have taken to the architecture and the design decisions is heavily inspired by Gatsby. Similar to Gatsby, we have a core Medusa API that ships as an NPM package to be installed in a Node project. Plugins and extensions ship as separate packages and can be configured in a medusa-config.js file. We really enjoyed this workflow and felt more powerful due to it so it made a lot of sense to us to draw some inspiration from them.

Are there any overarching goals of Medusa that drive design or implementation?

As mentioned earlier our overarching goal so to speak is to make it possible for developers to be more powerful when developing e-commerce solutions. This means that we listen a lot to what developers have to say and we focus on making our tooling cater to their needs. Of course, we also need to be driven by the needs of e-commerce businesses, but generally that comes naturally through the interactions with developers.

Where are you typically getting feedback from developers?

We have a Discord channel with developers in our community, which is a pretty good source for new ideas and features, but we also receive feedback from GitHub issues.

What is the most challenging problem that's been solved in Medusa, so far?

A major challenge of headless commerce in general is that a lot of systems have to communicate with each other. This first and foremost requires a good architecture that can easily support integrations to different tools, but it also requires a level of robustness to ensure that data doesn't get lost in between systems. A thing that we have spent quite a bit of effort on is support for idempotent operations in our API - this essentially allows you to retry an API request as many times as you want without triggering side effects more than once; an example of this can be found here: https://github.com/medusajs/medusa/blob/master/packages/medusa/src/api/routes/store/carts/complete-cart.js

Are there any competitors or projects similar to Medusa? If so, what were they lacking that made you consider building something new

The headless commerce space is growing and there are a couple of players who are doing similar things. Most of them focus on enterprise customers, which makes sense due to the fact that headless commerce traditionally is something that requires a lot of resources. Our approach is more focused on making developers capable of doing more with less, we therefore cater to a different set audience with the belief that putting the product in the hands of developers will make e-commerce businesses more capable of doing groundbreaking things.

Is Medusa intended to eventually be monetized if it isn't monetized already?

Right now we are focusing all efforts on the community and product development and have not begun any monetization path yet. Our thesis is that if we have a strong community with skilled people who are actively sharing ideas and helping each other we will be able to support that community in some way that will add value to them. One thing that we have considered is a hosting platform where developers can bring their own repo and then we will make sure to deploy to optimized servers and automatically upgrade Medusa as new versions are pushed. I think this is something that many developers would love as you get rid of the headaches that come with infrastructure and devops.

What is the best way for a new developer to contribute to Medusa?

We have a good handful of issues that are pretty straight forward for new developers to pick up. We are generally pretty good at using the good-first-issue label so I would suggest filtering the issues based on this label to find something that sparks an interest. There are also a lot of more advanced issues if you already feel comfortable with working with e-commerce.

Where do you see open-source heading next?

I believe that we will see a much bigger part of the software we use on a daily basis turn into open-source software. Historically speaking it is typically the building blocks of a system that are open-source primarily because you need to be able to trust that the libraries and tools you use will not damage the things that you build yourself. Furthermore, you need to be able to control the direction of your own projects and using closed source solutions restrict you in this regard. With this in mind it makes sense that many of the most successful open-source projects are OSes and databases, but as the scope of software expands we will begin looking at systems higher up the stack as fundamentals as well. A clear indication of this direction is the whole trend around Web 3.0, blockchains, open-state, etc.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Find a project that is scratching an itch you have for yourself. It is much more fun to be part of solving a problem that you can understand and where you can see the value of the solution.

Console #78 -- pyWhat, laser_control, and deptoolkit

Sai Phanindra — Sun, 07 Nov 2021 15:17:58 +0000

Sponsorship

Axolo

Still waiting to get your pull request reviewed?

Axolo is a GitHub - Slack app that helps tech teams resolve pull requests faster by creating 1 channel for 1 code review. Oh! and the best part? We don't require any code access! Get started in 2 minutes for free with no credit card at axolo.co

Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week. 😊

subscribe here 👉 Subscribe now

Already subscribed? Why not spread the word by forwarding Console to the best engineer you know?

Projects

30-seconds-of-code

30-seconds-of-code is short JavaScript code snippets for all your development needs.

language: JavaScript, stars: 87864, watchers: 2383, forks: 9314, issues: 6

last commit: November 03, 2021, first commit: November 29, 2017

social: https://twitter.com/30secondsofcode

pyWhat

Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is!

language: Python, stars: 4729, watchers: 60, forks: 224, issues: 19

last commit: November 01, 2021, first commit: March 19, 2021

social: https://twitter.com/bee_sec_san

Laser_control

Laser_control is an in-progress laser device for neutralizing mosquitoes, weeds, and other pests.

language: Python, stars: 324, watchers: 11, forks: 11, issues: 1

last commit: October 26, 2021, first commit: December 23, 2020

social: https://twitter.com/ildarRakhmatul1

deptoolkit

DEPToolkit is proof-of-concept software for researchers and small teams sifting through online material. With only one click of the mouse, the material will be archived in a framework demonstrating chain of custody and stored durably.

language: TypeScript, stars: 6, watchers: 5, forks: 3, issues: 6

last commit: August 23, 2021, first commit: May 12, 2021

social: https://twitter.com/deptoolkit

Donate To Console

An Interview With Basile Simon of The Digital Evidence Preservation Toolkit

Hey Basile! Thanks for joining us! Let's start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like?

My background is in journalism. I studied law and political science, but I worked in news organisations of different kinds, from the BBC (a large broadcaster), the Times & Sunday Times (a newspaper), and Reuters (a gigantic news agency). My thing is data journalism, which I mean in a really broad sense, but I think the industry calls it creative technology, or "data hyphen something".

I'm self-taught and I picked up some JS first to do imperative D3 for a long time. Then React came along, and I went down the R and tidy data route (look it up if this doesn't ring a bell). Recently, I've gone freelance and got to learn a huge amount from the declarative and dare I say functional world, and I've rediscovered JS through TypeScript and Svelte. This plays amazingly well with this D3-less way of writing bespoke data visualisations.

What are you currently learning?

For fun, I'm picking up a couple of projects in Clojure/Clojurescript, which I find fascinating due to the way writing code feels. It's concise and encourages thinking about isolating (or concentrating and deferring) risky steps. "Does this function do this one thing and this one thing only?" If yes, I can fold it away and stop worrying about it.

And with the REPL, you can test and play as you write. A fresh perspective!

What's the funniest GitHub issue you've received?

Not an issue per se, but a bug report. It's election night in the UK in 2017, and I was working at The Times and The Sunday Times. It's the big show: all the work goes out as the results start coming in, all of it was planned and built in only a few weeks. It's fairly high-traffic and the moment that the digital teams across news orgs look at each other's work very closely.

In the thick of it, we get word from the project's editor that his name had been changed to some kind of pun. Or at least that's what the man says his mates are telling him. That's at the top of the page, at its busiest. Some folks say they saw it, but I'm not sure I did. To this day we're still not quite sure what happened, and kind of think it was some sort of git history trick...

Why was the DEPToolkit started?

A buddy of mine who was outside Minsk in October last year during the protests asked me what they should set up to archive some of the content they had access to, so that Lukashenko would pay the price for his actions. I didn't know what to tell them. I knew of things that were a bit difficult to set up, or not really geared towards the individual researchers, or unable to capture content behind authentication (e.g. private groups).

Archiving in view of prosecution is tricky because preserving the chain of custody of a digital item involves a bit of care from the get-go. You can't move it about and make copies because at some point there will be a human error and the original file will be lost.

In 2014, when the West went all guns blazing in Iraq and then Syria to bomb ISIS, which the public was just discovering, I worked on setting up an NGO/ reporting project called Airwars.org, which was assessing claims of civilian harm as a result of these airstrikes. The project was really, really successful in terms of changing the way the military proactively reports these mistakes.

But, I really felt we were handling this "open source" material (as in, freely available, it's weird lingo but has somehow stuck) in a bit of a funny way. This was fine for talking to the media but I always wondered what would be the probative value of this stuff in court.

Can you give some examples of some of these "difficult to set up" tools?

Hunchly, which is expensive, SugarCube, which is the motherlode, or recently-Console'd ArchiveBox which we couldn't run on their machine.

Who, or what was the biggest inspiration for the DEPToolkit?

Without a doubt, the Syrian Archive. I met Jeff in 2015 when they were getting started properly and their work is just stellar. They're vacuuming social media platforms at a huge scale and verifying so much material... The North Star goal of theirs being to enable all that have a claim during an eventual international prosecution for what's been happening in Syria.

Are there any competitors or projects similar to the DEPToolkit? If so, what were they lacking that made you consider building something new?

Yeah, there's direct competition, and quite frankly some duplication. The Digital Evidence Vault, out of Carnegie Mellon University, does quite a few of the same things we do, but... it's closed source (rather: soon-to-be-open-source), you can't get to the guy (at least, I couldn't), and you know that's something that comes up time and time again when talking to their users.

There's real value in the openness of open source software. For small projects like this, particularly, there's nothing you can hide. So far, it seems that this has an appeal.

How do you intend to monetize the DEPToolkit?

Monetized is a bit too strong a word in this instance. The project got off the ground thanks to a grant by a German federal ministry and the Open Knowledge Foundation and I feel a responsibility towards this support to keep things out there and for free. That said, improvements and fixes ideally don't depend on one's happiness to throw some spare time at it -- that's not sustainable.

On the other hand, I've got first-hand experience of doing the philanthropy and grants and fellowships dance and reporting and matching funders' objectives. That's not my profession, and it's too much work for me, so I won't do that.

The leads I'm exploring right now are to fund the open source side through commercial engagements with enterprise software providers, some of whom appear to be interested in integrating some of what the Toolkit does in the workflows they have. More on that later (I really hope).

What was the most surprising thing you learned while working on the DEPToolkit?

There's still a huge amount of interest for anything that mentions blockchain or crypto (be it -graphy or -currency). I got the weirdest people reaching out of absolutely nowhere because this project does involve Merkle trees, which are the basis of immutable chains which are cryptographically verifiable.

I can't attest to the quality of these leads but yeah, buzzwords do have an effect.

Console #77 -- Janet, Little, and CSML

Sai Phanindra — Sat, 30 Oct 2021 13:12:08 +0000

Sponsorship

Fulton - the world's most comfortable, supportive, and sustainable insole

Fulton is a modern brand of arch support. They developed sustainable, supportive and comfortable insoles that improve your alignment and posture, preventing injury through:

A cork base, which absorbs shock and molds to your unique arch to provide custom support
A deep heel cup to provide maximum stability and prevent injury
Arch support and natural latex foam to improve overall comfort
Entirely natural and biodegradable materials to support not only you, but the planet*.*

Fulton is offering free shipping and free returns, so your order is risk free, learn more about Fulton here: walkfulton.com

Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week. 😊

subscribe here 👉 Subscribe now

Already subscribed? Why not spread the word by forwarding Console to the best engineer you know?

Jobs

Want to be paid for contributing to open-source? Check out the continually updated Console job board!

We've got some great open-source jobs from Canonical, SUSE, Brave, and many others. But, we're looking for more!

If you're an employer looking to get your job in front of thousands of the best engineers on the planet, you can request to post your job here.

Projects

janet

Janet is a functional and imperative programming language and bytecode interpreter.

language: C, stars: 2179, watchers: 54, forks: 163, issues: 33

last commit: October 26, 2021, first commit: February 09, 2017

social: https://janet-lang.org/

csml-engine

CSML is an easy-to-use chatbot programming language and framework.

language: Rust, stars: 487, watchers: 13, forks: 29, issues: 10

last commit: October 22, 2021, first commit: February 21, 2019

social: https://twitter.com/sayCSML

little-lang

Little was a statically typed C-like scripting language, ideated by BitKeeper.

language: Shell, stars: 205, watchers: 10, forks: 17, issues: 9

last commit: May 03, 2016, first commit: April 05, 2016

Donate To Console

An Interview With François of CSML

Hey François! Thanks for joining us! Let's start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like?

I have a slightly unusual background. I have a degree in musicology and studied cello professionally at the Paris conservatory, then decided I wanted to have a career as a photographer instead. After about 7 years of doing that I stumbled more or less by chance upon School 42 which happened to be right next to my home. 42 is a coding school created by a French billionaire with a revolutionary new model, open 24/7, with no teachers and where you learn with projects. It was the first one but there are now 35 campuses all over the world now! I loved it so much when I visited it one Saturday morning that I decided to apply while still working as a photographer. I kind of progressively did more of this while doing less of that and that's how I eventually changed careers, I guess! Then I worked as a devops for a while at Danone, which is a large non-tech company where I was lucky to get a lot of freedom to do very interesting things and have a global impact even though I was just a junior. Then I started an agency business with a friend from 42, and we ultimately founded another company, Clevy.io, where we launched a product for creating internal chatbots and conversational automation tools for large companies.

What's your most controversial programming opinion?

The DRY principle as it is usually presented is not a good one. It drives developers to find overcomplicated abstractions over simpler, readable, maintainable code. I get why you want to reuse components that you are using multiple times but the way it is often misinterpreted, especially by junior developers, is to over-engineer the simplest code "just in case you might need the abstraction later".

What is your favorite software tool?

I don't really care so much for the software I use. It should get the work done and not require too much configuration. I try to get productive with what I have, and not spend too much time thinking about the newest trend: most of my setup I've been using for 5+ years! I think the latest addition to my must-have stack was about 2 years ago, and it was a screenshotting tool called CleanShot X. I do a LOT of screenshots that I need to annotate, and the built-in tools in macOS are terrible. It might very well be the tool I use the most besides the obvious IDE, browser etc. It's been a life changer for me!

What IDE are you using?

I settled on VS Code about 5 years ago, and don't plan to change anytime soon. It's quite complete, extensible if needed (although I don't spend much time looking for new extensions), and quite versatile as I need to be productive in a few different languages at the same time (on a daily basis at least JS, Rust, Python, HTML/CSS...).

If you could teach every 12 year old in the world one thing, what would it be and why?

That it's ok to be wrong, and neither you nor your neighbor should feel bad for it - and it's definitely not OK to ridicule someone for being on the wrong side of things. Besides, "being wrong" is oftentimes a relative concept, and it never defines you as a person. Everybody is wrong all the time. Many things would be so much better if everyone was able to take a step aside and discuss without ego and pride stepping in: we're all in it together. If the "wrong" person "wins" the argument, you still have to live with them somehow.

But I guess that by 12 it's already a bit too late to have this discussion!

What are you currently learning?

I just moved to Sweden with my family. Fabulous place, especially if you have young kids. So I'm trying to learn all things Sweden - the language obviously, the politics, the geography, the pop culture, the administration...

What are some interesting things you've learned about Sweden?

One of the more interesting things I found about Sweden is how open the country is to immigrants - at least from other EU countries. But, from what I've seen it's also significantly easier to move to Sweden from most places in the world than it is in other EU countries, even France or other Nordic countries like Denmark. Everything is super centralized so it makes things relatively easy once you have been properly registered in the system. It took us less than 5-6 weeks after we set foot in Stockholm to receive our physical Swedish ID card! Every single person I have talked to so far in my quest has been extremely helpful, and quite efficient. Whenever you have a question, you will absolutely get a complete answer, probably right now in the same phone call, or they will tell you when they will call you back with an answer - and they always do. Otherwise, Swedes live outside most of the day even when it rains (they say that there is no bad weather, only bad clothes) so I try to do that as well, and I find myself in a much better (physical and psychological) shape than when I left Paris. Now we'll see how the winter goes...

What have you been listening to lately?

It really depends on what I need to get done. If I'm coding and it's late in the night and I need an extra kick, some form of electronic music that gets me going. Solomun, Weval, Rone, Archive, Modeselektor, SebastiAn... Otherwise, I listen to a lot of classical music. This one is even harder to classify because it's quite vast... Whenever one of my former colleagues or friends has a new album coming out, I try to give it a listen!

I feel that Spotify has made me listen to more music in terms of quantity, but I kind of find myself always listening to the same things, especially when it comes to classical music. I used to discover many more things before the streaming era than I do now, and that's made me a bit sad. Discoverability is still very much an unsolved problem. Recommendation algorithms just give you the same thing over and over by trying to optimize for time spent on the platform and making sure that you already know and like what you are going to hear/watch next.

How do you separate good project ideas from bad ones?

It's actually a question that I've been asking myself a lot recently. I don't have a good answer, because I don't think I'm really able to tell by myself what's a good or bad project, assuming we are talking about the project's eventual success (commercial, usage, fame, whatever) and not the biblical good/bad dichotomy. I think it all comes down to marketing at some point rather than just the idea. There are many cases where a product shouldn't exist but does, and is even successful for some reason. I think it was Justin Kan (twitch) who said: "Startups mostly don't compete against each other, they compete against no one giving a shit" and that sums it up: people didn't need your project until they found it 5 minutes ago, chances are they can continue their life without you. So my current answer to this question is that I don't, and sometimes I fail, sometimes I succeed, and most of the time there is a bit of both depending on how we look at the project. A project that eventually succeeds is usually just a project that gets worked on the most and the longest.

Why was CSML started?

We have been developing tons of chatbots - through our no-code NLP platform, but also by hand with regular programming languages (javascript, python, etc.). There are issues with both approaches: when using a GUI, you are ultimately limited to what the interface is letting you do. If you need something that's not available, too bad: you can request the feature, but that's about all you can do. And as a developer, it's always super frustrating to be locked into an incomplete tool, that you can't easily extend, not speaking about saving your own stuff or exporting/migrating your content to a different platform if you were to need it sometime down the line. There are advantages to no-code tools (it's supposedly easy to work on) but it just doesn't scale well, especially if you don't yet entirely know where you will be with your project in just a few months.

On the other hand, whilst using a more generic programming language will solve these problems as you can do basically anything, it requires so much work and effort that it's often not worth it (and it also takes so much more time).

We thought that there was a way to solve this problem by providing a developer experience that's as close as possible to just writing code with all of the advantages and flexibility, but also provide so much more comfort for developers by solving most of the complexities that are common in every chatbot. At its core, a chatbot is a fancy state machine: you need to know where you are in a conversation, remember what's been said before, give some context, connect with other APIs... so we have created a programming language that provides interfaces for all of that, and even the most complex scenarios are written in just a few lines of very simple, readable, and extensible code, that you can host just about anywhere, compatible with many database engines. A few years in, this language has become quite extensive, but remains very easy to pick up, with a very, very easy learning curve. It powers chatbots with millions of conversations, it is able to securely deal with super sensitive data if needed (we know of some really secure companies that are using it), and even works fully offline if needed.

You can try CSML on the playground here: https://play.csml.dev

Where did the name for CSML come from?

We are notoriously bad at naming things... The basic idea behind the CSML name was that we wanted to assert a few things about the language. It stands for Conversational Standard Meta Language. So you know that it's for creating Conversational experiences. We also aim to make it the de facto standard for writing cool chatbots, so that's where the S comes from. And it's a Meta Language - technically, it's a DSL, but it also makes it sound a bit like HTML, and basically we would like that people consider it like the HTML of chatbots.

Are there any overarching goals of CSML that drive design or implementation?

There is one main goal, and one secondary goal. The main goal is that everything should be more or less intuitive and about the same as describing your logic in plain English, so that it's really, really simple to pick up.

The secondary goal however is to make it as unlimited as possible. There should not be one single thing that's impossible to achieve.

The reason the goals are in that order is that 90% of the CSML users will use just 10% of the possibilities of CSML so we should make that as easy as possible, but it should never limit you in what you need to do. That's why, from the very first version of CSML, we have included a way to add external code in a CSML script, just so that we could bypass any inherent limitation, if needed, without waiting for the feature to be natively implemented (but you should definitely speak up and say what you need or why! Ideally through Slack or simply a github issue).

I would say that there are not many things from the first goal that are still missing, so we focus more on the second goal now. Also, some of our users are starting to be very proficient with CSML and to do things that we couldn't imagine ourselves, so we need to heavily invest in many niche features now. But we do find some "things that should be easy, but are hard AF" from time to time. For example: believe it or not, there is no for-loop/while-loop in CSML right now as of v1.7.0 (there are other ways to iterate over stuff) but it was only very recently, after all those years, that a user ever wondered about how to achieve something that would be so much easier with a for-loop. So that's going into the next release!

If so, what trade-offs have been made in CSML as a consequence of these goals?

I think the most important trade-off is that the code that you write in CSML can simply not be too smart. It's a language to write dumb code. This is super powerful for junior developers as they have a lot of freedom and not too many constraints, it's very easy to learn, but it can also be a bit frustrating for expert programmers who wish they would find some advanced concepts from C++ or Go or Rust, like better control over memory allocation, more optimization, strong types, etc. But it doesn't matter! We are not writing code that puts a rover on Mars: we are helping people create great chatbots. One of the most important parts of CSML is basically to add sleep(2000) more or less everywhere, because you actually do NOT want the chatbot to go too fast (it feels horrible to read otherwise)!

There is actually a vast majority of junior developers on Earth. We're happy to create something that's focused on making it the best experience for that audience and that actually helps them grow in their development career with a not-so-scary programming language that still gives them a lot of control.

Are there any competitors or projects similar to CSML? If so, what were they lacking that made you consider building something new?

There are basically 2 types of chatbot-related products out there.

Some like Rasa (also open source), Google Dialogflow, Amazon Lex, Microsoft Luis etc. focus mostly on the Natural Language Understanding part of the problem, which is in my opinion the wrong problem. It stops short of what makes a chatbot really valuable by only focusing on understanding the input, but not doing much to help the developer handle it, remember stuff from one session to another, connect to other APIs etc., which is what the end user actually needs. So either the developer has to write a LOT of code to make the chatbot useful, or the users are left with a funny chatbot that can do some nice small talk but is not capable of much. We address that by focusing on the developer experience to actually be able to build complex conversational assistants very easily but we consider that NLU is just a feature, not a product.

Other products are no-code platforms like Manychat, Chatfuel, Landbot etc. which let you make a simple chatbot in a few minutes but lack in ways to make your chatbot truly scale. It's usually a nightmare to work on larger chatbots, or with a team. Also, let's say you want to export your bot and re-import it elsewhere: since all you have is a fancy graphical interface, how are you exactly going to do that? All you can do is print it, and that is of no use. So those products are great for prototypes and quick and dirty experiences, but the tradeoffs at scale can be really an issue, which we obviously handle by creating a programming language that you can write with any text editor, host anywhere, run on an open-source engine and standard/non-locking dependencies, version as you like on git or any other version control tool...

What is the release process like for CSML?

We have 2 main branches, dev and master. On master we only have the code that goes in final releases, and we use dev to create pre-releases and merge all our feature branches. Each feature has its own branch and gets a full run of tests using Github actions before it can be merged. When we want to create a new release we also use a Github action which will compile a new version of the CSML interpreter then build a new binary and docker image for the server then publish it everywhere it needs to go. The great thing about Github actions (although the syntax is hideous compared to gitlab-ci.yml files) is that it's easy to compile things on multiple platforms/architectures! So we publish binaries for Mac and Linux, and are going to add Windows builds soon.

How is CSML eventually intended to be monetized?

We have created CSML Studio, a SaaS version available with various plans (free/pro/enterprise) on https://studio.csml.dev which includes a lot of extra features that are kind of "satellites" to CSML. For example one of our latest features on CSML Studio is a built-in, mini Customer Data Platform that allows you to save data from your chatbot users in a database that you can then export to a BI tool of your choice. It also contains connectors to all the most popular communication channels (while the CSML Engine is API-based only), and 100+ ready-to-use connectors with popular APIs and software. But at its core it will always be the exact same CSML engine as what you can find on Github!

How do you balance your work on open-source with your day job and other responsibilities?

We have decided early on that even though CSML was going to be a major part of our company (we sell no-code conversational automation products to large enterprise customers - which in the background uses the CSML engine), it would be open-source. So the decisions on what I spend time on in general are always business-driven, but if I'm spending time on CSML, it will mostly be open-source.

What is the best way for a new developer to contribute to CSML?

We don't actually expect most CSML users to contribute to CSML itself. The goal of CSML is to make it really easy, even for beginners/not-so-great developers, to develop complex chatbots. We do so by maintaining a chatbot engine in a very advanced, complex programming language, Rust. There is usually not a whole lot of intersection between beginners/junior developers who need a very simple but powerful chatbot engine AND understand and can write more complex, low-level code in Rust. So what we usually recommend when people ask us how they can contribute is that they report their issues or give us feedback and insights into how they are using CSML either in the repo or on Slack, and that they produce and publish content, entire chatbots or smaller modules that perform interesting tasks. And if they are missing a bit of documentation or something is unclear, this could also lead to a very valuable contribution!

If you plan to continue developing CSML, where do you see the project heading next?

We are focusing more and more on the "Automation" part of "Conversational Automation". We feel that CSML as a language is starting to look pretty good features-wise, and now we want to focus on making it easier to do more complex things - better serve our power-users. So there will be a powerful CSML CLI very soon, and more powerful ways to extend the language and connect to other APIs.

Also, I feel that we have not been doing error handling and logging correctly, so we might look at this soon.

Are there any other projects besides CSML that you're working on?

I have one toddler and a second one coming soon. I'm good for now...

Do you have any other project ideas that you haven't started?

A few actually. I used to do a lot of hackathons in the past and we had many very cool ideas back then, some are still quite relevant even 3-5 years later. But I think that if I have time to start working on something new, it might just be a personal, non code-related project, or just teaching myself something new.

Where do you see open-source heading next?

There have been some cool evolutions to the open source model recently, I think. There are still issues but I find that many fully open-source companies are funded now, and this was not so much the case just 2-3 years ago. It was very rare! So I believe there will be a lot of change in the business model of open-source projects, more companies will turn to OSS for their products and I'm quite excited about that.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Don't think too much about it. If you want to do something, you might as well publish it in open-source, unless you have a good reason not to. Otherwise, if you want to do something with an existing library and the feature does not exist, just propose creating a PR and see where it goes! Chances are the maintainers will be super happy - even if at the end you don't succeed, they will always love the boost to know that somebody is using their project in a way they didn't imagine and wants to push it further!

Also, asking "what can I do to help" is always a good way to start...

Console #76 -- Symbolica, Mastodon, and Uptime Kuma

Sai Phanindra — Thu, 28 Oct 2021 17:44:35 +0000

Sponsorship

Wynter Research Panels

Want to give back to the community while having a super low-key side hustle to fund your habit?

Wynter is looking for people to join its research panel! Participate in research surveys and get paid ($90-$180/hr) for your feedback and comments. The best part? It’s only 10-15 mins per survey. Sign up today!

Projects

mastodon

Mastodon is a free, open-source social network server based on ActivityPub where users can follow friends and discover new ones.

language: Ruby, stars: 25136, watchers: 639, forks: 4168, issues: 1975

last commit: October 21, 2021, first commit: February 20, 2016

social: https://twitter.com/joinmastodon

uptime-kuma

Uptime Kuma is a self-hosted monitoring tool like "Uptime Robot".

language: JavaScript, stars: 8221, watchers: 53, forks: 440, issues: 179

last commit: October 22, 2021, first commit: June 24, 2021

social: https://louislam.net/

Symbolica

Symbolica is symbolic execution as a service. Find bugs faster by exploring every reachable state of your program.

language: C#, stars: 30, watchers: 2, forks: 2, issues: 1

last commit: October 13, 2021, first commit: May 19, 2021

social: https://twitter.com/SymbolicaDev

Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week 😊

subscribe here 👉 console.substack.com/

An Interview With Matt and Iain of Symbolica

Hey guys! Thanks for joining us! Let's start with your backgrounds. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like?

Matt - I learnt to program on my first job after graduating with a physics degree. I joined a computational fluid dynamics team that had a huge C++ codebase. It was an initiation by fire because I'd only taken a single course in C and some computational labs in languages like Matlab as part of my degree. I was really frustrated and didn't enjoy programming to begin with, I just wanted to do physics (whatever that meant!). It was after I moved to a non-software team that I realised I really missed programming. I missed the feeling of building something out of nothing by incrementally solving little puzzles and at the same time I felt an urge to master this new skill.

Since then I've mostly worked in C# for the last 8 years, typically using a stack consisting of ASP.NET Core and React, so I've written a lot of TypeScript too. TypeScript and React gave me a taste for functional programming and recently I switched to F# as my primary language. In terms of frameworks, I've been enjoying using Giraffe as a thin, functional, F#-friendly wrapper around ASP.NET, but it's so lightweight that I'm not sure if it even really counts as a framework, which is why I like it. Fable is also very cool because it transpiles F# to JavaScript and so I can write React frontends in F# now too.

Iain - I first discovered programming at quite a young age, in computing class at school. I think this has now been replaced with something more like Computer Science, but at the time it was about general use of a computer, and only a fraction was spent on simple programming in Visual Basic. However, I found it engaging enough to try more interesting things in my own time, such as scripting some simple games. I then learned some C and C++ out of general interest and used them for small projects. My degree was in Information Engineering, but there was a lot of flexibility on module choices and I took a few programming and Software Engineering modules to try to formalize things a bit.

My first job as a Software Engineer was in defence, using C and C++. I then moved to finance and switched to Python and C#.

Who or what are your biggest influences as developers?

Domain Driven Design by Eric Evans has had a huge impact on us and how we think about composing large systems. We've found it interesting how it so often turns out that by finding the correct terminology to describe the domain, you usually end up with the cleanest and most optimal solution. One really striking example of this is that if you model your aggregates correctly then they should line up with the transactional boundaries in your system, which means you can avoid the need for things like distributed transitions and two phase commits. It also means your persistence layer is simpler because you can usually then get away with just writing the data to a document store (like Mongo), which avoids a lot of the overhead around SQL object mapping and the need for an ORM.

What's your most controversial programming opinion?

You probably don't need dependency injection (DI). We used to be big fans, but over time we've realised that DI is just one way to achieve inversion of control (IoC). These two things are often conflated. We definitely still want loose coupling with IoC, but find DI is often too "magic" and can make it very hard to understand the application at the top level. As we've written more F# we've found that because generally there's less ceremony, a dependency can be as simple as a single function, then actually it's easy enough to wire up the application by hand.

What is your favorite software tool?

It's got to be git, it's saved our bacon so many times when we thought we'd lost all our work, git reflog is a life saver. It's also a constant source of new tricks, it's got so many powerful features. For example, recently we had a difficult to debug issue with one of our dependencies, and we were able to use git bisect to automatically find the exact commit that introduced the regression by getting it to run a test on each commit it searched through.

If you could dictate that everyone in the world should read one book, what would it be?

Matt - Finite and Infinite Games comes to mind because I think too often people get trapped playing finite games and living in an artificial zero sum world, which is really limiting.

Iain - The Bonfire of the Vanities, because its social satire and understanding of human nature might somehow be even more relevant today than when it was written in the 80s.

If you had to suggest 1 person developers should follow, who would it be?

It's hard to pick just one person. We have a few people that have influenced us as developers, but we find that nobody gets everything right.

If you could teach every 12 year old in the world one thing, what would it be and why?

Statistics. The world is becoming increasingly data driven, and the ability to make sense of this in order to properly understand risks and to avoid disinformation campaigns is, and will continue to be, a critical skill. We're also entering a phase where people think they're making data driven decisions, but have misinterpreted it. Conversely, they also believe that without enough data they can't make a decision. We need to teach kids critical thinking skills and stats so they can sift through this data and make good decisions with or without it.

If I gave you $10 million to invest in one thing right now, where would you put it?

Right now we would have to say Symbolica. We've still got so many ideas in front of us to add to the project, and being able to hire some extra developers would be amazing. For example, we want to make testing more powerful and end-to-end by working directly with syscalls and assembly. We also want to incorporate some threading support to catch race conditions and deadlocks. These are big jobs and will take time and serious effort.

If I gave you $100 million to invest in one thing right now, where would you put it?

Quantum computing. Once this technology matures it will bring about a huge advancement in the quality of human life because it will allow us to simulate large complex molecules which will give us exponential improvements in technologies such as batteries, carbon capture and drug synthesis as well as unlocking new technologies that were previously out of reach.

What are you currently learning?

Matt - Over the last few years I've been making this broad shift from OOP to functional programming, so I'm definitely still learning lots about FP. More specifically I've been working through SICP lately, so that's taught me some Lisp and I've been trying to learn Haskell when I can.

Iain - I have been learning about LLVM as part of our current work. I hadn't previously looked into the internals of compilers in much detail, but it is quite fascinating to see some of the amazing things people have done.

Matt, would you say that learning Lisp has been worth the effort?

I think it's a language that's had quite a large impact on other languages so it's nice to have some familiarity with it and the influence it's had. I think at a high level it has a similar impact on your brain as learning any functional language in the different way they make you think compared to imperative languages.

What have you been listening to lately?

Matt - I'm a big fan of BBC Radio 6, so I often have that on. I recently rediscovered a band called Delorean, so I've had them on quite a bit. There's also a song called London Gangs by Sault which has been a real ear worm for me lately and according to my Spotify I seem to have had a Placebo revival too. Oh and with it being the 30th anniversary of Nirvana's Nevermind recently I've had that on quite a bit. In terms of podcasts I really like CoRecursive and Software Engineering Radio and Exponential View are regulars for me too.

Iain - I jump around a lot, but most recently Doves and White Lies.

How do you separate good project ideas from bad ones?

For personal projects just follow your interests and try not to worry too much about whether it's good or bad, just whether it's interesting, fun or helping learn something new.

If it's a work project then we think the best technique is to really understand the problem you're solving before you start building anything. Getting to know the user/customer, the pain they're currently facing and how much value this solution could generate for them is really important. As software developers we have a tendency to immediately start building because it's exciting, but often we can miss simpler solutions which might not even need us to build anything bespoke at all. We always try to look for an off-the-shelf solution first. If there's nothing already available or the current solutions don't work then we try to understand why. At this point you're usually in a much better position to design a solution and can be more sure it's going to be worth your time working on it.

Why was Symbolica started?

We wanted to write a tool that would let us prove that programs, before and after a refactoring, or written in different languages, were effectively equivalent. For instance, if someone wanted to port a program from a clean reference implementation to an efficient but complicated implementation, they could check it using this tool. We tried using existing proof tools for a while, such as KLEE, but we found that its abstractions weren't amenable to the extensions we wanted to make. A concrete example of this was that when we were trying to check that a small program written in Python was equivalent to one written in C, we needed to symbolically execute the Python interpreter and we weren't able to get this to work with KLEE. So that put us on the path of writing our own symbolic execution engine from scratch with some different design goals in mind.

Where did the name for Symbolica come from?

We were originally going to call it Symbolic, but the name was already associated with another company, so we settled on Symbolica as we liked how it sounded.

What was the biggest inspiration for Symbolica?

Some of the core technology is most similar to an existing symbolic executor called KLEE, and we have taken a lot of great ideas from this and its forks.

Are there any overarching goals of Symbolica that drive design or implementation?

We broadly have two design goals.

One is for it to be easily parallelizable. During symbolic execution, each time a fork in the code is found we can create smaller problems to represent each path, so we want to be able to run these subproblems in parallel, as well as apply some other optimizations to avoid going down paths we've explored before.

The second is to really focus on the usability from the developer's point of view. We want it to be really easy for developers to run this against their code so that we can reduce the friction people face when trying to test their software and get people writing more tests and shipping fewer bugs.

What trade-offs have been made in Symbolica as a consequence of these goals?

We have a lot of immutability throughout, to make parallelization as trivial as possible. Of course, this is generally good for simplicity and robustness too, but has led to some interesting design questions and efficiency problems. For instance, to implement these immutable data structures efficiently takes some care because we have to ensure we're not copying unnecessary data whenever there is a modification. We also need to be able distribute these data structures over multiple machines when parallelizing it across a compute cluster. So we have to be smart about how much data we're copying and when we do that. In many cases we can get away with lazily fetching the data on demand.

What is the most challenging problem that's been solved in Symbolica, so far?

So far, most of the work has gone into decomposing the overall problem into its fundamental abstractions, which we believe will allow us to support the features we need.

What sort of fundamental abstractions have you found thus far?

The necessary abstractions are probably not that surprising in hindsight, they're just those that describe how programs work at a very basic level, such as functions, instructions, operands and expressions etc. Really perfecting these and their relationships takes time, but it has paid off for us in that several features have since "fallen out in the wash". For example, it became self-evident that we could treat all data as fixed-size expressions, whether constant or symbolic, and we could build everything from a small set of operations on these types.

What was the most surprising thing you learned while working on Symbolica?

It's amazing how far a poor but minor design decision can cascade throughout a complex system.

What is your typical approach to debugging issues filed in the Symbolica repo?

As the project is pretty new we've not actually had any submitted yet. On other projects we've worked on together we'd try and write a failing test and then fix from there.

What is the release process like for Symbolica?

It's really simple at the moment, we just tag the master branch when we want to create a new release. If we want to patch an old version we'll just create a release branch from the tag that's broken and start creating patches on that, which we'll then propagate forward to newer versions if we need to. As it's a library we plan to follow semver, but we're not at v1 yet, so things are moving about a lot and so we're not being strict about not making breaking changes at the moment.

How is Symbolica intended to eventually be monetized?

Our intention is to offer a cloud-hosted version of Symbolica that people can pay for on a consumption basis. We believe our solution is amenable to fairly large parallelization, and so our goal is to make it efficient and cost-effective for everyone by running it on some elastic infrastructure in the cloud.

How do you balance your work on open-source with your day job and other responsibilities?

Luckily for us Symbolica is our day job. Before starting Symbolica we used to work together at another company and they were generally happy for us to open source things that we'd built internally, providing they weren't commercially sensitive. So we've both always been fortunate enough to be able to work on open-source projects during work time.

Do you think any of your projects do more harm than good?

I think this is one of the trickiest things with new technology. It can be really hard to predict the higher order effects it will have on society or how others will use it. I can't think of anything particularly harmful about Symbolica because our aim is to help people find more bugs in their code base and so hopefully we'll help to prevent some really bad outcomes like security vulnerabilities or loss of life.

What is the best way for a new developer to contribute to Symbolica?

The best way to contribute is to open an issue and let us know if there's anything you want to add to the project. We're friendly and we'd love external contributions. We are currently working on the hosted part, but we plan to spend some time on the open-source docs soon, so people know how to get started building and running the project on their machines. You can also email us at dev@symbolica.dev or message us on Twitter @SymbolicaDev.

Where do you see the project heading next?

We want to get an efficient and scalable version working in the cloud so that we can handle large problems. We also want to develop much better tooling around it so that it's easy for developers to integrate symbolic tests into their code base and development workflows.

What motivates you to continue contributing to Symbolica?

Well partly because we hope to be able to commercialize it and earn a living this way, but also because the problem is really interesting. We're both big proponents of software testing, but we also understand that many developers feel like they don't get a worthwhile return on investment when they write tests. We believe that with better tools, like Symbolica, we can change this.

Are there any other projects besides Symbolica that you're working on?

Symbolica takes up all of our focus at the moment, but we have carved some internal bits out into other open source libraries. For instance, we're building most of the cloud hosted version using F# and we recently wrote a small library to make configuration binding more type safe. The de facto .NET config binder was a common source of null values for us, which was frustrating when the rest of F#'s type system is really designed as if null isn't a valid value, so our library takes a more F# friendly approach and uses a Result type to model the fact that binding might fail.

Where do you see software development heading next?

We think there are a couple of big themes. The first one is that we're going to see this increase in low-code tools continue. We agree with Fred Brooks when he says there's no silver bullet, so we don't think these are some kind of magic solution, but we think we'll see domain-specific tools come out that lower the barrier to entry. So, we think we'll see this rise of people with specific domain expertise who architect and design domain-specific tools. They'll still have to put a whole load of thought into the information architecture and how different parts of the system interact, but they won't need to care about lower level details of the machine etc. So it will become much more about domain modeling and we might see people employed primarily for their domain knowledge and then trained on how to use whatever low-code platform it is that they build solutions in. Developers can get quite cagey about this because they think it's going to erode their value, but we disagree because there's still going to be a huge demand for people who know the lower level languages, in part to be able to build and support these low code tools / environments, and there'll still be plenty of other things that are better off written in lower level languages.

The second is an increase in popularity of statically typed and functional languages. We personally can't imagine working on a code base of any real size without having the compiler check our work and we think TypeScript, with its structural typing, has done a great job of showing that you can have static type checking with little ceremony and still retain the flexibility of dynamically typed languages. We think switching to F# has also really opened our eyes to just how much baggage OOP carries with it and how it's actually not a very natural mental model for most types of programming today, for example REST has a stateless architectural model, yet we often build REST APIs using OOP which is all about state. We think OOP still has a place for programs that really need to encapsulate mutative state (.e.g games) in long lived objects, but a lot of people only use it because it was the predominant paradigm in the first language they learnt.

Where do you see open-source heading next?

Personally we hope that the people that put in the large amount of effort and hours to maintain critical open source projects that are depended on by millions of developers get the recognition and compensation they deserve. Unfortunately, we've seen recently with projects like babel that it's often difficult to get enough funding to keep even a handful of developers working full time on something that's a critical piece of tooling to many companies. We're not sure what the exact solution is, but we expect to see more "open-core" business models. To us, this seems to strike a nice balance between developing in the open and also being able to capture some value from those efforts, where larger enterprises are willing to pay for a hosted version of the tool because it works out cheaper for them than running it themselves. Although, obviously this doesn't work in all cases.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Always open an issue and speak with the core project maintainers first. It's really frustrating if you spend your time building a feature or fixing a bug and then the PR gets rejected because it's not in keeping with the rest or project or something. It's always best to speak with the core maintainers first and scope the work out with them. You'll save yourself a lot of time and you'll end up with a better designed solution too as well as getting to learn a bit about how the core maintainers think and work.