TLDR:
Use your product, and talk to your users, as a developer there is no
better way to work on the things that matter. Almost every other path
leads to institutional imperative[1] and deepest technical debt that
is almost impossible to pay.
There is no better way to code than to work in the kitchen.
Hi,
I joined REKKI 8-9 months ago, REKKI is an ordering app that looks
like a chat app, that chefs use to order from their suppliers (kind of
like inverse deliveroo). 'Whatsapp for chefs' is how Ronen described
it when we met (best fucking CEO I have ever seen, it is pointless to
talk here about his vision and etc., but if you get a chance to meet
him, don't miss it). My mom has a small grocery shop in Sofia, and I
know how much pain is for her to order from her suppliers, and also I
believe that the whole producer->supplier->restaurant->consumer market
is completely non transparent and everything can be improved.
So as I said, I joined 8-9 months ago, and then I really did not
understand what is the actual difference between whatsapp and REKKI,
so I asked Ronen if he can set me up to work in a restaurant for 2
weeks. "2 weeks? are you crazy, who will write code?" he said, so he
gave me 2 days.
My first 10 hour shift was few days after that.
My expectations were fairly low, besides me being in complete panic
and outside of my comfort zone, I thought I will wash the dishes and
observe the dynamics.
I wanted to not be 'minus' to the staff, so I pre-trained myself how
to use the industrial dishwashers watching youtube videos[2]. BTW
those things have the best UX ever, the colors they use to convey
information, the way everything works is just incredible.
PANIC! The day has arrived! I calmed myself down by watching more
dish-washing videos from the metro on my way to Restaurant X. I was
there at 14:00 (as agreed).
The restaurant is fairly big, and the kitchen is in the middle, so
everyone can see you cooking, and the dishwasher was in the back in
smaller kitchen. You can't imagine my smile when it was one of the
dishwashers I have pre-trained for.
The staff was very welcoming, and when I told them I trained for the
dishwasher they were happy for me to wash the dishes while they do
some veg-prep (that's kitchen slang for "vegetable preparation" like
chopping potatoes haha). Shortly after that they gave me a knife and
some carrots to peel,I am fairly good with the knife (I do
leatherworking[3] and woodworking as a hobby), but in no way I am
comparable with a chef, for them knifes are more like body parts than
anything else, still I didn't do bad, and when they saw I didn't draw
any blood they gave me more and more veg-prep work.
The restaurant opens at 18:00, and the guests arrive around 18:30, we
ate around 18:00 and then the guests started coming, I was in kitchen
(wearing apron and everything, also I have just as much tattoos as the
chefs, so on the outside there was no question if I am a chef or not).
Because of my veg-prep and dish-washing performance(I suspect) they
gave me actual dish to cook: mushroom ravioli.
I think it involved those things:
- long mushroom stems (the ones I cut earlier)
- some other mushrooms
- truffle butter
- chicken broth one of the tricks I learned, everything tastes better if you use chicken broth instead of water
- some sauce I forgot how it is called
- stirring until mushrooms get golden
- some olive oil in the end
BTW, you can also check out my soup recipe
https://txt.black/~jack/soup.txt if you are interested in my cooking
abilities.
Turns out all people order food in the same time, I am telling you I
have been dealing with distributed system consensus for the last 6-7
years, and I never thought unorganized entities can act in this way
without behavior emerging from simple rule (like ducks following), its
CHAOS, I have to do so much mushroom ravioli.. PANIC...You know how
all chefs stir the pan by tossing the food in the air? Well I have to
use a spoon to stir; also all the guests can see me using a spoon..
Anyway I did ok, nobody returned their food! At 22:00 It is the first
time I can step outside for a breath of fresh air for 5 minutes.
Then things get easier, from 22:00 to 00:00 guests only drink, and we
started cleaning the kitchen around 23:30, my last metro leaves at
00:15, so if I don't leave at 00:00 I won't catch it, and of course
there are guests that won't leave, despite restaurant closing.
My legs hurt, from 14:00 to 00:00 nonstop, bloody hell this is some
brutal work. I couldn't take notes, or do anything, just chop chop
chop, stir stir stir.
BTW If the people of RestaurantX happen to read this post, thank you
so much for letting me cook! You rock!
Few weeks after that I got my second shift, in a small michelin star
restaurant. Things went in quite similar way, but I was more
confident, I did some dishes, a lot of veg-prep, which they thought
was boring, but I kind of liked it, there is not that many times in my
life when I know exactly what to do.
Again from 14:00 to 00:00, and again people did not leave when
restaurant is closing, fuck people just leave! Chefs are tired!
I will do a small divergence here to tell you about the tech stack
we had when I joined:Elixir was chosen, because whatsapp uses erlang, and we need
presence and "who is typing" feature as any serious chat app.In order to build a chat app, elixir makes a lot of sense, hot code
reloading can keep the sockets alive during deploys, you have native
presence with phoenix channels, etc. also super easy to build
channels and joining and leaving and everything.
So I came back to work, and let me tell you REKKI is not a Whatsapp
for chefs app. It is an ordering app that looks like a chat, but there
is no way that two chefs are in the app in the same time, if one is
ordering from suppliers, the others are cooking, or veg-prepping or
anything but not ordering.
Oh and I forgot to mention: the internet in the kitchen is total shit,
constantly switching wifi and 3g, and in some parts you have neither.
Phoenix channels(abstraction on top of websocket) are full duplex, but
serial, which means you basically can't block on your endpoints
because no new requests can come through the channel, so the payload
was sent to rabbitmq and then something else executed it. This lead to
incredibly interconnected and complex (especially with retries) system
in order to write a record in a database, but it was a requirement if
we want "someone is typing", except we don't!
And guess how good the websockets perform with the kitchen's internet.
Heartbeats, reconnects, flushing queues, pending actions etc etc, all
had to be taken care of.
This is true technical debt, you can only solve by doubling down on
the problem you already have, and of course, you make it work, and you
improve it, but you can only make incremental improvements.
You see if you work in the kitchen you see what your technology is
doing, and what your users do.
Now we are moving from elixir to go, we moved from channels to
endpoints, pulled up the queue actions in the endpoints so the chef
gets immediate feedback if there is any kind of error with their
request. I have to say I love the app now, how snappy and clean it is.
We moved away from elixir because we can't get devs and also every one
can read and write go (it has like ten reserved words..), and also
empowered frontend devs to become fullstack devs and make their
endpoints. (also it is super annoying to write business logic in a
functional language)
There is no better way to code than to work in the kitchen.
--
Fri 7 Feb 19:38:07 CET 2020
Borislav Nikolov (https://github.com/jackdoe)
References:
[1] https://www.berkshirehathaway.com/letters/1989.html
BERKSHIRE HATHAWAY 1989 shareholders letter
[2] https://www.youtube.com/results?search_query=dishwashing+training
Latest comments (2)
When it comes to buying appliances for your kitchen, the dishwasher sale is a big purchase. Not only do you need to consider the cost of the dishwasher, but also what features are important to you. Dishwashers can have a variety of features that can help you determine what’s right for your family’s needs. Knowing what to look for in a dishwasher will help you find the perfect one for your home and your budget.
Beam VM works all the time in sh**y unreliable 3G, 4G and 5G connections. The massively multi-user online gaming created by Nintendo is all based on BEAM VM, WhatsApp, Pepsico B2B is also full BEAM VM. Cisco, Ericsson, Klarna and T-Mobile mobile phone infrastructure is also based on switches that run BEAM VM, in fact 40% of the world mobile communication uses BEAM VM, and about 90% of internet switches. Even Motorola made a benchmark between its C implementation and its BEAM VM implementation and guess what: the BEAM outperformed the optimized C implementation.
It looks more an architectural disaster then the technology disaster that motivated the migration from Elixir to Go. And also a mindshift problem was solved: from function perspective (which takes time to grasp) to the good old structured programming (Go is 'almost' objected oriented but it's not there) and from Actor Model to CSP Model for concurrency.
But the most important thing: websockets were not created to be used on very unreliable networks, so avoid it on that kind of situation and you'll be fine.