![Cover image for I'm Charity Majors, Ask Me Anything! [FINISHED]](https://media2.dev.to/dynamic/image/width=1000,height=420,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fthepracticaldev.s3.amazonaws.com%2Fi%2Fa14csi1piwptc9ou45a3.jpg)
I am the cofounder and accidental CEO of honeycomb.io, where we are thinking hard about how to help you debug and understand the complex systems yo...
For further actions, you may consider blocking this person and/or reporting abuse
Hello Charity - thanks so much for doing this! I'm interested in how you got started - did you come from a Computer Science background? How'd you land your first job? (I'm always interested in origin stories 😂)
Also, any advice for programmers who are thinking about a future in management or leadership?
Thanks again! You're awesome :D
Hell no, I am a music major dropout (with side gigs in classical latin and greek, philosophy, etc). I never had computers growing up. I ran away to college when I was 15 though, and developed a crush on a boy who spent his time in the computer lab.
The lab stuck; the boy didn't.
Advice for those wanting to move to management: question yourself about why you want to do people management. Lots of people want it for the wrong reasons. And it's very different than wanting a future in leadership, which every single senior eng can and should have.
There's no scarcity of leadership work. I think it's terribly toxic to confuse management with leadership though.
Thank you - that's a very good point! Also high five for art backgrounds 🙌 I love that. Thanks again for doing this AMA!
Word. I think the liberal arts are the best education that 90% of humanity can possibly get. All hail philosophy majors. 👏
What is an area that new software engineers often overlook when writing code?
What happens to it after you deploy it.
It's not enough to compile and pass tests and not get paged about it. You should have muscle memories for going to check up on it -- did what you expected to happen, actually happen? Did anything else change?
What were some of the unknown unknowns you've had to deal with as a CEO?
oh jesus. much like observability, being ceo is all about the unknown-unknowns. I always feel like I'm failing, because I'm only ever working on whatever is broken in the org ... as soon as it works, it's someone else's job.
But the biggest unknown-unknown was having to be CEO at all. I distinctly did not plan on and did not want the job. I wanted to be CTO. It's always been my intention to have a primarily technical role for the rest of my career. And I knew CEO was a shit job and that I did not want it. But shit happens.
Ha! Is there anything you like about being CEO?
I read your interview with Business Insider where you mentioned how smooth the fundraising process went for honeycomb. What's your advice for folks that aren't quite as connected but are trying to raise?
I believe in what we're building, like a lot. That makes it worth it to me. There is very little about the job that I find rewarding or interesting on a day-by-day basis, and I'm still petrified about being relegated to PM roles after this -- what if nobody will ever hire me as an engineer again??? My 3 am self is in agony over this.
It was smooth this time because I have a COO who has been a VC and done startups before. It was not that smooth when I was trying to do this on my own. Moral of the story: admire business people and don't talk down to them, so the good ones flock to you. I think that's the moral of the story anyway.
My advice to anyone with fundraising questions is always, "talk to Ginsu". 😵 Dear god what a lifesaver good business people are.
Beyond that ... think big, pitch big, tell a story. I have to try very hard not to use my normal sardonic self-deprecating sense of humor around investors. Things that I think are fucking hilarious and realistic, they read as lack of ambition. Sigh.
What are your biggest goals for honeycomb's next 1-2 years?
First: nail user experience. We've built a service that's so fucking powerful that a few people will crawl over rusty nails and shards of glass to use it. Now it's time to circle back and make it more broadly available.
Instead of painstakingly instrumenting everything, we are building onramps to particular communities, so you can e.g. do "npm install honeycomb" and get all the basic stuff for free. I think this will help close the gap.
Second, I really deeply want to make honeycomb a mecca for distributed systems art and visualization. We need more visual metaphors for teasing out signal from noise, and more cutting edge art drawing on recent academic advances in data vis.
People are getting hit by this freight train of infra complexity, and I don't think A.I. is gonna save us in the next decade or so. What will save your ass is good design paired with intelligently plumbing your network. Turn it into a social graph question: when you get paged, what do you want to see? Obviously you want to see whatever questions were asked by the last person who got paged about this problem.
We have to look for ways to bring everyone up to the level of your best debugger, in every subject area. We have to get better at empowering teams, not individuals. Hell, I learned Linux by reading other people's .bash_history files. I want Honeycomb to feel like it's that embedded in your team and your social circle.
Third, I want to help software engineers demystify production. I want to take the CI/CD revolution another big step forward, and make it table stakes for every engineer to explore the consequences of their code in production, every time they deploy.
I think production should not just not be on the other side of a wall, but it should feel like the "fourth trimester" -- when you ship your code to prod, it's a blind, wailing, helpless, probably broken piece of shit when it first encounters real users, real data, real services, real network. You have to nurse it into growing up.
Software needs owners, not operators. Everything I'm consumed with is just details towards that goal.
Always count on a startup founder to have lots to say about these sorts of things 🙂
... i can keep going ... 😇
Damn, thanks for such a great response.
Do you have any articles you've written or talks where you've discussed something similar to the "trimesters" of development? I love that insight, and the analogy of needing to nurse your systems/tech to maturity.
And thank you for doing an AMA!
I have a draft. I hope to finish it this week. :)
Thanks for having me!
What can an organization do to be more inclusive of their DBA’s into their devops initiatives?
Use the same tools! The edges of tool usage is what creates silos.
Also, be willing to learn things like query optimization, etc so you don't have a DBA SPOF. People seem remarkably eager to wave a hand at it as tho it were black magic. IT's not. It's easy.
Which is more important - observability of systems, or bourbon?
Trick question: it's a false dichotomy. Everybody knows ops teams run on malt whiskey.
Do you think technologists make better CEOs than someone coming up through say marketing or finance ?
No, not necessarily. I've more often seen them be worse, because engineers tend to underestimate how important and/or challenging business stuff can be ("you just make the best product, and that will always win, right??"). Engineers have often underinvested in developing their powers of persuasion and other social skills as well.
But being a good CEO is all about surrounding yourself with people who are better than you, and knowing when and how to interfere.
For me, being on call is sort of nice and fun, but I think it's hard to make plans and sometimes affects my work-life balance. What do you miss about being on call?
A guilty little secret of mine is that I love firefighting. Some people jump out of airplanes ... I love the panic and glee of knowing everything is on fire, the company might not survive if I don't fix it correctly. I love high pressure and high stakes.
... But I will normally deny this (if sober), because in modern day infrastructure you are supposed to hate firefighting and want to write software all day.
On call sucks for a lot of people. It's a shame. I really believe it doesn't have to, it should be a net plus. Sounds like you're at least partway there. :)
I kind of also love firefighting. What's one of the worst fires you've dealt with? What did you learn?
Um.. the worst outage of my life was probably at Second Life around 7 years ago. We tried upgrading the primary from 4.1 to 5.0; all the secondaries had been upgraded painlessly, and all the benchmarks said 5.0 was faster. When we upgraded, the grid stumbled to recover; we ended up being mostly down for over 24 hours, and losing all that data when we had to roll back to the last good 4.1 secondary (due to binary incompatibility, couldn't roll back in place).
I spent a year developing capture/replay software for mysql and testing various configiurations and workloads before finally upgrading successfully.
And what did I learn? To be desperately paranoid of all database upgrades, and assume that anything that can go wrong, will go wrong.
Hah, totally feel you about loving firefighting. There's a real thrill of it for me, too, and it's really fun just being on the edge of the seat, where every solution you push out might be the one!!!
I might be partway there, not sure yet honestly. I can never reason out how it can be a net plus though, since (at least in my mind) it's a net-plus only if something broke and it was solved, right?
Net plus? Because for a week you have justifiable cause to run down any rabbit hole, do extravagant perf tuning, and other shit that normally isn't important enough to preempt your project work. :)
What are the biggest misconceptions about observability?
That it's a synonym for monitoring.
It's not -- although there's a lot of overlapping domain knowledge, and you might say that monitoring is a subset of observability.
Monitoring is heavily biased towards alerting, downtime, outages, and above all actionable alerts. I think of monitoring as being to ops what tests are to developers -- once you know about a problem, you can monitor for it, you can test for it. Monitoring is about known-unknowns.
Observability is a property of systems, and it primarily is concerned with unknown-unknowns. A system is observable if you can ask any new question you want out of it-- if you can understand the insides by interrogating the outputs, without needing to add new customt instrumentation for each question.
Can you suggest a talk or two to check out? By you or by someone else or both.
Oooohh boy. Do you like articles? Cindy Sridharan has been on fire lately, with her pieces on the subject. Honestly I had kind of stuck my neck out there, insisting this definition makes sense, and all the monitoring boys were coming down really hard on me saying it was bullshit. Cindy researched the topic, read a bunch of the stuff on both sides, and wrote some brilliant long pieces with even more detail than me where she came to basically the same conclusions. It was super validating. I think observability is an idea whose time has finally come.
As for talks ... I'm not aware of anyone else talking about this just yet, though I hope that changes soon. You might check out my Strangeloop 2017 talk. I was super specifically pitching it to software engineers, trying to meet them where they're at. Let me know if you like it or not, please!
Thanks for the tips
Hey Charity! What's the hardest part about dealing with the engineering stuff and the business stuff?
I don't really deal with the engineering stuff, right now. I would only annoy people if I stuck my nose in with opinions. I don't really deal with business stuff so much either though ... hmm.
I guess I would describe my role as always plugging the weakest important part of the ship, and paying lots of attention to product and strategy.
Year one was about, "do we have a product? is it different enough?"; year two was about .. "okay, now do we have a business, and will people pay for it?" and year three is indubitably going to be about user experience. So I'm spending most of my time learning things about that.
Not sure if this answers your question or not, sorry.
Note quite the same as accidentally becoming CEO, but thought you might like this article about a woman who joined a COBOL bootcamp, became a programmer, and ultimately found herself promoted out of programming.
I studied classical piano in college too! I always have Arabesque No. 1 by Debussy stuck in my muscle memory -- first thing that comes out when I get to the bench! Do you have a go to piece?
Niiiiice. :)
Rachmaninoff has always been my guilty pleasure. I spent two years learning the Revolutionary Etude when I was ~12, and I find myself compulsively fingering those long descending sequences when stressed. It's like a nervous tic.
YES!!! Everything c minor please. That one's so much fun but my fingers definitely aren't in shape for it anymore!
it's actually really depressing to play ... what comes out of my fingers doesn't sound anything like i remember it sounding 🤔
What's your favorite thing to do when you're not working, that's non-tech related?
For the last two years I've been walking 5-10 miles a day. I've always hated gyms and fitness in general, so it's nice tp find something that really works for me. It's practical, I can do it when I'm roaming strange cities, I can write emails and blog posts and do twitter and read ... basically I look down at my phone, and when I look up an hour has passed and I'm there. It's amazing.
I also love, love, love fancy whisky, and bonding with a good friend while sipping late into the night. I <3 my scotch malt whisky association pass. The tasting notes read like a markov's chain. _^
How do you price a software product/service?
Start by bleeding out your eyes and tearing out your hair for six months or so.
It's incredibly hard.
More seriously, ok, I guess I would say:
Thank you for such a thorough answer! Boy this is tough 😥.
so tough <3
What was it like being homeschooled? I'm thinking about homeschooling my future kids. Also, sometimes I wonder how I would've turned out if I was homeschooled.
Terrible. I was deeply isolated. I didn't really have any friends or learn to interact with other humans for years after leaving home.
There was also the religious fundamentalism, though.. it may be hard for me to disentangle the two.
But seriously: kids need strangers and peers and friends. Family isn't enough. Beyond that, I do think it's a crime how many hours/day kids spend sitting in classrooms reviewing stupid shit, and all having to go at the same dogged pace. It's not a good use of anyone's time. It's like a warehouse more than an institution designed to inspire love of learning.
Oh dear, sorry for all that...
100% agree. Thanks for the answer.
How'd honeycomb.io get it's name? I think it's pretty sweet.
...😉
The first name was bloodhound.sh.. we thought a detective dog would be cute, but it sounded vicious. So we shortened it to hound.sh. But then we got a legal challenge from hound-ci. So the four of us (myself, Christine, Ben, and Toshok) sat down with a lot of beers and tossed out all the ideas we could come up with one night.
The next morning we sat down with coffee and sifted through them for possible contenders. One was trufflepig.io, the other was honeycomb.io. But it was owned by Slack... :(
However, our lovely friends at Slack agreed to sell us the domain (and tossed in hny.co) for just the cost of the lawyer's fees! It was amazingly generous of them, and we are incredibly grateful.
I like honeycomb because of the associations with structured data, sweetness, hard work, and oh yes queen bees. <3
One of the early name ideas for Amazon was relentless.com. Check out what happens when you click that link 🙂
They too found it kind of vicious in the end.
OMG. TIL!
Yes! All great reasons! That makes a whole lot of sense. That's great that it worked out.
No question. Just wanted to say i loved reading your responses and you sound like a hella of a fun character :)
How is kubernetes evolving? Is it becoming easier to use?
Yeeeeeeeessss... slowly and painfully. Much more slowly than I would have thought. I actually recommend most folks use a hosted solution, sigh.