What advice can you give those of us who have been building UIs and CRUD backends for years but want to break out into different domains? For example, how do we follow in your footsteps and build "high-volume, low-latency, highly-available distributed systems" or "Big-Data Validation Framework?"
When I started out, I didn't plan on building these kinds of systems. I just wanted to learn everything I could and land a job. For several years, I had learned Data Structures & Algorithms but didn't get to apply them. However, I kept practicing implementing that stuff frequently. Once you get good at those, you can build standalone applications that are robust and efficient. You learn little things that go a long way. You become a performance junkie and try to get things done in fewer and fewer ticks and memory as possible. This is really your foundation for everything else. Later on, I was in a situation where I had to build a system that supported 100k operations per minute without crashing.
When I joined MSFT, I got to experience first-hand what engineering excellence really meant. However, the high-volume, low-latency, highly-available systems that I built were not my idea as such. It was a result of specific requirements at work. Our services had to support a certain number of queries per second. Over the course of time, I learned via observation and from code-reviews and picked up techniques for building better systems. I read a lot of architecture documents to understand how various search pipelines worked.
As for the Big-Data Validation Framework, I had cycled through a lot of teams in Bing. In the process, I was really tired of the redundant work I had to do across many of those teams. I noticed that most of my peers were in the same boat. I ended up building a tool to solve my problem and built it in a way that everyone else could contribute to it and all of us would benefit. This was partly a result of my laziness and partly boredom with the mundane tasks.
Most people don't decide to build stuff like this out of the blue. Perhaps you can think about joining one of the big companies where you can work on large-scale problems. It is something you should experience once in your engineering life as you get a chance to work with and learn some incredible stuff.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.