Hello! I'm Junade; by day I work at Cloudflare, focussing on running the Support Operations engineering group. By night, I work on a PhD in theoret...
For further actions, you may consider blocking this person and/or reporting abuse
Thanks for doing this! I have a couple of questions:
What's the most commonly overlooked actions those of us using Cloudflare can take to increase our cache hit rate?
When you wrote your php book, how did you decide the order of topics to cover?
Cache Question
Funny you should ask:
In that case there were some behind the scenes tweaks I used to optimise the cache behaviour in a way that (unless you're running a JS CDN) wouldn't be necessary for most sites on Cloudflare. These included ignoring the Origin header in the Cache Key and customising the Cache Topology to optimise traffic routing for cache hits.
In terms of general advice:
First and foremost, get your cache control headers right and make sure content is actually being cached (see the CF-Cache-Status headers). Use something like Bypass Cache on Cookie to cache dynamic content (when the viewers are anonymous).
Secondly enable Argo, Tiered Caching will make sure the stuff you requests stays cached by failing over to other Cloudflare data centres when something isn't in a local cache (whilst optimising the path around the internet that the request takes).
Also check out Origin Cache-Control it's a really cool and powerful feature for cache management.
Then measure and test continuously to improve your cache configuration.
Book Question
Instead of ordering the book chronologically, I ordered it logically by the information someone would most likely need to understand before moving on to the next topic. Building up complexity incrementally also makes it easier for someone to follow your book, even where the learning curve is more steep.
Thank you very much!
Thanks for doing this Junade, how's the PhD coming along?
Hi Ben;
It's going really well! I've been really lucky to have a great supervisor for both my Masters and my PhD and it's been awesome to be able to step away from practical development and get completely drawn into solving some theoretical problems. My PhD is largely focussed on exploring mathematical problems related to optimising coverage for mobile sensors and some other related Operational Research problems, which is an extension of the work I studied for my Masters. It's been fun to get exposed to theoretical problems which have substantial real-world implications; for example, during my Masters I was asked to solve a problem of a theoretical for a major multi-national defence company which was about how you can maximises the coverage of sensors attached to public transport vehicles by being intelligent in selecting which bus routes have vehicles with the sensors installed on. You can find some of the research in this paper.
The toughest learning curve so far has been the fact that I only have a Masters degree, I never finished school or did a Bachelors degree - so towards the end of my Masters I did have to spend a bit of time ensuring I had a strong general understanding of maths (in addition to the focus area I was studying). I also regularly have existential crises as when trying to solve particularly hard optimisation problems which haven't really been solved before (albeit with significant amounts of background research).
Funnily enough, I originally didn't want to get drawn into doing a degree, but at previous jobs, when I found myself in a formal engineering environment - I got drawn into a research project in collaboration with a university, which ultimately turned into a Masters project.
There is a high workload for writing in a PhD, both in terms of writing papers and getting an early start on working on my thesis, but I'm sure this will benefit me as I come closer to completion.
Obviously it will all come down to the thesis and a single viva at the end (and I have no real way of knowing objectively how those will go), but so far I'm enjoying it. Academic research isn't for everyone, but if you're looking to make your contribution to knowledge (as opposed to more superficial reasons), I would highly recommend it.
Good luck!
Any best-practices on balancing the workload with the rest of what you do?
How much of number one were you looking for before you started at Cloudflare? Where did this quality stand relative to other things they, or others, could offer?
I had a number of different offers from some quite notable companies (and I actually turned down some very good opportunities at some successful start-ups at the time), but I really had my eyes set of Cloudflare.
Different things matter to different people, and you need to figure out what you want. First and foremost of the things I value at Cloudflare is the freedom and trust that is put in you. You aren't pigeon holed and you are given the freedom to deliver stuff of value, no matter your rank.
Beyond that; you have the opportunity to make a big impact. The organisational structure is relatively flat, but growing. They were big enough to let me study whilst working. There are a ton of interesting problems that you can help solve here, that you won't see in a standard SaaS company or an agency.
The company is still growing and there's plenty of opportunity to help form that structure. When I started I was the only person in the Community team, in fact, I was the only person in the marketing organisation outside San Francisco. When I moved to Support Operations, my manager let me head up a team which was brand new in the organisation. You will be able to leave your mark quickly.
I'm webdev who do college at night, at my work I use PHP/Javascript/MySQL but I have much interest in learn other languages that I actually study and see in college like C/Python/Prolog on the other hand I feel like I must focus on use the knowledge from college to improve my work, what would you recommend me to focus on, the improvement on work or learn new things and put more focus on college stuff ?
Do what your enjoy more and would help you with your wider career aspirations (it's not unreasonable for salary exceptions or job security to play a part in this decision). An popular open-source project may be the thing that seals the deal for your next role. Consider if you need any specific technical skills for a particular software engineering discipline if you want to specialise; for example, a data scientist may find a language like R useful whilst a Dev Ops consultant may need to know Kubernetes well (and by extension, Go).
Either way; try and get a firm understanding on writing clean, tested and reliable code - above specific languages. In the words of Martin Fowler: "Any fool can write code that a computer can understand. Good programmers write code that humans can understand."
Language elitism is valued by some engineers, but the best software engineers I've encountered as Polyglots with firm understandings of the fundamentals.
If you're pursuing a career in PHP or a similarly aged language (as opposed to some newer buzzword-driven language) - you will spend substantial amounts of your time refactoring legacy code. I used to (and still do) enjoy this, but it can be an unfortunate reality that so much code is written poorly. If you keep doing greenfield work you generally get the privilege of making the mistakes before anyone else has a chance to notice (or notice that the new language everyone's talking about on HackerNews can't do fundamental things like Polymorphism properly).
thank you so much for your answer, i'm halfway on college and I feel very attracted to AI and Programming Languages paradigms, so I'll put more effort on college subjects and eventually try to use the concepts and the theoritcal knowledge on work since so much can be improved If you have certain knowledge, and yes 100x, work with PHP and legacy code put you in very Crazy code situations, I kinda like refactory and improve some things but after spent much time in a project just doing this you get bored, and thanks for mention the words of Martin, who I didn't knew, I'll take these to life.
Hey Junade! What's the coolest thing Cloudflare is working on?
So much cool stuff going on around the company. Cloudflare Resolver (1.1.1.1) has been cool recently, doing Edge Computing with Cloudflare Workers, the network team always are building and noticing cool things, the Data team are doing some cool stuff with ClickHouse, there's cool work going on to get Cloudflare to investigate running the Cloudflare edge software on ARM processors.
For me and my team:
Hello Junade! How did you enter a Master program without a bachelor degree 😮?
So I pretty much left school without any qualifications, let alone a BSc. When I left school I started working in a software engineering apprenticeship (a kind of qualification in the UK which is based on professional work and some classes which the government pays for). From there I focussed on my career for a few years; having been in a family where money was quite limited.
As I entered more formal engineering environments (such as the automotive industry or the), the desire for a formal degree became more apparent.
Most Masters courses in the UK have a backdoor "professional experience" route. I submitted my CV and a covering letter to a few different universities. Some weren't willing to take the risk of putting me on a Masters degree and instead offered me a Bachelors.
This was a hack. Some people tell me it's "a developers way of getting onto a Masters degree", other people have said it's "finding your own route like an entrepreneur".
One university offered me a position on an evening Masters course, geographically this choice made the most sense and was least likely to impact my career. Additionally, the university was working on a professional research project with my employer which I was involved with. I was impressed beyond all else by one of my class supervisors, who developed great rapport with me. He gave me a research problem to work on and eventually became my MSc supervisor.
Cloudflare is great at supporting part-time education, I finally graduated my Masters last year and have been working on my PhD since.
Hi Junade,
Thanks for the great resources. Love your writing.
I'm just digging into your blog.cloudflare post on adding DNS-Over-TLS on OpenWRT. My question is, can you point me to any docs that might help me set things up so that my DNS requests go first to a locally-attached pi-hole (ad-blocker) server, and then get forwarded to 1.1.1.1 using TLS?
Appreciate any pointers you might have to share.
How do you manage the balance betweeen day job and working on your PHD at night? Sounds super challenging. Keep being awesome!
First and foremost: have a supportive employer who values self-development. Part-time study is hard enough without an employer who is scared of employees becoming "too good and leaving".
Cloudflare is great at support self-development and my manager is very supportive. The flexible working makes things easier for those times when I need to have a meeting with my PhD Supervisor during work hours or something similar. PhDs are very flexible to study as (in the UK) they are very research focussed and there aren't classes. Work travel doesn't interfere with my degree as I am largely doing most my research work alone. so I still get to enjoy the fun of travelling to different counties to visit different Cloudflare offices.
When I was doing my Masters and at a previous job, it was hard work. I'd get up at 5:30 AM to drive 2.5 hours to work. I'd start early so I could leave early and drive to university where I'd study into the night and then I'd come home around midnight and work on writing my book, doing my degree project work or other personal projects. I coped with this for about 6 months during the toughest part of my degree, then gave into taking a term's break from my degree when I started Cloudflare.
Drive, self-determination and being focussed on finishing strong were all things which kept me going through that session.
Given your experience, how do you approach testing in the context of distributed systems? What have you found to be the most helpful in reducing production errors and ensuring stability?
Firstly, make sure you've nailed Test Driven Development. When developing, write the tests first and your software will be better architected alongside better tested. When a bug appears, write the test that would detect it before solving the problem itself.
Good software architecture goes a far way. Deploy fast and automate your build systems. Be careful of cascading failure and have a system that can operate when certain services drop, 5 services running at 95% uptime means the overall system has ~77% uptime. Message Queues will save you when Database-as-IPC bites.
In absolutely mission critical environments, you need redundancy. For example; in the systems I would work on, there would be two embedded systems with identical software. The other chip would be able to take control if something went wrong (and had the ability to detect erroneous behaviours) - and even speed up deployments by having one chip reprogram the other.
At Support Operations in Cloudflare, we are increasingly seeing these kinds of problems as optimisation problems and are doing some interesting work in practical applications of Search Based Software Engineering in distributed environments. Stay tuned to my Twitter feed or the Cloudflare Blog to learn more in due time.
What got you interested in development at such a young age, and how did you learn the necessary skills to land your first gig?
Largely tinkering and writing code from old software engineering books - like most developers, learning development the trial-and-error way. My first few jobs included a lot of technical debt and the book Refactoring by Martin Fowler was really important for me to learn those skills.
It was a culture shock moving into formal engineering environments (such as for the automotive industry or working on defence projects) as reliability became a key focus there and I quickly learnt about formal correctness of programs and TDD. My Masters degree was where I learnt about the academic process and basing knowledge of rigorous research of published work.
Anecdote: When I was doing a Cloudflare meet-up in Sydney a month or so ago; a 13 year old came to the meet-up and jokingly claimed he would achieve "world domination, one Google search at a time".
We're generally lucky that there's so much (mis)information around software development online, and there is so much content that will help you build basic websites and apps. When things get tougher, that's when you need to read deeper, gain more experience (and in my case, go back to school).
How do you approach Support Operations at Cloudflare? What do you find to be the most useful data to track and tools to use?
Support Operations is pretty much a regular software engineering team. There are a couple key differences which are important though. Firstly, as the manager for the Support Operations team, I have control over both the prioritisation of work and the engineering management; we are very good at understanding where business priorities lie and what will lead to the biggest impact against our metrics. Our work is heavily focussed on breaking constraints; whether this be providing better tooling to automate debugging for a new product or more intelligent market segmentation tools for our Marketing/Sales teams. The metrics I prioritise against usually come down to either lowering business costs, decreasing churn or increasing revenue.
Secondly we have very good flow without over-working people. We have a very large focus on reducing WIP (Work-in-Progress) in the team and there's a heavy focus on having a start-it/finish-it culture. I learnt about the importance of this when I worked in the automotive industry and was introduced to Japanese manufacturing systems like Lean and TPS. Unfortunately a lot of these systems/techniques have been misunderstood in the software engineering world; where Kanban was originally created as a mechanism for reducing WIP, it's now often misunderstood globally as meaning solely having a "Kanban board" to track work. I've been very keen to insist that Support Operations at Cloudflare does not treat it like it.
For engineers, the workflow closely resembles Extreme Programming; we have short iterations (with manageable work) and a heavy focus on automation in the software lifecycle itself (beyond the bread-and-butter things like Test Driven Development, we are also thinking more and more about practical implementations of Search Based Software Engineering). We are able to deliver phenomenal value whilst having the space to work on interesting problems.
We aren't closely bound to particular languages or tools - with the focus instead being on having clean, testable code that delivers value.
I would very highly recommend reading The Goal by Eliyahu Goldratt. Although about manufacturing instead of engineering, this book had a substantial impact in how I look at leading an engineering team. I insist everyone joining the team reads.
Hey Junade, what's the story behind the traffic signals from your Twitter bio?
So I spent some of my career working for a company which produces sensors that power things like Smart Motorways and pretty much every traffic signal in London (and most cities), amongst other technologies. You'll pass dozens of these sensors when driving around many parts of the world without realising it (whether it's counting you on and off a motorway for analytics purposes or the sensors which tell you the time until the next junction which use Bluetooth MAC addresses and Automatic Numberplate Recognition).
One of the things I was most proud of working on was some LED roadstuds ("LED cats eyes"), which would cut road accidents by up to 70% - most people wouldn't even realise they were installed on the roads they were driving down, but it would change their behaviour.
I worked on the software both on the sensors themselves and also the managed services that allows different bits of roadside equipment to be connected together (e.g. a SaaS service to configure how speed limit signs are controlled from accident detection systems). I even got to spend some time working in the Arctic for sensor testing!
How did you get interested in British constitutional law? Do you notice any similarities between the legal arena and programming?
Ages ago I used to be one of those teenagers who would knock on people's doors for political reasons. I was useful for political groups when they needed someone to make a CRM system or a website. I found myself being on the board of an organisation called The Reform Foundation - where I was introduced to the co-authors of this book (an MP and a professor) and I accidentally found myself being part of it.
I've come close to standing for local government elections in safe seats, but always backed off at the last minute. I've started to lose my reformist renegade spirit for politics (I guess I've moved that more towards the software development world?).
I've become far more cynical these past few years and have largely lost political interest - other than carrying a membership card for a political party in my wallet. Occasionally I still act as a "counting agent" in elections (someone who has the delight of watching ballot papers being counted and then arguing about how they should be counted).