"Yes, but how do you measure success?"
This is easily the most common phrase thrown around, not just in Developer Relations or Developer Experience, but in this agile world we live in today. The fact is, you can be really busy and, unfortunately, not deliver any real, measurable business value. This is typically a waste of your skills and a problem for your employer. No one wins.
Let's think about Developer Experience, which for the purpose of this post is roughly all things enabling developers. If the documentation exists, is well written, is used by developers, isn't that enough to say it's successful? Maybe it doesn't even have to be well written, it just needs to exist. If it's used by developers, isn't that successful?
Depending on your goals, sure, that may be successful. Having a single, unique, non-employee pageview may meet your criteria for success. Someone looked at it! Someone is enabled! We've done it! Success, in this case, is really just proving it is live.
This is enablement at its most basic, superficial sense. You have documentation and someone outside of your company can see it. But I highly doubt the person who asked this question is going to find this an acceptable answer. And long term, this shouldn't be an acceptable answer for you either!
What's my goal?
As I implied above, my goal with documentation is squarely in enablement.
@ASpittel Success to me is going to be enablement which is harder to measure.
Internally I want to see the "how do I" type support questions we can answer in docs drop. I want the feedback friction from consultants to drop. I want the docs experience to be so delightful no one complains.14:48 PM - 18 Mar 2021
Your first step is to admit to pretty much anyone and everyone, including yourself, that this is hard. You will need to iterate based on your community, your company's goals, and maybe even your industry.
The better you know your community, the better you can determine what is measurable. Copy-paste, cookie-cutter metrics will not help you as much as they will hurt you.
What do I measure?
I want my documentation to enable my developer community, existing enterprise customers, and my internal coworkers to get the information they need efficiently and delightfully.
This means I'm going to focus my success metrics around these topics and areas:
External
- Vanity metrics (pageviews, bounce, landing pages)
- Engagement metrics (star ratings, comments)
- Zero search results, search term keywords (comment if you want to see my thoughts on this in a future post)
- Percent of content updated per release, per quarter
Internal
- Partnering with tech support to decrease the number of 1-touch, "how do I" questions
- Partnering with consulting, customer success, field ops, etc. to incorporate UX-type feedback into docs experience (improve findability, searchability)
- Partnering with product management on success metrics for product/project
- Ability to execute doc-initiatives (gardening, re-platforming, etc.)
Data with no context is dangerous!
All of this comes with a big asterisk. Simply producing this data without context could lead to some thrash.
@ASpittel Vanity metrics are a good gut check, but remember for docs pageviews can be consistent per page and bounce may be high for things like glossary and low for things like conceptual content.
Having an engagement metric like comments or star rating can be helpful too.14:45 PM - 18 Mar 2021
Vanity metrics
Vanity metrics are a great gut check. Is your documentation live? Are your top-performing pages maintaining their standing?
Depending on the kind of documentation, these vanity metrics may be a little different from marketing pages. For example, glossary terms will probably have a low "time on page" and a high bounce. Conceptual pieces may have a higher "time on page" and perform similarly to landing page content. Both are functioning as expected.
At a previous company, our most consistent "lowest-performing page" was our uninstall instructions. This is also functioning as expected. The uninstall experience was so intuitive (and so unnecessary) that this was just an article for due diligence but answered the question without having to install it - "can I uninstall this software easily?".
Some other things to look at - what are the exit pages? What are the entrance pages? Are these what you expect? Adjustments to these lists can be great indicators of a successful documentation initiative.
Engagement metrics
Silent developers are happy developers or they are so confused they can't articulate what they need to get out of their confusion. This is where it comes back to knowing your community.
Certain developers and developer personas are known for quietly struggling through your documentation and others are going to make a big scene about it across social media.
Your developers may be a chatty group, willing to leave comments, questions, and concerns on your docs, forums, or with their field ops engagement.
Unfortunately, comment spaces on docs can turn into noise. To me, any comment shows engagement even if the engagement is "this is broken". They found the documentation article, found a way to engage, and potentially provided an opportunity to review the content for clarity, at a minimum.
Of course, more informative comments will inevitably give us more to go on, but that's for another blog.
Internal partners
You'll notice I wasn't super clear about measurements in my breakdown of internal partners. This was intentional.
You can share certain success metrics, for instance, lowering the number of tickets for 1-touch, "how do I" support requests. These create noise, but they also highlight pretty clear patterns in opportunities. By partnering with tech support, you can see these patterns (hopefully in the form of reports) create or modify existing documentation, and monitor the number of 1-touch tickets that come in on that topic, which should theoretically go down.
Now it's super important to partner here because depending on how support is measured, your work to successfully enable developers with better, clearer docs (that reduce 1-touch tickets) may impact not just the amount of tickets they get, but the rate at which they close those tickets. 1-touch tickets are fast to close but offer low business value. Removing 1-touch tickets free up cycles the technical support reps can spend on higher touch, higher complexity tickets, higher business value tickets.
Similarly, partnering with other externally facing teams to identify gaps in potential documentation is crucial to determine if there are any existing assets or material that can be reused or if not having this information publicly available is intentional. Perhaps this information is gated by an enterprise license or specific contract. Don't leak your special sauce recipe!
All in all, accidentally screwing an internal partner in the name of your initiative's success does not win friends.
Success is a healthy team
What is often lost in metrics that work for other teams or projects is that, just like in engineering, documentation has technical debt. Documentation corpus gardening in the form of reviewing, updating, archiving, and other honing activities is critical for the health of your docs. Whether your documentation is maintained by your product engineers or a dedicated team of technical writers, you need to factor in some time to burn down this doc tech debt.
Successful documentation doesn't mean the entire corpus is touched every release. With older, larger software platforms that simply isn't scalable. Measuring an initial baseline and setting a target from there is one way to see if your team is able to not just document features, but also tend to the gardening activities too.
I look at this as having an external and internal component. Externally, you'll see how many pages are updated for the new version or release maybe in a PR, release notes, or by the number of "updated" badges on your documentation. Internally, if you are maintaining a backlog, how many of those documentation-based backlog items were worked? Or how many were intended to be worked but were missed to support new feature documentation instead?
An undocumented or under-documented feature is only as good as a missing feature, and some documentation maintainers really internalize this. Balancing documenting new features and taking care of your documentation garden is critical to the health of your overall documentation corpus and team. Your documentation isn't successful if you are churning through burned-out maintainers.
Wrapping this up
There is more to this, but this is a great mindset to start with. My biggest caution is leaning too hard into the vanity metrics. Pageviews are good, but they tell your SEO story more than anything else. Great for awareness, possibly not as good for enablement, but it ultimately depends on your community, your company, and your goals.
I'm interested to hear from folks if this resonates. Given that documentation at Camunda is a joint effort between product engineering (writers) and Developer Experience (strategy) the success metrics are tied to the goal of enablement.
How do you measure success for documentation?
Big shoutout to Ali for this tweet that inspired my responses. Highly recommend reading through the responses there too.
Hey friends who work on documentation!
Does your team track any metrics to see how successful your documentation is? If so, what are those metrics?14:36 PM - 18 Mar 2021
Cover photo by patricia serna on Unsplash
Top comments (14)
Hey, This is super useful! :) Definitely interested to know more of your thoughts on "Zero search results, search term keywords" too :)
+1 Vote here also.
Same.
+3!
Great Read. It really saddens me to see supposedully great dev API's that are poorly documented but the core team flaunts it as if everything is super-ready for usage. While from a dev's point of view and experience - that lib/api isn't ripe enough for usage.
Thank you!
Some of this is truly tech debt, they built a great API, they know it very well because they are so close to it, then they have a hard time documenting it because they waited until it was "done" to do so. It can be hard to put yourself in the mindset of the API consumer when you are the API maintainer. Similar analogy to a backend dev trying to explain a frontend experience. The fundamentals might be there, but there are nuances that will impact the experience.
I didn't mention it here, but I strongly believe a dedicated technical writer should be included early in the lifecycle to help design an experience that can be well documented.
We actually had a feature that was going to be released out of lock-step with our product. It was going to be incredibly difficult to document and even hard for developers to understand. Thankfully the release was pushed, but this is a great example of something that should be caught by someone thinking about the documentation experience, whether it's a technical writer or a developer experience professional. We want the experience with the new feature to be good, or as you put it, ripe for usage!
I wonder what could motivate and drive founders of startups and engineering managers to utilize a tech writer or dev experience proffesionals early on.
What is your take on those 2 differnt paths and the importance of proper docs in each?
Outward-facing APIs that generate revenue will always be the easier case because it feels like you can directly attribute dollars lost/gained to the overall developer experience, including docs.
I worked in Enterprise IT and there was no concept of an internal technical writer. The team that built the project wrote just enough documentation to train the support team in maintaining it after a hand-off.
Two things could happen here.
The original project team could face a number of escalations from support (often due to the support team not understanding the project codebase or not having the proper skills to maintain the code base). This would typically hurt the project team more than the support team and result in a re-architecture or redesign of the project rather than a review of supporting assets like documentation, or lack thereof.
Alternatively, the support team could request additional training or a longer hand-off, blocking the project team from moving to the next project. This would usually result in the team leads or the most empathetic members of the team pulling together training. Could it still end in the above scenario? Yes. But often with training came tons of documentation and the number of escalations scaled at a slow enough rate a revamp of the project seemed to make more sense (changing business needs, chance to upgrade tech stack, implement new security features, etc.).
What I would rather advocate for is including technical writing and developer experience earlier in the design process. Stop waiting until a code freeze at the end of your release cycle to write your docs.
Just like you would get a UI mockup early on, why not discuss API scheme design, terms used in the API and product for consistency and clarity, etc. Theoretically, this should be part of UX, but having a dedicated stakeholder participating in design time on behalf of the developer community, the less complex developer experience technical debt you should incur, including docs.
Hi Amara,
Thanks a lot for this article. There's so little content out there still discussing the practicalities of DX work.
We've been scrabbling trying to see what insight we can get from metrics - there's a deep desire to be data-driven in our approach, but a dearth of useful data. Particularly gathering DX data for internal platforms seems to be quite a lot more difficult than for external facing systems. The number of customers is so much lower and it lends itself more to direct contact that objective measures of behaviour.
In that context some of your ideas here are exactly where we ended up, we're specifically restructuring our documentation to provide more direct answers to the "how do I?" questions and trying to establish a pipeline from support requests to documentation solutions (as well, as code solutions, of course).
It is deeply gratifying to see others coming to similar conclusions and to increased the shared knowledge in this space.
I love the idea of internal developer tooling teams, but I wondered if exactly what you mentioned was going to be a problem.
Even documentation seems silly to take a data-driven approach. All documentation, particularly externally facing documentation, is a value-add to the business.
Another option for internal is how fast or how friction-less onboarding developers is. Could be new developers, junior devs, or if you run some kind of rotation. Are you scoping or sizing less time/effort to onboard?
Funny you should mention that - onboarding time/complexity is our most pressing issue right now. Documentation is part of the problem, but automation is a bigger one. Setting up DX as a function and taking responsibility for discovering and fixing these kinds of problems is, in and of itself, the closest thing to a silver bullet.
FWIW, this is my second internal DX role (we actually called it DevCare last time around) and it's fair to say that the problems are quite different, as are the organisations. It strikes me that the way of thinking and the ownership of the domain are what matter most.
This is a really cool and thoughtful article - thanks for laying it out so nicely Amara. I always just think about my own documentation for design stuff and the scale of that with maybe like 20 devs/QAs max. Looking at it in this context, I'm mind blown, of course when things scale for larger teams and sharing across internal and external groups documentation quality/measuring etc become such a big deal.
The time wasted answering the same things again and again that could be a link to a Single Source Of Truth, could be an addition to metrics like number of questions but without a holistic view you won't go very far.
However, the ability to track the success of documentation might motivate some people who find it as the most boring thing on Earth LOL!
Love your focus on impact and not the 'easiest to measure' vanity metrics that so many of us get caught up on.