Since getting into the Developer Relations space, I take onboarding to new companies as an opportunity to evaluate what the overall product experience is like. This comes with all sorts of fun things like identifying broken links to fully broken internal processes. And everything in between
I am a strong believer that everyone is doing their best. I am also rational in understanding that priorities and focus change regularly. I want to make sure I approach this with curiosity and enthusiasm to propose a solution and fix it myself, if that's possible. After all, I probably have less meetings and less responsibilities than everyone else, simply by being new.
If I can't fix it myself, I propose a recommendation. A former manager of mine would be thrilled to see me bringing solutions, not problems. (To be fair to both of us, the platform I supported at the time didn't have a lot of solutions but certainly had a lot of problems.)
So with a use case in hand, I set off to see what the developer experience was like for Apryse.
But wait, where does Developer Experience fit in Developer Relations?
Before we get into the audit details, let's address the DevEx vs. the DevRel elephant in the room.
For me, Developer Relations is the broader umbrella that encompasses awareness (advocacy), enablement (developer experience), and engagement (community).
Some companies will bucket this a little different, including the one I'm at now, where Developer Experience is the umbrella but the core focus areas are still largely covered. At Arpyse, we also include AEO under Developer Experience which I think fits nicely in this era of AI-enabled or AI-assisted builders.
Now let's talk about audit process in terms of awareness, enablement, and engagement.
The process
I want to try to replicate, to the best of my ability, what the public facing developer experience is like. How do developers find the product, how can they try it out, and then how can they (or someone else in the company) start the purchasing and procurement process.
Doing this during onboarding should be the easiest time to ever do this because you probably have little to no deep product knowledge. you can literally approach it like an external developer because you are one. Unless your onboarding includes product training, which is cheating if it's only available to internals.
Discovery (awareness)
Can I find the product when I'm looking for it? Is it positioned well against competitors?
In today's world, discovery includes both a traditional search and AI tools, including LLMs like Claude, but also includes AI search summaries, like the ones you see at the top of an otherwise traditional Google search.
I approach this like any research project I did in school - lots of keywords, questions, and use case based researching. Then I see what kind of results I get.
If you think only your marketing website or your docs are your entry points for a developer audience, you have some work to do.
Consider other entry points, like if your product is available on a cloud or platform marketplace, as developers participating in that ecosystem may start there first - think a Salesforce or Appian developer starting in their marketplace rather than accidentally finding something not in the marketplace already and having to build it full custom. Is this information up-to-date? Well maintained and linked back to more granular information across your website and docs? Does it show up on the marketplace search with vague terms or only the company or product name?
Beyond marketplaces think of entry points like GitHub, package managers, or other places libraries and dependencies already live. Unless I'm starting from scratch or doing a re-platforming effort, the closer you are to where I'm already working, the easier it will be for me to pick it up. And here again, is this information up-to-date, well maintained, and linked back to more granular information? It's not enough to just be listed, it needs to look safe and maintained, often with very limited context.
Outside the scope of this audit, you can also evaluate broad, general awareness activities typically managed by marketing like events, webinars, blogs, etc. If I go to a developer conference, is Apryse there and how quickly can I understand what the product does in a conference setting?
All of this is part of the discovery process.
One thing I should mention before moving on - if your product doesn't come up when and where you expect it, that's a notable gap. Congrats, you have action items following your audit. To me, awareness gaps are the most critical to address immediately because if people (and AI) can't find you, they can't use you. Fix that!
Evaluation & trial (enablement)
I found the product I think I want to use, but I need to check on the features and eventually build a proof of concept (PoC).
You've been found, either by humans or AI, and they are moving forward with understanding and using your product, often in some kind of trial capacity.
Before they start building, they need to know if your features and capabilities will solve their problems. Maybe they've seen some marketing pages or blog describing this, but this is where they will start digging for more details.
This is probably where my process has changed the most in recent years. Using AI tools I can go from discovery to building really fast, sometimes even in the same chat.
And I did, with several LLMs on basic or free tiers with as little access to memories or prior knowledge as I could possibly give them. I didn't want them to taint my results by knowing I worked for Apryse because I wanted to experience what an external developer would experience, and a bias to Apryse because I was an employee was going to harm that.
During this evaluation phase is also where I check competitors and even PoC them too. You want to be choosy about this step because it's either time consuming or compute intensive (tokens), but it's a very realistic step for developers evaluating multiple options.
I'm evaluating the features, the ease of implementation, the clarity of documentation or other assets available for reference, and ultimately how much debugging did I have to do to stand up a working PoC. If this goes smoothly, I will literally go tell all my tech friends about it. (And this should be your goal, to have people outside the company talking about how great it is.)
Trial keys, licenses, and more
If your product has a trial process, it should be as frictionless as possible. I get that you want leads. You know what I, as an external developer, want? To figure out how many PoCs I have to do before I recommend a solution to my manager on a tight timeline. And know that when I build a PoC, I've timed it so it still works when I have to do a demo.
Your trial process should be super clear. How many days, weeks, credits, runs is the trial valid for? What features are included or not? Will I see uninformative error messages when using a trial key vs. a production key?
It does not matter if the friction experienced in this phase is 100% because of how trial keys or licenses are implemented. You need to fix that. External developers shouldn't have to absorb that.
A curated developer journey
This is the hill I'm willing to die on, sometimes daily. You cannot risk developers or AI defining their own developer journey. You need to own and nurture this. This stage of the audit is where this becomes alarmingly clear.
Taking a use case, I usually pick a programming language or framework along with it. In this case, I went Javascript (React + Node, with some Vite and Angular sprinkled in along the way. Remember, I ended up doing like 5 or 6 PoCs across Apryse and competitors.)
With both my own eyeballs and mouse pointer AND, separately, my crew of LLMs, I wanted to see if there was a clear, defined journey through Apryse for a Javascript developer. This was a well supported journey, but not necessarily a curated one depending on where I went or what questions I asked. For example, landing on the marketing website, we don't really link to a developer portal. In fact, we have about 4 pages I would consider someone external to Apryse calling a "developer portal," so we'll need to fix that.
A north star goal here would be for my various programming languages or frameworks, a developer in that space can move from discovery to building with clearly identified resources that we nurture, specific to their needs and use cases. For example, a Java developer would probably have different needs and use cases simply by living in the server space (Apryse Server SDK) compared to a developer using WebViewer (Web SDK). This could end up being the same developer, but they are solving different aspects of the problem and need an intro and understanding of our different SDKs.
Up and running, and beyond (engagement)
Is there anyone out there to support me? How can I get deeper knowledge?
Once I've successfully built a PoC, or like 5? 6...? in my case, I need to help whoever is actually going to do the purchasing make a final decision to make the purchase happen.
And in parallel, I need to make sure I know how and where to get support, either paid or community. Ideally, this is open and searchable because I'm both lazy and antisocial.
This is where I look for community support platforms, forums like Discourse, messaging platforms like Slack or Discord, GitHub communities, etc. I also evaluate their community health - is anyone actually posting? Are the only responses from employees or contributors? Does it look like a place I would feel safe asking a question?
If there are hundreds or thousands of people silently hanging out, this is a red flag. If only employees or contributors are responding, this is probably a yellow flag or maybe even green if you were looking for free (community) support. As a community grows, the community support should scale with it. If it doesn't, this might be something to look in to.
I also look at GitHub and forums outside of the direct control of the project or company. What's the sentiment here? Do issues remain open with no responses? Are PRs stale and neglected? Is there a whole subreddit bashing the project?
At this point, I also start looking for any signs of a champions program or certified developers in this space. This might come up on LinkedIn or a community support forum. These are usually really healthy indicators, that someone external to the company cares enough to go the extra mile with the technology. They may speak at conferences and share their experience too, which loops back to the general awareness activities.
With this information I can usually wrap up my recommendation and feel confident we are making a good choice.
The caveat
It's super important to stay as public and as focused on exactly what a given resource says. If you have access to internal experts or secret resources, this isn't a good replication of what an external user would have access to. If you overcome any and all friction with extensive research, that's not realistic with someone on a development team with a tight sprint timeline.
The AI caveat
I took this particular audit as an opportunity to use every LLM and AI chatbot I could reasonably get my hands on - Claude, Microsoft Copilot, GitHub Copilot, Gemini, ChatGPT + Codex. Occasionally I would use them a bit adversarially quoting one to the other. I also wanted to see what kind of project structure they delivered, what kind of experience I had pair programming or debugging with them, and what kind of AI-enabled or assisted experience they delivered.
Several times I had to reign them in because they could see my email in my directory file tree or path and infer Apryse as my employer. I get that they are being "smart" but this is not what I wanted. Microsoft Copilot had broad internal access and I actually ended up abandoning it because it would go find internal resources after I would explicitly tell it to only look on the public internet.
Speaking of public internet, GitHub Copilot out-of-the-box didn't have access to the public internet and neglected to share that with me. I wrote a whole thing about that, but TL;DR it was GUESSING old npm versions before I was finally like CITE YOUR SOURCES and it couldn't... because it didn't have access to the internet.
Wrapping up
So this was more info into my developer experience audit process. Have you done something similar? What does your process look like?
Again, and I cannot stress this enough, this isn't a finger pointing exercise. Some things may be accidentally neglected or forgotten, just fix them or recommend a fix and move on. Some things may be missing because it just wasn't the right time to do them, and maybe it's still not the right time.
The point is you now have a starting point for how and where to improve the developer experience, a great set of talking points to meet folks across the company and start building those cross-functional relationships, and you'll be able to directly impact the external developers trying to build a solution with your product. Lots of points. I think you get the point. 😉
Photo by Agence Olloweb on Unsplash
Top comments (2)
Very thorough overview ... !
Following the public path from discovery through procurement is a useful way to find handoff gaps. Recording the point where a first-time developer needs internal help would make the audit actionable without reducing it to a documentation checklist.