DEV Community: Pete Cheslock

Beyond Code Completion: Better Prompt Context to Supercharge Your AI Coding Workflow

Pete Cheslock — Tue, 12 Mar 2024 13:59:26 +0000

AI code completion tools like GitHub Copilot have revolutionized the way we write code. These tools not only speed up coding tasks but also help in adding features and fixing bugs more efficiently.

This post will show you how I use tracing, sequence diagrams, and other personal observability data with Generative AI, via a RAG (Retrieval-Augmented Generation) model to get better AI guidance for application development. Imagine feeding AI observability data like traces, performance timing, sequence diagrams, API calls, and other information that maps out your application’s interactions. The result? A hyper-personalized instructional guide on how your application operates and code suggestions empowering you to make informed decisions like a senior staff engineer or software architect.

With the strategic use of AI and runtime data, the path from code suggestion to architecture suggestion becomes not just accessible, but intuitive. This is more than AI coding assistance; it's creating a development process where information is synthesized and presented to you to elevate your code decision making and expertise within your group.
How could you quickly become an expert on your codebase?

Let’s show you how this works.

Shifting your AI from Code Suggestions to Thinking Partner

There are many tools that will statically index your code base and send them as context to the AI to provide code suggestions about your codebase. But there is still a big missing piece of context for generative AI coding assistants that can sometimes lead to bad code suggestions. Tools like Copilot and other AI coding assistants don’t know anything about your application when it runs. This results in between 50-70% of code suggestions being rejected by developers.

Adding tracing data to AI generated prompts can improve the quality of code suggestions. It can also help you get answers to different kinds of questions from generative AI coding tools which are trained on static open source code.

You can follow this process with any large token AI system like Claude by identifying tracing data relevant to the code you are working on, using it as context to prompt OpenAI or other LLMs. Generally, you’d generate tracing data by implementing OpenTelemetry (aka OTEL) libraries into your application, adding spans to your functions with Jaeger, or using commercial SaaS tools like Honeycomb and Datadog.

In this example, I’ll use AppMap to generate tracing data and other runtime data about my code in my code editor. This will provide a structured way of describing how my code executes when it runs.

I will also use Navie, the AI integration for LLMs from AppMap. By default, Navie uses OpenAI GPT-4 integration with pre-programmed prompts to consider runtime data. I will feed the AI a combination of code snippets, traces, and sequence diagrams as context of this particular code interaction I am working on.

In this case I’ve installed AppMap installed into a python powered web store which uses Django.

To get tracing data I use Request Recording to generate an individual sequence diagram for every API request I make. With the libraries installed, and by simply interacting with my application I get a complete view of the entire code sequence automatically without having to spend any additional work adding telemetry, traces, or spans to my code.

You could also use OTEL to instrument your app and capture similar data. But generally there is a larger development effort in identifying where in the code you need to add spans and traces too. The advantage of AppMap’s traces is that they are created without having to modify the codebase.

Now we’ve got some detailed traces and sequence diagrams and other runtime data. I could spend time looking through them to get a better understanding of how my new user registration process works, but instead let’s ask a question to our AI assistant, Navie.

Runtime data gives AI new powers to answer questions about how apps work

So using runtime data, let’s ask new kinds of questions beyond “how does this code work”. Let’s ask “how does this feature” or “how does this service works”, starting with the registration service.

AI has detailed information about the code and how the app works, so when you include code in the prompt to GenAI you can understand the code in the context of the app. This is in contrast to a generic explanation of what the code does using an internet search and static analysis as the only source of information.

Now I’ve learned how this API endpoint works, from API down to the database with runtime data.

So now my AI has context, and I can go further and do things like ask it to implement a new captcha service. I will get a very detailed solution that is relevant to my codebase.

Because the GenAI now has the sequence diagram and the specific functions and code snippets for the API interaction, it’s able to tell me exactly what files to go to and what functions to change.

And now when searching for that function in my code editor:

I know exactly where to go to add this line to which file.

Why can’t I just use Copilot or GPT-4 directly?

Now what would happen if I just asked Copilot or GPT-4 directly for information about how to add this simple captcha feature to my app? Even though tools like Copilot have ways of using your “workspace” as context in the question, it has no idea how the code runs and thus you would only get generic answers based on existing public knowledge.

For example this response from Copliot.

The only problem here is that there is no RegistrationForm class in my project. As Navie told us in the previous section, the class where I would actually need to add this is called EmailUserCreationForm.

Now you can see how adding tracing data and other runtime context improves the AI coding experience across your application. Now you have a software architect or a senior staff engineer pair programming with you. Without knowledge of your code’s runtime, these generic responses will leave you continuing to hunt and search through a complex code base. Adding runtime behavior providea context-aware insights to power hyper-personalized responses to your toughest software questions.

Check out this video demo watching me add this feature, with the help of AppMap Navie.

To learn more about getting started with AppMap Navie, head over to getappmap.com

How do you say... [insert tech lingo here]: A video series.

Pete Cheslock — Tue, 26 Sep 2023 19:24:44 +0000

I've been fascinated by how the industry has come to pronounce different tech words, lingo, products, and acronyms. Months ago, I asked my twitter followers for an idea of some good words to have people pronounce in a "game show" style lightning round.

// Detect dark theme var iframe = document.getElementById('tweet-1635342410053201921-731'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1635342410053201921&theme=dark" }

After 31(!!) individual recordings this summer, we shipped the series with friends from all over the tech industry in all kinds of roles.

We threw a slew of tech names and acronyms at them to see how they pronounced them because, hey, most of these words have been made up in the last decade or less to describe a new technology or name for something related to ye'ole computer.

The list:
/etc
AMI
k8s
chmod
curl
fsck
HAProxy
/proc/
NGINX
SQL
systemctl
SCSI
www
sudo
URL
/lib/
CIDR
epoch
jwt
strace

Wanna check out the supercuts? You can find them on YouTube Shorts or TikTok

Head over to this AppMap Youtube Playlist to check out all the full length "game show" style lightning rounds.

The best part is the conversations between humans - the relationships with folks from all walks of tech life - and the joy you get from watching the laughter or disgust ensue.

How do you say... "/etc"?

Pete Cheslock — Tue, 11 Jul 2023 17:38:22 +0000

How would you pronounce this file system directory name commonly found on linux and unix-like systems?

Contains system-wide configuration files and system databases; the name stands for et cetera[14] but now a better expansion is editable-text-configurations. Originally also contained "dangerous maintenance utilities" such as init,[6] but these have typically been moved to /sbin or elsewhere. Needs to be on the root filesystem itself.

/etc stands for Et Cetera which is a Latin expression used in English to mean and other (similar) things, or and so forth.

Translated literally from Latin, et means 'and', while cētera means 'the rest'; thus the expression translates to 'and the rest (of such things)'.

But if you had to tell another person to put a file in the /etc directory, how would you say it?

Listen to a few examples from technology professionals on the latest episode of "How Do You Say?"

Want to see more tech word pronunciations by tech professionals? Find them on Youtube. New videos ship every Friday.

How do you say... "k8s"?

Pete Cheslock — Fri, 16 Jun 2023 13:47:49 +0000

You may have seen the abbreviation for "Kubernetes" as k8s online. This is called a Numeronym.

A numeronym is a number-based word. Most commonly, a numeronym is a word where a number is used to form an abbreviation (albeit not an acronym or an initialism). Pronouncing the letters and numbers may sound similar to the full word, as in "K9" (pronounced "kay-nine") for "canine, relating to dogs". Alternatively, letters between the first and last letters of a word may be replaced by a number representing the number of letters omitted, such as in "i18n" for "internationalization".

But how would you SAY k8s if you had to tell it to another person? It's a word we often read or write but rarely say.

I asked over 30 technologists for their pronunciation of this numeronym. The results may surprise you!

Want to see more tech word pronunciations by tech professionals? Find them on Youtube. New videos ship every Friday.

How do you say... "fsck"?

Pete Cheslock — Tue, 30 May 2023 15:03:34 +0000

Have you ever had to boot into single user mode and run fsck?

It's a word we often read and write but rarely say. How would you pronounce it to a co-worker or say it out loud?

So how do you pronounce it?

I asked over 30 software engineers and technical operators exactly that question. Here are a handful of their responses.

There are no agreed upon pronunciations, but here are a handful from wikipedia.

"F-S-C-K", "F-S-check", "fizz-check", "F-sack", "fisk", "fizz-k", "fishcake", "fizik", "F-sick", "F-sock", "F-suck", "F-sek", "feshk", the sibilant "fsk", "fix", "farsk", "fosk" or "fusk".

Want to see more tech word pronunciations by tech professionals? Find them on Youtube. New videos ship every Friday.

How do you say... "jwt"

Pete Cheslock — Tue, 23 May 2023 20:21:44 +0000

Have you heard of a "jwt" before (JSON web token)?

It's a word we often read and write but rarely say.

So how do you pronounce it?

I asked over 30 software engineers and technical operators exactly that question. Here are a handful of their responses.

Turns out - there is an official pronunciation for "jwt" on the creators's (Auth0) website.


 web token (JWT), pronounced "jot", is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object.

But it's NOT the same way to pronounce "JWt (Java web toolkit)".


 (pronounced "jay-witty") is an open-source widget-centric web application framework for the Java programming language developed by Emweb.

Want to see more tech word pronunciations by tech professionals? Find them on Youtube. New videos ship every Friday.

The DevOps Hangover

Pete Cheslock — Fri, 19 May 2023 16:12:25 +0000

The greatest irony is that DevOps aimed to help developers talk with operations, and vice versa, yet developers are even further away from understanding how their code operates at runtime.

Engineering and development teams, from startups to large corporations, have faced hard economic events over the last 12 months. Large-scale layoffs and company restructuring have forced teams to “do more with less” while building, scaling, and operating large-scale systems. Yet progress still needs to be made, and value still needs to be shown higher up the chain.

After years of consuming countless tools and services to maintain high levels of performance and security, with little oversight for design or costs, the DevOps hangover is real. The average company continues to use well over 100+ SaaS tools, and many companies are still investing in new tools and technologies to help them weather the current economic storm (Battery Report 2023).

The teams who can take this time to assess and streamline their arsenal will be the ones who emerge on the other side, cutting costs and improving efficiency, while gaining a competitive advantage.

So, what’s the best medicine for our DevOps hangover? First, let’s understand how we got here.

DevOps then: Party time, excellent!

DevOps emerged during a time of unbridled economic expansion with financial resources and the easy-to-use, pay-as-you-go SaaS products to support it. The problem has been two-fold:

Developers continue to use solutions that aren't perfect, as long as they are useful enough for their needs. However, when budgets become tight (like today), these tools or services may cause unnecessary financial stress.
And, developer toil was largely ignored as we yelled over our shoulders that it’s ok to “move fast and break things!” But today, developers are spending more than 17 hours per week dealing with maintenance issues, such as debugging and refactoring, according to the Developer Coefficient report by Stripe.

DevOps now: Waking up to the hangover

Imagine you are now waking up to a request to do more with less, whether that is a reduced team size or a request to lower cloud and tools bills. With the loss of team members, you may be losing institutional knowledge that enables key applications to run. This loss can result in hundreds of thousands of dollars in outages, lost revenue, and regulatory fines. Despite these challenges, you may still be expected to maintain a five nines SLA to achieve aggressive growth targets.

It's not just engineers and operators who are waking up to the DevOps hangover. Executives, including CEOs, COOs, CFOs, and engineering leaders, are also taking a closer look at operational spending in relation to sales and growth. Many are now hiring consultants to help them map their cloud spending with the rest of their operational spending (like Observability and Data Platforms) in order to find ways to make necessary cost reductions.

What does the hangover cost us?

It depends on the conditions under which your development team is operating. Here are a few common environments:

Complex legacy codebases
A microservices platform that is growing exponentially
A hybrid, stalled-out digital transformation project that is a Frankenstein’s monster of monolith and microservices

Let’s break down each one.

Cost of a legacy codebase

When a company's codebase becomes a monolith that nobody can easily understand, new features take substantially longer to release because it’s much harder to add features and make changes when there are too many unknown upstream and downstream effects. This delay can be frustrating for both the development team and the end-users who are eagerly waiting for new features to be implemented.

Cost of a microservices platform

On the other end of the spectrum, Kubernetes and Lambda-native applications proliferated during the growth-at-all-costs era when cash was being burned at an astonishing rate. When running multiple products across shared underlying infrastructure, it becomes significantly more difficult to attribute per-product COGS (Cost of Goods Sold). This shift to a microservices architecture was made to improve the speed at which developers could deliver new applications. However, with shrinking budgets, already constrained Ops/DevOps/Platform teams are now pushing the ongoing management of these platforms onto the overburdened dev teams.

Cost of a hybrid monolith + microservices platform

And finally, in the middle of these two extremes are the teams stuck with both a legacy monolith, a constellation of microservices, and maintaining their five nines SLA with every code change. Teams are now faced with the challenges of breaking code changes, scaling issues, and the growing number and complexity of dependent services. And they are throwing all the observability tools at the problem to better understand how the code operates at runtime.

Autoscaling up is easy, Autoscaling down isn’t

While understanding how to scale up infrastructure as your usage grows is important, it can be challenging to determine how to scale down when usage decreases. And many teams never spent the time to plan for a scenario where they would need to reduce the infrastructure as usage decreased.

If the cost of delivering your application becomes detached from your sales and revenue, your company may end up spending more money than necessary. This can happen if your cloud and observability bills remain constant, even as your business slows down.

Curing the DevOps Hangover

Every software company is asking its development team to cut infrastructure costs. Yet, the same problems remain: bad code creates blockages, outages, and inefficiencies that cost hundreds of thousands, sometimes millions, to the business, annually.

First, let’s tackle operational costs.

Look beyond cloud spending as our only barometer for operational costs. The costs of delivering a product feature can exist over many services and across teams.

Start with a spreadsheet or list and account for all of the other applications you use to deliver the entire application.
Identify each feature or product's actual value to the user and business, not just what sales and marketing claim they do.
Find areas of overlap between applications and teams. This is a great opportunity to identify and break down technical and organizational barriers built up.

Some costs for delivering your application are hidden in third-party SaaS services that follow pay-per-use or subscription models but do not register as your typical cloud expense. In particular, observability tooling and data infrastructure lead the pack for non-cloud infrastructure spend.

Services like DataDog and NewRelic have extremely high costs related to custom metrics, and once a metric is added, it is unlikely to be removed later. Identify and attribute these underlying costs back toward the product or team supporting it.
Review and understand which team owns which data sources. Take note of where data is moving within your applications and services. Many cost overruns stem from developers not understanding the cost impacts of data transfer between services.
Identify where these tools are currently operating. In expensive production environments where it is costly to maintain these tools and more expensive to cure the issues? Consider migrating these tools earlier in the development process where mistakes are cheap to fix, and the cost of running developer tools is lower.

Debugging in production may seem like the best way to solve the problem, but this will be after your users are already impacted, or after a critical code change balloons your AWS costs. It’s simply too late to be effective.

Next, let’s address the importance of code analysis and application design.

Code quality and design influence costs. Your senior developers know it, but it may be much less obvious to your CFO. Consider this:

A developer adds new code to your application, like a new feature or a bug fix.
That code change makes a repeated call to a database, increasing database queries.
Those increased queries cause application and database performance loss.
An SRE, siloed by a developer, is alerted to poor performance and increases the underlying infrastructure to support the workload.
The cost to deliver this application has now doubled as the SRE team threw the cloud at the problem and increased the database to a larger instance.

It is quick and easy to make a database larger, but in reality, we simply masked a code quality issue AND increased our overall spending with no additional value being provided to the user. Over time, the code becomes riddled with performance issues that, in reality, never get fixed. Code quality suffers, as well as your bottom line.

For example, this type of N+1 SQL query problem isn’t ONLY a problem for databases and slow-performing applications. Today's services are pay-as-you-go, where each API request has a defined cost, and inadvertently calling that same API repeatedly will rack up the bills.

Finally, let’s acknowledge this will take continual improvement.

Remember, all hangovers will eventually pass. The ideal scenario is that you and your team can begin taking stock and evaluating your tools and services, and then you start seeing an improvement in the performance and costs of delivering your application. What can you then do to ensure things stay this way?

Remember that this isn’t a static process. That’s what got us into this mess. Make taking stock and evaluating your tools and services a quarterly step. Even better, include this work in your ongoing product development workflows. A little bit of low-friction software governance can go a long way toward reigning in wasteful and costly code changes, and incorporating it into your CI/CD means catching problems earlier in the process.

Even if you are unable to make a large investment into optimization today, understanding how your software is changing when it changes will be critical to stem the tide of growing operational expenses. Use tools to help you analyze your software during the development and testing phase in order to identify performance issues early and put those tools into the hands of the developers so they can better understand the operational impact of the code changes they are making.

How Do You Say... "Epoch"

Pete Cheslock — Mon, 08 May 2023 17:24:09 +0000

As a technologist, there are a lot of words and acronyms we often only read or type out but don't say out loud (commands/services/etc). So, I asked a bunch of folks how they pronounce some of them and recorded the result. For example...

For example: "Epoch"

In computing, an epoch is a date and time from which a computer measures system time. Most computer systems determine time as a number representing the seconds removed from particular arbitrary date and time.

How do you pronounce it?

Top 5 VS Code extensions

Pete Cheslock — Thu, 04 May 2023 18:58:33 +0000

As a co-builder of AppMap — a runtime code analysis platform for developers to better visualize, interact with, and understand their code — I spend a lot of time helping people in their code editor.

When our users are working in VS Code, they like to use a few key extensions to make life easier and more fun. We’re all about developer happiness around here, so I thought I’d share the top 5 VS Code extensions that the AppMap team loves.

[Out now! Early access to AppMap for GitHub Actions]

What are yours?

Vim

Our team likes to use the vim keybindings so they don’t have to take their fingers off the keyboard.
Prettier

This one is very popular, so it won’t come as a surprise to anyone that it’s on the list. Our dev team appreciates not having to worry about formatting their code, and with Prettier, their code stays linted and formatted. They say it works best with JS and JSON.
GitHub Copilot

This new extension really is as great as the hype. It drastically reduces the need for the team to look up language and API docs. Plus, we’re kind of into planes over here, so we love the name.

That's Kevin, AppMap CTO 🛫
Live Preview

This VS Code extension serves a workspace as HTTP, which is useful when the team wants to use, preview, test, or debug an HTML page (including JS code such as Vue) that’s part of the project.
GitHub Markdown Preview

It’s really helpful to have a preview of how the GitHub markdown features will display before the team commits a change, so this extension is one they use often.

BONUS: AppMap

Of course!

We just updated our marketplace page today as we’ve been building and improving AppMap for a few years now.

We built AppMap to run in the VS Code editor so it can analyze runtime code behavior before you commit changes and go to production. It works by recording detailed traces of how your application runs and analyzing it based on frameworks and techniques. Interactive, visual sequence diagrams and views present your bugs, loops, calls, dependencies, security issues, and other things that might keep your code from being ready to ship.

AppMap can also be used to help you understand a legacy codebase, which someone in our community recently likened to trying to understand “a crime scene.” It can also instantly generate OpenAPI documentation based on the API calls it observes at runtime.

AppMap supports the following programming languages: Java, Python, Ruby, Typescript, and JavaScript. And, these web applications and API frameworks: Ruby on Rails, Django, Flask, Express, and Spring.

In terms of data usage, AppMap runtime recordings and diagrams are created and stored locally on your machine. AppMap does not require any permissions to your web-hosted code repo in order to run.

How to get started with AppMap

Install from the VS Code marketplace.

Visit the documentation for guides and videos.

Use AppMap to generate sequence diagrams of your code.

Check us out and leave a ⭐ in GitHub.

Read the blog for user stories and product announcements.

Join us in Slack or email us for support: support@appmap.io

Code-First Auto-Generated OpenAPI Docs for Django Applications

Pete Cheslock — Tue, 28 Feb 2023 19:23:23 +0000

Using the open source e-commerce framework Django-Oscar as an example.

API documents are usually a developer’s first introduction to a project; however, they can be out of date or incorrect because developers on the project have to go in and update the documents manually each time the code changes. This is not only a waste of developers’ time, but it is easy to forget to do it altogether. The industry says "API first" - starting with an API Document - is the way to build applications, but this approach can be overlooked when there are tight deadlines.

AppMap is a free, dynamic runtime analysis tool that makes it easy to automatically generate API documents. Automatic OpenAPI generation can also be incorporated into your CI/CD system to prevent the documents from ever being out of date or incorrect. The days of manual API documentation writing are over. Write your code, run your code, and the document will appear in your repository.

Add AppMap to Django-Oscar

In Part One of this series, you learned how to generate AppMaps for the open-source project Django-Oscar. AppMaps are visual representations of code that allow developers to dig into the trace of their HTTP requests, function calls, and database writes. To install AppMap libraries into the Django-Oscar codebase, add the AppMap PyCharm extension.

If you are not using VS Code or Jetbrains, or if you want to incorporate OpenAPI generation into your CI/CD system, you can install the AppMap CLI and generate OpenAPI docs on the command line.

Run the Django-Oscar Test Suite

Set your environment variable as shown below, and run the PyTest command to execute the Django-Oscar test suite. The test coverage is extensive and will take a while to run, so feel free to run a subset of the tests first just to see how the API generation works. You can learn more about generating AppMaps from tests in Python in the AppMap Documentation.

APPMAP_LOG_LEVEL=info pytest -svv

After running the full test suite, you’ll have a collection of code-accurate AppMaps with all the up to date API request information within.

Click on one of the AppMaps in the sidebar to expand it and explore the dependencies within the project.

Generate the OpenAPI Spec

Now that our API requests have been recorded within our AppMaps, you can generate the OpenAPI v3 supported specification file via either PyCharm or using the AppMap CLI. From there you can import the file into Postman, other API management tools, or share with your team.

After the Django-Oscar test cases complete, click the “Generate OpenAPI” button.

This will open the OpenAPI generator. Then click the “Generate OpenAPI Documents” button and your OpenAPI specification will be displayed in the editor. Save this data anywhere you need it for later usage or distribution.

For example: I loaded this API Specification into Postman as a new collection.

Generate OpenAPI Document with AppMap CLI

If you are not a PyCharm user, or if you want to incorporate OpenAPI generation as part of your CI process, you can run the AppMap CLI command after the tests have completed to similarly output the API documents to a user-defined output file.

The AppMap CLI is written in JavaScript, which makes it easy to run across a variety of platforms. To install the AppMap CLI, you need Node.js installed, or you can grab pre-built binaries for @appland/appmap on Github. To install AppMap CLI after Node.js is installed, run the following command.

npx @appland/appmap@latest openapi --output-file openapi.yml

Import the OpenAPI documents into other tools to more easily interact with your APIs, test new or updated features, and finally export the OpenAPI documents to your users so that they will know how to interact with your application as well.

Now that you've generated the API document locally, add this command to your CI system to update your OpenAPI docs as your code changes.
You can learn more about other features of AppMap by reviewing our documentation or you can join the developers in our community Slack.

Optimizing Mastodon Performance: Finding and Fixing its N+1 Queries

Pete Cheslock — Thu, 12 Jan 2023 18:34:06 +0000

N+1 queries happen in a variety of ways, out of sight to many developers who are using an ORM. With the level of abstraction that ORMs provide, it can be easy for developers to accidentally incorporate a query that is executed on each result of the previous query. ORMs can make development faster and more secure, but the automatic generation of SQL queries can make it hard to predict the code's performance when it gets deployed.

Developers can be unaware their functions even query the database and a simple feature like checking user permissions could lead to querying for the same exact data multiple times.

Active Record is a Ruby on Rails framework used to easily save and retrieve objects from a database in programming frameworks. It automatically generates SQL statements to do this. ORMs like Active Record can make it easier for developers to write database-backed applications, but they can also make it harder to understand exactly how their code makes database queries.

In this final part of our Mastodon Series, you’ll learn how to analyze your AppMaps to uncover and fix an N+1 SQL query in an active code base.

In the previous posts you have learned:

In our previous post, you learned how AppMap uses the runtime code data to automatically generate accurate OpenAPI documentation. AppMap contains all the relevant HTTP request information including status codes, headers, and the session operations and AppMap builds complete and accurate OpenAPI documentation by analyzing how the code executes.

This same AppMap data can be used in other ways, for example in this post you’ll learn how to use AppMap Analysis to locate and fix a performance issue. Since the AppMaps shows function calls, SQL queries, exceptions, and more, they can be analyzed to find design flaws that contributes to performance and security issues.

Design related security flaws are rising in the OWASP Top 10 in recent years. Runtime data helps us to uncover design flaws that are contributing to slow performance such as an N+1 query.

Generate Code Analysis Findings

With our rspec tests passing and our AppMaps created back in Part 3, findings are now shown in the section of our VSCode extension titled “Runtime Analysis”. Since AppMap records the runtime application behavior, it analyzes the output of those recordings and scans for code behavior issues that you can’t find with a static analyzer.

AppMap can scan for a variety of Performance, Security, Scalability, and Maintainability issues. These rules are extensible for developers to create and contribute to the open source community.

On the left side of the VS Code extension, you can see a section called “Runtime Analysis”. This is where AppMap will display the results of the analysis scan. This scan should happen within seconds of AppMaps being created or updated.

When clicking on any one of the lines of code referenced in the findings, a new tab will open with additional details about the event, including links to the relevant CWE. The findings page will also show the stack trace leading up to the event, and a list of the AppMaps this flaw is present in.

By navigating to the base controller in Mastodon, AppMap identifies this line of code (Line 92) as making a database call (via User.find) when the doorkeeper_token variable is false. But by looking closer at the stack trace, AppMap shows line 109 is the caller function which checks to see if the current user is known and properly validated. This is correctly reported by AppMap as an N+1 query.

https://github.com/mastodon/mastodon/blob/623d3d2e32ac8ec2819f2cd99e6565d06c9b0023/app/controllers/api/base_controller.rb#L111-L141

A simple fix would be to just fetch current_user once in require_user!, or to memoize it in the current_user function.

AppMap visualizes this function and the repeated SQL queries.

Understanding the Before and After

Before implementing the code change, save the original AppMap in a “Baseline” collection so that AppMap can visually compare the before and after impact of the change.

This code is found in 6 AppMaps and they are listed on the specific findings page.

Now select one of these AppMaps, right click them, and choose the option to “Save To Collection”. Collections can be used to create copies of AppMaps which is useful for saving the current code behavior as a baseline before you implement code changes.

From there we can click <create> to create a new collection of AppMaps or select a previously created collection to add to.

After copying that AppMap to a new collection, you’ll see it included within the list of existing AppMaps.

It’s important to note that this makes a copy of the AppMap in the state it was in when it was added to the Collection. After updating the code and the AppMap, the collection will still contain the previous version of the code.

Implementing the Performance Fix

This poorly performing code is improved by breaking out the user validation into its own function so it will not repeatedly query the database for the same information.

https://github.com/mastodon/mastodon/pull/23057

Identify Out of Date AppMaps

After making this code change (either committed to your project or not) AppMap will display both the baseline and other maps that are listed as out of date because we’ve changed the functions these maps had previously recorded. AppMap knows when the underlying code has been changed and can notify when the AppMaps no longer match the code.

Open the Command Palette in VS Code (View -> Command Palette OR Shift+⌘+P on Mac OR Ctrl+Shift+P on Windows), and search for the option to “Copy Out-of-Date Tests”.

You can choose to copy the file names into the clipboard and paste them into the rspec command.

After successfully running only the tests needed to update our AppMaps the baseline AppMap will continue to be out of date and the other AppMaps will be updated. It’s important to note that when you make a code change to a function recorded in the “Baseline” AppMap, that AppMap will always display as “out of date” since it will always represent the original test case run.

Generating Sequence Diagram Diffs

Another feature of AppMap is the ability to automatically create sequence diagrams of your code’s runtime behavior. This can be extremely useful information to include in a pull request or a code review to help other members of your team understand how the code behavior has changed.

Since a baseline AppMap exists prior to our code change, AppMap can create a sequence diagram diff of the before and after. Assuming the local VS Code environment is configured to support sequence diagram creation, AppMap will be able to both generate a sequence diagram, or compare two sequence diagrams.

To generate the sequence diagram diff, right click on the latest version of the AppMap and select “Compare Sequence Diagrams”

Hit enter twice then select the original baseline AppMap saved in our Baseline collection.

From here a new browser window will open showing a sequence diagram diff visualizing the change in behavior of the code from before and after our changes. The red items in sequence diagram will indicate code behavior that has been removed, and green items will show new code paths created.

Now the changes in the sequence diagram show the code is not longer making similar repeated SQL queries. By reducing these unnecessary queries the speed of this code function has increased.

Summary

In this post, you’ve learned how to find and fix a performance issue in the Mastodon code base using AppMap. The automated code analysis will work for other projects written in languages other than Ruby like Java, Python, and JavaScript/TypeScript.

To get started finding potential issues in your code base, download and install the AppMap VS Code Extension, or the JetBrains Marketplace in your code editor and follow the similar instructions taken in this project. For more language specific information, check out the AppMap documentation, or join the AppMap Community Slack to ask a developer for more information.

Automatically Generating OpenAPI Docs for Mastodon

Pete Cheslock — Wed, 14 Dec 2022 20:26:15 +0000

As the Mastodon open source community grows, many developers are coming into the project for the first time and learning more by viewing the documentation. One of the common pieces of documentation that users in both the Mastodon Discourse and GitHub issues are looking for is an up-to-date version of API documentation.

One user proposes writing rswag specs to allow for automatically documenting the APIs and generating the OpenAPI schema. The downside to this approach is the additional engineering time it takes for adding this net new code to the project.

The OpenAPI specification, which is formerly known as the Swagger Specification, is a community-driven open standard for documenting APIs. OpenAPI allows both humans and computers to discover and understand the capabilities of a service without requiring access to the service’s source code.

In the previous parts of this series, we showed you:

By now you’ll now have AppMaps recorded from the Mastodon tests. The AppMap data specification includes all of the API calls processed including the schema of each request and response. Using the AppMap CLI (or the VS Code or JetBrains extension) developers can automatically output OpenAPI specifications for Mastodon in just 3 steps.

Add AppMap to Mastodon
Run the Mastodon Test suite
Generate the OpenAPI Spec

Add AppMap to Mastodon

Check out part 1 of this series to install AppMap libraries into the Mastodon code base, and to install the VS Code or JetBrains extension.

If you are not using VS Code or Jetbrains, or if you want to incorporate OpenAPI generation into your CI/CD system, you can install the AppMap CLI and generate OpenAPI docs on the command line.

Run the Mastodon Test suite

In Part 3 of this series you learned how to run the entire Mastodon rspec test suite. After running the full test suite you’ll have a collection of code-accurate AppMaps with all the up-to-date API request information within.

Generate the OpenAPI Spec

Now that AppMaps exist with our API requests, you can generate the OpenAPI v3 supported specification file via either VS Code or using the AppMap CLI. From there you can import this into Postman, other API management tools, or share with your team.

Generate in VS Code

After the Mastodon test cases complete - click the “Generate OpenAPI” button on the left column.

This will open the OpenAPI introduction page. Then click the “Generate OpenAPI Definitions” button and your OpenAPI specification will be displayed in the editor. Save this data anywhere you need it for later usage or distribution.

For example: I loaded this API Specification into Postman as a new collection.

Generate with AppMap CLI

If you are not a VS Code user, or if you want to incorporate OpenAPI generation as part of your CI/CD build process, you can run the AppMap CLI command after the rspec tests have completed to similarly output the API documentation to a user-defined output file.

The AppMap CLI is written in JavaScript which makes it easy to run across a variety of platforms. You’ll need Node.js installed or alternatively you can grab pre-built binaries for @appland/appmap on Github.

npx @appland/appmap@latest openapi --output-file openapi.yml

Now you can import this OpenAPI documentation into other tools to more easily interact with your APIs, test new or updated features, and finally export these OpenAPI specifications to your users so that they will know how to interact with your application as well.

You can learn more about other features of AppMap by reviewing our documentation or you can join the developers in our community Slack.