DEV Community: AppMap

How AppMap Navie solved the SWE bench AI coding challenge

Kevin Gilpin — Tue, 25 Jun 2024 16:11:31 +0000

AppMap Navie is an AI coding assistant that you can use directly in your VSCode or JetBrains code editor.

SWE Bench is a benchmark from Princeton University that assesses AI language models and agents on their ability to solve real-world software engineering issues. It's made up of 2,294 issues from 12 popular Python repositories, along with their human-coded solutions and test cases. It is considered to be the most difficult of the well-known coding benchmarks.

AppMap Navie recently posted 14.6% on SWE Bench, ahead of Amazon Q and 8 other tools. We were able to process the entire benchmark in under 4 hours, and at the lowest recorded cost of operation - up to 95% less expensive than other solvers.

How did we do it? Read on for useful techniques that you can apply to your own AI programming.

Why basic solvers fail

The easiest way to use an LLM to solve a code issue is simply to send the issue description to the LLM along with all the code files and prompt the LLM to generate a patch file. This is basically what the first generation of SWE bench solvers attempted to do. However, the solution rate of this approach is very low (single digit percents). Why?

1) Wrong context Most LLMs have a context limit which is too small to load the entire codebase. So, some guesses have to be made about which files to give the LLM. And when those guesses are wrong, the LLM fails to generate the right solution.

2) Failed patching LLMs are not good at creating patch files. Most LLM-generated patch files will fail to apply.

3) Broken code LLMs will generate code that is malformed. It won't even run cleanly, never mind pass the test cases.

4) Wrong code design The LLM does not understand the project design and architecture. So, it tries to solve the problem in the wrong place; or it fixes one problem while creating another.

You can see some of these early solvers on the SWE bench leaderboard:

Generation 2 - Agentic architecture

The next generation of solvers adopted a more complex architecture, in an effort to solve the problems above.

Basically, the idea of an "agent" is to give the LLM a wider variety of tools, and then run a loop in which the LLM chooses a tool and examines the results of using it.

Tools do things like:

Search the code for a keyword
Inspect a file
Run a program and examine the console output
Edit a file

Agents do substantially better on the benchmark:

However, most of the "agentic" solutions only appear on the SWE Bench "Lite" leaderboard. Why is that?

1) 💸 Cost Every tool an agent uses consumes tokens. Tokens cost money. Agentic loops use tokens over and over.
2) 🐢 Speed By design, agentic solvers can take a lot of time to explore the problem space. They can backtrack and repeat things they've already done. They can get stuck.

AppMap Navie architecture - Semi-agentic

Agents have a higher pass rate than Basic solvers, but they are slow and expensive. AppMap Navie takes an intermediate architecture, in which the solver is provided with powerful capabilities:

Rich, but selective, code context to enable retrieval-augmented generation (RAG) architecture.
A powerful and reliable tool for making file edits.
Self-healing feedback for fixing code.

Code context

It's inefficient and expensive to send huge amounts of code context to the LLM. Embeddings are slow and expensive to generate. But without the right code context, the LLM-generated solution will fail.

Navie uses a technique called "client-side RAG", in which the code is organized and searched locally. Client-side compute is fast, cheap, and much more efficient than sending huge token payloads to the LLM or building expensive embeddings.

Planning

With the right context selected, it's time to get right to code generation - right?

Wrong. Before code generation comes Planning. Human developers don't dive into coding without some kind of plan. Building an understanding of the system architecture is an essential step, it can't be skipped over by humans, and it shouldn't be skipped by AI coders either.

So, Navie performs an explicit planning step, in which the the issue description is combined with the context to produce a detailed plan. The plan includes:

A restatement of the problem.
A high level solution description.
A list of files to be modified.
For each file, a description (no code, yet), of how that file will be changed.

Here's an example of a Navie-generated plan.

File editing

Now, with the plan in hand, the LLM is ready to change code files.

Navie doesn't ask the LLM to generate patch files; it doesn't work. Instead, the LLM generates a "search / replace" pair of code snippets. This works most of the time, and a simple retry loop fixes up most of the occasions when it doesn't.

Here are the Navie-generated code changes that implement the Plan.

Lint repair

The LLM still might get something wrong. Common cases include:

Mistakes with indenting (particularly with Python).
Missing imports.

The Navie solver runs a linter, then feeds the linter errors back into the AI code editor. Most lint errors can be fixed this way.

An example of lint errors fixed by Navie.

Test repair

Still not done! If the solution generated by Navie breaks existing tests, it's probably not going to fix the issue properly. So the Navie solver runs the application test cases to try and catch and fix any incompatibilities that may have been introduced.

Now, it's ready

Now a patch file is created by simply diff-ing the AI-edited code with the Git base revision. This patch file is submitted to the SWE Bench harness for evaluation.

How is this so efficient?

The Navie solver runs for about 1/3 the cost of most other solvers; and it's 95% cheaper than some of the most intensive agentic solvers on the benchmark (of those that post their costs; many don't 🙁).

Efficient client-side RAG context saves $$ on embeddings and LLM tokens.
Lint repair and test repair fixes solutions that might be almost, but not quite, correct.
A smaller "tool" suite and a linear approach to solving the problem prevents the LLM from wandering down dead ends or getting lost in pointless loops.

Try Navie yourself!

Navie is available today, with no wait list. Here's how to get Navie, or learn more about AppMap:

⬇️ Download AppMap Navie for VSCode and JetBrains: https://appmap.io/get-appmap
⭐ Star AppMap on GitHub: https://github.com/getappmap
📰 Follow on LinkedIn: https://www.linkedin.com/company/appmap
💬 Join AppMap Slack: https://appmap.io/slack
ℹ️ Read the AppMap docs: https://appmap.io/docs

Beyond Code Completion: Better Prompt Context to Supercharge Your AI Coding Workflow

Pete Cheslock — Tue, 12 Mar 2024 13:59:26 +0000

AI code completion tools like GitHub Copilot have revolutionized the way we write code. These tools not only speed up coding tasks but also help in adding features and fixing bugs more efficiently.

This post will show you how I use tracing, sequence diagrams, and other personal observability data with Generative AI, via a RAG (Retrieval-Augmented Generation) model to get better AI guidance for application development. Imagine feeding AI observability data like traces, performance timing, sequence diagrams, API calls, and other information that maps out your application’s interactions. The result? A hyper-personalized instructional guide on how your application operates and code suggestions empowering you to make informed decisions like a senior staff engineer or software architect.

With the strategic use of AI and runtime data, the path from code suggestion to architecture suggestion becomes not just accessible, but intuitive. This is more than AI coding assistance; it's creating a development process where information is synthesized and presented to you to elevate your code decision making and expertise within your group.
How could you quickly become an expert on your codebase?

Let’s show you how this works.

Shifting your AI from Code Suggestions to Thinking Partner

There are many tools that will statically index your code base and send them as context to the AI to provide code suggestions about your codebase. But there is still a big missing piece of context for generative AI coding assistants that can sometimes lead to bad code suggestions. Tools like Copilot and other AI coding assistants don’t know anything about your application when it runs. This results in between 50-70% of code suggestions being rejected by developers.

Adding tracing data to AI generated prompts can improve the quality of code suggestions. It can also help you get answers to different kinds of questions from generative AI coding tools which are trained on static open source code.

You can follow this process with any large token AI system like Claude by identifying tracing data relevant to the code you are working on, using it as context to prompt OpenAI or other LLMs. Generally, you’d generate tracing data by implementing OpenTelemetry (aka OTEL) libraries into your application, adding spans to your functions with Jaeger, or using commercial SaaS tools like Honeycomb and Datadog.

In this example, I’ll use AppMap to generate tracing data and other runtime data about my code in my code editor. This will provide a structured way of describing how my code executes when it runs.

I will also use Navie, the AI integration for LLMs from AppMap. By default, Navie uses OpenAI GPT-4 integration with pre-programmed prompts to consider runtime data. I will feed the AI a combination of code snippets, traces, and sequence diagrams as context of this particular code interaction I am working on.

In this case I’ve installed AppMap installed into a python powered web store which uses Django.

To get tracing data I use Request Recording to generate an individual sequence diagram for every API request I make. With the libraries installed, and by simply interacting with my application I get a complete view of the entire code sequence automatically without having to spend any additional work adding telemetry, traces, or spans to my code.

You could also use OTEL to instrument your app and capture similar data. But generally there is a larger development effort in identifying where in the code you need to add spans and traces too. The advantage of AppMap’s traces is that they are created without having to modify the codebase.

Now we’ve got some detailed traces and sequence diagrams and other runtime data. I could spend time looking through them to get a better understanding of how my new user registration process works, but instead let’s ask a question to our AI assistant, Navie.

Runtime data gives AI new powers to answer questions about how apps work

So using runtime data, let’s ask new kinds of questions beyond “how does this code work”. Let’s ask “how does this feature” or “how does this service works”, starting with the registration service.

AI has detailed information about the code and how the app works, so when you include code in the prompt to GenAI you can understand the code in the context of the app. This is in contrast to a generic explanation of what the code does using an internet search and static analysis as the only source of information.

Now I’ve learned how this API endpoint works, from API down to the database with runtime data.

So now my AI has context, and I can go further and do things like ask it to implement a new captcha service. I will get a very detailed solution that is relevant to my codebase.

Because the GenAI now has the sequence diagram and the specific functions and code snippets for the API interaction, it’s able to tell me exactly what files to go to and what functions to change.

And now when searching for that function in my code editor:

I know exactly where to go to add this line to which file.

Why can’t I just use Copilot or GPT-4 directly?

Now what would happen if I just asked Copilot or GPT-4 directly for information about how to add this simple captcha feature to my app? Even though tools like Copilot have ways of using your “workspace” as context in the question, it has no idea how the code runs and thus you would only get generic answers based on existing public knowledge.

For example this response from Copliot.

The only problem here is that there is no RegistrationForm class in my project. As Navie told us in the previous section, the class where I would actually need to add this is called EmailUserCreationForm.

Now you can see how adding tracing data and other runtime context improves the AI coding experience across your application. Now you have a software architect or a senior staff engineer pair programming with you. Without knowledge of your code’s runtime, these generic responses will leave you continuing to hunt and search through a complex code base. Adding runtime behavior providea context-aware insights to power hyper-personalized responses to your toughest software questions.

Check out this video demo watching me add this feature, with the help of AppMap Navie.

To learn more about getting started with AppMap Navie, head over to getappmap.com

Announcing AppMap for GitHub - Runtime Code Reviews for Every Pull Request

Kevin Gilpin — Thu, 09 Nov 2023 01:17:20 +0000

Most of us have a love-hate relationship with code reviews. We rely on code reviews to ensure that the code we send to production is clean, performs well, and is free of security flaws. But code reviews can also add a lot of overhead and delays to shipping code.

Some people suggest that code reviews should be eliminated altogether. Well, I'm not in that camp, and here's why: Unexpected code behavior and deep runtime defects are responsible for a staggering 40% of performance problems¹ and 50% of security concerns² in software development. But, I do think that it's important to make code reviews easier for developers, and at AppMap we've created AppMap for CI to help do that.

“Security and performance reviews are part of the process on any engineering team, and ours is no different. It can take months to find issues and weeks to fix problems using static analysis and code reads. With AppMap, that time is reduced to minutes.”

-Padraig Coffey, CEO at Zartis.

If you know AppMap already, you know us for our VS Code and JetBrains extensions that automatically generate complete, accurate, interactive diagrams of your code. So now, with AppMap for CI, we've built on the solid foundation of AppMap to make code reviews easier, better, and faster.

The first CI platform we support is GitHub Actions. Our solution combines an AppMap GitHub App with GitHub Actions. When you build your project, the AppMap Analysis Action analyzes AppMaps from before and after each code change. It's able to identify critical performance, security, and stability issues and annotate specific lines of code in the pull request. It also performs deep analysis of test case failures and web service API changes, giving you a big head start on code review. And by the way, the performance and security flaws that AppMap finds are not discoverable by static analysis tools - because modern code is too dynamic for static analysis tools to understand.

“When we initially adopted AppMap, we were only using it for visualizing N+1 queries locally. We later integrated it into our CI run, and we are finding the GitHub comments posted by AppMap to be very useful. The OpenAPI integration, in particular, was a surprise: it’s nice to see an automatically generated summary of what endpoints have changed in a PR."

-Paul Kuruvilla, CTO at CodeCrafters.

Your data stays with you

We know that your code is sensitive, and you don't want it to be transmitted to 3rd parties. All AppMaps are stored as artifacts in your GitHub build system and are sent directly to your browser when you view them. The data only resides in two places - the GitHub Actions environment, and your machine. Learn more at appmap.io/security.

A tour of AppMap's "runtime code review"

✅ AppMap issues a summary report of its runtime analysis as a comment on every PR. The PR comment includes four key categories:

Failed tests

Identifies the root cause of test failures with detailed insights and direct links to the line of code which caused the error. AppMap also includes source diffs and behavioral sequence diffs for efficient debugging.

API Changes

Detects API route differences, including new, deleted, or modified routes, as well as alterations in response attributes like body, content, and descriptions.

Security flaws

Uncovers issues such as broken or missing access control, deprecated cryptography, improper session management, and missing authentication. AppMap focuses on the runtime security flaws that dominate the OWASP Top 10, and can't be reliably detected by static analysis.

Performance problems

Identifies performance problems like N+1 SQL queries, very complex queries, RPC anti-patterns, and plain old slow code.

Example - Performance problems

Here's an example of a code change that introduces four new performance problems.

The AppMap runtime code review includes details about the flaws detected, a stack trace, and links to the offending source code.

AppMap highlights specific code changes that are contributing to the flaw. It's able to focus on the specific code changes that are relevant to the newly introduced problem.

AppMap also links to an interactive "diff" diagram showing how the code change has introduced the flaw.

Example - Security flaw

Here is an example of AppMap describing a security flaw and the code change that produced it.

Included in the comment is a stack trace showing what led to the security flaw, including links to the source code.

AppMap also provides a full set of interactive visualizations that describe the code behavior that resulted in the security flaw.

Wrap-up

To get started, install the AppMap GitHub App from the GitHub Marketplace.

Not ready to try it for yourself? We have example projects on our GitHub so you can check out AppMap for GitHub Actions at work.

Links

🤖 AppMap for GitHub Actions: https://getappmap.com
⬇️ Download AppMap for VSCode and JetBrains: https://appmap.io/download
⭐ Star AppMap on GitHub: https://github.com/getappmap
📹 Follow on: https://youtube.com/@appmap
💬 Join AppMap Slack: https://appmap.io/slack
ℹ️ Read the AppMap docs: https://appmap.io/docs

References

Unlock developer creativity with GitHub Actions

Kevin Gilpin — Tue, 24 Oct 2023 16:27:48 +0000

Since its release in November 2019, GitHub Actions has continued to gain in popularity. Fully 72% of AppMap users tell us that their team is using GitHub Actions! I've been personally using GitHub Actions intensively for about six months, and it feels to me like a substantial leap forward in automation technology. Why? As CI/CD has gained importance, it has increasingly become disconnected from the productivity needs of developers, primarily due to security concerns. I find that GitHub Actions allows me to automate code in innovative ways that are not possible within the confines of traditional CI/CD pipelines.

How GitHub Actions works

In case you're new to GitHub Actions, here's a quick overview:

A GitHub Action is a code snippet that runs on GitHub-hosted infrastructure (or you can provide your own compute).
GitHub Actions can be combined together into sequences called Workflows.
A GitHub action can be defined in a GitHub repository, and imported by other projects. So they are highly sharable and reusable.
A Workflow is a YAML file that lives in the .github folder of the code itself.
Because a workflow is a git-managed file, each version of the code can have its own workflows. That makes it easy to try out new workflows on a branch, and then merge them to main when they are debugged.
There are lots of different ways to trigger workflows. Common triggers include "run on pull request", "run on push", and "run on schedule".

With this combination of features, GitHub Actions gives developers the power to easily create just about any kind of code automation.

How we got here

I'm, um, old enough that when I started my career, manual builds were the norm. Build automation tools like Jenkins came along pretty soon afterwards, and brought repeatability to the build + test + package cycle.

Deployment was still almost always a manual step. The next big step was CI/CD, which moved build automation to cloud-hosted infrastructure and also automated the deploy steps - this, of course, was made possible by cloud computing infrastructure: Amazon EC2 and all the services that derived from it.

Trouble in paradise

CI/CD became the core of DevOps, which expanded automation into many new areas. Most of this growth and expansion has been very positive, but the rigor of the automation has actually had some drawbacks for developers. Why? Because CI/CD has such so much power to package and deploy code, it's also become an area of high security risk.

Hackers quickly identified and exploited multiple avenues of attack:

Inject malicious code directly into the target software as it's being built.
Use a compromised CI job is a stepping-stone for access to production systems.
Use inside knowledge of the code to build the perfect exploit or backdoor.
Attack the CI/CD vendor directly to obtain credentials in bulk. (Four well-known vendors in the CI/CD space have all been breached in the last decade).

Security is a real concern with CI/CD systems. As a result, they are now highly protected. This is necessary - but it's also meant that CI/CD is not a developer-friendly environment anymore. CI/CD is too tightly controlled to be a good place for developers to try out new experiments in code automation.

GitHub Actions to the rescue

This is where GitHub Actions comes in. A GitHub Action does not need to be part of the "blessed" CI/CD process. It can perform any kind of ad-hoc job that a developer might want it to. For example, there are GitHub Actions to report test status, inject comments into commits and PRs, synchronize with other systems, generate code, send notifications, look for bugs, etc. And it can do these things "off to the side", out of the way of the official pipeline job. It's no longer necessary to modify the master build file (think: travis.yml, Jenkinsfile, .circleci) in order to try out some new code automation.

Since Actions are controlled by YAML files that any developer can create and modify - and they run in the cloud with direct and easy access to project source code - GitHub Actions combine power and safety in a way that's not found either in local developer environments or in traditional CI/CD tools.

Powerful, tune-able performance

As a developer, you have a good understanding of performance considerations such as CPU capabilities, memory use, and parallelization. So, you'll be glad to know that GitHub Actions is powerful enough to let you control and optimize exactly how your workflow uses and consumes compute and storage.

Each Workflow runs on a pre-defined runner instance, which is basically a container running with a pre-defined allocation of CPU cores and amount of memory. So you can run small, lightweight jobs with little cost.
Is your Workflow slow and sluggish? You've got options; several options in fact. First, you can bump the workflow up to a larger runner instance. How does 64 cores and 256GB of RAM sound? You can get that. What if the code you are running doesn't scale that well with cores? Well, you can apply matrix build strategies as well.
You can store and retrieve data within and across workflows. Both a cache store and artifact store are available. The cache is really fast, and it can only be accessed from within Workflows. Artifacts are slower, but they can be accessed from anywhere using the GitHub REST API. Cache expiration times and artifact retention times are both customizable, so you can tune the amount of money you are spending on these storage resources.

Proceed! (With caution)

While it is separate from the official CI/CD pipeline, the privileged operation of GitHub Actions does still pose a security risk. GitHub Actions have access to the code; can commit and push code; can be given access to secrets that may be mishandled. Recognizing this, GitHub provides repository and organization settings that can be used to control which GitHub Actions are allowed to run, and even whether the feature is enabled at all.

Furthermore, each GitHub Action Workflow runs with its own explicit permissions. So, the security capabilities of each Workflow are clearly stated, and less-trusted Actions can be run with lower privilege settings than more-trusted Actions. This type of "least-privilege" access management is a highly effective security strategy, and it's not possible in monolithic and/or legacy CI/CD jobs.

As always, freedom comes with responsibility and we've heard of at least one organization that has locked its developers out of GitHub Actions after incidents of careless, insecure coding. So, be aware and mindful of the security controls that are available, and use them!

How we're using GitHub Actions and AppMap together to improve code quality

AppMap is a tool that records code traces into JSON data files, which can be visualized and analyzed to improve code understanding and quality. GitHub Actions offers a way to create a comprehensive AppMap data set for every pull request, providing a complete profile of the runtime architecture, performance, and security characteristics of your application. It also offers a way to compare the AppMap data sets of two different code versions.

Comparing AppMaps of two code versions (a baseline and a proposed change set) is an amazingly powerful technique for analyzing code for architecture changes, security flaws and performance problems. With AppMap for GitHub Actions, we are offering exactly this capability. Our Early Access program is wrapping up now, and we are transitioning to general availability. Come check it out!

Because AppMap runs in your GitHub account, the data is processed within your account and it stays within the boundaries of it. You don't have to share source code or sensitive data with any third party. This is the type of win/win/win capability that GitHub Actions offers. A high level of developer control, good performance, and good security characteristics.

How are you using GitHub Actions?

Has your organization adopted GitHub Actions? How enabled are developers to add and change GitHub Actions? Did you encounter any problems or mishaps along the way that others could benefit from hearing about? We would love to hear about it in the comments below.

Links

🤖 Get Early Access to AppMap for GitHub Actions: https://getappmap.com
⬇️ Download AppMap for VSCode and JetBrains: https://appmap.io/download
⭐ Star AppMap on GitHub: https://github.com/getappmap
📹 Follow on: https://youtube.com/@appmap
💬 Join AppMap Slack: https://appmap.io/slack
ℹ️ Read the AppMap docs: https://appmap.io/docs

How do you say... [insert tech lingo here]: A video series.

Pete Cheslock — Tue, 26 Sep 2023 19:24:44 +0000

I've been fascinated by how the industry has come to pronounce different tech words, lingo, products, and acronyms. Months ago, I asked my twitter followers for an idea of some good words to have people pronounce in a "game show" style lightning round.

// Detect dark theme var iframe = document.getElementById('tweet-1635342410053201921-731'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1635342410053201921&theme=dark" }

After 31(!!) individual recordings this summer, we shipped the series with friends from all over the tech industry in all kinds of roles.

We threw a slew of tech names and acronyms at them to see how they pronounced them because, hey, most of these words have been made up in the last decade or less to describe a new technology or name for something related to ye'ole computer.

The list:
/etc
AMI
k8s
chmod
curl
fsck
HAProxy
/proc/
NGINX
SQL
systemctl
SCSI
www
sudo
URL
/lib/
CIDR
epoch
jwt
strace

Wanna check out the supercuts? You can find them on YouTube Shorts or TikTok

Head over to this AppMap Youtube Playlist to check out all the full length "game show" style lightning rounds.

The best part is the conversations between humans - the relationships with folks from all walks of tech life - and the joy you get from watching the laughter or disgust ensue.

6 super nerdy podcasts & channels

Jen Wike Huger — Wed, 09 Aug 2023 02:29:34 +0000

Last week, the AppMap team shared their 8 favorite tech and software podcasts.

We're developers, so the media we consume covers the complexity of the software industry... and a ton of NERDY stuff, too! Like Linux, web dev, and the 2000s.

Let's get into it.

[Out now! Early access to AppMap for GitHub Actions]

Shop Talk Show

Dave Rupert and Chris Coyier produce a weekly podcast about building websites - front end web design, development, and UX. They have guests and answer listener submitted questions. This one features a banjo! and thoughts on long-dead Google Reader.

GIANT ROBOTS

... SMASHING INTO OTHER GIANT ROBOTS. Rotating between three hosts, this podcast is "about the design, development, and business of great software." Recently they interviewed Brittany Martin, a co-host of The Ruby on Rails Podcast, a podcast featured in our last post. Oh, and here's my friend Lauren Maffeo, the author of Designing Data Governance from the Ground Up.

Cathode Ray Dude

The host of this YouTube channel quit his day job a month ago and is now much happier spending his energy on his video channel (not really a podcast) about "cameras, computer mice, and the 2000s."

PROOF

This podcast is all about in-depth NFT coverage. Venture capitalist Kevin Rose and team "interview NFT artists, both up-and-coming and industry icons. [They] also cover the generative art scene, the NFT gaming/metaverse, and founders building new tools for creators and collectors." I like cool domains so will be checking out this one with David Greenstein, co-founder of sound.xyz.

The Linux Experiment

The host, Nick, treats this as a safe space with "no techno babble, no super technical content" that focuses on "Linux desktop news, simple tutorials, application spotlights, and opinion pieces trying to stay positive, without gatekeeping." It's no surprise that his step-by-step walkthrough of how to switch to Linux has a gazillion views.

The a16z Podcast

The team at Andreessen Horowitz, a silicon valley venture capital firm, interviews "industry experts, business leaders, and other interesting thinkers and voices from around the world." Interestingly, they are looking for the next host of the podcast!

Got any super nerdy podcasts or video channels to share with us? We need more, clearly.

8 great software & tech podcasts 2023

Jen Wike Huger — Thu, 03 Aug 2023 20:40:40 +0000

Podcasts come and go. Time and energy to listen to them also. And what a wonderful innovation they are, especially when you are learning or working in a field like software and tech.

Millions of developers means the complexity of the industry, business, culture, news, and let's not forget... the technology itself, is seemingly endless.

All you must ask is "What do I need to learn today?" and the answer will reveal itself quickly with a few keystrokes. Yet, with countless options and podcasters talking about their perspective on everything from AI to apple pie, how will you narrow it down?

To help with that, today we're sharing the AppMap team's favorite podcasts. Enjoy, and let us know what you're listening to!

[Out now! Early access to AppMap for GitHub Actions]

Software & tech podcasts

Software Engineering Radio
The self-proclaimed "authority on translating software theory into practice."

Arrested DevOps
Want to maximize your DevOps-ing? Check it out. One of our team has been a guest here, too!

Data Engineering Podcast
Get the goods on data management with guest engineers and entrepreneurs who are shaping the industry.

Language-specific podcasts

airhacks.fm podcast with Adam Bien
Stellar discussions on Java, serverless, cloud, architecture, and web!

The Ruby on Rails Podcast
Still going strong, this podcast is a weekly conversation about Ruby on Rails, open source software, and the programming profession.

The Python Podcast.init
Another one where we've been a guest, this is a podcast about Python and the people who make it great.

Deeper stuff

Darknet Diaries
The description says it all: "Explore true stories of the dark side of the Internet with host Jack Rhysider as he takes you on a journey through the chilling world of hacking, data breaches, and cyber crime."

The Lex Fridman Podcast
One for that higher level, about the nature of intelligence, consciousness, love, and power. Oooo!

What would you add to this list? Any favorites to share with us? We'd love to hear about them and give them a listen!

Improve application performance: How to find and fix slow function calls, HTTP requests, and SQL queries

Garrett Hamelin — Fri, 21 Jul 2023 16:53:43 +0000

Runtime analysis is the latest and greatest way for developers to find, flag, and fix software flaws before production. Our series on rules and their impact is focused in this article on Slow Function Calls, HTTP Requests, and SQL Queries.

I will review what these rules mean and how to enhance your application performance using AppMap.

If you learn better through video you can check out a walkthrough of these rules here:

Slow function calls

In this first example, let’s use AppMap to locate a performance finding related to slow function calls and use the new flame graph view to see the duration of each function call. It's crucial to optimize those long-running functions, as they can significantly impact performance. Often slow function calls are caused by poor code practices, inadequate algorithms, or excessive resource consumption, leading to scalability issues, bottlenecks in system responsiveness, and even user frustration

To illustrate this, let's examine code snippets related to slow renders and highlight the usage of partials. Repeated rendering of partials within loops can significantly impact performance.

For some context, the example we are dissecting today is an e-commerce merch store. To make sure we are all on the same page, the AppMap we are looking at outlines the user interaction of a user selecting featured products to view. The function in question ActionController::Renderers#render_to_body can be plainly seen in the flame graph.

When we dig a little deeper and see what is actually happening.

We can see that Render_to_body calls render_to_body_with_renderer which gets passed a series of options. Looking into the function itself we see the _renderers.each do ...which, when passed a file with a lot of partials that need to be rendered, some containing caching operations themselves, we start to understand exactly why this function runs for such a long time.

Looking at the trace view, we can see just how many children are spawned and will complete before we can return from the original calling function.

To mitigate this we may want to reduce the number of partials that we are individually working with. Good coding practices encourage abstraction and code reuse but doing so may have introduced a performance concern. We need to be careful that we don’t abstract which can impact the performance of our application at runtime.

Slow HTTP requests and SQL queries

Slow HTTP requests and SQL queries often stem from a range of situations, ranging from suboptimal coding practices, inefficient schemas and configuration, or lack of indexing to lack of resources available, and high network congestion. We need to be aware of potential causes, such as the lack of caching HTTP requests or leveraging a content delivery network (CDN) for faster content delivery.

Find and fix performance issues

To address slow function calls, HTTP requests, and SQL queries, we can implement effective mitigation strategies by updating our coding practices and improving resource management. For slow renders, alternative home or index views can be used to minimize the number of function calls. Additionally, you can leverage caching techniques and pre-render pages that don't require dynamic data, as well as load-balancing techniques to reduce the pressure on our network.

In summary

To improve application performance, it’s important to identify and optimize slow function calls, HTTP requests, and SQL queries.

AppMap’s flame graphs and other tools like the interactive sequence diagram make it easy to pinpoint performance bottlenecks and implement appropriate mitigation strategies.

Links

⬇️ Download AppMap for VSCode and JetBrains: https://appmap.io/download

⭐ Star AppMap on GitHub: https://github.com/getappmap

🐦 Follow on Twitter: https://twitter.com/getappmap

💬 Join AppMap Slack: https://appmap.io/slack

ℹ️ Read the AppMap docs: https://appmap.io/docs

📺 Watch AppMap Tutorials: https://www.youtube.com/@appmap

📸 Cover Photo by Robert Linder on Unsplash

Garrett HamelinFollow

I love technology and everything about it. I'm a software engineer who enjoys helping others succeed and building communities of like minded people.

Introducing Flame graphs: It’s getting hot in here

Garrett Hamelin — Fri, 14 Jul 2023 18:47:50 +0000

Flame graphs help developers identify code bottlenecks and understand code execution patterns, so we’re excited to announce they are now available within the AppMap extension for VS Code and JetBrains editors like IntelliJ.

“Flame graphs are a visualization of hierarchical data, created to visualize stack traces of profiled software so that the most frequent code-paths to be identified quickly and accurately.”

In this article, I explore how they work, and how to generate and use them with AppMap.

How to read a flame graph

Flame graphs are read from bottom to top, left to right, providing a holistic view of function execution. In this video, you can see how I identify time-consuming functions and visualize the sequence of operations.

To make sense of the flame graphs, we must first understand the color scheme.

Purple represents SQL queries
Blue indicates classes, methods, or functions that follow in the execution flow
Yellow denotes external service calls (absent in this video)
Teal is the name of the AppMap

What you get with flame graphs

Flame Graphs reveal crucial insights about application performance. Each function in the graph is labeled with its corresponding execution time in milliseconds or microseconds. This detailed breakdown allows developers to pinpoint long-running queries and functions, optimizing code efficiency.

Zooming in on the graph or selecting a specific function provides a deeper understanding of the call stack and execution sequence. For example, the demonstration showcased the occurrence of a selector before an active support call, shedding light on the execution flow. This level of granularity empowers developers to tackle complex performance issues head-on.

Revealing performance issues

By examining the graph, developers can detect common issues like N+1 queries, a notorious problem in application development. The ability to visualize the duration of each query or function provides actionable insights, highlighting areas for improvement.

Flame graphs also expose the impact of long-running functions on overall performance. While some functions, like renders to body, are intended to run longer, others such as support services and callbacks, should be optimized for efficiency. The graph's tally of execution times offers a comprehensive overview, ensuring developers can focus their efforts where they matter most.

AppMap improves on flame graphs by highlighting the flaws in your application and shows you the direct impact of a specific flaw on your application. No more searching to figure out where an N+1 query happened or how many times it ran!

Flame graphs, sequence diagrams, and more

Try AppMap today to help you identify issues with your code or refactor an existing code base. Use our new flame graph and sequence diagram views. We also have traditional dependency and trace views.

Links

⬇️ Download AppMap for VSCode and JetBrains: https://appmap.io/download

⭐ Star AppMap on GitHub: https://github.com/getappmap

🐦 Follow on Twitter: https://twitter.com/getappmap

💬 Join AppMap Slack: https://appmap.io/slack

ℹ️ Read the AppMap docs: https://appmap.io/docs

📺 Watch AppMap Tutorials: https://www.youtube.com/@appmap

Garrett HamelinFollow

I love technology and everything about it. I'm a software engineer who enjoys helping others succeed and building communities of like minded people.

How do you say... "/etc"?

Pete Cheslock — Tue, 11 Jul 2023 17:38:22 +0000

How would you pronounce this file system directory name commonly found on linux and unix-like systems?

Contains system-wide configuration files and system databases; the name stands for et cetera[14] but now a better expansion is editable-text-configurations. Originally also contained "dangerous maintenance utilities" such as init,[6] but these have typically been moved to /sbin or elsewhere. Needs to be on the root filesystem itself.

/etc stands for Et Cetera which is a Latin expression used in English to mean and other (similar) things, or and so forth.

Translated literally from Latin, et means 'and', while cētera means 'the rest'; thus the expression translates to 'and the rest (of such things)'.

But if you had to tell another person to put a file in the /etc directory, how would you say it?

Listen to a few examples from technology professionals on the latest episode of "How Do You Say?"

Want to see more tech word pronunciations by tech professionals? Find them on Youtube. New videos ship every Friday.

Exploring" Too Many Updates": How to find and fix a code issue for a large number of outward calls

Garrett Hamelin — Fri, 07 Jul 2023 18:59:37 +0000

Whether you're a software engineer looking to optimize your code or just eager to understand runtime analysis, this article will help you think about rules as they relate to code analysis.

In this article, I explore a maintenance rule called "Too Many Updates" and discuss its implications, causes, and mitigation strategies.

Understanding the rule

At AppMap, we define the "Too Many Updates" rule as a verification process that examines the number of SQL and RPC updates executed by a command function or method within your application. This rule sets a default threshold of five updates in our default.yml file. Any function or method that exceeds this threshold triggers the "Too Many Updates" finding to appear in the runtime analysis results.

In technical terms, this finding corresponds to CWE-1048, which refers to an “invokable control element with a large number of outward calls” [sic]. For simplicity, we'll refer to it as "Too Many Updates" throughout this article.

Applying the rule

To illustrate the concept, let's consider a real-world example using a Spring Pet Clinic application implemented with Java and the Spring framework. This application allows us to manage owners, pets, and visits for a veterinary clinic. Imagine we have created an owner, and a pet, a bird we have given the name of Jimmy, and added a recent visit to the veterinarian. Now, let's take a closer look at the code and explore the consequences of the "Too Many Updates" finding.

Setting custom overrides

Before diving into the code, let's first understand how to customize the default threshold for the "Too Many Updates" rule. By creating an appmap-scanner.yml file, and running the command, you can override the default settings.

$ npx @appland/scanner \
    --appmap-dir tmp/appmap \
    --config appmap-scanner.yml \
    ci

In the example provided, the warning limit is set to one for demonstration purposes.

checks:
  - rule: authzBeforeAuthn
  - rule: http500
  - rule: illegalPackageDependency
    properties:
      callerPackages:
        - equal: actionpack
      calleePackage:
        equal: app/controllers
  - rule: insecureCompare
  - rule: missingAuthentication
  - rule: missingContentType
  - rule: nPlusOneQuery
  - rule: secretInLog
  - rule: slowFunctionCall
    properties:
      timeAllowed: 0.2
      functions:
        - match: Controller#create$
  - rule: slowHttpServerRequest
    properties:
      timeAllowed: 0.5
  - rule: slowQuery
    properties:
      timeAllowed: 0.05
  - rule: tooManyJoins
  - rule: tooManyUpdates
        properties:
            warningLimit: 1
  - rule: unbatchedMaterializedQuery
  - rule: updateInGetRequest

However, in practical scenarios, it's recommended to set a more appropriate threshold based on your application's requirements. Now that we can control our threshold let's jump over and view our findings.

Analyzing findings

To analyze the findings, we'll use AppMap to navigate to the runtime analysis section. The findings table provides a comprehensive overview of all the identified issues.

You can also group the findings by project, date, or category to gain better visibility into your application's maintainability.

Examining Example 1

Open the findings report for the "Too Many Updates" finding to understand its causes and implications. In this example, the command performs 47 SQL or RPC updates, as indicated in the event summary.

When we examine the map, we can observe a sequence diagram representing the execution flow. The presence of this finding suggests potential poor coding practices or architectural flaws in the application.

It highlights the need to consolidate multiple calls into a more efficient and maintainable structure. In this particular application, this is the result of database migrations being performed during the setup phase for development.

Examining Example 2

Now, let's explore another example to compare and contrast the findings. In this case, the threshold is set to two, and we examine the sequence diagram for the corresponding map finding an action that performs to “updates” in the same function call.

This specific scenario involves adding a pet to an owner. We can observe that two updates are performed: one to INSERT the pet's information…

INSERT INTO pets (id, birth_date, name, type_id)`

…and another to associate the pet with its owner.

UPDATE pets SET owner owner_id=? WHERE id=?

Unlike the previous example, this finding emphasizes a design choice rather than inherent flaws in the application's architecture. It prompts us to evaluate if these two calls can be consolidated into a single call for improved efficiency and simplicity.

Selecting a strategy

To address the "Too Many Updates" finding, we have several mitigation strategies at our disposal.

The first and simplest approach is refactoring the code to consolidate multiple updates into a single call. Often we find that “Too Many Updates” isn’t the result of some large flaw in the design of our application, but rather the result of poor implementation of coding standards and common best practices.

By reviewing the code and analyzing the sequence diagrams, we can identify opportunities to optimize the number of updates performed. Additionally, if we do find a larger issue at play we can consider architectural changes. Such as reevaluating the structure of services and abstractions, to ensure a more streamlined approach.

Alternatively, if these issues uncover a security risk in our application, we can employ the use of rate limiting to ensure our application remains performant and secure. These mitigation strategies enhance maintainability, improve code readability, and reduce potential security risks.

Watch the video:

Summary

In this article, we explored the "Too Many Updates" maintenance rule and its impact on software engineering practices.

By understanding the causes, implications, and mitigation strategies, you can effectively optimize your code and ensure a more efficient and maintainable application.

Remember to review your findings, analyze sequence diagrams, and implement the appropriate refactoring techniques and architectural changes. By doing so, you'll enhance the overall quality and performance of your software.

Links
📸 Cover Photo by Karl Pawlowicz on Unsplash

⬇️ Download AppMap for VSCode and JetBrains: https://appmap.io/download

⭐ Star AppMap on GitHub: https://github.com/getappmap

🐦 Follow on Twitter: https://twitter.com/getappmap

💬 Join AppMap Slack: https://appmap.io/slack

ℹ️ Read the AppMap docs: https://appmap.io/docs

📺 Watch AppMap Tutorials: https://www.youtube.com/@appmap

Garrett HamelinFollow

I love technology and everything about it. I'm a software engineer who enjoys helping others succeed and building communities of like minded people.

How to streamline and focus your sequence diagrams

Garrett Hamelin — Wed, 28 Jun 2023 19:38:01 +0000

AppMap’s new feature gives developers greater control over their sequence diagrams to enhance code reviews.

The value of sequence diagrams, especially for code reviews, is growing. And AppMap is at the cutting edge, creating new tools and capabilities for developers to generate and use sequence diagrams for designing and improving code quality from writing to reviewing. In this article, I’ll share a new filtering feature of our sequence diagram tool in AppMap.

Taming the complexity of sequence diagrams

As software projects grow in size and complexity, so do the sequence diagrams generated from them that represent their inner workings. These diagrams often become extensive and overwhelming, making it challenging to decipher the flow of interactions and identify critical components. The AppMap user community has requested a way to declutter their automatically-generated sequence diagrams in order to focus on the essential elements of their application.

We heard you, and we’re excited to share this new feature release. Read on for the details, and watch this walkthrough video to see it in action.

Advanced filtering for unparalleled control

With our latest release, we added an enhanced filtering system that empowers developers to take control of their sequence diagrams by eliminating unnecessary noise and tailoring your diagrams to display only the information that matters most to you and your team.

3 key functionalities

Live filtering: Instantly reduce complexity by hiding specific components, such as external code or framework-related elements. With just a few clicks, you can create a clean, focused view of your sequence diagram.
Customizable views: Once you have refined your diagram to your liking, you can save that view as a filter. This means you can apply the filter to future app maps, whether you want it as a default setting for all diagrams or selectively choose where to apply it.
Seamless switching: Switching between different views is effortless. Just select the desired filter, and the diagram will dynamically update to display your preferred elements. No need to rebuild or navigate complex menus.

Benefits and advantages

AppMap's enhanced filtering feature goes beyond simplifying your sequence diagrams.

Persistence: Your custom filter views persist as long as the diagram remains open. This allows you to revisit and analyze your diagrams without losing your customized settings. If a diagram has been closed, getting that view back is as simple as reapplying the filter. If you find yourself applying the same filter to a majority of your sequence diagrams, set a previously saved filter as the default, providing a seamless experience across any sequence diagram you choose to open or share. Changing or setting a default view becomes as simple as clicking a button.
Intuitive interface: Our user-friendly interface ensures that applying filters and customizing your view is a breeze. With our configure-as-you-go model, developers spend less time configuring and more time gaining insights from their diagrams.
Increased efficiency: By decluttering your sequence diagrams and focusing on the relevant elements, you can make quick decisions on things that matter most, improving your code analysis efficiency and gaining a deeper understanding of your application's behavior on whatever scale you choose to view.

Getting Started + community

Sequence diagrams are particularly useful for designing and testing software systems that involve multiple components, asynchronous events, or complex control flows. Learn more about the common use cases for sequence diagrams and how to interpret them in this post.

To try out our new filtering feature, download AppMap into your favorite IDE and use our handy docs to get started. We hope you enjoy exploring the possibilities, experimenting with different views, and revolutionizing how you analyze and understand your code!

To chat with us and other users about it and get your questions answered, join our community.

Cover Photo by Luca Bravo on Unsplash

Garrett HamelinFollow

I love technology and everything about it. I'm a software engineer who enjoys helping others succeed and building communities of like minded people.