Automating CodeCommit PR Reviews with Amazon Bedrock: 986 Reviews in 10 Weeks, $0.18 Each, and What We Could Not Measure

TL;DR

• A team of 5 ran 986 reviews over about 10 weeks on a single CDK infrastructure repository. Each review took about 60 seconds on average, and the actual Bedrock bill totaled $180.60 (about $0.18 per review)
• Review count, cost, and duration are now visible as numbers, but we still can't measure whether the findings were correct, or how lead time changed before and after adoption

Table of Contents

• Introduction
• Architecture Overview
• How the PR Review Works
• Measuring the Effect
• Visualization
• Thinking About Cost
• Results in Production
• Design Decisions
• Limitations and Future Improvements
• Conclusion

Introduction

CodeCommit is a very convenient service that lets you keep your whole development environment inside AWS. However, compared with GitHub, its surrounding ecosystem is limited. An OSS tool such as Qodo AI's PR-Agent has a provider for CodeCommit, but its features are limited and it assumes you use it from the CLI.

So I built a mechanism that reviews CodeCommit pull requests (PRs) with Amazon Bedrock and posts the result as comments. I did not use the OSS PR-Agent. Instead, I built it from scratch with EventBridge and Lambda so that it runs automatically whenever a PR is created or updated. Many of the articles I referred to ended at "post the comment", but once you think about operating it, you also need to make the following visible:

• How much does it cost per month?
• Whose PRs are they, and how many tokens does each model use?
• Are findings actually being raised, and if so, which severity is most common?
• Has the time until a PR is merged changed?

This article also explains how to collect and report these numbers.

Architecture Overview

The overall architecture looks like this.

Full architecture diagram (including CDK test CI, usage dashboard, and Slack/Teams notifications)

This is the review flow after a PR is made.

EventBridge receives PR creation and update events, and a Lambda function (Python) builds the diff and passes it to Bedrock. The result is posted to CodeCommit as a PR comment. The posted comments look like the following (only the beginning is shown). Code and documents are posted as separate comments.

Code review

Doc review

The system keeps three kinds of data needed for visualization.

Layer	What you learn	Source data	Storage
① Review content	Findings (by severity), LGTM, agreement between models, duration, errors	Bedrock structured output	CloudWatch custom metrics
② Cost and usage	Token counts and estimated cost by repository, PR author, and model	Bedrock invocation logs	CloudWatch Logs
③ Development flow	Time from PR creation to merge	PR status change events	JSON in S3

How the PR Review Works

Building the diff from blobs

CodeCommit's GetDifferences returns the list of changed files and their blob IDs, but not the diff text. So I fetch the before and after blobs with GetBlob and assemble a unified diff with Python's difflib.

diff_lines = list(difflib.unified_diff(
    before_content.splitlines(),
    after_content.splitlines(),
    fromfile=f"a/{path}",
    tofile=f"b/{path}",
    n=5,          # lines of context around each change
    lineterm="",
))

Here is how each change type is handled. Only the before/after diff is passed to the model, so unchanged lines consume no tokens.

Change type	What is passed to the model
Modified (M)	Unified diff (with 5 lines of context before and after)
Added (A)	The whole file
Deleted (D)	Path only (noted as "deleted")
Binary / empty	No body (noted as such)

I also set an upper limit on the number of diff lines handled in one review. When the limit is reached, the rest is cut off per file, and the PR comment states that it was truncated.

Routing by path

Changed files are classified by path into code / document / skip. Code and documents are reviewed separately, and paths that match skip are excluded from the review. The patterns are set with the Lambda environment variables DOCUMENT_PATH_PATTERNS and SKIP_PATH_PATTERNS.

Skipping a review

I also added a feature to skip the review of a whole PR when the title contains [skip-review]. Excluding by path is also possible, but writing it in the title skips the review immediately. In this repository, documentation is generated as HTML with MkDocs, so regenerating it doesn't trigger a review of a huge number of files.

Combining multiple models with Agents-as-Tools

You can specify multiple models through parameters. When you specify multiple models for review, you can also specify a separate model that summarizes their results. For example, you can use two models, Nova Pro and Claude Sonnet 4.6, for review, and Claude Haiku 4.5 to summarize the results.

In the per-environment parameter file, set reviewMode to multi and write the model IDs in multiModelConfig. To run with a single model, use reviewMode: "single" and singleModelConfig: { modelId }.

reviewMode: "multi",
multiModelConfig: {
  modelIds: ["apac.amazon.nova-pro-v1:0", "jp.anthropic.claude-sonnet-4-6"],  // reviewers
  summaryModelId: "jp.anthropic.claude-haiku-4-5-20251001-v1:0",              // merges the results
},

The reviewer models are exposed as @tools and called by the summary agent. At first, I passed the diff to each reviewer tool as an argument. However, since the one calling the tool is the summary agent (an LLM), it had to generate the diff character by character as an argument. The diff is large, so this generation took a long time and caused timeouts. Now the tools take no arguments. The prompt containing the diff is held in a variable outside the tool, and the tool references it directly. Because the diff no longer passes through the LLM, this problem doesn't occur.

def _make_reviewer(mid: str):
    @tool
    def reviewer_tool() -> str:  # no arguments: the diff never passes through the LLM
        if mid not in reviewer_results:  # each reviewer runs only once
            sub_agent = Agent(model=_build_bedrock_model(mid, repository_name, pr_author))
            reviewer_results[mid] = str(sub_agent(prompt_text))  # prompt_text holds the diff
        return reviewer_results[mid]
    return reviewer_tool

Comment limit: 10,240 bytes

CodeCommit comments have a limit of 10,240 bytes. Code reviews and document reviews are posted as separate comments, and if a comment still exceeds the limit, the end is truncated with a note that content was omitted.

Avoiding duplicate posts for the same commit

An HTML comment containing the commit ID is appended to the end of each posted comment. Before reviewing, the Lambda reads the existing comments and skips the review if it finds the same marker and commit ID.

suffix = f"\n\n<!-- commit:{commit_id} -->"  # appended to every review comment

# before reviewing: look through the existing comments on the PR
if DUPLICATE_MARKER in content and source_commit_id in content:
    return True  # already reviewed this commit, skip

Measuring the Effect

This time, I designed and implemented the system with the goal not only of building the review mechanism but also of reporting the adoption effect inside the company. So I decided which numbers to collect up front, in parallel with implementing the review.

That said, not everything was in place from the start. I added the metrics I needed little by little while operating the system.

① Receive review content as structured data, not free text

I make the summary agent always call a tool named report_review_result at the end.

@tool
def report_review_result(
    summary_markdown: str,   # the PR comment body
    findings: list[dict],    # [{"severity": "critical|warning|info", "count": 2, "detected_by": ["nova", "sonnet"]}]
    lgtm: bool,
) -> str:
    """Report the final review result."""

You could extract severity from the comment's Markdown with a regular expression, but even a small change in the body's format would break it. If you receive it as tool arguments, the number of findings and LGTM can be used directly as numbers.

The received values are sent to the PRAgent/Review namespace with PutMetricData.

put_metric("FindingCount", count, dimensions={
    "Repository": repo, "ReviewGroup": group,
    "Severity": severity, "Environment": env,
})

These are the main metrics sent.

Metric	Description
FindingCount	Number of findings (by repository, review type, severity, and environment)
ReviewCount / ReviewLatencyMs	Number of reviews and duration
ReviewWithinSLACount / ReviewExceededSLACount	Whether the review finished within the target time
InputTokenCount / OutputTokenCount	Token counts
MultiModelAgreementCount / MultiModelUniqueCount	Findings detected by 2 or more models / findings detected by only 1 model
ToolUseFallbackCount / KbFallbackCount / ErrorCount	Fallbacks and errors
ChangedFileCount / ChunkCount / SkipCount	PR size, number of chunks, and number of skips

Sending can be switched on and off with the deploy parameter ENABLE_METRICS. If sending fails, the review itself does not fail; the error is only logged.

Agreement between models

With two models, you can tell whether both raised the same finding or only one did. A finding with two or more entries in detected_by counts as agreement, and one with a single entry counts as uniquely detected. A high agreement rate is a reason to cut down to one model and lower the cost, and a low rate suggests that using two is worthwhile. Note, however, that "findings both models agree on are correct" is not guaranteed (see the limitations below).

② Cost and usage: add metadata to the invocation log

If you enable Bedrock's model invocation logging, the token count of each call is stored in the log. However, as it is, you can't tell which repository and whose PR a call belongs to.

Bedrock's Converse has requestMetadata, and the values you put there are written to the invocation log. With Strands' BedrockModel, you can pass it through additional_args.

BedrockModel(
    model_id=model_id,
    additional_args={"requestMetadata": {
        "repository": repo, "prAuthor": author, "environment": env,
    }},
)

The reporting Lambda aggregates this log with Logs Insights.

fields @timestamp
| filter ispresent(requestMetadata.repository)
| stats sum(input.inputTokenCount) as inTok, sum(output.outputTokenCount) as outTok
  by requestMetadata.repository, modelId

There is an upper limit on the number of fields you can specify in stats ... by in Logs Insights, so "repository × model" and "PR author × repository" are split into separate queries.

Cost is estimated by multiplying the aggregated token counts by the configured unit prices. The unit prices are given in modelPricing in the parameter file (USD per 1,000 tokens for input and output). This is only an estimate, so check the exact amount in Cost Explorer or similar.

③ Record the time until merge

A separate Lambda receives the PR status change event (pullRequestStatusChanged) and records the lead time.

Note that this event alone cannot tell you whether the PR was merged or closed without merging. So the Lambda calls GetPullRequest and checks pullRequestTargets[].mergeMetadata.isMerged. PRs that were closed without being merged are not recorded.

Records are saved to S3 as JSON, partitioned by date.

reports/pr_lead_time/YYYY/MM/DD/<repository>_pr-<id>.json

{
  "recordType": "pr_merge_lead_time",
  "repositoryName": "sample-repo",
  "pullRequestId": "123",
  "author": "alice",
  "mergeOption": "SQUASH_MERGE",
  "createdAt": "2026-01-01T00:00:00Z",
  "mergedAt": "2026-01-01T03:30:00Z",
  "leadTimeSeconds": 12600,
  "leadTimeHours": 3.5
}

Visualization

The collected numbers can be viewed in the following three ways.

CloudWatch dashboard

The metrics from ① are shown as graphs such as the number of reviews per repository, findings by severity, agreement and unique detections between models, and token usage per model. Repositories come and go, so the widgets are built with SEARCH expressions and no repository name is hard-coded.

CloudWatch dashboard (excerpt showing findings by severity and token usage per model)

Alarms are set on review errors, SLA breaches, messages in the DLQ, and so on.

Daily, weekly, and monthly reports

A report-generating Lambda is started on a schedule, runs Logs Insights, and saves JSON like the following to S3.

reports/daily/YYYY/MM/DD/report.json
reports/weekly/YYYY/Www/report.json
reports/bedrock/...        # Bedrock usage by model and PR author
reports/monthly/...

The content is sent as a notification through SNS.

Monthly report

Daily report

{
  "reportType": "daily",
  "environment": "dev",
  "periodStart": "2026-09-29T00:00:00Z",
  "periodEnd": "2026-09-30T00:00:00Z",
  "generatedAt": "2026-09-30T00:05:40Z",
  "summary": {
    "reviewCount": 40,
    "lgtmCount": 0,
    "totalInputTokens": 1073657,
    "totalOutputTokens": 167242,
    "avgLatencySeconds": 72.403,
    "totalReviewedFiles": 188,
    "totalReviewedLines": 12995
  },
  :
}

Usage dashboard (optional)

For those who want to see the JSON in S3 on a web page, I also built a static site on CloudFront + S3, and an API Gateway (HTTP API) protected by Cognito JWT authentication. The API reads the reports in S3 through a Lambda function and shows them on two screens: an overall view and a per-PR-author view.

• Authentication uses Authorization Code + PKCE, and self sign-up is disabled
• The static site is locked down in S3 with OAC and is reachable only through CloudFront

This can be enabled as an option.

Usage dashboard: overall (top part excerpted)

Usage dashboard: individual (top part excerpted; review lead time not yet supported)

Thinking About Cost

If you have a few hundred PRs a month, the main cost is Bedrock token charges. The rough formula is as follows.

Monthly cost ≈ number of PRs × reviews per PR × (input tokens × input price + output tokens × output price) × number of models

The extra cost of collecting metrics is determined by the number of custom metrics (one per combination of dimensions) and the number of PutMetricData calls. The count grows with the combinations of repository, severity, and environment, so check it if you have many repositories. Check the latest CloudWatch pricing page for unit prices.

Results in Production

These are the results of a development team (5 people) running the system for about 10 weeks on a single CDK infrastructure repository. The numbers come from the report JSON in S3 and the billed amounts in Cost Explorer. This environment also includes a few reviews of other repositories, but most are for the repository under evaluation. The setup used a single model (reviewMode: "single"), Claude Sonnet 4.6 (cross-region inference). This is not an operating record of multiple models.

Month	Reviews	Findings (Critical / Warning / Info)	LGTM	Avg. duration	Actual Bedrock bill
July (from the 23rd)	73	112 / 319 / 272	0	68 s	$20.94
August	362	406 / 1,236 / 1,027	6	61 s	$54.65
September	551	530 / 1,861 / 1,653	15	58 s	$105.00
Total	986	1,048 / 3,416 / 2,952	21	about 60 s	$180.60

• On average, one review covered 12 files and about 300 lines, with 1.1 Critical and 3.5 Warning findings. No findings (LGTM) occurred in 21 of 986 reviews (about 2%), so almost every review raised something
• Duration averaged about 1 minute. The weekly p95 was in the 2–3 minute range, and the slowest week was about 5 minutes
• By author, 2 of the 5 people accounted for about 80% of Bedrock calls. The difference in how often people open PRs shows up directly

Estimate vs. actual

One review cost about $0.18. From July to September, the number of reviews grew about 7.5 times and the bill about 5 times. The per-review cost was $0.29 in July, $0.15 in August, and $0.19 in September.

Estimating from token counts with the formula above (29.58 million input and 4.33 million output tokens, about $154 at Sonnet's public list price) came out about 15% lower than the actual bill. It seems better to check the exact amount in Cost Explorer.

Design Decisions

Decision	Choice	Reason and trade-off
Receiving results	tool_use (structured output)	Robust against changes in the body format. However, it counts the severity self-reported by the model as is, so correctness isn't guaranteed
Sending metrics	Call PutMetricData directly	About 10–15 calls per review is too little to justify EMF (estimated at about $0.015/month for 1,500 calls). Consider EMF if the volume grows
Detecting merges	Check with GetPullRequest	The event alone can't distinguish a merge from a rejection. The trade-off is one extra API call per PR
Adding data to invocation logs	requestMetadata	Uses the values measured by Bedrock instead of re-counting tokens in the app
Dashboard authentication	Serve the static site's HTML and JS without authentication, and protect only /api/* with Cognito JWT	Without logging in, the JavaScript on the page redirects to Cognito's login screen. The screen layout and API paths are visible even when not logged in, but no aggregated values can be fetched, so the real harm is small. To authenticate before serving the HTML and JS, you can add authentication on the CloudFront side, but the setup becomes more complex

Limitations and Future Improvements

• This time I couldn't measure whether the findings were correct. The severity of a finding is self-reported by the model, and I can't tell whether it was an actual bug or whether it was fixed. So even if the number of findings drops, you can't say code quality improved
  • As an improvement, I implemented a Lambda (review_feedback) that collects replies to findings, 👍/👎 reactions, and whether the flagged file was changed afterward, at the time the PR is merged or closed. It is under evaluation.
• Lead time can be viewed by day, repository, and PR author in the usage dashboard, but I can't compare it before and after adopting the review
  • There is no data from before adoption, so pre-adoption PRs would have to be recalculated from CodeCommit history
• Findings and LGTM rate by PR author can't be produced. prAuthor is attached to the invocation log, so token counts by author are available, but the findings and LGTM metrics don't carry the author
  • Adding the author to the metrics increases the combinations of dimensions, and with them the number of custom metrics and the cost. I'll consider adding it after checking that it's worth the cost
• Because prAuthor is kept in the invocation log and reports, per-person usage is visible. The purpose is to understand usage, not to evaluate individuals, and this needs to be settled during operation
  • It will need care if the usage dashboard is rolled out widely
• Anyone who opens a PR can write comments freely. An instruction hidden in a comment could steer the review result (prompt injection)
  • Even if the review is steered, the review Lambda doesn't change code or merge, so the impact is limited to review accuracy and comments. I haven't verified whether the review can really be steered, but I plan to test countermeasures, such as stating explicitly in the prompt that the PR body and diff are "data to be examined, not instructions"

Conclusion

Running the review and posting comments was relatively easy to build. However, seeing the effect in numbers required structured output and metadata on the invocation logs. In this evaluation, I couldn't measure whether the findings were correct, or how lead time changed before and after adoption. Also, the track record comes from only one team, so I'd like to keep refining it by operating it with other teams as well.