DEV Community: SpecialBroccoli

Build Dynamic Markdown Summaries for GitHub Actions

SpecialBroccoli — Wed, 06 Nov 2024 20:48:53 +0000

Introducing Markdown Builder – a GitHub Action that enables developers to generate rich markdown summaries directly from Handlebars templates with custom data. Whether you're posting updates to GitHub Actions summaries or adding comments on pull requests, Markdown Builder simplifies the process by allowing you to dynamically inject data into templates.

Check out the GitHub Markdown Builder on GitHub!

🌟 Why Markdown Builder?

Static markdown often doesn’t capture the full picture. For CI/CD workflows, test reporting, and documentation, dynamic markdown summaries that adapt to context and data are powerful. Markdown Builder allows you to create these adaptable summaries without extensive scripting by combining Handlebars templates with optional JSON data. Inspired by the GitHub Test Reporter, which allows users to build dynamic test reports, Markdown Builder brings similar flexibility to markdown any type of markdown summaries.

🔑 Key Features

Easy Template Integration: Provide a Handlebars template, and Markdown Builder will render it for you.
Dynamic Data: Pass JSON data to your template, enabling custom summaries tailored to your specific use case.
Flexible Posting Options: Choose where you want your markdown displayed:
- GitHub Action Summary: Perfect for summarizing key information within the workflow.
- Pull Request Comment: Great for adding context or test results directly on your PRs.

🛠 How It Works

1. Setup the Action

In your GitHub Actions workflow file, add a step for Markdown Builder. Here’s an example configuration:

- name: Generate Markdown Summary
  uses: Ma11hewThomas/markdown-builder@v1
  with:
    template-file-path: './path/to/your/template.hbs'
    json-file-path: './path/to/data.json'  # Optional
    summary: true  # Set to true to post on GitHub Actions summary
    pull-request: false  # Set to true if you want it commented on the PR

2. Configure Your Inputs

Markdown Builder requires a template file, and you can optionally provide JSON data to fill in the template’s placeholders. Here’s what each input does:

template-file-path: Path to your Handlebars template file. This file will define the structure and layout of your markdown.
json-file-path (optional): Path to a JSON file that provides data to your template.
summary: If set to true, posts the rendered markdown to the GitHub Action summary.
pull-request: If set to true, comments the markdown on the associated pull request.

3. Customizing the Template with Handlebars

You can use any Handlebars syntax in your template to include data fields, loops, or conditional formatting. This makes Markdown Builder adaptable for a variety of scenarios, from summarizing test results to generating release notes.

# Build Summary
- Status: {{data.status}}
- Date: {{data.date}}
{{#each data.tests}}
  - Test: {{name}} - Result: {{result}}
{{/each}}

4. See It in Action

After setting up the Action, you'll get instant, visually rich markdown output in your GitHub workflow summary or PR comments. No more manual updates or formatting struggles – Markdown Builder handles it all.

💡 Real-World Use Cases

Test Reporting: Automatically post test results as markdown on PRs to keep the team informed.
Build Summaries: Capture key build metrics or error logs right in the workflow summary.
Release Notes: Generate release summaries for stakeholders without manually formatting each item.

💬 Share Your Templates!

One of the great aspects of Markdown Builder is its flexibility with Handlebars templates. Whether you’re using it for test summaries, build metrics, or custom release notes, we’d love to see how you’re leveraging Markdown Builder in your workflows!

How to Share

If you’ve created a template that others might find useful, consider contributing it to the community templates section.

🤝 Support and Feedback

If you try Markdown Builder, we’d love to hear from you! Feel free to leave a GitHub star if you find it helpful, and don’t hesitate to reach out with feedback or ideas for enhancements.

Happy building! ✨

AI Test Report - Summarize Failed Tests

SpecialBroccoli — Tue, 05 Nov 2024 19:10:56 +0000

The AI Test Reporter is a powerful tool that works with any test framework and leverages your existing accounts with AI language models like OpenAI GPT, Azure and Anthropic Claude to create concise, human-readable summaries of test results, highlighting likely causes and suggesting possible solutions to help your team resolve issues faster.

TL:DR — Check out the AI Test Reporter to generate an AI test report with any testing framework!

Can You Help Us Grow? 🙏

Before diving into how this works, we really need your help to grow! Please consider giving our GitHub organization a follow and support our mission to enhance test reporting with AI. 🙌

Key Features

Immediate Insights: Print summaries directly to the stdout after running your tests.
Seamless Integration: Integrate with developer tools like GitHub, Slack and more for effortless collaboration.
Universal Compatibility: Works with all major testing frameworks via standardized CTRF reports.
AI Summary JSON Report: Add an AI summary to your CTRF JSON report, enabling further processing and integration with other tools and workflows.

Here is an example of the stdout:

In this blog post, we’ll walk you through how to set up and use the AI Test Reporter

OK, Let’s Go!!

To begin using the AI Test Reporter, you’ll need a funded API account with OpenAI, Anthropic, or another leading AI provider that offers API access. Ensure your account allows you to generate an API access token, as this token is essential for integrating the AI capabilities into your testing workflow.

Step 1: Install the CTRF Reporter

First, you need to generate a CTRF test report. CTRF reporters are available for most modern test frameworks, and you can also convert a JUnit report.

Step 2: Generate an AI Summary

Now that you have your CTRF test report, you can use the AI Test Reporter to generate AI-powered summaries for any failed tests. While the tool supports multiple AI providers, we’ll use OpenAI in this example.

Execute the following command in your terminal:

npx ai-ctrf openai <path-to-ctrf-report>

Replace with the path to your CTRF report file, typically ctrf-report.json.

Since the package interacts with the OpenAI API, you will need to set the OPENAI_API_KEY environment variable. For further details on configuration options, check out the AI Test Reporter documentation.

After running the command, the AI Test Reporter will analyze your CTRF report and summarise any failed tests. The summary will be printed directly to your console, highlighting likely causes and suggesting possible solutions to help you debug more efficiently.

Seamless Integration with CI/CD Tools

One of the standout features of AI Test Reporter is that the AI-generated summary becomes a part of your CTRF JSON report. This opens up limitless opportunities for integrating the report into various stages of your CI/CD pipelines and developer workflows. With a standardized data exchange format, you have access to all the relevant test information, including the AI summary, which you can easily share across your toolchain. Whether it’s posting the summary to pull requests, tickets, or communication platforms, the possibilities are vast.

CTRF comes with built-in support for many platforms. In this post, I’ll focus on two: GitHub and Slack.

We’d love to hear how you’re using your CTRF JSON report in your workflows — share your experience in the comments!

Publish AI Test Summary to GitHub Actions

You can display the AI-generated test summary in your GitHub Actions workflow by using the GitHub Test Reporter. Add the following to your workflow YAML file:

- name: Publish CTRF AI Test Summary Results
  run: npx github-actions-ctrf ai <path-to-ctrf-report>
  if: always()

You will have an AI-generated failed test summary, seamlessly integrated into your GitHub Actions workflow summary

If you’d like to add a comment with your AI summary to an associated pull request, simply use the --pr argument:

- name: Publish CTRF AI Test Summary Results
  run: npx github-actions-ctrf ai <path-to-ctrf-report> --pr
  if: always()

You’ll see an AI summary as a comment on the related pull request — ensuring the insights are visible where they’re most needed.

Send AI test summary to your Slack Channel

Use the Slack Test Reporter to send AI generated test summaries directly to your Slack channel. This integration allows team members to see failed test details quickly and suggested solutions without leaving their communication platform.

npx slack-ctrf ai <path-to-ctrf-report>

More CI/CD integrations are being built, checkout the AI Test Reporter documentation for all integrations

How Does This Work?

CTRF (Common Test Report Format) is a standardized test report schema that serves as a bridge between various testing frameworks and developer tools. It includes a collection of tooling to integrate with platforms like GitHub Actions and more. Among its various properties, the CTRF schema supports an ai property for AI-generated summaries.

Here’s an example of what a CTRF report test object looks like with the AI property

{
  "name": "should display title",
  "status": "failed",
  "duration": 1200,
  "message": "error message",
  "trace": "error trace",
  "ai": "AI generated test summary"
}

In this schema:

name: The name of the test.
status: The result of the test (e.g., failed).
duration: The time the test took to run (in milliseconds).
message: The error message from the failed test.
trace: The stack trace of the error.
ai: The AI-generated test summary, provides a concise explanation of the likely cause of failure and possible solutions.

By using CTRF, you can easily integrate AI-generated test summaries into your workflow. The ai property is filled with the summary provided by AI models such as OpenAI or Anthropic, which can then be displayed in your GitHub Actions summary or added as a comment to a pull request.

You can do so much with CTRF, and the tooling is constantly evolving to better support your testing and development workflows.

If you found this useful, remember to follow the organization and give a star to the repositories you find useful.

🧡 Your support helps the library grow and is truly appreciated!