DEV Community

Cover image for Notes on Datadog Continuous Testing
Kenta Takeuchi
Kenta Takeuchi

Posted on • Originally published at bmf-tech.com

Notes on Datadog Continuous Testing

#datadog

This article was originally published on bmf-tech.com.

Overview

Notes on what I researched about Datadog continuous testing.

What is Datadog Continuous Testing

  • No-code
    • Tests can be set up with simple clicks
  • Self-healing E2E
    • Tests with "resilience"
    • Minimizes false positives
  • Supports cross-browser testing
  • Covers major integrations
    • CircleCI, Github Actions, Jenkins, etc.

It seems to have mechanisms to ease the operation of E2E and maintain test reliability.

Introduction

The Datadog test recorder Chrome extension is required.

No other preparation is needed, and you can start using it immediately.

Creating Tests

  1. Create a Browser Test
  2. Record the test case

It seems better to open the recording in a popup (since the UI in the recording screen is an iframe).

Test Configuration

Setting various options related to test execution.

You can handle Basic authentication, set cookies, request headers, etc.

cf. Test Configuration

Test Items

Let's look at the test items available in Browser Tests.

cf. https://docs.datadoghq.com/ja/synthetics/browser_tests/actions/?tab=%E3%82%A2%E3%82%AF%E3%83%86%E3%82%A3%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%A7%E8%A6%81%E7%B4%A0%E3%82%92%E3%83%86%E3%82%B9%E3%83%88%E3%81%99%E3%82%8B

Assertion

Assertions to check around the DOM.

Navigation

Transition-related.

Special Actions

UI-related operations.

Variables

You can define arbitrary variables.

Subtest

You can reuse existing browser tests within another browser test.
You can nest up to two levels for reuse.

HTTP Request

You can execute HTTP requests within a browser test.

Self-healing

It's not clear how much repair is actually possible.
It is stated that changes in the UI are automatically detected and explored.

If there is a UI change that modifies an element (e.g., moves it to another location), the test will automatically locate the element again based on the points of reference that were not affected by the change. Once the test completes successfully, Datadog will recompute, or “self-heal,” any broken locators with updated values. This ensures that your tests do not break because of a simple UI change and can automatically adapt to the evolution of your application’s UI.
In the next section, we’ll look at how you can fine-tune your test notifications to ensure that you are only notified of legitimate failures.

cf. https://www.datadoghq.com/ja/blog/test-maintenance-best-practices/

It's best not to raise expectations too high as it is limited to "simple UI changes."

When the test runs successfully, the browser test will recalculate (or "self-heal") broken locators with updated values, ensuring that the test does not break due to a simple UI update and automatically adapts to the application's UI.

cf. https://docs.datadoghq.com/ja/synthetics/browser_tests/advanced_options/

Parallelization

You can set parallelization from Synthetic settings.

cf. https://docs.datadoghq.com/ja/continuous_testing/settings/

Dashboard

Explore

You can search for results of Synthetic Monitoring and Continuous Testing (CI Batches) and test execution results (Test Runs).
cf. https://docs.datadoghq.com/ja/continuous_testing/explorer/?tab=cibatches

Test coverage

Provides insights into the overall test coverage of RUM applications using browser data collected from RUM and Synthetic browser test results
cf. https://docs.datadoghq.com/ja/synthetics/test_coverage/

It seems possible to analyze what is not being tested. It could help improve the comprehensiveness of test cases.
Track the transition of coverage

CI Integration

Synthetic tests can integrate with various CIs.

Available integrations:

  • Azure DevOps Extension
  • CircleCI Orb
  • Github Actions
  • GitLab
  • Jenkins
  • NPM package

cf. https://docs.datadoghq.com/continuous_testing/cicd_integrations/

Codification

cf. https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/synthetics_test

Since it is a DOM-based test, it seems difficult to create test cases from code...

Notifications

Since it is managed as Synthetics, there are no particular concerns about notifications.

Cost

1000 times/$12 (on-demand $18)
cf. https://www.datadoghq.com/ja/pricing/?product=continuous-testing#continuous-testing

I think it's cheap.
Is it free for less than 1000 times??

Scenario Management Method

Will it break down when many scenarios are prepared?

As a policy for managing test scenarios:

  • Views
  • tag It seems to be in the form of using. There doesn't seem to be any other management UI.

As a policy for managing test cases, it is recommended to create tests in a DRY manner, and it seems better to actively use subtests.
cf. https://docs.datadoghq.com/ja/synthetics/browser_tests/actions/?tab=%E3%82%A2%E3%82%AF%E3%83%86%E3%82%A3%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%A7%E8%A6%81%E7%B4%A0%E3%82%92%E3%83%86%E3%82%B9%E3%83%88%E3%81%99%E3%82%8B#%E3%82%B5%E3%83%96%E3%83%86%E3%82%B9%E3%83%88

Notes on Points of Interest

  • Since Browser Test can obtain the same information as RUM, it might be useful as reference information for debugging
  • When creating a Browser Test, you need to set the test frequency, but it cannot be set to 0, so it is assumed to be executed regularly?
    • It seems okay if you set it to PAUSED
  • Parallelization seems essential
  • Be careful with data cleanup
    • In cases where data is generated during test execution, if the test fails midway, data will be generated in an incomplete state.
      • Even if you create test cases with cleanup in mind, data will remain if the test fails.
        • How to ensure cleanup?
  • Things that seem untestable
    • Selecting the OS on the test execution device
    • You can select the type of browser and browser size, but not the OS
  • SMS authentication is not supported
  • Native apps are not supported
  • Verification of push notifications is also difficult
  • Not limited to E2E, but it seems important to have a mechanism to observe parts that change frequently or where tests are prone to breakage. The dashboard seems to allow for visualization of that.

References

Top comments (0)