DEV Community: Dan Gross

Dive into Mocking Your Microservice Dependencies with Skyramp

Dan Gross — Thu, 14 Dec 2023 15:34:30 +0000

Welcome back, fellow developers! In today's blog post, we're diving into the world of microservices and discussing a crucial aspect of their development and testing: mocking dependencies. Distributed applications built on microservices offer unparalleled flexibility and scalability, but they also bring challenges when it comes to managing dependencies during development and testing.

Fear not! Skyramp provides effective ways to mock your microservice dependencies, making your development process smoother and more efficient. This blog post will cover the necessary inputs and many choices you have at your disposal for mocking dependencies with Skyramp.

Mocking for the Uninitiated

Mocking is a technique used in software development to simulate the behavior of certain components, usually external dependencies, during the testing phase. The purpose of mocking is to isolate the unit of code being tested and control the inputs and outputs of external dependencies. Instead of using the actual implementations of these dependencies, developers create "mock" objects or functions that mimic the expected behavior without executing the real logic.

Why Do Developers Mock Services?

In a microservices architecture, where a system is composed of multiple independently deployable services, mocking becomes particularly important for several reasons:

Isolation: Microservices often rely on external services, databases, or APIs. When developers are testing a specific microservice, they want to isolate it from these external dependencies. Mocking allows them to simulate the behavior of these dependencies, focusing solely on the code under test.
Consistency: Given that microservices have dependencies on other microservices or third-party APIs, mocking enables developers to test each microservice without relying on the availability or consistency of external services. This is crucial for maintaining a fast and reliable testing pipeline.
Controlled Testing Environment: Mocking provides control over the responses from external services, allowing developers to test different scenarios, error conditions, or edge cases that might be challenging to reproduce with real services. This helps uncover potential issues and improve the overall robustness of the microservices.
Parallel Development: In a microservices environment, different teams might be working on different services simultaneously. Mocking enables parallel development by allowing teams to develop and test their microservices independently, even if other services they depend on are still under development.
Cost and Resource Efficiency: Connecting to real services, especially external services, during every test can be resource-intensive and slow down the development and testing process. Mocking eliminates the need to make actual requests, improving the efficiency of the testing cycle.
Fault Injection: Mocking allows developers to simulate error conditions or faults in external services, helping them assess how well their microservices handle failures or unexpected behaviors in the dependencies. This is crucial for building resilient microservices.
Early Testing: Mocking facilitates testing early in the development process when certain dependencies might not be fully implemented or available. It allows developers to start testing components even before all external services are ready for integration.

Mocking microservice dependencies is a key practice in developing distributed applications to ensure efficient and effective testing, isolate units of code, and streamline the development workflow in a distributed and complex architecture. Let's take a look at all the ways you can mock dependencies for your microservices with Skyramp.

Mocking with Skyramp

Mocking dependencies with Skyramp all starts with Skyramp's Mocker. Mocker is a solution for creating mock API services which run in-cluster for either Kubernetes or Docker Compose. In the architecture diagram below, you can see how mock descriptions can be applied to the Skyramp Worker to run the mocks.

Skyramp Clients

Skyramp offers multiple interfaces for mocking dependencies depending on your preference or workflow. Your options include:

The command-line interface (CLI) for using Skyramp in a terminal.
The Visual Studio Code extension for using Skyramp in VSCode.
The Skyramp libraries for Python or JavaScript for making Skyramp calls from your own code.

Supported Environments

To get started with mocking dependencies with Skyramp, your microservice must be running in one of these environments:

Docker Compose: Docker Compose is used for defining and running multi-container Docker applications. You can configure your application's services in YAML and then start the services with a simple command.
Kubernetes: Orchestrating and managing containerized distributed applications becomes more manageable with Kubernetes, providing scalability and resilience to your microservices.

Service Protocols

Skyramp supports a number protocols for mocks to accommodate different microservices architectures. Some of the key protocols include:

gRPC: If your microservices communicate using the gRPC protocol, Skyramp provides comprehensive mocking capabilities, ensuring a smooth testing experience.
REST: For microservices adhering to a RESTful architecture, Skyramp supports robust mocking to facilitate thorough testing and development.
JSON-RPC WebSocket: Skyramp extends its support to JSON-RPC over WebSocket, enabling developers to mock dependencies in real-time communication scenarios.
JSON-RPC HTTP: When your microservices utilize JSON-RPC over HTTP for communication, Skyramp is equipped to support mocks for this protocol, offering flexibility in testing various scenarios.

Choices for Mocking

Generate a mock from an API definition
Skyramp simplifies the process of mocking by allowing you to generate mocks directly from your API definitions, such as OpenAPI or Protobuf. This means you can easily create realistic mocks that mimic the behavior of your actual microservices. With the Skyramp CLI, it's as easy as running skyramp mocker generate ... with the relevant inputs. See the Skyramp Docs for which flags to use based on your protocol.
Create a mock from scratch
For more control over your mocks, Skyramp enables you to create mocks from scratch. This is useful when you need to tailor your mocks to specific scenarios or edge cases that may not be fully covered by the API definitions. With the Skyramp CLI, it's as easy as running skyramp init mock ... and providing a protocol. See the Skyramp Docs for the specific sub-commands.
Configure mocks with data
Mocking isn't just about mimicking endpoints, it's also about replicating real-world scenarios. With Skyramp, you can configure your mocks with realistic data, ensuring that your microservices are tested thoroughly under various conditions. See the Skyramp Docs on how to configure responses for your mocks.

What's in a Mock?

At a fundamental level, a Skyramp mock consists of an endpoint, a response, and a mock configuration. Below is a sample mock description file as an example. You can read more details about Skyramp mock descriptions in the Skyramp Docs.

version: v1
mock:
    description: routeguide
    responses:
        - responseName: ListFeatures
        - responseName: RecordRoute
        - responseName: RouteChat
          lossPercentage: 50
          delayConfig:
              minDelay: 1000
              maxDelay: 2000
    proxies:
        - endpointName: routeguide-svc-RouteGuide
          methodName: GetFeature

Getting our Feet Wet: A Simple Example

Let's walk through a simple example to illustrate how easy it is to mock microservice dependencies with Skyramp. We will use a sandbox environment that already has Docker and the Skyramp CLI installed, so there is nothing you need to install locally. You may recognize elements of this example from previous blog posts. Start by launching the sandbox environment in GitHub Codespaces:

In this example, we have microservices that communicate via REST, and we want to mock one of the dependent microservices for testing purposes. This particular distributed application is an e-commerce store and we want to mock the payment service for the checkout flow.

Once the Codespace launches, you will see a VSCode window in your browser as shown above. The repo source files are pre-loaded and a terminal pane is available. Let's start by changing directories:

cd skyramp/docker-compose

Next, let's bring up the containers, which include the Skyramp Worker container and the application containers.

docker compose up &

Now that the application services are running, let's take a look at the mock we generated previously for our payment service. It is called payment-service-k8s.yaml under the mocks folder, which you can navigate to and open in VSCode. Notice the configuration parameters of the mock, including services, endpoints, and responseValues as discussed earlier.

Let's apply the mock to the Skyramp Worker:

skyramp mocker apply payment-service-k8s --address localhost:35142

Then, we can verify the mock is running:

skyramp mocker status --address localhost:35142

You should see output like this:

Applied mocks at the address 'localhost:35142': 
+----+-----------------+-----------------+----------+-------------+
| ID | ADDRESS         | SERVICE         | ENDPOINT | METHOD      |
+----+-----------------+-----------------+----------+-------------+
| 1  | localhost:35142 | payment-service | charge   | charge-POST |
+----+-----------------+-----------------+----------+-------------+

Finally, we can run a test against the mock with a checkout test that has been pre-defined:

skyramp tester start checkout-test --address localhost:35142

You will see that the test scenario passes because the mock payment service is in place and the dependency for the checkout functions as expected. If you want to see the test fail by not having the mock in place, you can delete the running mocks and try the test again:

skyramp mocker delete -a localhost:35142

You are free to experiment with this scenario to introduce new configurations and then adapt them to your own environment.

This simple example demonstrates how quickly you can set up a mock for your microservice dependencies using Skyramp. For more information, including advanced features of Mocker like gRPC Proxying and Dynamic Responses, be sure to visit the Skyramp Docs.

Open Swim

In conclusion, Skyramp provides a robust solution for mocking microservice dependencies, offering flexibility, ease of use, and support for various environments and protocols. Whether you are generating mocks from API definitions or creating them from scratch, Skyramp empowers you to streamline your development and testing processes.

We encourage you to please join the Skyramp Community Discord for any questions, comments, or feedback.

Happy mocking!

Jump into Microservices Testing with Docker Compose and Skyramp

Dan Gross — Thu, 30 Nov 2023 17:21:29 +0000

Distributed applications leveraging microservices have become a predominant approach for building scalable and maintainable software systems. As developers embrace the microservices paradigm, the need for effective testing solutions becomes increasingly vital. Skyramp, a powerful testing tool designed specifically for distributed systems, offers a robust solution to automate the testing of applications built on microservices.

My previous blog posts have focused on the array of options Skyramp provides for testing distributed applications deployed to Kubernetes clusters. However, Kubernetes is not required to reap the benefits of using Skyramp for test automation. You can also setup and deploy your system-under-test using Docker Compose. In this article, we'll explore how you can leverage Skyramp in conjunction with Docker Compose to streamline your microservices testing process.

For a quick overview on all the points this article will cover, you can check out this short video, "Skyramp & GitHub Codespaces with Docker," as a reference:

A Fly-By of Skyramp and Docker Compose

Before diving into the details, let's briefly introduce Skyramp and Docker Compose.

Skyramp: A Testing Powerhouse for Microservices

Skyramp is a testing tool specifically tailored for distributed systems built on microservices architecture. It provides a comprehensive set of features for automating the testing of various components within a microservices ecosystem. This includes: test generation, functional integration and performance testing, mocking services, hot code reload, and an integrated dashboard. While previous posts have centered around Kubernetes integration, it's important to note that Skyramp is versatile and can be employed with Docker Compose in place of Kubernetes.

Below is an architecture diagram showing where the Skyramp components fit in a Docker Compose setup for conducting automated testing:

Docker Compose: Simplifying Orchestration

Docker Compose is a tool for defining and running multi-container Docker applications. It allows developers to define a multi-container environment in a single file, making it easy to set up and manage complex microservices architectures. Docker Compose is an excellent choice for local development and testing, providing a lightweight alternative to full-scale orchestration solutions.

Setting the Stage

To set some context before jumping into the walkthrough, let's first look at a few key concepts:

Example Repo from Skyramp

Skyramp provides a sample project, sample-microservices, which serves as an excellent starting point for demonstrating testing and mocking with a full-featured distributed application. The application is based on Google's Online Boutique repo, which is an e-commerce store consisting of 11 different microservices. The docker-compose-demo branch referenced above showcases how Skyramp can be seamlessly integrated with Docker Compose for testing microservices with no local setup required. You can also clone the repository and explore the structure of the microservices setup for your own purposes.

Mocks for Simulating Dependencies

One powerful aspect of Skyramp is its ability to handle dependencies and external services through mocks. Define mocked services with a mock description for Skyramp's Mocker to simulate the behavior of external dependencies. This allows you to isolate and test individual microservices in a controlled environment.

Dashboard for Test Results

Skyramp includes a dashboard for organization and analysis of your test results. Integrate the Skyramp dashboard into your testing setup to gain insights into tests, performance metrics, and potential issues. The dashboard enhances visibility, making it easier to identify and address issues in your microservices deployment. To spin up the dashboard for Docker locally with the CLI, simply run skyramp dashboard up --docker from your terminal.

Docker-Specific Docs

Because Docker Compose is markedly different than Kubernetes in terms of setup and certain behaviors, the Skyramp docs have been tailored to provide Docker-specific instructions. There you can discover the many ways Skyramp can help you with your testing strategy for Docker Compose deployments.

A Simple Walkthrough

As mentioned above, we will use the Online Boutique distributed application for our example system-under-test. We can easily create a running test environment using GitHub Codespaces. To launch your own environment so that you may follow along, simply click the link below and then click "Create codespace."

Bringing Up Services

Once the Codespace launches, you will see a VSCode window in your browser. The repo source files are pre-loaded and a terminal pane is available. In the root folder of the repo, you will see a shell script called workflow-1.sh. Executing this script will bring up the system-under-test using a pre-configured docker-compose.yml file under the skyramp/docker-compose folder. Go ahead and give it a spin from the command line:

sh workflow-1.sh &

At this point you should see a screen similar to this one, which shows the output from the running docker containers:

The system-under-test deployment is comprised of the services we need to conduct a test of the checkout process, namely the product catalog, cart, payment, and checkout services.

Our pre-defined test scenario in this case will aim to test the checkout flow. First, change directories in the terminal:

cd skyramp/docker-compose

Next, let's go ahead and run our test. The test itself is defined under tests as checkout-test.yml, which you can review in the VSCode window from the codespace or by navigating here. To run, execute:

skyramp tester start checkout-test --address localhost:35142

Uh-oh, it looks like the test failed because the checkout process could not connect to the payment service.

Truth be told, we did that on purpose. This allows us to demonstrate how we can bring up a mock of the payment service using Skyramp Mocker. Simply run:

skyramp mocker apply --address localhost:35142

The mock is defined under mocks as payment-service-k8s.yaml, which you can review in the VSCode window from the codespace or by navigating here.

With the payment service mock in place, let's re-run our checkout test:

skyramp tester start checkout-test --address localhost:35142

Here we see the test scenario passes, so we have successfully tested our checkout components in concert to validate that the microservices are behaving as expected in the flow. Great work, give yourself a pat on the back.

Wrapping Up

Incorporating Skyramp with Docker Compose offers a seamless and efficient approach to testing microservices locally. By leveraging Docker Compose's simplicity for setting up multi-container environments and Skyramp's comprehensive testing capabilities, developers can ensure the reliability and performance of their microservices architectures. Whether you're new to microservices testing or looking to enhance your existing workflows, the combination of Skyramp and Docker Compose provides a powerful solution for robust and automated testing. As always, happy testing.

Finally, please join our Discord for any questions, comments, or feedback.

Observe This: One Step from Trace to Test with Skyramp

Dan Gross — Thu, 16 Nov 2023 14:44:34 +0000

KubeCon North America 2023 in Chicago last week was a barrage of technology and innovation all aimed at running distributed applications at scale with Kubernetes. In particular, the observability topic featured prominently in technical sessions, vendor booths, and in hallway conversations.

Observability plays a key role in gaining insights about a system's behavior in order to ensure the reliability and performance of deployed applications. Because of the complex nature of distributed applications leveraging microservices, an important aspect of observability is tracing. In this blog, we'll explore how you can harness observability tracing to create effective test scenarios easily with Skyramp.

The Anatomy of Tracing

Observability tracing involves capturing and analyzing the flow of requests as they traverse through the different components of a distributed system. Tools like Jaeger, OpenTelemetry, and Pixie can be used in Kubernetes environments to trace the execution of requests across microservices.

Two Key Components of Tracing

Spans: Spans represent individual units of work or operations within a distributed system. Spans will include metadata such as the operation name, start and end timestamps, and contextual information like tags and logs.
Traces: Traces consist of a collection of spans, forming a sequence that traces the journey of a request. Often, spans are collected and assembled into traces based on their shared trace ID.

Traces Versus Logs

Traditional logging is still valuable from an observability standpoint. Logs offer detailed information about individual events within an application based on time, aiding in debugging and troubleshooting. However, if you've combed through the logs produced from a distributed application, you know it can be difficult to gain end-to-end visibility of a request. Observability tracing focuses on providing a higher-level view of the flow of requests through a distributed system, aiding in performance analysis and system optimization. Think of traces like enhanced logs. Both can and should be used in tandem for your observability strategy in Kubernetes.

A Sample Trace

Below is an short excerpt from a sample trace file, which can be found here in Skyramp's public repo, sample-microservices.

  {
    "Source": "px-operator/93bc2253481b44a32e04a523286b130a01e7a6e2168db1be5c02993f1978q75",
    "Destination": "product-catalog-service",
    "RequestBody": null,
    "ResponseBody": {
      "categories": [
        "accessories"
      ],
      "description": "Add a modern touch to your outfits with these sleek aviator sunglasses.",
      "id": "OLJCESPC7Z",
      "name": "Sunglasses",
      "picture": "/static/img/products/sunglasses.jpg",
      "priceUsd": {
        "currencyCode": "USD",
        "nanos": 990000000,
        "units": 19
      }
    },
    "Headers": {
      "Accept-Encoding": "gzip",
      "Host": "product-catalog-service:60000",
      "User-Agent": "Go-http-client/1.1"
    },
    "Method": "GET",
    "Path": "/get-product?product_id=OLJCESPC7Z",
    "Status": 200,
    "ResponseMsg": "",
    "Port": 0
  },

The excerpt above represents one request in a series of requests in the trace.json file. You can see quite a bit of extended information in the form of metadata about this request operation as made available from a span.

Generating Tests from Traces

Observability traces can give us valuable insights into the behavior of our distributed applications for troubleshooting and optimization. So, can we also use traces for generating test scenarios? With Skyramp, you can! Creating tests based on observability traces provides several advantages:

Realistic Scenarios: Traces capture real interactions between microservices, allowing you to recreate realistic scenarios for testing.
Performance Analysis: Identify performance bottlenecks and latency issues by analyzing trace data during test executions.
Error Detection: Leverage traces to detect errors and exceptions, facilitating the creation of robust test cases to handle various failure scenarios.
Regression Testing: Use trace data to establish a baseline for application behavior, enabling effective regression testing as your application evolves.
Early Detection: Catching issues early before deploying to production saves time and money. Using traces a basis for test scenarios helps avoid "testing in prod."

Furthermore, these kinds of functional integration tests, known as service tests or component tests, are often overlooked and yet an important layer in the test pyramid. This is also the area of testing where Skyramp shines.

The above image is from an article related to testing on martinfowler.com.

Skyramp on the Scene

Skyramp provides an array of features to help developers and platform engineers create, manage, and run test scenarios for microservices in Kubernetes clusters.

Previous blog posts have shown how to generate tests based on API schema definitions with OpenAPI or Protocol Buffers. For example, this blog covers test generation with OpenAPI: Test Generation for Distributed Apps Made Easy with Skyramp. However, there are many options available for generating tests, as you can see from the flag options in the Skyramp CLI:

skyramp tester generate <flags>
      --address string              destination address of tests
      --alias string                Kubernetes service / Docker alias name
      --api-schema string           path to API schema file, or URL (URL support for OpenAPI 3.x only)
      --cluster-id string           cluster id from telemetry provider
  -n, --namespace string            Kubernetes namespace where Skyramp worker resides
      --openai                      (experimental) use OpenAI to generate test values (the 'OPENAI_API_KEY' environment variable must be set with an OpenAI API token)
      --openai-model string         (experimental) Optional, GPT model to use for OpenAI (one of [gpt-3.5-turbo gpt-4]). Note that some models may not accesible based on the API token (default "gpt-3.5-turbo")
      --openapi-tag string          tag to filter on (for openapi protocol)
      --output-prefix string        prefix for generated files
      --port int                    port number for the service
      --proto-service string        proto service to utilize (for protobuf protocol)
      --protocol string             protocol to use for the test configuration (one of [grpc rest])
      --start-time string           start time to retrieve traces from
      --telemetry-provider string   telemetry provider, currently only pixie is supported
      --trace-file string           trace file path

For this blog, we will focus on generating tests from observability traces. You can accomplish this based on trace data captured directly from a telemetry provider or using an exported trace file as the input.

One Step Generation

You will see in the skyramp/rest-demo folder from the sample-microservices repo referenced above that there is a trace folder containing an example trace file discussed earlier. This particular trace file was created by calling the running system in the cluster with curl commands and exporting the resulting trace.

Furthermore, the tests/test-trace-bOl4.yaml demo test description as well as the scenarios/scenario-trace-bOl4.yaml file were automatically generated from the example trace file using Skyramp. To illustrate, this how to generate the same test scenario with the Skyramp CLI:

skyramp tester generate \
--trace-file trace/trace.json \
--protocol rest

Output:

Successfully created/updated the test description configuration files: 
        scenarios/scenario-trace-XI0m.yaml
        tests/test-trace-XI0m.yaml

It really is that easy!

Another way to capture traces for generating test scnearios is by passing in a telemetry provider to skyramp generate. In the following example, we are using Pixie as the provider:

skyramp tester generate \
--telemetry-provider pixie \
--cluster-id cid1 \
--namespace ns1 \
--start-time "-5m"

Whichever way you choose, generating tests with Skyramp is just one step.

Executing the Tests

Once the test scenarios are generated from observability traces, we can easily execute them to playback the sequence in the trace with Skyramp. Here, we run the test that we generated above against our system-under-test:

skyramp tester start test-trace-bOl4

And we get our results of the test run:

Starting tests
Tester finished
Test trace-test------
 [Status: finished] [Started at: 2023-11-15 15:46:00 PST] [End: 2023-11-15 15:46:00 PST] [Duration: 0s]
  - pattern0.scenario_x2Dk
    [Status: finished] [Started at: 2023-11-15 15:46:00 PST] [Duration: 0s]
  - pattern0.scenario_x2Dk.0.POST_q9FJ
    [Status: finished] [Started at: 2023-11-15 15:46:00 PST] [Duration: 0s]
    Executed: {"success":"200 OK"}
  - pattern0.scenario_x2Dk.1.assert
    [Status: finished] [Started at: N/A]
    Assert: requests.POST_q9FJ.code == 200
    Passed: true
  - pattern0.scenario_x2Dk.2.GET_D0WG
    [Status: finished] [Started at: 2023-11-15 15:46:00 PST] [Duration: 0s]
    Executed: {"user_id":"eeeee","items":[{"product_id":"OLJCESPC7Z","quantity":2}]}
  - pattern0.scenario_x2Dk.3.assert
    [Status: finished] [Started at: N/A]
    Assert: requests.GET_D0WG.code == 200
    Passed: true
  - pattern0.scenario_x2Dk.4.POST_qwWE
    [Status: finished] [Started at: 2023-11-15 15:46:00 PST] [Duration: 0s]
    Executed: {"order_id":"149bbe85-c41c-443a-b273-53551c1b52f3","shipping_tracking_id":"00be572e-02d4-41a5-b1ac-f50c87c2cc8a","shipping_cost":{"currency_code":"USD","units":10,"nanos":100},"shipping_address":{"street_address":"1600 Amp street","city":"Mountain View","state":"CA","country":"USA"},"items":[{"item":{"product_id":"OLJCESPC7Z","quantity":2}}]}
  - pattern0.scenario_x2Dk.5.assert
    [Status: finished] [Started at: N/A]
    Assert: requests.POST_qwWE.code == 200
    Passed: true

All the tests passed, so this was a successful test run. Of course, you can modify the test scenarios manually if you choose. You can introduce new steps or inject alternative data. Modifications can be valuable for troubleshooting behavior of the microservices in the system-under-test.

All of the examples above are available for you to try for yourself. Simply follow the links provided or visit the Skyramp Docs for additional guidance.

Leaving with a Trace

Observability tracing is a powerful mechanism for understanding the intricacies of distributed systems. By leveraging traces to create test scenarios in Kubernetes clusters, you can enhance the robustness and reliability of your applications. As we demonstrated, generating tests with Skyramp based on traces is easy. This proactive approach to testing ensures that your system not only meets current requirements but is also well-prepared for future challenges.

Embrace observability tracing as a fundamental aspect of your testing strategy with Skyramp, and watch as your Kubernetes applications become more resilient and performant in the face of evolving demands. Keep exploring and remember the sky's the limit. Happy tracing!

P.S. Please join the Skyramp Community Discord for any questions, comments, or feedback.

3 Tools to Improve Your Code Quality: GitHub Codespaces, Kubernetes, and Skyramp

Dan Gross — Thu, 02 Nov 2023 13:46:01 +0000

In the world of developing distributed applications with microservices, code quality is paramount. It can make the difference between a well-functioning application, and one that's prone to bugs and vulnerabilities. To help you maintain and improve your code quality, we'll explore three powerful tools in this blog: GitHub Codespaces, Kubernetes, and Skyramp. You’ll be able to try this combo for yourself with an instant dev environment.

Putting the Pieces Together

In previous blog posts, we covered using GitHub Codespaces for debugging, generating tests in a Kubernetes cluster, and running Skyramp's built-in dashboard to manage test results. In this blog, we’ll bring the pieces together to show how you can use all these tools to construct and run test scenarios for your system-under-test.

A Word about Component Testing

Within the space beyond unit testing and prior to end-to-end testing, there is a critical stage known as component testing. This infodeck on Martin Fowler's website covering testing strategies for microservices describes component testing this way:

"In a microservice architecture, the components are the services themselves. By writing tests at this granularity, the contract of the API is driven through tests from the perspective of a consumer."

When you're developing a distributed application with microservices, it's essential to ensure that each component functions correctly, both in isolation as well as when integrated with other components. Component testing is the key to ensuring this, and Skyramp is here to assist you in this endeavor.

Instant Dev Environment

GitHub Codespaces can be used to provide a cloud-based, integrated development environment that makes it easier to set up and run your tests with Skyramp. You can clone your repository, create a dedicated Codespace for your microservices, and then execute tests. A straightforward testing workflow is crucial to identify and fix issues early in the development process, saving you time and reducing headaches in the long run.

Here are some benefits of using GitHub Codespaces for component testing:

On-Demand Environments: Easily spin up a development environment for your microservices project, reducing setup time and eliminating conflicts between dependencies.
Consistency: Ensure that all developers work in identical environments, reducing the "it works on my machine" problem.
Collaboration: Multiple developers can collaborate on the same project without worrying about where to setup and run environments.
Easy Test and Debug: Run tests and debug seamlessly within the Codespace environment using a familiar VSCode interface.

We've prepared a GitHub Codespace from a branch of the Skyramp sample-microservices repo on GitHub. This branch is based on the Online Boutique demo project from Google Cloud. The project is a web-based e-commerce store that consists of an 11-tier microservices application. Skyramp added support for REST, which we will use for the example in this blog.

The GitHub Codespace can be launched from the link below and then clicking "Create Codespace".

That will setup a VSCode environment within the browser that looks like this:

Kubernetes Clusters in GitHub Codespaces

For microservices to function as a cohesive system, you need a way to test their interaction in a realistic environment. This is where Kubernetes comes into play. Kubernetes allows you to manage containerized applications in clusters, making it a perfect fit for deploying and testing microservices.

We will be using the Online Boutique application mentioned earlier for our example, which is built around Kubernetes. Skyramp can then be utilized to easily deploy and test the distributed app consisting of various microservices. We have created tests in the repo for testing 4 primary components of the application - Product Catalog, Cart, Payment, and Checkout.

Let's get started by changing the directory in the terminal of the Codespace to skyramp/rest-demo:

cd skyramp/rest-demo

Then, we'll create a new Kubernetes cluster with the Skyramp CLI:

skyramp cluster create -l

Expected output:

Creating local cluster     [##############################################################] 100 %
Successfully created local cluster.
Starting/restarting Resolver     [##############################################################] 100 %
Resolver manages DNS changes needed to access the cluster and needs sudo access.
[sudo] updating file /etc/resolv.conf

Next, let's bring up the system using Skyramp Deployer:

skyramp deployer up rest-app

Expected output:

email-service-745ddcd9c4-6zf8s                 ready
frontend-6f8fc574f7-w6hgb                      ready
product-catalog-service-5467566c8f-5n86m       ready
skyramp-worker-6c8bfdf587-bpf9m                ready
cart-service-56cd75c447-kshpg                  ready
currency-service-5b966cbfb8-tffff              ready
ad-service-7cb787976d-ldjm7                    ready
payment-service-5f9b47cfd4-26r9d               ready
redis-6fc54f68c4-fwg75                         ready
recommendation-service-6f8cd558b8-6cppj        ready
checkout-service-c5546db56-vdxld               ready
shipping-service-599947db74-flqhq              ready
All pods are ready.

The system-under-test is now up and running. We have prepared several test scenarios which will be easy to run and then review the results with Skyramp.

A Complement of Tests

Quality assurance is not limited to early unit tests and final end-to-end tests. You need a comprehensive suite of tests, which include functional integration and performance tests, as well as the component tests highlighted earlier. This is where Skyramp completes the picture. Skyramp is an automated testing platform that can help you create and manage a wide range of tests for this stage of development.

Skyramp takes a modular approach to testing, as illustrated below with examples taken from the testing scenario we used for this blog.

Endpoints: Skyramp will use endpoint definitions for networking details of services used in tests as shown in this endpoint YAML file:

version: v1
services:
    - name: product-catalog-service
      port: 60000
      alias: product-catalog-service
      protocol: rest
endpoints:
  - name: product-catalog-service-get-products
    path: "/get-products"
    serviceName: product-catalog-service
    methods:
      - type: 'GET'
        name: get-products

Scenarios: Skyramp uses scenario definitions for request behavior of service methods or chains of requests, and specific asserts within test scenarios:

version: v1
scenarios:
  - name: scenario_123
    steps:
      - requestName: listProductsRequest
      - asserts: requests.listProductsRequest.res[0].id == "OLJCESPC7Z"
      - requestName: addCartRequest
      - asserts: requests.addCartRequest.res.success == "200 OK"
      - requestName: getCartRequest
      - asserts: requests.getCartRequest.res.user_id == "abcde"
      - requestName: chargeRequest
      - asserts: requests.chargeRequest.res.transaction_id != null
      - requestName: checkoutRequest
      - asserts: requests.checkoutRequest.res.items[0].item.product_id == "OLJCESPC7Z"

Tests: Skyramp uses test definitions as a starting point to execute the proper behavior of tests:

version: v1
test:
  name: "[REST] Checkout system load test testcase"
  target: rest-app
  testPattern:
    - startAt: 1s
      atOnce: 2
      scenarioName: scenario1
      duration: 10s

Running the Test, Checking the Results

We’ve found it incredibly valuable to have a central dashboard that aggregates and visualizes the results of your tests so you can get the most out of these tools, and ensure that your code quality is continually improving. Skyramp provides just that, and it can be run in GitHub Codespaces! Let's fire up the Skyramp Dashboard and run our test scenario against the services in the cluster.

Back to the Codespaces terminal, run this command:

skyramp dashboard up &

Click "Open in Browser" from the dialog box popup regarding port forwarding and you will see the Skyramp Dashboard interface in a new browser tab. Click on "Tests". There will be no tests yet to display, but we can change that. Back to the Codespace tab, run Skyramp Tester in the terminal to execute our full-system test:

skyramp tester start full-system -n test-rest-demo

Now back to the Skyramp Dashboard tab in the browser and you will see the results of the test run there:

Looks like a passing grade! Notice the cells in the table can be expanded to view details of the test results.

You can explore further and even tweak the tests by checking out the YAML files in the endpoints, scenarios, and tests folders under skyramp/rest-demo. Also, visit the Skyramp Docs to learn more about all the functionality available for testing with Skyramp.

The Power of 3

By combining GitHub Codespaces, Kubernetes, and Skyramp, especially with a robust testing dashboard, you can ensure that your code quality remains high throughout the development lifecycle. As a result, your applications will be more reliable, ultimately leading to greater user satisfaction and a stronger competitive edge in the landscape of microservices-based distributed apps. As always, happy testing!

For more info, visit: Skyramp.dev & Join the Skyramp Community Discord

Test Generation for Distributed Apps Made Easy with Skyramp

Dan Gross — Fri, 27 Oct 2023 20:05:25 +0000

In the ever-evolving world of software development, the importance of testing to ensure quality cannot be overstated. This rings especially true when it comes to distributed applications with loosely coupled microservices. These kinds of distributed applications bring a host of complexities that require a thoughtful approach to testing. With the advent of testing tools provided by Skyramp, combined with technology building blocks like Kubernetes and OpenAPI, test generation for distributed apps is easier and more efficient than ever before. This blog will dive into the process of generating tests for your distributed applications by showing a step-by-step example using Skyramp and the aforementioned standards.

Talking About Test Generation

Before we delve into the step-by-step walkthrough of generating tests, let's first understand why test generation is beneficial for distributed applications. Distributed applications, most often composed of various microservices, can be particularly intricate in terms of communication, dependencies, and data flow. Manually writing tests for each component can be time-consuming and prone to human error. Automated test generation can streamline this process, ensuring comprehensive coverage and robust testing.

OpenAPI, formerly known as Swagger, is a set of specifications that defines a standard, language-agnostic interface to RESTful APIs. It's often used to document and describe the capabilities of a microservice, including the available endpoints, input/output parameters, and response formats. This can act as the foundation for automated test generation by providing a clear and structured definition of a service.

Leveraging OpenAPI, Skyramp can analyze your microservices and automatically create test cases based on your API specifications and service dependencies. This allows you to verify that the various components of your distributed application work seamlessly together, saving you time. It should be noted that Skyramp supports other protocols for test generation like gRPC and Protocol Buffers, but we'll save that for another day.

Generating Tests: A Step-By-Step Walkthrough

Now, let's get into the nitty-gritty of how to generate tests for your distributed applications using Skyramp. We will use our tried and true sample-microservices repo on GitHub, which is an 11-tier distributed app with REST support already configured for deployment to a Kubernetes cluster. The app itself is a complete web-based e-commerce store created by Google, known as Online Boutique.

Below is the step-by-step walkthrough to guide you along the journey. These steps were tested on Mac, but should work on Linux or Windows with a bash shell.

Step 1: Install the Skyramp CLI

The first step is to install the Skyramp Command Line Interface (CLI). See the Skyramp Docs for specifics, or you can run the following command in your terminal:

bash -c "$(curl -fsSL https://skyramp.dev/installer.sh)"

NOTE: You will also need Docker installed to continue.

Step 2: Clone the Repository

Next, clone the sample repository discussed above. This contains the code and configuration files necessary for your distributed application.

git clone https://github.com/letsramp/sample-microservices.git

Step 3: Create a Cluster

Now, create a cluster using the Skyramp CLI. This cluster will serve as the environment in which your microservices will be deployed and tested.

skyramp cluster create -l

Step 4: Change to the Demo Directory

Navigate to the directory containing the prepared REST demo project under "rest-demo".

cd sample-microservices/skyramp/rest-demo

Step 5: Deploy the Cluster with Skyramp Deployer

Use the Skyramp Deployer to bring up the microservices within the cluster.

skyramp deployer up rest-app

The output will look like this:

ad-service-7cb787976d-l49n5                    ready
currency-service-5b966cbfb8-q7dvf              ready
email-service-745ddcd9c4-7wdvq                 ready
cart-service-56cd75c447-xzdlk                  ready
checkout-service-c5546db56-lsdtm               ready
product-catalog-service-5467566c8f-sl6wb       ready
skyramp-worker-6c8bfdf587-jr2tq                ready
payment-service-5f9b47cfd4-cw4j2               ready
recommendation-service-6f8cd558b8-dmvh8        ready
shipping-service-599947db74-8dtkn              ready
frontend-6f8fc574f7-nv76g                      ready
redis-6fc54f68c4-v6mqb                         ready
All pods are ready.

Step 6: OpenAPI Configuration

In this project, there is a pre-defined OpenAPI specification file (demo.yaml). This file describes the API endpoints and their expected behavior. Take a peek at the contents of this file.

cat openapi/demo.yaml

Notice that the definition centers around charging a credit card. We can use this definition to test the payment service running in our cluster.

Step 7: Generate a Test

Now comes the fun part - generating a test using the Skyramp CLI. You can specify the protocol, API schema, alias, and port for your test. Run the command below to see how this works.

skyramp tester generate \
--protocol openapi \
--api-schema openapi/demo.yaml \
--alias test-rest-demo \
--port 60000

This command will automatically generate test cases for your distributed application based on the provided OpenAPI schema.

Step 8: Tune Your Test

Upon running the test generation command from the step above, Skyramp will create three files that contain your test cases and configuration. You can review and customize these files to fit your specific testing needs. The output will look something like this:

Creating test files     [##############################################################] 100 %
Successfully created/updated the test description configuration files:
    endpoints/test-rest-demo-tDPT.yaml
    scenarios/scenario-test-rest-demo-tDPT.yaml
    tests/test-test-rest-demo-tDPT.yaml

Let's make a few edits to the files that were generated.

1) Change the serviceName value in the generated "endpoint" file to use the payment service running in the cluster:

serviceName: payment-service

2) Change the requestName parameter in the generated "test" file to scenarioName and assign the value "scenario_123".

 - scenarioName: scenario_123

3) Modify the generated "scenario" file to include a scenarios section at the end of the file that looks like this:

scenarios:
  - name: scenario_123
    steps:
      - requestName: chargeRequest
      - asserts: requests.chargeRequest.res.transaction_id != null

4) Also modify generated parameters and values in the "scenario" file under the requests section as shown below. Many of the parameters will be auto-generated based on the OpenAPI definition, but we can improve the values.

    - name: chargeRequest
      headers:
        Content-Type: "application/json"
      blob: |-
        {
          "amount": {
            "currency_code": "USD",
            "units": 100
          },
          "credit_card": {
            "credit_card_cvv": 100,
            "credit_card_expiration_month": 10,
            "credit_card_expiration_year": 2030,
            "credit_card_number": "4432-8015-6152-0454"
          }
        }

Step 9: Run and Refine

After completing the edits from above, we are ready to run our test against the payment service. Simply run Skyramp Tester to execute the test scenario we composed.

skyramp tester start test-test-rest-demo-2VHL -n test-rest-demo

The output should indicate a passed test such as this:

Starting tests
Tester finished
Test test-rest-demo------
 [Status: finished] [Started at: 2023-10-26 09:32:56 PDT] [End: 2023-10-26 09:32:57 PDT] [Duration: 1s]
  - pattern0.scenario_123
    [Status: finished] [Started at: 2023-10-26 09:32:57 PDT] [Duration: 0s]
  - pattern0.scenario_123.0.chargeRequest
    [Status: finished] [Started at: 2023-10-26 09:32:57 PDT] [Duration: 0s]
    Executed: {"transaction_id":"e740f1b8-3618-4253-94ae-9be0de082f68"}
  - pattern0.scenario_123.1.assert
    [Status: finished] [Started at: N/A]
    Assert: requests.chargeRequest.res.transaction_id != null
    Passed: true

You can continue to fine-tune your tests by modifying the generated files. Working with these files, you can see the modular approach Skyramp takes to testing. Adjust the test cases, add custom scenarios, and configure test environments as needed to ensure comprehensive testing for your distributed applications.

This Was Only a Test

In conclusion, automated test generation with Skyramp simplifies the testing process for distributed applications. It enhances the efficiency and reliability of testing scenarios, ensuring that your microservices work harmoniously to deliver a robust and error-free application.

By following the step-by-step walkthrough outlined in this blog, you can see how to effectively generate and customize tests for your distributed apps, saving time and ensuring the quality of your software. Happy testing with Skyramp!

Turbocharge Your Debugging with Skyramp's Hot Code Reload

Dan Gross — Fri, 20 Oct 2023 15:06:42 +0000

Are you tired of the traditional, cumbersome debugging process that requires re-deployments of microservices every time you want to test a code change? Skyramp now offers a game-changing feature known as Hot Code Reload. With this feature, you can debug your services in real-time without the hassle of re-deploying your application. You can attach your debugger, step through code, and make changes while your service continues to run without interruption.

In this blog post, we'll explore how to enable and use Skyramp's Hot Code Reload feature to turbocharge your development and debugging workflow for distributed applications. We'll first cover how to setup Hot Code Reload for any Kubernetes environment, then we'll step through a specific example that you can follow along using GitHub Codespaces.

In addition, we also have a YouTube video that walks through a quick summary of the Hot Code Reload feature from Skyramp:

How to Setup Hot Code Reload

The Hot Code Reload feature is available as a configuration option for the Skyramp Deployer tool. By using Deployer to deploy services to your Kubernetes cluster, you can take advantage of this feature.

To setup the Hot Code Reload feature, you need to include a 'debug' section in your target description file under the 'containers' section. This 'debug' section should specify various parameters to configure the debugging process. Here's an example of what this looks like:

debug:
  - containerName: Deployment/my-service
    runtimeType: go
    command: myservice
    debugPort: 33333
    mountPaths:
      - /workspaces/myproject/src/myservice

Let's break down what each of these parameters means:

containerName: This is the name of the container you want to use in debug mode. You should specify the container associated with the service you wish to debug.
runtimeType: Here, you specify the runtime type of your service, such as 'go,' 'node,' or 'python.' This information is crucial for Skyramp to set up the debugging environment correctly.
command: This parameter represents the application's entry point that will run in a loop. It should be relative to the first path specified in the 'mountPaths'. It is the core of the debugging process, allowing you to inject code changes without disrupting the service.
debugPort: The 'debugPort' is the local port of the running service that you want to debug. This port is where your debugger will connect to interact with your service.
mountPaths: You can specify one or more paths that should be mounted to the remote container. These paths are the locations where your code changes will be applied. It's essential to ensure that your code and any necessary dependencies are mounted correctly.

The information above is also covered in the Skyramp Docs, where you can learn more about how to utilize all the Skyramp capabilities for testing distributed applications.

Looking under the Hood

Once you've configured the 'debug' section in your target description file, Skyramp's Hot Code Reload feature comes to life. Now, let's take a closer look at how it works:

Real-time Debugging
With Hot Code Reload, you can attach your debugger to the running service. This allows you to inspect the current state of your application and set breakpoints as needed.

Code Changes
As you debug your service, you can make code changes on the fly. These changes are instantly injected into the running application, giving you the flexibility to fix issues or implement new features without restarting the service.

Interactive Debugging
You can step through your code, inspect variables, and watch how your changes affect the behavior of your service. This interactive debugging experience greatly accelerates the troubleshooting and development process.

Streamlined Workflow
Hot Code Reload streamlines your debugging process by eliminating the need for frequent re-deployments. It saves you time and effort, enabling faster iteration and more efficient debugging.

Give It a Spin with Codespaces

In this section, we are going to step through an example of using Hot Code Reload to debug a sample application within a pre-configured deployment. Don't worry, though, no local setup is required because we are going to use GitHub Codespaces from a browser.

For our scenario, let's say a problem was reported with the cart service in our distributed application. We will attempt to find and fix the issue by setting up a Codespace environment to test and debug the cart service using Skyramp.

Note: the specific steps outlined below may be refined since the Skyramp product is rapidly evolving.

Step 1 - Fire Up the Codespace

Our starting point is the hot-code-reload-demo branch in Skyramp's letsramp/sample-microservices GitHub repo. You can use your browser to navigate to the correct branch in the repo here. The sample-microservices repo contains a demo project based on GCP's Online Boutique with added support for REST and Thrift APIs. This sample e-commerce application is perfect for demonstrating cloud-native development and testing, including debugging with Hot Code Reload with Skyramp.

To launch the Codespace, click the "Open in GitHub Codespaces" link from the README.md. You can also click the link below:

From the "Create a new codespace" page on GitHub, simply click "Create codespace". It will take a few moments to set up your Codespace. When the process finishes, you will see a VSCode environment like this:

Step 2 - Setup the Cluster and Deploy

Next, we will use the Skyramp CLI to create a Kubernetes cluster in the Codespace, and then deploy the services we need in order to test and debug.

Create a Kubernetes cluster from the Terminal in VSCode:

skyramp cluster create -l
cp ~/.skyramp/kind-cluster.kubeconfig ~/.kube/config

For debug purposes, we will need to install this SMB driver with Helm into the Codespace:

helm repo add csi-driver-smb https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/charts
helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.11.0

Also for debugging, the binary for the cart service needs to be built:

cd /workspaces/sample-microservices/src/cartservice
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o cartservice -gcflags "all=-N -l" .

Finally, we deploy the services to the cluster using Skyramp Deployer:

cd /workspaces/sample-microservices/skyramp/rest-demo
skyramp deployer up checkout-system

You should now see that all pods are ready.

Step 3 - Test and Debug

We will be using a specific port for the debugger, so open the
src/cartservice/.vscode/launch.json file in the repo and change the port parameter value to 2345.

Then, create a second Terminal in VSCode using the drop down and selecting "bash". From there, we will start the debugger with the following commands:

POD=$(kubectl get pods -n test-rest-demo | grep cart-service | awk '{ print $1}')
kubectl exec -it $POD -n test-rest-demo -- /mnt/debug/debuggers/dlv exec /mnt/debug/cart-service/cartservice/cartservice --wd /mnt/debug/cart-service/cartservice  --listen :2345 --log --api-version=2  --headless --continue --accept-multiclient

Now we can run our test scenario from the first Terminal window using Skyramp Tester:

skyramp tester start checkout-test -n test-rest-demo

We see that there is a failed test. It looks like a problem with the user ID.

Create a third Terminal to setup port forwarding for the debugger, and then execute the following:

POD=$(kubectl get pods -n test-rest-demo | grep cart-service | awk '{ print $1}')
kubectl port-forward $POD -n test-rest-demo 2345:2345

Now we can attach the debugger in VSCode to step through the code. Open the src/cartservice/rest.go file and set breakpoint on line 33 at the post function. Since the error was related to the user ID, there might be an issue in this function.

Click the "Run and Debug" button on the left nav and then click the green play arrow to attach the debugger. You will see the status bar at the bottom of VSCode turn red. From the first Terminal, re-run the test as shown earlier. Execution will stop at line 33 where we set our breakpoint.

Using the debug controls to step over the code, we see that user_id is not populated. It looks like the code is using UserID instead of user_id for the parameter. That's most likely the issue, so let's fix it. We can continue execution and then disconnect the debugger in VSCode using the debug controls. Also, we can kill the debugger by running the following command in the first Terminal:

POD=$(kubectl get pods -n test-rest-demo | grep cart-service | awk '{ print $1}')
kubectl exec -it $POD -n test-rest-demo -- killall dlv

Step 4 - Fix and Validate

Now on to the best part -- fixing the bug. First, update the code in rest.go to use user_id instead of UserID on line 33. Then, rebuild the binary as we did earlier from the first Terminal:

cd /workspaces/sample-microservices/src/cartservice
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o cartservice -gcflags "all=-N -l" .

Now, we restart the debugger in the second Terminal as we did earlier:

POD=$(kubectl get pods -n test-rest-demo | grep cart-service | awk '{ print $1}')
kubectl exec -it $POD -n test-rest-demo -- /mnt/debug/debuggers/dlv exec /mnt/debug/cart-service/cartservice/cartservice --wd /mnt/debug/cart-service/cartservice  --listen :2345 --log --api-version=2  --headless --continue --accept-multiclient

Switching back to the first Terminal, we can re-run our test with Skyramp Tester:

cd /workspaces/sample-microservices/skyramp/rest-demo
skyramp tester start checkout-test -n test-rest-demo

Look at that, the previously failed test now passes!

We found the bug, validated the fix, and did not have to re-deploy our cluster. Excellent work!

Benefits of Hot Code Reload

Skyramp's Hot Code Reload feature brings several benefits to your development and debugging workflow:

Time Savings
Say goodbye to the time-consuming process of re-deploying your application every time you make a code change. Hot Code Reload lets you make changes on the fly, saving you valuable development time.

Enhanced Productivity
With real-time debugging and interactive code changes, you can identify and fix issues more efficiently. This increased productivity helps you deliver high-quality software faster.

Reduced Downtime
Hot Code Reload minimizes service downtime for your system-under-test by allowing you to fix issues without restarting the entire application. This is particularly useful for services that must maintain continuous availability in a shared environment.

Improved Collaboration
Debugging is more accessible and collaborative with Hot Code Reload. Team members can work together to troubleshoot and enhance your services in real-time.

Flexible Testing
Quickly test new features or experiments by applying changes while your service is running. This flexibility encourages experimentation and innovation.

Summing It Up

In conclusion, Skyramp's Hot Code Reload feature is a game-changer for developers. It empowers you to debug your services in real-time, eliminate the need for re-deployments, and significantly accelerate your development and testing workflow. Simply configuring Skyramp Deployer and following the steps outlined in this blog post, you can start using Hot Code Reload to turbocharge your debugging process for distributed applications today. Happy debugging!

All your Tests and Mocks in One Place with Skyramp Dashboard

Dan Gross — Tue, 10 Oct 2023 17:32:17 +0000

Skyramp gives you the tools you need to make integration and performance testing for distributed applications easier. One way Skyramp helps you save time and be more productive is with the Skyramp Dashboard. The Skyramp Dashboard allows you to view the details of your test results, see your active mocks, and track all of your test runs.

In this post, we will take a deep dive into using the Skyramp Dashboard for testing microservices in a Kubernetes cluster. If you're a developer or devops engineer looking to streamline your testing workflow for distributed applications, you've come to the right place.

Setting Up

Before we get started, let's make sure you have a suitable testing environment in place.

Install the Skyramp CLI

First things first, you need to have the Skyramp Terminal Client installed. If you haven't already done this, you can find the install steps here.

Register your Kubernetes Cluster

Next, you will need to have a Kubernetes cluster registered with Skyramp. You have two choices. From the Skyramp CLI, you can either: (1) register an existing cluster with skyramp cluster register, or (2) create a new cluster with skyramp cluster create. More details about these Skyramp CLI Commands can be found here.

Tip: For an easy way to follow along with this post, you can create a local Kubernetes cluster by stepping through this simple walkthrough. The tests and mocks referenced in this post will use the examples from the walkthrough. As shown in the steps from the walkthrough, we can create a local Kubernetes cluster from the sample project that is registered with Skyramp by running this command:

skyramp cluster create --local

Once you have a Kubernetes cluster registered with Skyramp, you can double-check the current cluster by running:

skyramp cluster current

This will produce output like:

current: kind-cluster
kubeconfig: ~/.skyramp/kind-cluster.kubeconfig

With your testing environment in place, let's move on to deploying the Skyramp Dashboard to the cluster.

Bringing Up the Dashboard

To deploy the Skyramp Dashboard to your currently registered Kubernetes cluster, execute the following command:

skyramp dashboard up

This command automatically creates the Kubernetes assets for the dashboard like the client, the server, and a MongoDB Kubernetes operator within the Skyramp namespace of your cluster. It also sets up port forwarding for local access to the dashboard.

Once the dashboard is ready, the terminal client will provide you with a URL for access and then open the dashboard page in a new browser tab. The terminal output should look something like this:

The dashboard is accessible here: http://localhost:50326/testresults
Port forwarding is started for the dashboard, hit Ctrl+c (or Cmd+c) to stop

Browsing the Dashboard

Using your browser to access the URL provided, you should see the Skyramp Dashboard main page. If you're starting from scratch without any test runs, you will just see an empty page with no tests. Once you run some tests, you will see a list of tests like this:

As you can see, this list shows the start and stop times of the tests along with the status of the results. The tests in the list above were run from the examples in the walkthrough referenced earlier. One such test from the walkthrough that we are using is a test of the checkout system, which can be run with Tester like this:

skyramp tester start checkout-test

Try it yourself!

Viewing Individual Test Results

Once you have a list of tests in the dashboard, you can click on each test to view the details, including the specific test results for that run of the test.

In this case, we are viewing the test results from a checkout test run that passed. You can expand the output to review the details. Here, our second assert that checks the quantity matched the output from the service and so there is no error.

Viewing Load Tests

You can also use the Skyramp Dashboard to view load test results, which have their own characteristics. Here is the test output from a load test we ran, again using an example from the walkthrough:

For load tests, the results will show stats such as: latency numbers, error rates, requests per second (RPS), and service utilization. The graphs above are autogenerated based on the metrics from the test run.

See your Active Mocks

Finally, the Skyramp Dashboard also displays any active mocks that have been applied by Mocker in the cluster. If you mocked the payment service from the walkthrough, as an example, you will see that mock in place as shown below:

If you have multiple mocks applied, you will see a list of all your active mocks.

Stopping the Dashboard

To stop the Skyramp Dashboard and clean up resources, run the following command:

skyramp dashboard down

The dashboard will then terminate. You can always bring the dashboard back with skyramp dashboard up.

Summary

Congratulations! You are now equipped with the knowledge to use the Skyramp Dashboard on your Kubernetes cluster, making testing easier and saving you time. Whether it's tracking test results or managing active mocks, Skyramp Dashboard simplifies the process, making your development workflow smoother than ever.

If you have any questions or run into any issues, refer to the official Skyramp Dashboard documentation for further assistance. Happy testing!

Skyramp Feature Highlight: Using Dynamic Python for Tests and Mocks

Dan Gross — Tue, 03 Oct 2023 17:15:31 +0000

Python developers, rejoice! Skyramp now supports Python code for dynamically testing and mocking your distributed applications. Previously, Skyramp supported only JavaScript for dynamic requests and responses within test and mock description files. In this blog post, we'll walk you through Skyramp's new capability of employing dynamic Python code in requests and responses for testing and mocking microservices, right from a running Kubernetes cluster.

Unpacking Skyramp Tests

Before diving into the step-by-step example, let's first lay the foundation by understanding what constitutes a Skyramp test. At the heart of every Skyramp test is a test description file. This file can either be auto-generated using skyramp tester generate or created manually as a .yaml file.

What's Inside a Test Description?

Within a Skyramp test description file, you define the essential components of your test. This includes the scenarios for testing, the services to be tested, the requests you'll make to them, the expected responses in the form of asserts, and any custom logic or handlers to process these interactions. Let's dissect the important elements.

Here's a simple scenario in a test description:

scenarios:
  - name: scenario1
    steps:
    - request: request1
    - asserts: requests.request1.res.message == "abc"

The scenario above simply tests if the message returned from a service is equal to "abc". Below is an example request to support the scenario above. We can assume this hypothetical service takes an id and returns a message. Both of these YAML sections are contained within a Skyramp test description file.

requests:
  - name: request1
    endpoint: our-service
    method: GetMessage
    requestValue:
      blob: |
        {
            "id": "123"
        }

Notice the block under requestValue. It is configured as a static blob in this example, but it can also be configured as dynamic code that the Skyramp Worker will execute during a test. The language used in a dynamic handler had been limited to JavaScript, but now you can use Python! Let's look at how.

A Hands-on Example: Testing with Python Code

To illustrate how Python can now be harnessed in Skyramp tests, let's embark on a simple journey you can follow yourself. We'll clone a sample project from a Skyramp-provided repo on GitHub, which conveniently contains microservices that can be swiftly deployed to a Kubernetes cluster, all primed and ready for testing.

Paving the Way with the Skyramp CLI

First, make sure you have the Skyramp CLI installed. The Skyramp CLI enables the Deployer, Mocker, and Tester tools. You can follow these easy steps from the Skyramp Docs for installation.

Cloning the Project

Start with cloning Skyramp's letsramp/sample-microservices repo, which is a sample e-commerce system that includes a number of microservices. See the GitHub repo Readme for additional information. The Skyramp repo is based on Google Cloud's Online Boutique, adding test demos and support for REST and Thrift APIs.

From a terminal, you can execute this command:

git clone https://github.com/letsramp/sample-microservices.git

Start your Cluster

Now that the project is in place, you can navigate to the grpc-demo directory in your terminal and bring up a local cluster for testing by executing these three commands:

cd sample-microservices/skyramp/grpc-demo
skyramp cluster create --local
skyramp deployer up checkout-system

Putting Python to the Test

With the system-under-test up and running, let's edit one of the tests to show where we can insert Python code that will be run dynamically during testing. The tests/checkout-test.yaml test description file contains a requests section with a static blob. Edit the section under requestValue for request1 as follows:

requests:
  - name: request1
    endpoint: cart-service
    method: AddItem
    requestValue:
      python: |
        def handler():
          uid = "abcde"
          qty = 5 * 5
          pid = "OLJCE"
          pid += "SPC7Z"
          request_value = {
            "userId": uid,
            "item": {
                "productId": pid,
                "quantity": qty
            }
          }
          return SkyrampValue(value=request_value)

You can see we replaced the blob attribute with python and are now creating the request values in code. To learn more about the specific details of how you can use Python in test descriptions, visit the Skyramp Docs that cover Python Dynamic Requests.

You will notice that the tests/checkout-test.yaml test description file has two requests. The first request, shown above as request1, adds an item to the cart. The second request, request2, places an order. We will leave the requestValue in request2 as a static blob.

For our test scenario under scenarios, we first make a request to request1 and then make a request to request2. The existing assert checks that the productId matches the first item returned in the response from request2. Let's also add an asserts line in the same file to check the quantity value returned. This should match the quantity we set in request1. Notice the type is a number and not a string.

scenarios:
  - name: scenario1
    steps:
      - request: request1
      - request: request2
      - asserts: requests.request2.res.order.items[0].item.productId == "OLJCESPC7Z"
      - asserts: requests.request2.res.order.items[0].item.quantity == 25

We are almost ready to run our new test scenario with Python, but we are missing one final piece. We know our checkout depends on the payment service, but it is not running in our system-under-test. Luckily, we have a mock for that.

Yes, Mocks Too!

Not only is Python available for dynamic requests in the test description, but Python is also available for dynamic responses in the mock description.

Open mocks/payment-service-k8s.yaml for editing, which is also under the skyramp/grpc-demo folder in the letsramp/sample-microservices repo.

Change the section under responseValues to include the python attribute and the code shown below:

responseValues:
    - name: ChargeResponse
      python: |
        def handler(req):
          key = "transaction_id"
          value = "default-string"
          return SkyrampValue(value={key: value})

Here, we now return values from Python-generated variables instead of a static blob as before. To learn more about the specific details of how you can use Python in mock descriptions, visit the Skyramp Docs that cover
Python Dynamic Responses

Mocking and Testing with Skyramp

With our new test description and mock description updated to include dynamic Python, we can proceed with our test session. Skyramp makes running tests a breeze.

First, apply the mock that was updated with Python code:

skyramp mocker apply -n test-grpc-demo

You should see this output in the terminal:

Applying mock config     [##############################################################] 100 %
Successfully applied mock configurations.

Next, run the test that was updated with Python code:

skyramp tester start checkout-test -n test-grpc-demo

The test results should produce the following output:

 - pattern0.scenario1.2.assert
    [Status: finished] [Started at: N/A]
    Assert: requests.request2.res.order.items[0].item.productId == "OLJCESPC7Z"
    Passed: true
  - pattern0.scenario1.3.assert
    [Status: finished] [Started at: N/A]
    Assert: requests.request2.res.order.items[0].item.quantity == 25
    Passed: true

Nice job, you did it! This was just a simple taste to show how to use Python for tests and mocks, but there is much more to explore. Additional capabilities include:

Utilizing the req parameter for incoming requests (see docs)
Using the pythonPath attribute for external source code files (see docs)
Installing NPM-based packages
Chaining and Overrides

Wrapping Up

That concludes our exploration of using Python code in Skyramp tests and mocks. Now Python developers can leverage their skills to create comprehensive and robust testing scenarios. We hope you've enjoyed this glimpse into using Python for dynamic requests and responses in testing distributed applications with Skyramp.

Stay tuned for more updates and enhancements as Skyramp continues to evolve, enabling developers to optimize their testing processes like never before. Happy testing!

Test Your Microservices with Skyramp in IntelliJ IDEA

Dan Gross — Fri, 08 Sep 2023 19:42:10 +0000

Hello devs! This is Dan from Skyramp. When we say Skyramp meets devs in any workflow, we mean it. In this blog, we will demonstrate iterative testing and development for microservices in a local Kubernetes cluster performed directly from the IntelliJ IDEA with Skyramp. It’s as easy as 1, 2, 3.

What Are We Testing?

This blog features the Skyramp fork of Google’s popular cloud-native microservices demo app, Online Boutique. Online Boutique is a web-based e-commerce app containing microservices that mimic real-world services, such as a product catalog, shopping cart, ad service, recommendation service, payment service, and others. The services use gRPC APIs by default, but Skyramp has also added support for REST and Thrift APIs.

This walkthrough will focus on the ad service component of the system to demonstrate how you can easily generate test descriptions and run tests during the development stage using Skyramp. The ad service is a microservice written in Java, so IntelliJ IDEA is a perfect fit for our development environment.

Pre-Flight Checklist

Here’s what you need in order to follow along with this walkthrough:

Install Docker
Install the Skyramp Terminal Client
Install JetBrains IntelliJ IDEA and download a JDK
Clone the Skyramp project, sample-microservices, to a folder of your choice. From a terminal, run:

git clone https://github.com/letsramp/sample-microservices.git

Let’s Get Testing

First, start the IntelliJ IDEA you installed and open the Skyramp sample-microservices project from the local folder where it was cloned. Navigate to the src/adservice/src/main/java/hipstershop folder and open the AdService.java file to view the ad service source code.

Next, open the Terminal pane in the IDE by clicking the Terminal button in the lower left corner. You can run all the skyramp commands from this Terminal. Change to the Skyramp demo directory for gRPC in the terminal and check your installed version of skyramp:

cd skyramp/grpc-demo
skyramp version

At this point, you should see something like this:

Next, we will create a local Kubernetes cluster for deploying and testing the microservices. This is easily done with the Skyramp client:

skyramp cluster create --local

Deploy all the services to the cluster that will make up our system under test using Skyramp Deployer:

skyramp deployer up grpc-app

Once the services are deployed, you can check the status of the services, also with Skyramp Deployer:

skyramp deployer status

Now we will generate a test description for the ad service using Skyramp Tester. This will allow us to execute tests against the running service.

skyramp tester generate --protocol protobuf \
--api-schema pb/demo.proto --alias ad-service \
--port 9555 --proto-service AdService

How do you know what values to provide to Skyramp Tester in order to generate a test description? In this case, we know we will be using protobuf as the protocol for our distributed application. We already have a proto file defined in the repo at pb/demo.proto, so we can utilize that for the api-schema flag. The AdService is specified in the proto file, which is the service we want to test, so that should be used for the proto-service flag. The values for the alias and port flags are derived from the deployment status as seen in the screenshot above.

Once you run the Skyramp Tester command above, the generated test description file will be under the tests folder called ad-service.yaml. Next, we simply need to define our specific scenario for testing.

The ad service behavior we want to test is whether the service returns exactly two ads from its response payload, and if those ads are non-zero length. Furthermore, we want to test that the two ads are not the same, because we do not want duplicate ads returned. We can define all of this by adding a scenarios section with asserts in the file as seen below:

Here is the scenario section with the asserts to add to the YAML file if you would like to copy and paste:

scenarios:
 - name: scenario1
   steps:
     - request: EAdServiceMGetAds
     - asserts: requests.EAdServiceMGetAds.res.ads.length == 2
     - asserts: requests.EAdServiceMGetAds.res.ads[0].text.length >= 1
     - asserts: requests.EAdServiceMGetAds.res.ads[1].text.length >= 1
     - asserts: requests.EAdServiceMGetAds.res.ads[0].text != requests.EAdServiceMGetAds.res.ads[1].text

Now save the file. Running the test is easy with Skyramp Tester. From the same skyramp/grpc-demo directory, just execute:

skyramp tester start ad-service -n test-grpc-demo

Notice from the test results that the first three asserts passed, but the last assert failed. We appear to be getting duplicate ads from our ad service. A side note: this failed test may take several tries to surface, since the ads returned are randomized and may not always be the same.

If we navigate back to the source file and examine the code, we see that getRandomAds() does not account for duplicates.

We can fix that with a LinkedHashSet to force uniqueness. Here is the new getRandomAds() source with a few lines changed:

private List<Ad> getRandomAds() {
  List<Ad> ads = new ArrayList<>(MAX_ADS_TO_SERVE);
  Collection<Ad> allAds = adsMap.values();
  Set<Ad>set = new LinkedHashSet<Ad>();
  while (set.size() < MAX_ADS_TO_SERVE) {
    set.add(Iterables.get(allAds, random.nextInt(allAds.size())));
  }
  ads.addAll(set);
  return ads;
}

We update the code in the editor and then rebuild the ad service from src/adservice in a new terminal pane by running:

cd src/adservice
./gradlew installDist

Once the ad service is rebuilt, simply bring all the services down and back up again with Skyramp Deployer:

skyramp deployer down grpc-app
skyramp deployer up grpc-app

Now, when you re-run the test with Skyramp Tester, all asserts will pass because the ad service no longer produces duplicate ads in the payload.

skyramp tester start ad-service -n test-grpc-demo

Once you are finished with your test session, you can remove the test cluster completely with Skyramp:

skyramp deployer down grpc-app 
skyramp cluster remove --local

Wrapping Up

And there you have it. You can use this setup to explore more capabilities Skyramp provides with Mocker, Tester, and Deployer. We give you all the tools you need to mock, test, and deploy your own microservices from IntelliJ IDEA easily with Skyramp.

Happy testing!

Have questions or comments? Join the Skyramp Community Discord.

Debugging FreeRTOS with QEMU in VSCode

Dan Gross — Wed, 15 Feb 2023 23:06:15 +0000

In this post, I will step through an example setup that shows how to debug a FreeRTOS application in the Visual Studio Code (VSCode) editor using QEMU to emulate an Arm Cortex-M3 microcontroller. No actual hardware is required for this example, except a laptop or desktop as your development environment.

First off, what is QEMU and why would we want to debug with it? QEMU is a machine emulator that allows you to virtualize hardware types, even across different architectures. This can be very helpful for embedded development because you can run applications against hardware targets that you may not have immediate access to. This might be because (a) you are still in the process of selecting a chipset for an embedded product design, (b) your boards are somewhere else like a lab, (c) you want to run automated tests without physical hardware, or (d) the real hardware is simply in short supply. If you recognize any of these constraints, and the software development must go on, then QEMU can help.

QEMU supports many different hardware platforms, from x86 to ARM to RISC-V. FreeRTOS also supports many different hardware platforms. So, it is worth pointing out that the example I will cover below is not limited to just this hardware platform for FreeRTOS. You can adapt the same methods if you have a target other than the Arm Cortex-M3.

With those considerations, how do we begin? There are a number of prerequisites and dependencies in order to get up and running with the example. I have laid these out in 7 steps below, which you can follow. The order is not that important except the last step should stay the last step. You may have some or all of these tools installed on your development machine already, so just double-check that they are up-to-date and present in your path if that is the case.

Download and install VSCode from here.
Download and install QEMU from here.
Download and install the Arm GNU Toolchain from here.
Download and install CMake from here.
For Windows, download and install Git (includes Bash terminal) from here. Git should already be installed on MacOS and Linux.
Make sure 'make' is installed on your system. There are different ways to do this depending on your platform.

a. On Windows, you can install 'make' using Chocolatey at the Command Prompt with this command: > choco install make

b. On MacOS, you can install 'make' using Homebrew in the Terminal app with this command $ brew install make

c. On Linux, you can install 'make' using the package manager from a terminal with this command: $ sudo apt install build-essential
Add the QEMU, Arm GNU Compiler, CMake, and 'make' installation paths to your PATH environment variable. OSes and environments have different ways to do this, so use the appropriate method for your system.

Once you have the prerequisite tools installed, proceed by cloning the source code we will be working with. From the command line, navigate to a folder where you would like to clone the FreeRTOS repository with the demo code, such as $ cd $HOME/Projects as an example. Then, go ahead and clone the FreeRTOS repository from GitHub like this:

$ git clone https://github.com/FreeRTOS/FreeRTOS.git --recurse-submodules
When cloning the repo has finished, launch VSCode. In VSCode, select 'File > Open Folder' in the menu. Navigate to the FreeRTOS repository you just cloned and select this subfolder: '.../FreeRTOS/FreeRTOS/Demo/CORTEX_MPS2_QEMU_IAR_GCC'

After VScode loads the demo folder, open '.vscode/launch.json' in the editor. Find the miDebuggerPath parameter and change the value to the path where 'arm-none-eabi-gdb' is located on your machine.

Now open 'main.c' and make sure that mainCREATE_SIMPLE_BLINKY_DEMO_ONLY is set to 1. This will generate only the simple blinky demo.

Next, press the 'Run and Debug' button from the left side panel in VSCode. Select 'Launch QEMU RTOSDemo' from the dropdown at the top and press the 'play' button. This will build the code, run the program, and attach the debugger.

From there, you can 'Continue', 'Step Over', 'Step Into', 'Step Out', and 'Stop' from the button bar. You can also add breakpoints in the code by right clicking next to the line number.

Pressing the 'Continue' button will execute the code, which will produce output like this in the Terminal:

QEMU RTOSdemo started
Message received from task
Message received from task
Message received from task
Message received from task
Message received from task
Message received from task
Message received from task
Message received from task
Message received from task
Message received from software timer
...

And there you have it, debugging FreeRTOS with QEMU in VSCode. You can use this example to play around and explore what is possible. Happy debugging!