Artillery -- Quick check your site's performance

#performance #beginners #node #testing

Jmeter is a great performance testing tool but has a steep learning curve and could take long to set up. Sometimes, there is just not enough time for that. Come in Artillery !

Quick facts:

Scripts written in YAML: which greatly reduces the code knowledge you need to have to set up tests.
Designed for testing backend systems, such as API services, e-commerce backends, chat systems, game backends, databases, message brokers and queues, and anything else that can be communicated with over a network.
Can't test frontends
Two options: Artillery Core (free) and Artillery Pro (paid)
Functional and load testing could be performed in one package
Artillery Pro is used to run the performance tests on the Cloud i.e., AWS.

Simple installation using NPM:

npm install -g artillery@1.6

A sample test

This test (let's say it's called test-artillery.yml) :

runs against the URL: https://blazedemo.com/,
follows a phased ramp-up/ramp-down set of steps:

       Warm-up : creates 1 virtual user each second for 5 seconds
       Ramp-up : creates 1 virtual user per second ramping up to 5 virtual users per second for 1 minute (60 seconds)
       Ramp-down : ramps down the number of concurrent virtual users to 0 over 15 seconds

asserts the aggregate p95 (95th percentile) latency is 200ms or less, and that the maximum error rate was less than 1%.
includes a scenario which is used to test a virtual user flow

config :
  target: "https://blazedemo.com/"
  plugins: 
    expect : {}
  phases:
    - duration: 5
      arrivalRate : 1
      name : Warm-up
    - duration: 60 
      arrivalRate: 1
      rampTo: 5
      name: Ramp up load
    - duration: 15
      arrivalRate: 1
      rampTo: 0
      name: Kill
  ensure:
    p95: 200
    maxErrorRate: 1
scenarios:
  - name: 'test scenario'
    flow:
      - get: 
          url: "/vacation.html"
          expect: 
            - statusCode: 200
      - think: 2

Run the test

artillery run test-artillery.yml

A section of the full console output

Console output is not the most easily understood report of results so Artillery has a built-in html reporting feature which is easier on the eyes.

First, create the report:

artillery run --output report.json test-artillery.yml

Sample JSON generated

Next, display that JSON as HTML:

artillery report --output report.html report.json

Sample HTML report

There is a ton of other stuff that Artillery can do including more complex user scenarios, running large-scale load tests, adding synthetic traffic in production to maintain a margin of safety against traffic spikes etc.
This post only skimmed over Artillery Core; Artillery Pro which is a paid service, provides a seamless upgrade path from tests running on a developer’s machine, to scaling up & running the same test scripts from your organization’s AWS account.

What's to like about Artillery

Scripts written in YAML; easy to learn
Easy to setup and good documentation
Good starting point for performance evaluation

What could improve

Could do with more options for reporting; it currently supports Datadog (via agent or HTTP API), StatsD and InfluxDB with Telegraf + StatsD plugin while others like Prometheus are in progress.

Happy testing!