DEV Community

Cover image for Artillery -- Quick check your site's  performance

Posted on

Artillery -- Quick check your site's performance

Jmeter is a great performance testing tool but has a steep learning curve and could take long to set up. Sometimes, there is just not enough time for that. Come in Artillery !

Quick facts:

  • Scripts written in YAML: which greatly reduces the code knowledge you need to have to set up tests.
  • Designed for testing backend systems, such as API services, e-commerce backends, chat systems, game backends, databases, message brokers and queues, and anything else that can be communicated with over a network.
  • Can't test frontends
  • Two options: Artillery Core (free) and Artillery Pro (paid)
  • Functional and load testing could be performed in one package
  • Artillery Pro is used to run the performance tests on the Cloud i.e., AWS.

Simple installation using NPM:

npm install -g artillery@1.6
Enter fullscreen mode Exit fullscreen mode

A sample test

This test (let's say it's called test-artillery.yml) :

       Warm-up : creates 1 virtual user each second for 5 seconds
       Ramp-up : creates 1 virtual user per second ramping up to 5 virtual users per second for 1 minute (60 seconds)
       Ramp-down : ramps down the number of concurrent virtual users to 0 over 15 seconds

  • asserts the aggregate p95 (95th percentile) latency is 200ms or less, and that the maximum error rate was less than 1%.
  • includes a scenario which is used to test a virtual user flow
config :
  target: ""
    expect : {}
    - duration: 5
      arrivalRate : 1
      name : Warm-up
    - duration: 60 
      arrivalRate: 1
      rampTo: 5
      name: Ramp up load
    - duration: 15
      arrivalRate: 1
      rampTo: 0
      name: Kill
    p95: 200
    maxErrorRate: 1
  - name: 'test scenario'
      - get: 
          url: "/vacation.html"
            - statusCode: 200
      - think: 2
Enter fullscreen mode Exit fullscreen mode

Run the test

artillery run test-artillery.yml
Enter fullscreen mode Exit fullscreen mode

A section of the full console output

Screen Shot 2021-03-18 at 12.41.11 PM

Console output is not the most easily understood report of results so Artillery has a built-in html reporting feature which is easier on the eyes.

First, create the report:

artillery run --output report.json test-artillery.yml
Enter fullscreen mode Exit fullscreen mode

Sample JSON generated

Screen Shot 2021-03-18 at 8.46.42 PM

Next, display that JSON as HTML:

artillery report --output report.html report.json
Enter fullscreen mode Exit fullscreen mode

Sample HTML report

There is a ton of other stuff that Artillery can do including more complex user scenarios, running large-scale load tests, adding synthetic traffic in production to maintain a margin of safety against traffic spikes etc.
This post only skimmed over Artillery Core; Artillery Pro which is a paid service, provides a seamless upgrade path from tests running on a developer’s machine, to scaling up & running the same test scripts from your organization’s AWS account.

What's to like about Artillery

  • Scripts written in YAML; easy to learn
  • Easy to setup and good documentation
  • Good starting point for performance evaluation

What could improve

Could do with more options for reporting; it currently supports Datadog (via agent or HTTP API), StatsD and InfluxDB with Telegraf + StatsD plugin while others like Prometheus are in progress.

Happy testing!

Top comments (1)

reevosolutions profile image

very useful