Instrumentation for Laravel: visualize operational data and performance metrics

Offloading tasks for background processing is such a great resource in software engineering. And a good job queue does the job of abstracting this complex setup in a safe and maintainable way.

But as the number of jobs and data grows around the queue, we might wonder how much information we have about what is going on in the queue workers.

What instrumentation can answer

• How many jobs ran last hour?
• What were the lengthiest tasks during that period of time?
• When did that job stop processing?
• What's the job run rate (how many per hour) and what's its average, max and minimum run times?

Those questions will introduce themselves especially during performance incidents, system malfunctions, data loss, etc - so imagine how useful it is to have an informed dataset about the system health.

Horizon: mileage may vary

For the Laravel Queue there is the good Horizon Dashboard, which is free and readily integrates to redis queues. But if you need customized dashboards or to smash data in order to answers more questions then custom instrumentation comes handy.

Determining what to instrument

For in-house instrumentation where we consume the data ourselves I've been targeting to gather initially:

• name of the procedure (the Job class name in this case)
• time spent processing
• procedure params
• caught exception description

There is a tendency to log out as much information as possible. For optimal instrumentation, don't do that. Include in the instrumentation payload only what's most relevant for the system operation.

Ingestion and data analysis

Instrumentation data will be really useful only with log ingestion and visualization tools. There are many projects out there in the market, such as the open-source Elastic Stack (logstash + kibana) and the paid service LogDNA.

Laravel Queue Instrumentation: a implementation

Over the past year, I've solved lots of performance and data loss situations with a similar approach. Above I'm sharing a gist with a sample Laravel Queue instrumentation.

How about your experience, have you ever instrumented system runtime? What tools and process do your team makes use?

Comments

[Sergey Telpuk]

Hello, look at spiral/framework, You will be very surprised because that one has lots of features out-of-the-box which simplify your developer's life.

Features

• Battle-tested since 2013
• Lightning fast full-stack PHP framework
• PSR-{2,3,4,6,7,11,15,16,17} compliant
• Powerful application server and resident memory application kernel
• Native support of queue (AMQP, SQS, Beanstalk) and background PHP workers
• GRPC server and client
• Pub/Sub, event broadcasting
• HTTPS, HTTP/2+Push, encrypted cookies, sessions, CSRF-guard
• MySQL, MariaDB, SQLite, Postgres, SQLServer support, auto-migrations
• The ORM you will use for the next 25 years
• Intuitive scaffolding and prototyping (it literally writes code for you)
• Helpful class discovery via static analysis
• Authentication, RBAC security, validation, and encryption
• Dynamic template engine to create your own HTML tags (or just use Twig)
• MVC, HMVC, CQRS, Queue-oriented, RPC-oriented, CLI apps... any apps

[Deonte Jones]

I need your help if you can show me how to recover my money u accidentally sent litecoin to bitcoin