DEV Community: Timber Staff

PromQL For Humans

Timber Staff — Wed, 24 Oct 2018 20:05:25 +0000

PromQL is a built in query-language made for Prometheus. Here at Timber we've found Prometheus to be awesome, but PromQL difficult to wrap our heads around. This is our attempt to change that.

Basics

Instant Vectors

Only Instant Vectors can be graphed.

http_requests_total

This gives us all the http requests, but we've got 2 issues.

There are too many data points to decipher what's going on.
You'll notice that http_requests_total only goes up, because it's a counter. These are common in Prometheus, but not useful to graph.

I'll show you how to approach both.

It's easy to filter by label.

http_requests_total{job="prometheus", code="200"}

You can check a substring using regex matching.

http_requests_total{status_code=~"2.*"}

If you're interested in learning more, here are the docs on Regex.

Range Vectors

Contain data going back in time.

Recall: Only Instant Vectors can be graphed. You'll soon be able to see how to visualize Range Vectors using functions.

http_requests_total[5m]

You can also use (s, m, h, d, w, y) to represent (seconds, minutes, hours, ...) respectively.

Important functions

For Range Vectors

You'll notice that we're able to graph all these functions. Since only Instant Vectors can be graphed, they take a Range Vector as a parameter and return a Instant Vector.

Increase of `http_requests_total` averaged over the last 5 minutes.

rate(http_requests_total[5m])

Irate

Looks at the 2 most recent samples (up to 5 minutes in the past), rather than averaging like rate

irate(http_requests_total[5m])

It's best to use rate when alerting, because it creates a smooth graph since the data is averaged over a period of time. Spikey graphs can cause alert overload, fatigue, and bad times for all due to repeatedly triggering thresholds.

HTTP Requests in the last hour.

This is equal to the rate * # of seconds

increase(http_requests_total[1h])

These are a small fraction of the functions, just what we found most popular. You can find the rest here.

For Instant Vectors

Broken by Status Code

sum(rate(http_requests_total[5m]))

You'll notice that rate(http_requests_total[5m]) above provides a large amount of data. You can filter that data using your labels, but you can also look at your system as a whole using sum (or do both).

You can also use min, max, avg, count, and quantile similarly.

This query tells you how many total HTTP requests there are, but isn't directly useful in deciphering issues in your system. I'll show you some functions that allow you to gain insight into your system.

Sum by Status Code

sum by (status_code) (rate(http_requests_total[5m]))

You can also use without rather than by to sum on everything not passed as a parameter to without.

Now, you can see the difference between each status code.

Offset

You can use offset to change the time for Instant and Range Vectors. This can be helpful for comparing current usage to past usage when determining the conditions of an alert.

sum(rate(http_requests_total[5m] offset 5m))

Remember to put offset directly after the selector.

Operators

Operators can be used between scalars, vectors, or a mix of the two. Operations between vectors expect to find matching elements for each side (also known as one-to-one matching), unless otherwise specified.

There are Arithmetic (+, -, *, /, %, ^), Comparison (==, !=, >, <, >=, <=) and Logical (and, or, unless) operators.

Vector Matching

One-to-One

Vectors are equal i.f.f. the labels are equal.

API 5xxs are 10% of HTTP Requests

rate(http_requests_total{status_code=~"5.*"}[5m]) > .1 * rate(http_requests_total[5m])

We're looking to graph whenever more than 10% of an instance's HTTP requests are errors. Before comparing rates, PromQL first checks to make sure that the vector's labels are equal.

You can use on to compare using certain labels or ignoring to compare on all labels except.

Many-to-One

It's possible to use comparison and arithmetic operations where an element on one side can be matched with many elements on the other side. You must explicitly tell Prometheus what to do with the extra dimensions.

You can use group_left if the left side has a higher cardinality, else use group_right.

Examples

Disclaimer: We've hidden some of the information in the pictures using the Legend Format for privacy reasons.

CPU Usage by Instance

100 * (1 - avg by(instance)(irate(node_cpu{mode='idle'}[5m])))

Average CPU Usage per instance for a 5 minute window.

Memory Usage

node_memory_Active / on (instance) node_memory_MemTotal

Percentage of memory being used by instance.

Disk Space

node_filesystem_avail{fstype!~"tmpfs|fuse.lxcfs|squashfs"} / node_filesystem_size{fstype!~"tmpfs|fuse.lxcfs|squashfs"}

Percentage of disk space being used by instance. We're looking for the available space, ignoring instances that have tmpfs, fuse.lxcfs, or squashfs in their fstype and dividing that by their total size.

HTTP Error Rates as a % of Traffic

rate(http_requests_total{status_code=~"5.*"}[5m]) / rate(http_requests_total[5m])

Alerts Firing in the last 24 hours

sum(sort_desc(sum_over_time(ALERTS{alertstate="firing"}[24h]))) by (alertname)

You can find more useful examples here.

3 Pillars of Observability

It's important to understand where metrics fit in when it comes to observing your application. I recommend you take a look at the 3 pillars of observability principle. Metrics are an important part of your observability stack, but logs and tracing are equally so.

We're a cloud-based logging company at Timber that seamlessly augments your logs with context. We've got a great product built, and you can check it out for free!

A Gentle Introduction to GraphQL with Elixir and Phoenix

Timber Staff — Wed, 24 Oct 2018 20:04:49 +0000

Getting a working GraphQL application!

Latest Versions

Elixir: v1.6.5
Hex: v0.17.7
Phoenix: v1.3.2
Ecto: v2.2.10 (phoenix_ecto version is v3.2)
Absinthe: v1.4
Absinthe Plug: v1.4

This tutorial was built on a Mac OS X 10.11.6 system.

Assumptions

You understand a minimum of Elixir syntax and concepts
You already have a development database ready to go
You have all of the pre-requisite software installed. Pre-reqs are:
- Postgresql with an appropriate development username/password (Installation Guide)
- Elixir (Installation Guide)
- Phoenix (Installation Guide)
- A code editor or IDE (I'll be using Visual Studio Code)

About this guide

This is a guest post bought to you by your friends @ Timber. Reach out on Twitter if you're interested in writing for us.

Throughout this guide, you'll see a lot of useful stuff, whether it's descriptions, code, or shell commands! Any time you see some code like this:

$ do the thing

These blocks represents shell commands being run (specifically, anything with a $ in front of it). Any additional lines will be output from the command run. Sometimes ... will be used to indicate long blocks of text cut out in interest of brevity.

IEx (Elixir's Interactive REPL) has commands will be represented like this:

iex(1)> "do the thing"

And any additional lines will be output from that operation. Don't worry about the number inside of the parantheses. These just indicate what command number your IEx shell is currently on and these may not match up.

Code will be represented in code blocks, like this:

name = "brandon"
name
|> String.upcase()
|> String.reverse()

The best way to get the most out of this tutorial is to follow along as you build the same project as us (or something equivalent). The code will be available on Github with checkpoint tags at each step allowing you to double-check your code against the finished product per each checkpoint!

Moving on to the application

We'll start off by actually talking about what problem this project is designed to solve. It's always good to think about what you're building before you start thinking about what you're building with; it prevents some chicken-and-egg scenarios where you have a great technology that you're using and the project you're trying to build with it is totally incompatible.

The project we're building is going to be an application that will store/retrieve Event Logs. You could use this for something like tracking requests you're making, or tracking audit events in a log, or...well, anything that would require storing some arbitrary events with types, messages, and payloads. We'll start off by actually creating our project and getting everything set up before we start talking more about the design.

Next, our tech choice: Elixir and Phoenix are absolutely fantastic for building extremely high performance, low maintenance systems, and GraphQL's ability to fetch huge sets of data in varied ways makes Elixir a particularly great symbiotic fit!

Our project will be called "Greenfy" short for "Green Faerie" (yes, a silly Absinthe reference), but slightly more startup-sounding.

Creating a new project

This is what happened:

    $ mix phx.new greenfy

    Fetch and install dependencies? [Yn] y
    ...
    We are all set! Go into your application by running:

    $ cd greenfy

    Then configure your database in config/dev.exs and run:

    $ mix ecto.create

    Start your Phoenix app with:

    $ mix phx.server

    You can also run your app inside IEx (Interactive Elixir) as:

    $ iex -S mix phx.server

Run our tests, verify green.

Setting up our app

Next, we'll want to follow the instructions given to us by Phoenix upon creating a new project, so we'll need to do the following:

$ cd greenfy
$ mix ecto.create
The database for Greenfy.Repo has been created

You should already have your database set up with a user account that can create and modify databases as necessary, so when you run the mix ecto.create command, Ecto will helpfully create the development database for us to run our migrations on later!

Adding in Absinthe

We know we're going to be building a GraphQL project, so we're going to add Absinthe nice and early in the process. We'll need to add both libraries for absinthe and absinthe_plug (we'll need absinthe_plug since we're using Phoenix to build out our application).

Open up mix.exs and add the following line to the defp deps do function block:

    {:absinthe, "~> 1.4"},
    {:absinthe_plug, "~> 1.4"}

Make sure you add a comma at the end of what was previously the last line in that list of tuples or you'll get a syntax error! At the top, in the block for the application function, you'll want to add in support for Absinthe Plug via adding one more atom into the extra_applications list, :absinthe_plug. The resulting function body should look like this:

  def application do
    [
      mod: {Greenfy.Application, []},
      extra_applications: [:logger, :runtime_tools, :absinthe_plug]
    ]
  end

Then run:

    $ mix do deps.get, compile
    ...
    Generated greenfy app

You may be wondering why we're adding both absinthe and absinthe_plug. The reason for this is that first of all, the absinthe dependency handles most of the scenarios we need handled for a smooth integration between GraphQL and Elixir,

Creating our first GraphQL query

So, you're probably coming into this tutorial already having an inkling of what GraphQL is and what it does and what it provides for both the developer and the end-user. A pitfall I've run into repeatedly over the course of my career is that most REST APIs over time become less RESTful. There are more and more exceptions you need to build into your application, and some of your endpoints become too bloated and they're difficult to modify out or...

Well, lots of things happen. Especially if you have a mobile or client application that is reliant on your data! The shape of what you might fetch for a desktop client is likely going to look radically different from what you would fetch for a mobile client, especially on a slower connection.

GraphQL recognizes two things:

The shape of your data will change
The shape of your data needs to be different for each client

Based on that, GraphQL implements more of a client-focused method of getting data out of a system. The client will tell the server what data it needs, and more importantly, HOW it needs that data to be shaped! This allows you to iterate a little faster, to build better filters and functionality, and easily add support for more data in your system without having to build/version a whole brand new endpoint that you will also need to support over time.

Another awesome feature is that GraphQL supports querying for schema information, which means your users can do more to figure out the shape of the data and what they can do with it, which is incredibly helpful for both the client AND the server!

GraphQL is not perfect, by any means. It's pretty complicated and you'll likely run into a lot of people with lots of REST experience not understanding why a particular piece of a GraphQL query works the way it does. You'll need to train your users to be able to use this new API effectively, but the end result is allowing the client to get whatever they want, however they want it!

With that past us, let's move on to talk more about the actual implementation.

Designing the shape of our data

Our application is a simple event logging platform, so we're just going to have a very simple table structure (to start).

events
___
id
event_type
message
payload
inserted_at
updated_at

id, inserted_at, and updated_at will all be included as default columns for any Ecto tables we create, so we don't need to worry about how we're going to define those.

For the event_type, payload, and message columns, we're going to use those to track what types of events (unsurprisingly) each event is and message will store a condensed version of the payload. Separately, we have another column called payload, whose responsibility is to store the full payload when necessary. This allows the message column to stay smaller and easier to search.

Getting our database tables running in Ecto

Let's get started by building out this table using Phoenix's built-in generators:

$ mix phx.gen.context Log Events events event_type:string message:string payload:text

creating lib/greenfy/log/events.ex
creating priv/repo/migrations/20180614191428_create_events.exs
creating lib/greenfy/log/log.ex
injecting lib/greenfy/log/log.ex
creating test/greenfy/log/log_test.exs
injecting test/greenfy/log/log_test.exs

Remember to update your repository by running migrations:

    $ mix ecto.migrate

Next, let's run the migrate command:

$ mix ecto.migrate
[info] == Running Greenfy.Repo.Migrations.CreateEvents.change/0 forward
[info] create table events
[info] == Migrated in 0.0s

This is actually enough for us to start building out the GraphQL portion of our application!

Building support for our first query in GraphQL

Let's take a look at what a sample query for our Log.Events schema might look like:

{
    events {
        id
        event_type
        message
        payload
        inserted_at
        updated_at
    }
}

To make this work, we'll need to define both the Types and the Schemas for GraphQL to understand how to turn data in the database, and then data in Elixir data structures, into GraphQL data structures. We'll start off by defining the types. Create a new file, under lib/greenfy_web/schema, called data_types.ex (you'll need to create the schema directory), and we'll start defining our first object type:

defmodule Greenfy.Schema.DataTypes do
  use Absinthe.Schema.Notation

  object :event do
    field :id, :id
    field :event_type, :string
    field :message, :string
    field :payload, :string
  end
end

You'll probably notice that :inserted_at and :updated_at are not being included in our data types notation yet! The reason for this is that those are not easily represented by the standard built-in types for Absinthe, so we'll need to define some fancy scalar types later! (Don't worry, we'll describe this all in great detail later!)

Sidebar: GraphQL Definitions

We're using A LOT of terms here pretty liberally when talking about GraphQL but not doing a lot to explain what they actually are, so let's take a really quick sidebar talking about some of the terms we've discussed so far (and the ones we'll be doing later).

First off, we have Types. Types are exactly what they sound like: definitions of the various data types we'll be defining for our GraphQL application. A good example of this is the object definition above: An event is a GraphQL object type, with four fields defined on it so far (we'll get to fields next).

A Field is essentially a bit of queryable information. An object, like the data type we defined previously, stores a collection of different fields together. Each field should define, at a minimum, its name (or key) and its type.

Next, we'll be working with Schemas. If Types define what data is queryable in our GraphQL API and the shape of that data, the Schema defines how we actually get that data OUT of our GraphQL API.

Later on, you'll see something called Resolvers. If the Schema tells the client how to query the data out of our application, the Resolver tells the server how to react to that query and how to interpret those GraphQL queries.

Finally, this will be something we'll use much later, but you'll hear it mentioned before that point: Mutations! Mutations are how the data in our GraphQL application is modified! Instead of your POST, PUT, and PATCH endpoints in your REST API, you just define Mutations instead.

Back to our implementation

Next, we'll need to create our Schema. Create lib/greenfy_web/schema.ex and give it the following contents:

defmodule Greenfy.Schema do
  use Absinthe.Schema

  import_types Greenfy.Schema.DataTypes

  query do
    @desc "Get a list of events"
    field :events, list_of(:event) do
      resolve fn _parent, _args, _resolution ->
        {:ok, Greenfy.Log.list_events()}
      end
    end
  end
end

We use the provided Absinthe.Schema macro to build out our Schema body. Next, we define our query body, which starts off with a @desc statement, which sets a module variable called desc (unsurprisingly) which describes the general use for this query. Think of it as a bit of documentation for the client.

Next, in our query, we have a field. Again, our field is queryable data, so we say that our "query" has a key in it called events. We're also telling Absinthe that we should expect events to return a list of events, so we use the function list_of(:event) (note the singular version of event and that it is an atom, here). This tells Absinthe that when someone queries events, we should be fetching a list of the Data Type Event that we defined in our Data Types file earlier.

Finally, we have a Resolver, which tells Absinthe that when a client queries for a list of events, this function's return statement is what data should be passed back to the client (after some transformation, of course). A resolvestatement takes in three arguments; the parent, the args (the arguments to our query, so whatever we might be looking for or filtering on), and the resolution, which tells us what to do when we've fetched all of the data and what to do with that data at the end! We'll build on this later by moving our resolvers off into their own code, but for right now we're going to stick with a simple implementation.

One final step before we can start testing

Before we get too deep, though, let's modify our router so we can start testing out our application and refactoring as appropriate. Open up lib/greenfy_web/router.ex, delete the following block:

  scope "/" do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
  end

Uncomment out the /api scope section, and remove the mention of the Application module, "Greenfy" from the scope block. Then, we'll want to add the GraphiQL route to give us a playground to test out our application. The scope statement should look something like this:

  scope "/api" do
    pipe_through :api

    forward "/graphiql", Absinthe.Plug.GraphiQL, schema: Greenfy.Schema
  end

And finally, testing it out

In your terminal, start up your server by running:

$ iex mix -S phx.server

And then open up your browser window and point it at http://localhost:4000/api/graphiql. You should see a window that looks like this:

You can see in this screenshot that we have our initial query running, selecting the primary key, eventType (note the difference from event_type; GraphQL expects JavaScript-style Camel-case whereas Elixir by default expects snake_case style. Absinthe handles this conversion for us!) We're not selecting the inserted_at and updated_at columns yet; we still have time to get to that later!

Before You Go

Just wanted to catch you before you finished reading to let you know we're a cloud-based logging company that's looking to make debugging easier by seamlessly augmenting logs with context. We've got a great product built and you can check it out for free!

Summary

So now we're at a point where we have a base GraphQL API and support for VERY limited operations to get the data out! We understand a bit more about GraphQL, the terminology used, and how it all fits together, so where do we go from here?

Easy: we'll make this application more robust! For one thing, we can't verify this works super well unless we can also get data into it. In addition, testing is already hard, so how on earth are we going to test something so completely malleable?

Stay tuned! We have a lot more to talk about, a lot more to understand, and hopefully we can eventually land at being experts at implementing GraphQL APIs in Elixir.

Following along? You can check your work against the tutorial repo at Github. Tag/Release v0.1.0 is the complete version for this phase of the tutorial.

Decorators in Python: What you need to know

Timber Staff — Fri, 10 Aug 2018 20:07:54 +0000

Python decorators are a powerful concept that allow you to "wrap" a function with another function.

The idea of a decorator is to abstract away something that you want a function or class to do, besides its normal responsibility. This can be helpful for many reasons such as code reuse, and sticking to curlys law.

By learning how to write your own decorators, you can significantly improve readability of your own code. They can change how the function behaves, without needing to actually change the code (such as adding logging lines). They are a fairly common tool in python, familiar to those who use frameworks such as flask or click, but many only know how to use them, not how to write their own.

This is a guest-post brought to you by your friends @ Timber. If you're interested in writing for us, feel free to reach out on Twitter.

How it works

First off, let's show an example of a decorator in python. The following is a very basic example of what a decorator would like like if you were using it.

@my_decorator
def hello():
    print('hello')

When you define a function in python, that function becomes an object.

The function hello above is a function object. The @my_decorator is actually a function that has the ability to use the hello object, and return a different object to the interpreter. The object that the decorator returns, is what becomes known as hello. Essentially, it's the same thing as if you were going to write your own normal function, such as: hello = decorate(hello). Decorate is passed the function -- which it can use however it wants -- then returns another object. The decorator has the ability to swallow the function, or return something that is not a function, if it wanted.

Writing your own decorator

As mentioned, a decorator is simply a function that is passed a function, and returns an object. So, to start writing a decorator, we just need to define a function.

def my_decorator(f):
    return 5

Any function can be used as a decorator. In this example the decorator is passed a function, and returns a different object. It simply swallows the function it is given entirely, and always returns 5.

@my_decorator
def hello():
    print('hello')

>>> hello()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable
'int' object is not callable

Because our decorator is returning an int, and not a callable, it can not be called as a function. Remember, the decorator's return value replaces hello.

>>> hello
5

In most cases, we want the object our decorator returns, to actually mimic the function we decorated. This means that the object the decorator returns, needs to be a function itself. For example, let's say we wanted to simply print every time the function was called, we could write a function that prints that information, then calls the function. But that function would need to be returned by the decorator. This usually leads to function nesting such as:

def mydecorator(f):  # f is the function passed to us from python
    def log_f_as_called():
        print(f'{f} was called.')
        f()
    return log_f_as_called

As you can see, we have a nested function definition, and the decorator function is returning that function. This way, the function hello can still be called like a standard function, the caller does not need to know that it is decorated. We can now define hello as the following:

@mydecorator
def hello():
    print('hello')

We get the following output:

>>> hello()
<function hello at 0x7f27738d7510> was called.
hello

(Note: the number inside the <function hello at 0x7f27738d7510> reference will be different for everyone, it represents the memory address)

Wrapping functions correctly

A function can be decorated many times, if desired. In this case, the decorators cause a chain effect. Essentially, the top decorator is passed the object from the former, and so on and so forth. For example, if we have the following code:

@a
@b
@c
def hello():
    print('hello')

The interpreter is essentially doing hello = a(b(c(hello))) and all of the decorators would wrap each other. You can test this out yourself by using our existing decorator, and using it twice.

@mydecorator
@mydecorator
def hello():
    print('hello')

>>> hello()
<function mydec.<locals>.a at 0x7f277383d378> was called.
<function hello at 0x7f2772f78ae8> was called.
hello

You will notice that the first decorator, wrapped the second, and printed it separately.

One interesting thing you might have noticed here, is that first line printed said <function mydec.<locals>.a at 0x7f277383d378> instead of what the second line printed, which is what you would expect: <function hello at 0x7f2772f78ae8>. The is because the object returned by the decorator is a new function, not called hello. This is fine for our trivial example, but can often break tests and things that might be trying to introspect the function attributes. If the idea is for a decorator to act like the function it decorates, it needs to also mimic that function. Luckily, there is a python standard library decorator called wraps for that in functools module.

import functools
def mydecorator(f): 
    @functools.wraps(f)  # we tell wraps that the function we are wrapping is f
    def log_f_as_called():
        print(f'{f} was called.')
        f()
    return log_f_as_called

@mydecorator
@mydecorator
def hello():
    print('hello')

>>> hello()
<function hello at 0x7f27737c7950> was called.
<function hello at 0x7f27737c7f28> was called.
hello

Now, our new function appears just like the one its wrapping/decorating. However, we are still reliant on the fact that it returns nothing, and accepts no input. If we wanted to make that more general, we would need to pass in the arguments and return the same value. We can modify our function to look like this:

import functools
def mydecorator(f): 
    @functools.wraps(f)  # wraps is a decorator that tells our function to act like f
    def log_f_as_called(*args, **kwargs):
        print(f'{f} was called with arguments={args} and kwargs={kwargs}')
        value = f(*args, **kwargs)
        print(f'{f} return value {value}')
        return value
    return log_f_as_called

Now we are printing everytime the function is called, including all the inputs the function receives, and what it returns. Now you can simply decorate any existing function and get debug logging on all its inputs and outputs without needed to manually write the logging code.

Adding variables to the decorator

If you were using the decorator for any code that you wanted to ship, and not just for yourself locally, you might want to replace all the print statements with logging statements. In which case, you would need to define a log level. It might be safe to assume you always want the debug log level, but what that also might depend on the function. We can provide variables to the decorator itself, to define how it should behave. For example:

@debug(level='info')
def hello():
    print('hello')

The above code would allow us to specify that this specific function should log at the info level instead of the debug level. This is achieved in python by writing a function, that returns a decorator. Yes, a decorator is also a function. So this is essentially saying hello = debug('info')(hello). That double parenthesis might look funky, but basically, debug is function, that returns a function. Adding this to our existing decorator, we would need to nest one more time, now making our code look like the following:

import functools
def debug(level): 
    def mydecorator(f)
        @functools.wraps(f)
        def log_f_as_called(*args, **kwargs):
            logger.log(level, f'{f} was called with arguments={args} and kwargs={kwargs}')
            value = f(*args, **kwargs)
            logger.log(level, f'{f} return value {value}')
            return value
        return log_f_as_called
    return mydecorator

The changes above turn debug into a function, which returns a decorator which uses the correct logging level. This is starting to get a bit ugly, and overly nested. A cool little trick to solve that, which I like to do, is simply add a default kwarg level to debug and return a partial. A partial is a "non-complete function call" that includes a function and some arguments, so that they are passed around as one object without actually calling the function yet.

import functools
def debug(f=None, *, level='debug'): 
    if f is None:
        return functools.partial(debug, level=level)
    @functools.wraps(f)   # we tell wraps that the function we are wrapping is f
    def log_f_as_called(*args, **kwargs):
        logger.log(level, f'{f} was called with arguments={args} and kwargs={kwargs}')
        value = f(*args, **kwargs)
        logger.log(level, f'{f} return value {value}')
        return value
    return log_f_as_called

Now the decorator can work as normal:

@debug
def hello():
    print('hello')

And use debug logging. Or, you could override the log level:

@debug('warning')
def hello():
    print('hello')

Logging to the Cloud

Writing your logs to a hosted log aggregation service makes your life easier, so you can focus on building better software, not debugging.

Disclaimer: I’m a current employee @ Timber. This section is entirely optional, but I sincerely believe that logging to the cloud will make your life easier (and you can try it for completely free).

pip install timber

import logging
import timber

logger = logging.getLogger(__name__)

timber_handler = timber.TimberHandler(api_key='...')
logger.addHandler(timber_handler)

That’s it. All you need to do is get your API key from timber.io, and you’ll be able to see your logs. We automatically capture them from the logging module (and seamlessly add context), so you can continue to log normally.

An Intro to Web Scraping with lxml and Python

Timber Staff — Mon, 11 Jun 2018 14:59:26 +0000

Why should you even bother learning how to web scrape? If your job doesn't require you to learn it, then let me give you some motivation. What if you want to create a website which curates cheapest products from Amazon, Walmart and a couple of other online stores? A lot of these online stores don't provide you with an easy way to access their information using an API. In the absence of an API, your only choice is to create a web scraper which can extract information from these websites automatically and provide you with that information in an easy to use way.

Here is an example of a typical API response in JSON. This is the response from Reddit:

There are a lot of Python libraries out there which can help you with web scraping. There is lxml, BeautifulSoup and a full-fledged framework called Scrapy. Most of the tutorials discuss BeautifulSoup and Scrapy, so I decided to go with lxml in this post. I will teach you the basics of XPaths and how you can use them to extract data from an HTML document. I will take you through a couple of different examples so that you can quickly get up-to-speed with lxml and XPaths.

If you are a gamer, you will already know of (and likely love) this website. We will be trying to extract data from Steam. More specifically, we will be selecting from the "popular new releases" information. I am converting this into a two-part series. In this part, we will be creating a Python script which can extract the names of the games, the prices of the games, the different tags associated with each game and the target platforms. In the second part, we will turn this script into a Flask based API and then host it on Heroku.

Step 1: Exploring Steam

First of all, open up the "popular new releases" page on Steam and scroll down until you see the Popular New Releases tab. At this point, I usually open up Chrome developer tools and see which HTML tags contain the required data. I extensively use the element inspector tool (The button in the top left of the developer tools). It allows you to see the HTML markup behind a specific element on the page with just one click. As a high-level overview, everything on a web page is encapsulated in an HTML tag and tags are usually nested. You need to figure out which tags you need to extract the data from and you are good to go. In our case, if we take a look, we can see that every separate list item is encapsulated in an anchor (a) tag.

The anchor tags themselves are encapsulated in the div with an id of tab_newreleases_content. I am mentioning the id because there are two tabs on this page. The second tab is the standard "New Releases" tab, and we don't want to extract information from that tab. Hence, we will first extract the "Popular New Releases" tab, and then we will extract the required information from this tag.

Step 2: Start writing a Python script

This is a perfect time to create a new Python file and start writing down our script. I am going to create a scrape.py file. Now let's go ahead and import the required libraries. The first one is the requests library and the second one is the lxml.html library.

import requests
import lxml.html

If you don't have requests installed, you can easily install it by running this command in the terminal:

$ pip install requests

The requests library is going to help us open the web page in Python. We could have used lxml to open the HTML page as well but it doesn't work well with all web pages so to be on the safe side I am going to use requests.

Now let's open up the web page using requests and pass that response to lxml.html.fromstring.

html = requests.get('https://store.steampowered.com/explore/new/')
doc = lxml.html.fromstring(html.content)

This provides us with an object of HtmlElement type. This object has the xpath method which we can use to query the HTML document. This provides us with a structured way to extract information from an HTML document.

Step 3: Fire up the Python Interpreter

Now save this file and open up a terminal. Copy the code from the scrape.py file and paste it in a Python interpreter session.

We are doing this so that we can quickly test our XPaths without continuously editing, saving and executing our scrape.py file.

Let's try writing an XPath for extracting the div which contains the 'Popular New Releases' tab. I will explain the code as we go along:

new_releases = doc.xpath('//div[@id="tab_newreleases_content"]')[0]

This statement will return a list of all the divs in the HTML page which have an id of tab_newreleases_content. Now because we know that only one div on the page has this id we can take out the first element from the list ([0]) and that would be our required div. Let's break down the xpath and try to understand it:

// these double forward slashes tell lxml that we want to search for all tags in the HTML document which match our requirements/filters. Another option was to use / (a single forward slash). The single forward slash returns only the immediate child tags/nodes which match our requirements/filters
div tells lxml that we are searching for divs in the HTML page
[@id="tab_newreleases_content"] tells lxml that we are only interested in those divs which have an id of tab_newreleases_content

Cool! We have got the required div. Now let's go back to chrome and check which tag contains the titles of the releases.

Step 4: Extract the titles & prices

The title is contained in a div with a class of tab_item_name. Now that we have the "Popular New Releases" tab extracted we can run further XPath queries on that tab. Write down the following code in the same Python console which we previously ran our code in:

titles = new_releases.xpath('.//div[@class="tab_item_name"]/text()')

This gives us with the titles of all of the games in the "Popular New Releases" tab. Here is the expected output:

Let's break down this XPath a little bit because it is a bit different from the last one.

. tells lxml that we are only interested in the tags which are the children of the new_releases tag
[@class="tab_item_name"] is pretty similar to how we were filtering divs based on id. The only difference is that here we are filtering based on the class name
/text() tells lxml that we want the text contained within the tag we just extracted. In this case, it returns the title contained in the div with the tab_item_name class name

Now we need to extract the prices for the games. We can easily do that by running the following code:

prices = new_releases.xpath('.//div[@class="discount_final_price"]/text()')

I don't think I need to explain this code as it is pretty similar to the title extraction code. The only change we made is the change in the class name.

Step 5: Extracting tags

Now we need to extract the tags associated with the titles. Here is the HTML markup:

Write down the following code in the Python terminal to extract the tags:

tags = new_releases.xpath('.//div[@class="tab_item_top_tags"]')
total_tags = []
for tag in tags:
    total_tags.append(tag.text_content())

So what we are doing here is that we are extracting the divs containing the tags for the games. Then we loop over the list of extracted tags and then extract the text from those tags using the text_content() method. text_content() returns the text contained within an HTML tag without the HTML markup.

Note: We could have also made use of a list comprehension to make that code shorter. I wrote it down in this way so that even those who don't know about list comprehensions can understand the code. Eitherways, this is the alternate code:

tags = [tag.text_content() for tag in new_releases.xpath('.//div[@class="tab_item_top_tags"]')]

Lets separate the tags in a list as well so that each tag is a separate element:

tags = [tag.split(', ') for tag in tags]

Step 6: Extracting the platforms

Now the only thing remaining is to extract the platforms associated with each title. Here is the HTML markup:

The major difference here is that the platforms are not contained as texts within a specific tag. They are listed as the class name. Some titles only have one platform associated with them like this:

<span class="platform_img win"></span>

While some titles have 5 platforms associated with them like this:

<span class="platform_img win"></span>
<span class="platform_img mac"></span>
<span class="platform_img linux"></span>
<span class="platform_img hmd_separator"></span>
<span title="HTC Vive" class="platform_img htcvive"></span>
<span title="Oculus Rift" class="platform_img oculusrift"></span>

As we can see these spans contain the platform type as the class name. The only common thing between these spans is that all of them contain the platform_img class. First of all, we will extract the divs with the tab_item_details class, then we will extract the spans containing the platform_img class and finally we will extract the second class name from those spans. Here is the code:

platforms_div = new_releases.xpath('.//div[@class="tab_item_details"]')
total_platforms = []

for game in platforms_div:
    temp = game.xpath('.//span[contains(@class, "platform_img")]')
    platforms = [t.get('class').split(' ')[-1] for t in temp]
    if 'hmd_separator' in platforms:
        platforms.remove('hmd_separator')
    total_platforms.append(platforms)

In line 1 we start with extracting the tab_item_details div. The XPath in line 5 is a bit different. Here we have [contains(@class, "platform_img")] instead of simply having [@class="platform_img"]. The reason is that [@class="platform_img"] returns those spans which only have the platform_img class associated with them. If the spans have an additional class, they won't be returned. Whereas [contains(@class, "platform_img")] filters all the spans which have the platform_img class. It doesn't matter whether it is the only class or if there are more classes associated with that tag.

In line 6 we are making use of a list comprehension to reduce the code size. The .get() method allows us to extract an attribute of a tag. Here we are using it to extract the class attribute of a span. We get a string back from the .get() method. In case of the first game, the string being returned is platform_img win so we split that string based on the comma and the whitespace, and then we store the last part (which is the actual platform name) of the split string in the list.

In lines 7-8 we are removing the hmd_separator from the list if it exists. This is because hmd_separator is not a platform. It is just a vertical separator bar used to separate actual platforms from VR/AR hardware.

Step 7: Conclusion

This is the code we have so far:

import requests
import lxml.html

html = requests.get('https://store.steampowered.com/explore/new/')
doc = lxml.html.fromstring(html.content)

new_releases = doc.xpath('//div[@id="tab_newreleases_content"]')[0]

titles = new_releases.xpath('.//div[@class="tab_item_name"]/text()')
prices = new_releases.xpath('.//div[@class="discount_final_price"]/text()')

tags = [tag.text_content() for tag in new_releases.xpath('.//div[@class="tab_item_top_tags"]')]
tags = [tag.split(', ') for tag in tags]

platforms_div = new_releases.xpath('.//div[@class="tab_item_details"]')
total_platforms = []

for game in platforms_div:
    temp = game.xpath('.//span[contains(@class, "platform_img")]')
    platforms = [t.get('class').split(' ')[-1] for t in temp]
    if 'hmd_separator' in platforms:
        platforms.remove('hmd_separator')
    total_platforms.append(platforms)

Now we just need this to return a JSON response so that we can easily turn this into a Flask based API. Here is the code:

output = []
for info in zip(titles,prices, tags, total_platforms):
    resp = {}
    resp['title'] = info[0]
    resp['price'] = info[1]
    resp['tags'] = info[2]
    resp['platforms'] = info[3]
    output.append(resp)

This code is self-explanatory. We are using the zip function to loop over all of those lists in parallel. Then we create a dictionary for each game and assign the title, price, tags, and platforms as a separate key in that dictionary. Lastly, we append that dictionary to the output list.

In a future post, we will take a look at how we can convert this into a Flask based API and host it on Heroku.

This article was written by Yasoob from Python Tips. I hope you guys enjoyed this tutorial. If you want to read more tutorials of a similar nature, please go to Python Tips. I regularly write Python tips, tricks, and tutorials on that blog. And if you are interested in learning intermediate Python, then please check out my open source book here.

Have a great day!

Just a disclaimer: we're a logging company here @ Timber. We'd love it if you tried out our product (it's seriously great!), but that's all we're going to advertise it.