DEV Community: Timber

Vector - A High-Performance Logs & Metrics Router Written In Rust

Ben Johnson — Tue, 02 Jul 2019 16:00:43 +0000

Hi everyone, we just released Vector, an open-source logs and metrics collector built in Rust.

vectordotdev / vector

A high-performance observability data pipeline.

Quickstart • Docs • Guides • Integrations • Chat • Download • Rust Crate Docs

What is Vector?

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data Collect, transform, and route all your logs and metrics to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.

To get started, follow our quickstart guide or install Vector.

Principles

Reliable - Built in Rust, Vector's primary design goal is reliability.
End-to-end - Deploys as an agent or aggregator. Vector is a complete platform.
Unified - Logs, metrics (beta), and traces (coming…

View on GitHub

Basically, Vector rides along with your app collecting useful data (logs and metrics) and forwards it to a service of your choice. Ex: Elasticsearch, S3, CloudWatch Logs, and so on. This decouples your app from any specific vendor, giving you control, data ownership, and flexibility. Not to mention it's quite a bit faster :)

The docs and website provide a deeper dive:

Github: https://github.com/timberio/vector
Docs: https://docs.vector.dev/
Website: https://vector.dev

Let us know what you think!

PromQL For Humans

Timber Staff — Wed, 24 Oct 2018 20:05:25 +0000

PromQL is a built in query-language made for Prometheus. Here at Timber we've found Prometheus to be awesome, but PromQL difficult to wrap our heads around. This is our attempt to change that.

Basics

Instant Vectors

Only Instant Vectors can be graphed.

http_requests_total

This gives us all the http requests, but we've got 2 issues.

There are too many data points to decipher what's going on.
You'll notice that http_requests_total only goes up, because it's a counter. These are common in Prometheus, but not useful to graph.

I'll show you how to approach both.

It's easy to filter by label.

http_requests_total{job="prometheus", code="200"}

You can check a substring using regex matching.

http_requests_total{status_code=~"2.*"}

If you're interested in learning more, here are the docs on Regex.

Range Vectors

Contain data going back in time.

Recall: Only Instant Vectors can be graphed. You'll soon be able to see how to visualize Range Vectors using functions.

http_requests_total[5m]

You can also use (s, m, h, d, w, y) to represent (seconds, minutes, hours, ...) respectively.

Important functions

For Range Vectors

You'll notice that we're able to graph all these functions. Since only Instant Vectors can be graphed, they take a Range Vector as a parameter and return a Instant Vector.

Increase of `http_requests_total` averaged over the last 5 minutes.

rate(http_requests_total[5m])

Irate

Looks at the 2 most recent samples (up to 5 minutes in the past), rather than averaging like rate

irate(http_requests_total[5m])

It's best to use rate when alerting, because it creates a smooth graph since the data is averaged over a period of time. Spikey graphs can cause alert overload, fatigue, and bad times for all due to repeatedly triggering thresholds.

HTTP Requests in the last hour.

This is equal to the rate * # of seconds

increase(http_requests_total[1h])

These are a small fraction of the functions, just what we found most popular. You can find the rest here.

For Instant Vectors

Broken by Status Code

sum(rate(http_requests_total[5m]))

You'll notice that rate(http_requests_total[5m]) above provides a large amount of data. You can filter that data using your labels, but you can also look at your system as a whole using sum (or do both).

You can also use min, max, avg, count, and quantile similarly.

This query tells you how many total HTTP requests there are, but isn't directly useful in deciphering issues in your system. I'll show you some functions that allow you to gain insight into your system.

Sum by Status Code

sum by (status_code) (rate(http_requests_total[5m]))

You can also use without rather than by to sum on everything not passed as a parameter to without.

Now, you can see the difference between each status code.

Offset

You can use offset to change the time for Instant and Range Vectors. This can be helpful for comparing current usage to past usage when determining the conditions of an alert.

sum(rate(http_requests_total[5m] offset 5m))

Remember to put offset directly after the selector.

Operators

Operators can be used between scalars, vectors, or a mix of the two. Operations between vectors expect to find matching elements for each side (also known as one-to-one matching), unless otherwise specified.

There are Arithmetic (+, -, *, /, %, ^), Comparison (==, !=, >, <, >=, <=) and Logical (and, or, unless) operators.

Vector Matching

One-to-One

Vectors are equal i.f.f. the labels are equal.

API 5xxs are 10% of HTTP Requests

rate(http_requests_total{status_code=~"5.*"}[5m]) > .1 * rate(http_requests_total[5m])

We're looking to graph whenever more than 10% of an instance's HTTP requests are errors. Before comparing rates, PromQL first checks to make sure that the vector's labels are equal.

You can use on to compare using certain labels or ignoring to compare on all labels except.

Many-to-One

It's possible to use comparison and arithmetic operations where an element on one side can be matched with many elements on the other side. You must explicitly tell Prometheus what to do with the extra dimensions.

You can use group_left if the left side has a higher cardinality, else use group_right.

Examples

Disclaimer: We've hidden some of the information in the pictures using the Legend Format for privacy reasons.

CPU Usage by Instance

100 * (1 - avg by(instance)(irate(node_cpu{mode='idle'}[5m])))

Average CPU Usage per instance for a 5 minute window.

Memory Usage

node_memory_Active / on (instance) node_memory_MemTotal

Percentage of memory being used by instance.

Disk Space

node_filesystem_avail{fstype!~"tmpfs|fuse.lxcfs|squashfs"} / node_filesystem_size{fstype!~"tmpfs|fuse.lxcfs|squashfs"}

Percentage of disk space being used by instance. We're looking for the available space, ignoring instances that have tmpfs, fuse.lxcfs, or squashfs in their fstype and dividing that by their total size.

HTTP Error Rates as a % of Traffic

rate(http_requests_total{status_code=~"5.*"}[5m]) / rate(http_requests_total[5m])

Alerts Firing in the last 24 hours

sum(sort_desc(sum_over_time(ALERTS{alertstate="firing"}[24h]))) by (alertname)

You can find more useful examples here.

3 Pillars of Observability

It's important to understand where metrics fit in when it comes to observing your application. I recommend you take a look at the 3 pillars of observability principle. Metrics are an important part of your observability stack, but logs and tracing are equally so.

We're a cloud-based logging company at Timber that seamlessly augments your logs with context. We've got a great product built, and you can check it out for free!

A Gentle Introduction to GraphQL with Elixir and Phoenix

Timber Staff — Wed, 24 Oct 2018 20:04:49 +0000

Getting a working GraphQL application!

Latest Versions

Elixir: v1.6.5
Hex: v0.17.7
Phoenix: v1.3.2
Ecto: v2.2.10 (phoenix_ecto version is v3.2)
Absinthe: v1.4
Absinthe Plug: v1.4

This tutorial was built on a Mac OS X 10.11.6 system.

Assumptions

You understand a minimum of Elixir syntax and concepts
You already have a development database ready to go
You have all of the pre-requisite software installed. Pre-reqs are:
- Postgresql with an appropriate development username/password (Installation Guide)
- Elixir (Installation Guide)
- Phoenix (Installation Guide)
- A code editor or IDE (I'll be using Visual Studio Code)

About this guide

This is a guest post bought to you by your friends @ Timber. Reach out on Twitter if you're interested in writing for us.

Throughout this guide, you'll see a lot of useful stuff, whether it's descriptions, code, or shell commands! Any time you see some code like this:

$ do the thing

These blocks represents shell commands being run (specifically, anything with a $ in front of it). Any additional lines will be output from the command run. Sometimes ... will be used to indicate long blocks of text cut out in interest of brevity.

IEx (Elixir's Interactive REPL) has commands will be represented like this:

iex(1)> "do the thing"

And any additional lines will be output from that operation. Don't worry about the number inside of the parantheses. These just indicate what command number your IEx shell is currently on and these may not match up.

Code will be represented in code blocks, like this:

name = "brandon"
name
|> String.upcase()
|> String.reverse()

The best way to get the most out of this tutorial is to follow along as you build the same project as us (or something equivalent). The code will be available on Github with checkpoint tags at each step allowing you to double-check your code against the finished product per each checkpoint!

Moving on to the application

We'll start off by actually talking about what problem this project is designed to solve. It's always good to think about what you're building before you start thinking about what you're building with; it prevents some chicken-and-egg scenarios where you have a great technology that you're using and the project you're trying to build with it is totally incompatible.

The project we're building is going to be an application that will store/retrieve Event Logs. You could use this for something like tracking requests you're making, or tracking audit events in a log, or...well, anything that would require storing some arbitrary events with types, messages, and payloads. We'll start off by actually creating our project and getting everything set up before we start talking more about the design.

Next, our tech choice: Elixir and Phoenix are absolutely fantastic for building extremely high performance, low maintenance systems, and GraphQL's ability to fetch huge sets of data in varied ways makes Elixir a particularly great symbiotic fit!

Our project will be called "Greenfy" short for "Green Faerie" (yes, a silly Absinthe reference), but slightly more startup-sounding.

Creating a new project

This is what happened:

    $ mix phx.new greenfy

    Fetch and install dependencies? [Yn] y
    ...
    We are all set! Go into your application by running:

    $ cd greenfy

    Then configure your database in config/dev.exs and run:

    $ mix ecto.create

    Start your Phoenix app with:

    $ mix phx.server

    You can also run your app inside IEx (Interactive Elixir) as:

    $ iex -S mix phx.server

Run our tests, verify green.

Setting up our app

Next, we'll want to follow the instructions given to us by Phoenix upon creating a new project, so we'll need to do the following:

$ cd greenfy
$ mix ecto.create
The database for Greenfy.Repo has been created

You should already have your database set up with a user account that can create and modify databases as necessary, so when you run the mix ecto.create command, Ecto will helpfully create the development database for us to run our migrations on later!

Adding in Absinthe

We know we're going to be building a GraphQL project, so we're going to add Absinthe nice and early in the process. We'll need to add both libraries for absinthe and absinthe_plug (we'll need absinthe_plug since we're using Phoenix to build out our application).

Open up mix.exs and add the following line to the defp deps do function block:

    {:absinthe, "~> 1.4"},
    {:absinthe_plug, "~> 1.4"}

Make sure you add a comma at the end of what was previously the last line in that list of tuples or you'll get a syntax error! At the top, in the block for the application function, you'll want to add in support for Absinthe Plug via adding one more atom into the extra_applications list, :absinthe_plug. The resulting function body should look like this:

  def application do
    [
      mod: {Greenfy.Application, []},
      extra_applications: [:logger, :runtime_tools, :absinthe_plug]
    ]
  end

Then run:

    $ mix do deps.get, compile
    ...
    Generated greenfy app

You may be wondering why we're adding both absinthe and absinthe_plug. The reason for this is that first of all, the absinthe dependency handles most of the scenarios we need handled for a smooth integration between GraphQL and Elixir,

Creating our first GraphQL query

So, you're probably coming into this tutorial already having an inkling of what GraphQL is and what it does and what it provides for both the developer and the end-user. A pitfall I've run into repeatedly over the course of my career is that most REST APIs over time become less RESTful. There are more and more exceptions you need to build into your application, and some of your endpoints become too bloated and they're difficult to modify out or...

Well, lots of things happen. Especially if you have a mobile or client application that is reliant on your data! The shape of what you might fetch for a desktop client is likely going to look radically different from what you would fetch for a mobile client, especially on a slower connection.

GraphQL recognizes two things:

The shape of your data will change
The shape of your data needs to be different for each client

Based on that, GraphQL implements more of a client-focused method of getting data out of a system. The client will tell the server what data it needs, and more importantly, HOW it needs that data to be shaped! This allows you to iterate a little faster, to build better filters and functionality, and easily add support for more data in your system without having to build/version a whole brand new endpoint that you will also need to support over time.

Another awesome feature is that GraphQL supports querying for schema information, which means your users can do more to figure out the shape of the data and what they can do with it, which is incredibly helpful for both the client AND the server!

GraphQL is not perfect, by any means. It's pretty complicated and you'll likely run into a lot of people with lots of REST experience not understanding why a particular piece of a GraphQL query works the way it does. You'll need to train your users to be able to use this new API effectively, but the end result is allowing the client to get whatever they want, however they want it!

With that past us, let's move on to talk more about the actual implementation.

Designing the shape of our data

Our application is a simple event logging platform, so we're just going to have a very simple table structure (to start).

events
___
id
event_type
message
payload
inserted_at
updated_at

id, inserted_at, and updated_at will all be included as default columns for any Ecto tables we create, so we don't need to worry about how we're going to define those.

For the event_type, payload, and message columns, we're going to use those to track what types of events (unsurprisingly) each event is and message will store a condensed version of the payload. Separately, we have another column called payload, whose responsibility is to store the full payload when necessary. This allows the message column to stay smaller and easier to search.

Getting our database tables running in Ecto

Let's get started by building out this table using Phoenix's built-in generators:

$ mix phx.gen.context Log Events events event_type:string message:string payload:text

creating lib/greenfy/log/events.ex
creating priv/repo/migrations/20180614191428_create_events.exs
creating lib/greenfy/log/log.ex
injecting lib/greenfy/log/log.ex
creating test/greenfy/log/log_test.exs
injecting test/greenfy/log/log_test.exs

Remember to update your repository by running migrations:

    $ mix ecto.migrate

Next, let's run the migrate command:

$ mix ecto.migrate
[info] == Running Greenfy.Repo.Migrations.CreateEvents.change/0 forward
[info] create table events
[info] == Migrated in 0.0s

This is actually enough for us to start building out the GraphQL portion of our application!

Building support for our first query in GraphQL

Let's take a look at what a sample query for our Log.Events schema might look like:

{
    events {
        id
        event_type
        message
        payload
        inserted_at
        updated_at
    }
}

To make this work, we'll need to define both the Types and the Schemas for GraphQL to understand how to turn data in the database, and then data in Elixir data structures, into GraphQL data structures. We'll start off by defining the types. Create a new file, under lib/greenfy_web/schema, called data_types.ex (you'll need to create the schema directory), and we'll start defining our first object type:

defmodule Greenfy.Schema.DataTypes do
  use Absinthe.Schema.Notation

  object :event do
    field :id, :id
    field :event_type, :string
    field :message, :string
    field :payload, :string
  end
end

You'll probably notice that :inserted_at and :updated_at are not being included in our data types notation yet! The reason for this is that those are not easily represented by the standard built-in types for Absinthe, so we'll need to define some fancy scalar types later! (Don't worry, we'll describe this all in great detail later!)

Sidebar: GraphQL Definitions

We're using A LOT of terms here pretty liberally when talking about GraphQL but not doing a lot to explain what they actually are, so let's take a really quick sidebar talking about some of the terms we've discussed so far (and the ones we'll be doing later).

First off, we have Types. Types are exactly what they sound like: definitions of the various data types we'll be defining for our GraphQL application. A good example of this is the object definition above: An event is a GraphQL object type, with four fields defined on it so far (we'll get to fields next).

A Field is essentially a bit of queryable information. An object, like the data type we defined previously, stores a collection of different fields together. Each field should define, at a minimum, its name (or key) and its type.

Next, we'll be working with Schemas. If Types define what data is queryable in our GraphQL API and the shape of that data, the Schema defines how we actually get that data OUT of our GraphQL API.

Later on, you'll see something called Resolvers. If the Schema tells the client how to query the data out of our application, the Resolver tells the server how to react to that query and how to interpret those GraphQL queries.

Finally, this will be something we'll use much later, but you'll hear it mentioned before that point: Mutations! Mutations are how the data in our GraphQL application is modified! Instead of your POST, PUT, and PATCH endpoints in your REST API, you just define Mutations instead.

Back to our implementation

Next, we'll need to create our Schema. Create lib/greenfy_web/schema.ex and give it the following contents:

defmodule Greenfy.Schema do
  use Absinthe.Schema

  import_types Greenfy.Schema.DataTypes

  query do
    @desc "Get a list of events"
    field :events, list_of(:event) do
      resolve fn _parent, _args, _resolution ->
        {:ok, Greenfy.Log.list_events()}
      end
    end
  end
end

We use the provided Absinthe.Schema macro to build out our Schema body. Next, we define our query body, which starts off with a @desc statement, which sets a module variable called desc (unsurprisingly) which describes the general use for this query. Think of it as a bit of documentation for the client.

Next, in our query, we have a field. Again, our field is queryable data, so we say that our "query" has a key in it called events. We're also telling Absinthe that we should expect events to return a list of events, so we use the function list_of(:event) (note the singular version of event and that it is an atom, here). This tells Absinthe that when someone queries events, we should be fetching a list of the Data Type Event that we defined in our Data Types file earlier.

Finally, we have a Resolver, which tells Absinthe that when a client queries for a list of events, this function's return statement is what data should be passed back to the client (after some transformation, of course). A resolvestatement takes in three arguments; the parent, the args (the arguments to our query, so whatever we might be looking for or filtering on), and the resolution, which tells us what to do when we've fetched all of the data and what to do with that data at the end! We'll build on this later by moving our resolvers off into their own code, but for right now we're going to stick with a simple implementation.

One final step before we can start testing

Before we get too deep, though, let's modify our router so we can start testing out our application and refactoring as appropriate. Open up lib/greenfy_web/router.ex, delete the following block:

  scope "/" do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
  end

Uncomment out the /api scope section, and remove the mention of the Application module, "Greenfy" from the scope block. Then, we'll want to add the GraphiQL route to give us a playground to test out our application. The scope statement should look something like this:

  scope "/api" do
    pipe_through :api

    forward "/graphiql", Absinthe.Plug.GraphiQL, schema: Greenfy.Schema
  end

And finally, testing it out

In your terminal, start up your server by running:

$ iex mix -S phx.server

And then open up your browser window and point it at http://localhost:4000/api/graphiql. You should see a window that looks like this:

You can see in this screenshot that we have our initial query running, selecting the primary key, eventType (note the difference from event_type; GraphQL expects JavaScript-style Camel-case whereas Elixir by default expects snake_case style. Absinthe handles this conversion for us!) We're not selecting the inserted_at and updated_at columns yet; we still have time to get to that later!

Before You Go

Just wanted to catch you before you finished reading to let you know we're a cloud-based logging company that's looking to make debugging easier by seamlessly augmenting logs with context. We've got a great product built and you can check it out for free!

Summary

So now we're at a point where we have a base GraphQL API and support for VERY limited operations to get the data out! We understand a bit more about GraphQL, the terminology used, and how it all fits together, so where do we go from here?

Easy: we'll make this application more robust! For one thing, we can't verify this works super well unless we can also get data into it. In addition, testing is already hard, so how on earth are we going to test something so completely malleable?

Stay tuned! We have a lot more to talk about, a lot more to understand, and hopefully we can eventually land at being experts at implementing GraphQL APIs in Elixir.

Following along? You can check your work against the tutorial repo at Github. Tag/Release v0.1.0 is the complete version for this phase of the tutorial.

Decorators in Python: What you need to know

Timber Staff — Fri, 10 Aug 2018 20:07:54 +0000

Python decorators are a powerful concept that allow you to "wrap" a function with another function.

The idea of a decorator is to abstract away something that you want a function or class to do, besides its normal responsibility. This can be helpful for many reasons such as code reuse, and sticking to curlys law.

By learning how to write your own decorators, you can significantly improve readability of your own code. They can change how the function behaves, without needing to actually change the code (such as adding logging lines). They are a fairly common tool in python, familiar to those who use frameworks such as flask or click, but many only know how to use them, not how to write their own.

This is a guest-post brought to you by your friends @ Timber. If you're interested in writing for us, feel free to reach out on Twitter.

How it works

First off, let's show an example of a decorator in python. The following is a very basic example of what a decorator would like like if you were using it.

@my_decorator
def hello():
    print('hello')

When you define a function in python, that function becomes an object.

The function hello above is a function object. The @my_decorator is actually a function that has the ability to use the hello object, and return a different object to the interpreter. The object that the decorator returns, is what becomes known as hello. Essentially, it's the same thing as if you were going to write your own normal function, such as: hello = decorate(hello). Decorate is passed the function -- which it can use however it wants -- then returns another object. The decorator has the ability to swallow the function, or return something that is not a function, if it wanted.

Writing your own decorator

As mentioned, a decorator is simply a function that is passed a function, and returns an object. So, to start writing a decorator, we just need to define a function.

def my_decorator(f):
    return 5

Any function can be used as a decorator. In this example the decorator is passed a function, and returns a different object. It simply swallows the function it is given entirely, and always returns 5.

@my_decorator
def hello():
    print('hello')

>>> hello()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable
'int' object is not callable

Because our decorator is returning an int, and not a callable, it can not be called as a function. Remember, the decorator's return value replaces hello.

>>> hello
5

In most cases, we want the object our decorator returns, to actually mimic the function we decorated. This means that the object the decorator returns, needs to be a function itself. For example, let's say we wanted to simply print every time the function was called, we could write a function that prints that information, then calls the function. But that function would need to be returned by the decorator. This usually leads to function nesting such as:

def mydecorator(f):  # f is the function passed to us from python
    def log_f_as_called():
        print(f'{f} was called.')
        f()
    return log_f_as_called

As you can see, we have a nested function definition, and the decorator function is returning that function. This way, the function hello can still be called like a standard function, the caller does not need to know that it is decorated. We can now define hello as the following:

@mydecorator
def hello():
    print('hello')

We get the following output:

>>> hello()
<function hello at 0x7f27738d7510> was called.
hello

(Note: the number inside the <function hello at 0x7f27738d7510> reference will be different for everyone, it represents the memory address)

Wrapping functions correctly

A function can be decorated many times, if desired. In this case, the decorators cause a chain effect. Essentially, the top decorator is passed the object from the former, and so on and so forth. For example, if we have the following code:

@a
@b
@c
def hello():
    print('hello')

The interpreter is essentially doing hello = a(b(c(hello))) and all of the decorators would wrap each other. You can test this out yourself by using our existing decorator, and using it twice.

@mydecorator
@mydecorator
def hello():
    print('hello')

>>> hello()
<function mydec.<locals>.a at 0x7f277383d378> was called.
<function hello at 0x7f2772f78ae8> was called.
hello

You will notice that the first decorator, wrapped the second, and printed it separately.

One interesting thing you might have noticed here, is that first line printed said <function mydec.<locals>.a at 0x7f277383d378> instead of what the second line printed, which is what you would expect: <function hello at 0x7f2772f78ae8>. The is because the object returned by the decorator is a new function, not called hello. This is fine for our trivial example, but can often break tests and things that might be trying to introspect the function attributes. If the idea is for a decorator to act like the function it decorates, it needs to also mimic that function. Luckily, there is a python standard library decorator called wraps for that in functools module.

import functools
def mydecorator(f): 
    @functools.wraps(f)  # we tell wraps that the function we are wrapping is f
    def log_f_as_called():
        print(f'{f} was called.')
        f()
    return log_f_as_called

@mydecorator
@mydecorator
def hello():
    print('hello')

>>> hello()
<function hello at 0x7f27737c7950> was called.
<function hello at 0x7f27737c7f28> was called.
hello

Now, our new function appears just like the one its wrapping/decorating. However, we are still reliant on the fact that it returns nothing, and accepts no input. If we wanted to make that more general, we would need to pass in the arguments and return the same value. We can modify our function to look like this:

import functools
def mydecorator(f): 
    @functools.wraps(f)  # wraps is a decorator that tells our function to act like f
    def log_f_as_called(*args, **kwargs):
        print(f'{f} was called with arguments={args} and kwargs={kwargs}')
        value = f(*args, **kwargs)
        print(f'{f} return value {value}')
        return value
    return log_f_as_called

Now we are printing everytime the function is called, including all the inputs the function receives, and what it returns. Now you can simply decorate any existing function and get debug logging on all its inputs and outputs without needed to manually write the logging code.

Adding variables to the decorator

If you were using the decorator for any code that you wanted to ship, and not just for yourself locally, you might want to replace all the print statements with logging statements. In which case, you would need to define a log level. It might be safe to assume you always want the debug log level, but what that also might depend on the function. We can provide variables to the decorator itself, to define how it should behave. For example:

@debug(level='info')
def hello():
    print('hello')

The above code would allow us to specify that this specific function should log at the info level instead of the debug level. This is achieved in python by writing a function, that returns a decorator. Yes, a decorator is also a function. So this is essentially saying hello = debug('info')(hello). That double parenthesis might look funky, but basically, debug is function, that returns a function. Adding this to our existing decorator, we would need to nest one more time, now making our code look like the following:

import functools
def debug(level): 
    def mydecorator(f)
        @functools.wraps(f)
        def log_f_as_called(*args, **kwargs):
            logger.log(level, f'{f} was called with arguments={args} and kwargs={kwargs}')
            value = f(*args, **kwargs)
            logger.log(level, f'{f} return value {value}')
            return value
        return log_f_as_called
    return mydecorator

The changes above turn debug into a function, which returns a decorator which uses the correct logging level. This is starting to get a bit ugly, and overly nested. A cool little trick to solve that, which I like to do, is simply add a default kwarg level to debug and return a partial. A partial is a "non-complete function call" that includes a function and some arguments, so that they are passed around as one object without actually calling the function yet.

import functools
def debug(f=None, *, level='debug'): 
    if f is None:
        return functools.partial(debug, level=level)
    @functools.wraps(f)   # we tell wraps that the function we are wrapping is f
    def log_f_as_called(*args, **kwargs):
        logger.log(level, f'{f} was called with arguments={args} and kwargs={kwargs}')
        value = f(*args, **kwargs)
        logger.log(level, f'{f} return value {value}')
        return value
    return log_f_as_called

Now the decorator can work as normal:

@debug
def hello():
    print('hello')

And use debug logging. Or, you could override the log level:

@debug('warning')
def hello():
    print('hello')

Logging to the Cloud

Writing your logs to a hosted log aggregation service makes your life easier, so you can focus on building better software, not debugging.

Disclaimer: I’m a current employee @ Timber. This section is entirely optional, but I sincerely believe that logging to the cloud will make your life easier (and you can try it for completely free).

pip install timber

import logging
import timber

logger = logging.getLogger(__name__)

timber_handler = timber.TimberHandler(api_key='...')
logger.addHandler(timber_handler)

That’s it. All you need to do is get your API key from timber.io, and you’ll be able to see your logs. We automatically capture them from the logging module (and seamlessly add context), so you can continue to log normally.

Multiprocessing vs. Multithreading in Python: What you need to know.

Sid Panjwani — Mon, 02 Jul 2018 20:49:52 +0000

TLDR: If you don't want to understand the under-the-hood explanation, here's what you've been waiting for: you can use threading if your program is network bound or multiprocessing if it's CPU bound.

We're creating this guide because when we went looking for the difference between threading and multiprocessing, we found the information out there unnecessarily difficult to understand. They went far too in-depth, without really touching the meat of the information that would help us decide what to use and how to implement it.

What is Threading? Why might you want it?

By nature, Python is a linear language, but the threading module comes in handy when you want a little more processing power. While threading in Python cannot be used for parallel CPU computation, it's perfect for I/O operations such as web scraping because the processor is sitting idle waiting for data.

Threading is game-changing because many scripts related to network/data I/O spend the majority of their time waiting for data from a remote source. Because downloads might not be linked (i.e., scraping separate websites), the processor can download from different data sources in parallel and combine the result at the end. For CPU intensive processes, there is little benefit to using the threading module.

Fortunately, threading is included in the standard library:

import threading
from queue import Queue
import time

You can use target as the callable object, args to pass parameters to the function, and start to start the thread.

def testThread(num):
    print num

if __name__ == '__main__':
    for i in range(5):
        t = threading.Thread(target=testThread, arg=(i,))
        t.start()

If you've never seen if __name__ == '__main__': before, it's basically a way to make sure the code that's nested inside it will only run if the script is run directly (not imported).

The Lock

You'll often want your threads to be able to use or modify variables common between threads but to do that you'll have to use something known as a lock. Whenever a function wants to modify a variable, it locks that variable. When another function wants to use a variable, it must wait until that variable is unlocked.

Imagine two functions which both iterate a variable by 1. The lock allows you to ensure that one function can access the variable, perform calculations, and write back to that variable before another function can access the same variable.

When using the threading module, this can also happen when you're printing because the text can get jumbled up (and cause data corruption). You can use a print lock to ensure that only one thread can print at a time.

print_lock = threading.Lock()

def threadTest():
    # when this exits, the print_lock is released
    with print_lock:
        print(worker)

def threader():
  while True:
    # get the job from the front of the queue
    threadTest(q.get())
    q.task_done()

q = Queue()
for x in range(5):
    thread = threading.Thread(target = threader)
    # this ensures the thread will die when the main thread dies
    # can set t.daemon to False if you want it to keep running
    t.daemon = True
    t.start()

for job in range(10):
    q.put(job)

Here, we've got 10 jobs that we want to get done and 5 workers that will work on the job.

Multithreading is not always the perfect solution

I find that many guides tend to skip the negatives of using the tool they've just been trying to teach you. It's important to understand that there are both pros and cons associated with using all these tools. For example:

There is overhead associated with managing threads, so you don't want to use it for basic tasks (like the example)
Increases the complexity of the program, which can make debugging more difficult

What is Multiprocessing? How is it different than Threading?

Without multiprocessing, Python programs have trouble maxing out your system's specs because of the GIL (Global Interpreter Lock). Python wasn't designed considering that personal computers might have more than one core (shows you how old the language is), so the GIL is necessary because Python is not thread-safe and there is a globally enforced lock when accessing a Python object. Though not perfect, it's a pretty effective mechanism for memory management. What can we do?

Multiprocessing allows you to create programs that can run concurrently (bypassing the GIL) and use the entirety of your CPU core. Though it is fundamentally different from the threading library, the syntax is quite similar. The multiprocessing library gives each process its own Python interpreter and each their own GIL.

Because of this, the usual problems associated with threading (such as data corruption and deadlocks) are no longer an issue. Since the processes don't share memory, they can't modify the same memory concurrently.

Let's Get Started:

import multiprocessing
def spawn():
  print('test!')

if __name__ == '__main__':
  for i in range(5):
    p = multiprocessing.Process(target=spawn)
    p.start()

If you have a shared database, you want to make sure that you're waiting for relevant processes to finish before starting new ones.

for i in range(5):
  p = multiprocessing.Process(target=spawn)
  p.start()
  p.join() # this line allows you to wait for processes

If you want to pass arguments to your process, you can do that with args

import multiprocessing
def spawn(num):
  print(num)

if __name__ == '__main__':
  for i in range(25):
    ## right here
    p = multiprocessing.Process(target=spawn, args=(i,))
    p.start()

Here's a neat example because as you notice, the numbers don't come in the order you'd expect (without the p.join()).

As with threading, there are still drawbacks with multiprocessing ... you've got to pick your poison:

There is I/O overhead from data being shuffled around between processes
The entire memory is copied into each subprocess, which can be a lot of overhead for more significant programs

What should you use?

If your code has a lot of I/O or Network usage:

Multithreading is your best bet because of its low overhead

If you have a GUI

Multithreading so your UI thread doesn't get locked up

If your code is CPU bound:

You should use multiprocessing (if your machine has multiple cores)

Just a disclaimer: we're a logging company here @ Timber. We'd love it if you tried out our product (it's seriously great!), but that's all we're going to advertise it.

The Pythonic Guide To Logging

Sid Panjwani — Mon, 25 Jun 2018 16:08:43 +0000

When done properly, logs are a valuable component of your observability suite. Logs tell the story of how data has changed in your application. They let you answer questions such as: Why did John Doe receive an error during checkout yesterday? What inputs did he provide?

The practice of logging ranges from very simple static string statements to rich structured data events. In this article, we'll explore how logging in Python will give you a better view of what's going on in your application. In addition, I'll explore some best practices that will help you get the most value out of your logs.

print('Why not just use the print statement?')

Many Python tutorials show readers how to use print as a debugging tool. Here's an example using print print an exception as a script:

>>> import sys
>>>
>>> def captureException():
...     return sys.exc_info()
...
>>> try:
...     1/0
... except:
...     print(captureException())
...
(<class 'ZeroDivisionError'>, ZeroDivisionError('division by zero',), <traceback object at 0x101acb1c8>)
None

While this is useful in smaller scripts, as your application and operation requirements grow, print becomes a less viable solution. It doesn't offer you the flexibility to turn off entire categories of output statements and it only allows you to output to the stdout. It also misses information such as line number and time that it was generated that can assist with debugging. While print is the easiest approach since it doesn't require setup, it can quickly come back to bite you. It's also bad practice to ship a package that prints directly to stdout because it removes the user's ability to control the messages.

logging.info("Hello World to Logging")

The logging module is part of Python's standard library and has been since version 2.3. It automatically adds context, such as line number & timestamps, to your logs. The module makes it easy to add namespace logging and severity levels to give you more control over where and what is output.

I'm a believer that the best way to learn something is to do it, so I'd encourage you to follow along in the Python REPL. Getting started with the logging module is simple, here's all you have to do:

>>> import logging
>>> logging.basicConfig()
>>> logger = logging.getLogger(__name__)
>>>
>>> logger.critical('logging is easier than I was expecting')
CRITICAL:__main__:logging is easier than I was expecting

What just happened? getLogger provided us with a logger instance. We then gave it the event 'logging is easier than I was expecting' with a level of critical.

Levels

The Python module allows you to differentiate events based on their severity level. The levels are represented as integers between 0 and 50. The module defines five constants throughout the spectrum, making it easy to differentiate between messages.

Each level has meaning attached to it, and you should think critically about the level you're logging at.

>>> logger.critical("this better be bad")
CRITICAL:root:this better be bad
>>> logger.error("more serious problem")
ERROR:root:more serious problem
>>> logger.warning("an unexpected event")
WARNING:root:an unexpected event
>>> logger.info("show user flow through program")
>>> logger.debug("used to track variables when coding")

Notice the info and debug didn't print a message. By default, the logger will only print warning, error, or critical messages. You can customize this behavior, and even modify it at runtime to activate more verbose logging dynamically.

>>> # should show up; info is a higher level than debug
>>> logger.setLevel(logging.DEBUG)
>>> logger.info(1)
INFO:root:1
>>>
>>> # shouldn't show up; info is a lower level than warning
... logger.setLevel(logging.WARNING)
>>> logger.info(2)

Formatting the Logs

The default formatter is not useful for formatting your logs, because it doesn't include critical information. The logging module makes it easy for you to add it in by changing the format.

We'll set the format to show the time, level, and message:

>>> import logging
>>> logFormatter = '%(asctime)s - %(levelname)s - %(message)s'
>>> logging.basicConfig(format=logFormatter, level=logging.DEBUG)
>>> logger = logging.getLogger(__name__)
>>> logger.info("test")
2018-06-19 17:42:38,134 - INFO - test

Some of this data, such as time and levelname can be captured automatically, but you can (and should) push extra context to your logs.

Adding Context

A generic log message provides just about as little information as no log message. Imagine if you had to go through your logs and you saw removed from cart. This makes it difficult to answer questions such as: When was the item removed? Who removed it? What did they remove?

It's best to add structured data to your logs instead of string-ifying an object to enrich it. Without structured data, it's difficult to decipher the log stream in the future. The best way to deal with this is to push important metadata to the extra object. Using this, you’ll be able to enrich your messages in the stream.

>>> import logging
>>> logFormatter = '%(asctime)s - %(user)s - %(levelname)s - %(message)s'
>>> logging.basicConfig(format=logFormatter, level=logging.DEBUG)
>>> logger = logging.getLogger(__name__)
>>> logger.info('purchase completed', extra={'user': 'Sid Panjwani'})
2018-06-19 17:44:10,276 - Sid Panjwani - INFO - purchase completed

Performance

Logging introduces overhead that needs to be balanced with the performance requirements of the software you write. While the overhead is generally negligible, bad practices and mistakes can lead to unfortunate situations.

Here are a few helpful tips:

Wrap Expensive Calls in a Level Check

The Python Logging Documentation recommends that expensive calls be wrapped in a level check to delay evaluation.

if logger.isEnabledFor(logging.INFO):
    logger.debug('%s', expensive_func())

Now, expensive_func will only be called if the logging level is greater than or equal to INFO.

Avoid Logging in the Hot Path

The hot path is code that is critical for performance, so it is executed often. It's best to avoid logging here because it can become an IO bottleneck, unless it's necessary because the data to log is not available outside the hot path.

Storing & Accessing These Logs

Now that you've learned to write these (beautiful) logs, you've got to determine what to do with them. By default, logs are written to the standard output device (probably your terminal window), but Python's logging module provides a rich set of options to customize the output handling. Logging to standard output is encouraged, and platforms such as Heroku, Amazon Elastic Beanstalk, and Docker build on this standard by capturing the standard output and redirecting to other logging facilities at a platform level.

Logging to a File

The logging module makes it easy to write your logs to a local file using a "handler" for long-term retention.

>>> import logging
>>> logger = logging.getLogger(__name__)
>>>
>>> handler = logging.FileHandler('myLogs.log')
>>> handler.setLevel(logging.INFO)
>>>
>>> logger.addHandler(handler)
>>> logger.info('You can find this written in myLogs.log')

If you've been following along (which you should be), you'll notice that your log won't show up in your file because it is level info. Make sure to use setLevel to change that.

Now it's easy to search through your log file using grep.

grep critical myLogs.log

Now you can search for messages that contain keywords such as critical or warning.

Rotating Logs

The Python logging module makes it easy to log in a different file after an interval of time or after the log file reaches a certain size. This becomes useful if you automatically want to get rid of older logs, or if you're going to search through your logs by date since you won’t have to search through a huge file to find a set of logs that are already grouped.

To create a handler that makes a new log file every day and automatically deletes logs more than five days old, you can use a TimedRotatingFileHandler. Here’s an example:

logger = logging.getLogger('Rotating Log by Day')

# writes to pathToLog
# creates a new file every day because `when="d"` and `interval=1`
# automatically deletes logs more than 5 days old because `backupCount=5`
handler = TimedRotatingFileHandler(pathToLog, when="d", interval=1,  backupCount=5)

The Drawbacks

It’s important to understand where logs fit in when it comes to observing your application. I recommend readers take a look at the 3 pillars of observability principle, in that logs should be one of a few tools you reach for when observing your application. Logs are a component of your observability stack, but metrics and tracing are equally so.

Logging can become expensive, especially at scale. Metrics can be used to aggregate data and answer questions about your customers without having to keep a trace of every action, while tracing enables you to see the path of your request throughout your platform.

Wrapping Up

If you take anything away from this post, it should be that logs serve as the source of truth for the user’s actions. Even for ephemeral actions, such as putting an item into and out of a cart, it’s essential to be able to trace the user’s steps for a bug report, and the logs allow you to determine their actions.

The Python logging module makes this easy. It allows you to format your logs, dynamically differentiate between messages using levels, and ship your logs externally using "handlers". Though not the only mechanism you should be using to gain insight into user actions, it is an effective way to capture raw event data and answer unknown questions.

Instead of doing this yourself, we’ve got a pretty awesome service here @ Timber (it’s seriously great) that automatically captures context with your logs to make debugging easier. Try us out for (completely) free; you don’t even need a credit card!

Logging to the Cloud

Writing your logs to a hosted log aggregation service seriously makes your life easier, so you can focus your time on what’s important.

Disclaimer: I’m a current employee @ Timber. This section is entirely optional, but I sincerely believe that logging to the cloud will make your life easier (and you can try it for completely free).

pip install timber

import logging
import timber

logger = logging.getLogger(__name__)

timber_handler = timber.TimberHandler(api_key='...')
logger.addHandler(timber_handler)

That’s it. All you need to do is get your API key from timber.io, and you’ll be able to see your logs. We automatically capture them from the logging module, so you can continue to follow best-practices and log normally, while we seamlessly add context.

Note: If you're creating the application on Flask, I'd recommend you also check out their logging documentation.

An Intro to Web Scraping with lxml and Python

Timber Staff — Mon, 11 Jun 2018 14:59:26 +0000

Why should you even bother learning how to web scrape? If your job doesn't require you to learn it, then let me give you some motivation. What if you want to create a website which curates cheapest products from Amazon, Walmart and a couple of other online stores? A lot of these online stores don't provide you with an easy way to access their information using an API. In the absence of an API, your only choice is to create a web scraper which can extract information from these websites automatically and provide you with that information in an easy to use way.

Here is an example of a typical API response in JSON. This is the response from Reddit:

There are a lot of Python libraries out there which can help you with web scraping. There is lxml, BeautifulSoup and a full-fledged framework called Scrapy. Most of the tutorials discuss BeautifulSoup and Scrapy, so I decided to go with lxml in this post. I will teach you the basics of XPaths and how you can use them to extract data from an HTML document. I will take you through a couple of different examples so that you can quickly get up-to-speed with lxml and XPaths.

If you are a gamer, you will already know of (and likely love) this website. We will be trying to extract data from Steam. More specifically, we will be selecting from the "popular new releases" information. I am converting this into a two-part series. In this part, we will be creating a Python script which can extract the names of the games, the prices of the games, the different tags associated with each game and the target platforms. In the second part, we will turn this script into a Flask based API and then host it on Heroku.

Step 1: Exploring Steam

First of all, open up the "popular new releases" page on Steam and scroll down until you see the Popular New Releases tab. At this point, I usually open up Chrome developer tools and see which HTML tags contain the required data. I extensively use the element inspector tool (The button in the top left of the developer tools). It allows you to see the HTML markup behind a specific element on the page with just one click. As a high-level overview, everything on a web page is encapsulated in an HTML tag and tags are usually nested. You need to figure out which tags you need to extract the data from and you are good to go. In our case, if we take a look, we can see that every separate list item is encapsulated in an anchor (a) tag.

The anchor tags themselves are encapsulated in the div with an id of tab_newreleases_content. I am mentioning the id because there are two tabs on this page. The second tab is the standard "New Releases" tab, and we don't want to extract information from that tab. Hence, we will first extract the "Popular New Releases" tab, and then we will extract the required information from this tag.

Step 2: Start writing a Python script

This is a perfect time to create a new Python file and start writing down our script. I am going to create a scrape.py file. Now let's go ahead and import the required libraries. The first one is the requests library and the second one is the lxml.html library.

import requests
import lxml.html

If you don't have requests installed, you can easily install it by running this command in the terminal:

$ pip install requests

The requests library is going to help us open the web page in Python. We could have used lxml to open the HTML page as well but it doesn't work well with all web pages so to be on the safe side I am going to use requests.

Now let's open up the web page using requests and pass that response to lxml.html.fromstring.

html = requests.get('https://store.steampowered.com/explore/new/')
doc = lxml.html.fromstring(html.content)

This provides us with an object of HtmlElement type. This object has the xpath method which we can use to query the HTML document. This provides us with a structured way to extract information from an HTML document.

Step 3: Fire up the Python Interpreter

Now save this file and open up a terminal. Copy the code from the scrape.py file and paste it in a Python interpreter session.

We are doing this so that we can quickly test our XPaths without continuously editing, saving and executing our scrape.py file.

Let's try writing an XPath for extracting the div which contains the 'Popular New Releases' tab. I will explain the code as we go along:

new_releases = doc.xpath('//div[@id="tab_newreleases_content"]')[0]

This statement will return a list of all the divs in the HTML page which have an id of tab_newreleases_content. Now because we know that only one div on the page has this id we can take out the first element from the list ([0]) and that would be our required div. Let's break down the xpath and try to understand it:

// these double forward slashes tell lxml that we want to search for all tags in the HTML document which match our requirements/filters. Another option was to use / (a single forward slash). The single forward slash returns only the immediate child tags/nodes which match our requirements/filters
div tells lxml that we are searching for divs in the HTML page
[@id="tab_newreleases_content"] tells lxml that we are only interested in those divs which have an id of tab_newreleases_content

Cool! We have got the required div. Now let's go back to chrome and check which tag contains the titles of the releases.

Step 4: Extract the titles & prices

The title is contained in a div with a class of tab_item_name. Now that we have the "Popular New Releases" tab extracted we can run further XPath queries on that tab. Write down the following code in the same Python console which we previously ran our code in:

titles = new_releases.xpath('.//div[@class="tab_item_name"]/text()')

This gives us with the titles of all of the games in the "Popular New Releases" tab. Here is the expected output:

Let's break down this XPath a little bit because it is a bit different from the last one.

. tells lxml that we are only interested in the tags which are the children of the new_releases tag
[@class="tab_item_name"] is pretty similar to how we were filtering divs based on id. The only difference is that here we are filtering based on the class name
/text() tells lxml that we want the text contained within the tag we just extracted. In this case, it returns the title contained in the div with the tab_item_name class name

Now we need to extract the prices for the games. We can easily do that by running the following code:

prices = new_releases.xpath('.//div[@class="discount_final_price"]/text()')

I don't think I need to explain this code as it is pretty similar to the title extraction code. The only change we made is the change in the class name.

Step 5: Extracting tags

Now we need to extract the tags associated with the titles. Here is the HTML markup:

Write down the following code in the Python terminal to extract the tags:

tags = new_releases.xpath('.//div[@class="tab_item_top_tags"]')
total_tags = []
for tag in tags:
    total_tags.append(tag.text_content())

So what we are doing here is that we are extracting the divs containing the tags for the games. Then we loop over the list of extracted tags and then extract the text from those tags using the text_content() method. text_content() returns the text contained within an HTML tag without the HTML markup.

Note: We could have also made use of a list comprehension to make that code shorter. I wrote it down in this way so that even those who don't know about list comprehensions can understand the code. Eitherways, this is the alternate code:

tags = [tag.text_content() for tag in new_releases.xpath('.//div[@class="tab_item_top_tags"]')]

Lets separate the tags in a list as well so that each tag is a separate element:

tags = [tag.split(', ') for tag in tags]

Step 6: Extracting the platforms

Now the only thing remaining is to extract the platforms associated with each title. Here is the HTML markup:

The major difference here is that the platforms are not contained as texts within a specific tag. They are listed as the class name. Some titles only have one platform associated with them like this:

<span class="platform_img win"></span>

While some titles have 5 platforms associated with them like this:

<span class="platform_img win"></span>
<span class="platform_img mac"></span>
<span class="platform_img linux"></span>
<span class="platform_img hmd_separator"></span>
<span title="HTC Vive" class="platform_img htcvive"></span>
<span title="Oculus Rift" class="platform_img oculusrift"></span>

As we can see these spans contain the platform type as the class name. The only common thing between these spans is that all of them contain the platform_img class. First of all, we will extract the divs with the tab_item_details class, then we will extract the spans containing the platform_img class and finally we will extract the second class name from those spans. Here is the code:

platforms_div = new_releases.xpath('.//div[@class="tab_item_details"]')
total_platforms = []

for game in platforms_div:
    temp = game.xpath('.//span[contains(@class, "platform_img")]')
    platforms = [t.get('class').split(' ')[-1] for t in temp]
    if 'hmd_separator' in platforms:
        platforms.remove('hmd_separator')
    total_platforms.append(platforms)

In line 1 we start with extracting the tab_item_details div. The XPath in line 5 is a bit different. Here we have [contains(@class, "platform_img")] instead of simply having [@class="platform_img"]. The reason is that [@class="platform_img"] returns those spans which only have the platform_img class associated with them. If the spans have an additional class, they won't be returned. Whereas [contains(@class, "platform_img")] filters all the spans which have the platform_img class. It doesn't matter whether it is the only class or if there are more classes associated with that tag.

In line 6 we are making use of a list comprehension to reduce the code size. The .get() method allows us to extract an attribute of a tag. Here we are using it to extract the class attribute of a span. We get a string back from the .get() method. In case of the first game, the string being returned is platform_img win so we split that string based on the comma and the whitespace, and then we store the last part (which is the actual platform name) of the split string in the list.

In lines 7-8 we are removing the hmd_separator from the list if it exists. This is because hmd_separator is not a platform. It is just a vertical separator bar used to separate actual platforms from VR/AR hardware.

Step 7: Conclusion

This is the code we have so far:

import requests
import lxml.html

html = requests.get('https://store.steampowered.com/explore/new/')
doc = lxml.html.fromstring(html.content)

new_releases = doc.xpath('//div[@id="tab_newreleases_content"]')[0]

titles = new_releases.xpath('.//div[@class="tab_item_name"]/text()')
prices = new_releases.xpath('.//div[@class="discount_final_price"]/text()')

tags = [tag.text_content() for tag in new_releases.xpath('.//div[@class="tab_item_top_tags"]')]
tags = [tag.split(', ') for tag in tags]

platforms_div = new_releases.xpath('.//div[@class="tab_item_details"]')
total_platforms = []

for game in platforms_div:
    temp = game.xpath('.//span[contains(@class, "platform_img")]')
    platforms = [t.get('class').split(' ')[-1] for t in temp]
    if 'hmd_separator' in platforms:
        platforms.remove('hmd_separator')
    total_platforms.append(platforms)

Now we just need this to return a JSON response so that we can easily turn this into a Flask based API. Here is the code:

output = []
for info in zip(titles,prices, tags, total_platforms):
    resp = {}
    resp['title'] = info[0]
    resp['price'] = info[1]
    resp['tags'] = info[2]
    resp['platforms'] = info[3]
    output.append(resp)

This code is self-explanatory. We are using the zip function to loop over all of those lists in parallel. Then we create a dictionary for each game and assign the title, price, tags, and platforms as a separate key in that dictionary. Lastly, we append that dictionary to the output list.

In a future post, we will take a look at how we can convert this into a Flask based API and host it on Heroku.

This article was written by Yasoob from Python Tips. I hope you guys enjoyed this tutorial. If you want to read more tutorials of a similar nature, please go to Python Tips. I regularly write Python tips, tricks, and tutorials on that blog. And if you are interested in learning intermediate Python, then please check out my open source book here.

Have a great day!

Just a disclaimer: we're a logging company here @ Timber. We'd love it if you tried out our product (it's seriously great!), but that's all we're going to advertise it.

Creating A Real-World CLI App With Node

Jason Maurer — Mon, 21 May 2018 17:00:16 +0000

This was first posted on Timber.io. Come check us out, try our product, or read some of our other posts. We're a cloud-based logging platform built for developers, by developers. Spend less time debugging and more time shipping.

The command line is a user interface that doesn't get enough attention in the world of Javascript development. The reality is that most dev tools should have a CLI to be utilized by nerds like us, and the user experience should be on par with that of your meticulously-created web app. This includes a nice design, helpful menus, clean error messages and outputs, loading indicators and progress bars, etc.

There aren't many real-world tutorials out there when it comes to building command-line interfaces with Node, so this is the first of a series that will go beyond a basic "hello world" CLI app. We'll be creating an app called outside-cli, which will give you the current weather and 10-day forecast for any location.

Note: There are several libraries out there which aid in creating complex CLI's such as oclif, yargs and commander, but we'll be keeping our dependencies slim for the sake of this example so you can better understand how things are working under the hood. This tutorial assumes you have a basic working knowledge of Javascript and Node.

Setting up the project

As with all Javascript projects, creating a package.json and an entry file is the best way to kick things off. We can keep it simple--no dependencies are needed yet.

package.json

{
  "name": "outside-cli",
  "version": "1.0.0",
  "license": "MIT",
  "scripts": {},
  "devDependencies": {},
  "dependencies": {}
}

index.js

module.exports = () => {
  console.log('Welcome to the outside!')
}

Creating a bin file

We'll need a way to invoke our newly minted app and show the welcome message, as well as add it to the system path so it can be called from anywhere. A bin file is the way to do that.

bin/outside

#!/usr/bin/env node
require('../')()

Never seen #!/usr/bin/env node before? It's called a shebang. It basically tells the system this isn't a shell script and it should use a different interpreter.

It's important to keep the binary file slim, as it's only purpose is to invoke the app. All our code should live outside the binary so it can remain modular and testable. It will also help if we want to provide programatic access to our library in the future.

In order to run the bin file directly, we'll need to give it the correct filesystem permissions. If you're on UNIX, this is as easy as running chmod +x bin/outside. If you're on Windows, do yourself a favor and use the Linux Subsystem.

Next, we'll add our binary to the package.json file. This will automatically place it onto the user's system path when they install our package as a global (npm install -g outside-cli).

package.json

{
  "name": "outside-cli",
  "version": "1.0.0",
  "license": "MIT",
  "bin": {
    "outside": "bin/outside"
  },
  "scripts": {},
  "devDependencies": {},
  "dependencies": {}
}

We can now call our bin file directly by running ./bin/outside. You should see the welcome message. Running npm link in the root of your project will symlink your binary file to the system path, making it accessible from anywhere by running outside.

Parsing commands and arguments

When you run a CLI app, it consists of arguments and commands. Arguments (or "flags") are the values prepended with one or two hyphens (such as -d, --debug or --env production) and are useful for passing options to our app. Commands are all the other values that don't have a flag. Unlike commands, arguments don't need to be specified in any particular order. For example, we could run outside today Brooklyn and just assume that the second command will always be the location--but wouldn't it be better to run outside today --location Brooklyn in case we want to add more options in the future?

In order for our app to be useful at all, we'll need to parse those commands ands arguments, and turn them into an object. We could always jump into process.argv and try to do it ourselves, but let's install our first dependency called minimist to take care of this one for us.

$ npm install --save minimist

index.js

const minimist = require('minimist')

module.exports = () => {
  const args = minimist(process.argv.slice(2))
  console.log(args)
}

Note: The reason we remove the first two arguments with .slice(2) is because the first arg will always be the interpreter followed by the name of the file being interpreted. We only care about the arguments after that.

Now running outside today should output { _: ['today'] }. If you run outside today --location "Brooklyn, NY", it should output { _: ['today'], location: 'Brooklyn, NY' }. We'll go more in-depth with arguments later when we actually use the location, but for now this is enough to set up our first command.

Argument Syntax

To better understand how argument syntax works, you can read this. Basically, a flag can be single or double hyphened, and will take the value immediately following in the command or equal true when there is no value. Single-hyphen flags can also be combined for short-handed booleans (-a -b -c or -abc would give you { a: true, b: true, c: true }.)

It's important to remember that values must be quoted if they contain special characters or a space. Running --foo bar baz would give you { _: ['baz'], foo: 'bar' }, but running --foo "bar baz" would give you { foo: 'bar baz' }.

Running commands

It's a good idea to split up the code for each command and only load it into memory when it is called. This creates faster startup times and prevents unnecessary modules from loading. Easy enough with a switch statement on the main command given to us by minimist. Using this setup, each command file should export a function, and in this case, we're passing the arguments to each command so we can use them later.

index.js

const minimist = require('minimist')

module.exports = () => {
  const args = minimist(process.argv.slice(2))
  const cmd = args._[0]

  switch (cmd) {
    case 'today':
      require('./cmds/today')(args)
      break
    default:
      console.error(`"${cmd}" is not a valid command!`)
      break
  }
}

cmds/today.js

module.exports = (args) => {
  console.log('today is sunny')
}

Now if you run outside today, you'll see the message "today is sunny", and if you run outside foobar, it will tell you that "foobar" is not a valid command. Obviously we still need to query a weather API to get real data, but this is a good start.

Expected commands

There are a few commands and arguments that are expected to be in every CLI: help, --help and -h, which should obviously show help menus, and version, --version and -v which should output the current app version. We should also default to a main help menu if no command is specified.

This can be easily implemented in our current setup by adding two cases to our switch statement, a default value for the cmd variable, and implementing some if statements for the help and version argument flags. Minimist automatically parses arguments to key/values, so running outside --version will make args.version equal true.

const minimist = require('minimist')

module.exports = () => {
  const args = minimist(process.argv.slice(2))

  let cmd = args._[0] || 'help'

  if (args.version || args.v) {
    cmd = 'version'
  }

  if (args.help || args.h) {
    cmd = 'help'
  }

  switch (cmd) {
    case 'today':
      require('./cmds/today')(args)
      break

    case 'version':
      require('./cmds/version')(args)
      break

    case 'help':
      require('./cmds/help')(args)
      break

    default:
      console.error(`"${cmd}" is not a valid command!`)
      break
  }
}

To implement our new commands, follow the same format as the today command.

cmds/version.js

const { version } = require('../package.json')

module.exports = (args) => {
  console.log(`v${version}`)
}

cmds/help.js

const menus = {
  main: `
    outside [command] <options>

    today .............. show weather for today
    version ............ show package version
    help ............... show help menu for a command`,

  today: `
    outside today <options>

    --location, -l ..... the location to use`,
}

module.exports = (args) => {
  const subCmd = args._[0] === 'help'
    ? args._[1]
    : args._[0]

  console.log(menus[subCmd] || menus.main)
}

Now if you run outside help today or outside today -h, you should see the help menu for the today command. Running outside or outside -h should show you the main help menu.

Adding another command

This project setup is really awesome because if you need to add a new command, all you need to do is create a new file in the cmds folder, add it to the switch statement and add a help menu if it has one.

cmds/forecast.js

module.exports = (args) => {
  console.log('tomorrow is rainy')
}

index.js

// ...
    case 'forecast':
      require('./cmds/forecast')(args)
      break
// ...

cmds/help.js

const menus = {
  main: `
    outside [command] <options>

    today .............. show weather for today
    forecast ........... show 10-day weather forecast
    version ............ show package version
    help ............... show help menu for a command`,

  today: `
    outside today <options>

    --location, -l ..... the location to use`,

  forecast: `
    outside forecast <options>

    --location, -l ..... the location to use`,
}

// ...

Loading indicators

Sometimes a command can take a long time to run. If you're fetching data from an API, generating content, writing files to the disk or any other process that takes more than a few milliseconds, you want to give the user some feedback that your app hasn't frozen and is simply working hard. Sometimes you can measure the progress of your operation and it makes sense to show a progress bar, but other times it's more variable and makes sense to show a loading indicator instead.

For our app, we can't measure the progress of our API requests so we'll use a basic spinner to show something is happening. Install two more dependencies for our network requests and our spinner:

$ npm install --save axios ora

Now let's create a utility that will make a request to the Yahoo weather API for the current conditions and forecast of a location.

Note: The Yahoo API uses "YQL" syntax and it's a little funky--don't try to understand it, just copy and paste. This was the only weather API I could find that didn't require an API key.

utils/weather.js

const axios = require('axios')

module.exports = async (location) => {
  const results = await axios({
    method: 'get',
    url: 'https://query.yahooapis.com/v1/public/yql',
    params: {
      format: 'json',
      q: `select item from weather.forecast where woeid in
        (select woeid from geo.places(1) where text="${location}")`,
    },
  })

  return results.data.query.results.channel.item
}

cmds/today.js

const ora = require('ora')
const getWeather = require('../utils/weather')

module.exports = async (args) => {
  const spinner = ora().start()

  try {
    const location = args.location || args.l
    const weather = await getWeather(location)

    spinner.stop()

    console.log(`Current conditions in ${location}:`)
    console.log(`\t${weather.condition.temp}° ${weather.condition.text}`)
  } catch (err) {
    spinner.stop()

    console.error(err)
  }
}

Now if you run outside today --location "Brooklyn, NY", you'll see a quick spinner while it makes the request, followed by the current weather conditions.

Since the request happens so fast, it can be difficult to see the loading indicator. If you want to manually slow it down for the purpose of seeing it, you can add this line to the beginning of your weather util function: await new Promise(resolve => setTimeout(resolve, 5000)).

Great! Now let's copy that code over to our forecast command, and change the formatting a bit.

cmds/forecast.js

const ora = require('ora')
const getWeather = require('../utils/weather')

module.exports = async (args) => {
  const spinner = ora().start()

  try {
    const location = args.location || args.l
    const weather = await getWeather(location)

    spinner.stop()

    console.log(`Forecast for ${location}:`)
    weather.forecast.forEach(item =>
      console.log(`\t${item.date} - Low: ${item.low}° | High: ${item.high}° | ${item.text}`))
  } catch (err) {
    spinner.stop()

    console.error(err)
  }
}

You can now see a 10-day weather forecast when you run outside forecast --location "Brooklyn, NY". Looks good! Let's add one more utility to automatically get our location based off our IP address if no location is specified in the command.

utils/location.js

const axios = require('axios')

module.exports = async () => {
  const results = await axios({
    method: 'get',
    url: 'https://api.ipdata.co',
  })

  const { city, region } = results.data
  return `${city}, ${region}`
}

cmds/today.js & cmds/forecast.js

// ...
const getLocation = require('../utils/location')

module.exports = async (args) => {
  // ...
    const location = args.location || args.l || await getLocation()
    const weather = await getWeather(location)
  // ...
}

Now if you simply run outside forecast without a location, you'll see the forecast for your current location.

Errors and Exit Codes

I didn't go into a lot of detail on how to best handle errors (this will come in a later tutorial), but the most important thing to remember is to use the correct exit codes. If your CLI ever has a critical error, you should exit with process.exit(1). This lets the terminal know that the program did not exit cleanly--which will notify you from a CI service, for example. Let's create a quick utility that does this for us, so we can get the correct exit code when a non-existant command is run.

utils/error.js

module.exports = (message, exit) => {
  console.error(message)
  exit && process.exit(1)
}

index.js

// ...
const error = require('./utils/error')

module.exports = () => {
  // ...
    default:
      error(`"${cmd}" is not a valid command!`, true)
      break
  // ...
}

Publishing to NPM

The last step to getting our library out into the wild is to publish it to a package manager. Since our app is written in Javascript, it makes sense to publish to NPM. Let's fill out our package.json a bit more:

{
  "name": "outside-cli",
  "version": "1.0.0",
  "description": "A CLI app that gives you the weather forecast",
  "license": "MIT",
  "homepage": "https://github.com/timberio/outside-cli#readme",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/timberio/outside-cli.git"
  },
  "engines": {
    "node": ">=8"
  },
  "keywords": [
    "weather",
    "forecast",
    "rain"
  ],
  "preferGlobal": true,
  "bin": {
    "outside": "bin/outside"
  },
  "scripts": {},
  "devDependencies": {},
  "dependencies": {
    "axios": "^0.18.0",
    "minimist": "^1.2.0",
    "ora": "^2.0.0"
  }
}

Setting engine will ensure anyone installing our app has an updated version of Node. Since we are using async/await syntax without transpilation, we are requiring Node 8.0 or above.
Setting preferGlobal will warn the user if installing with npm install --save rather than npm install --global.

That's it! You can now run npm publish and your app will be available for download. If you want to take this a step further and release on other package managers (such as Homebrew), you can check out pkg or nexe, which help you bundle your app into a self-contained binary.

Takeaways and Conclusion

This is the structure we follow for all of our CLI apps here at Timber, and it helps keep things organized and modular. Some key takeaways from this tutorial for those who only skimmed it:

Bin files are the entry point for any CLI app, and should only invoke the main function
Command files shouldn't be required until they are needed
Always include help and version commands
Keep command files slim--their main purpose is to call functions and show user messages
Always show some kind of activity indicator
Exit with the correct error codes

I hope you now have a better understanding of how to create and organize CLI apps in Node. This is the first part of a series of tutorials, so come back later as we go more in-depth on adding design, ascii art and color, accepting user input, writing integration tests and more. You can see all the source code we wrote today on GitHub.

Wait for a Script to Load in Javascript

Zach Sherman — Mon, 30 Apr 2018 22:59:21 +0000

Ever needed to wait for a script to load before doing something with it? Here's a quick and easy way to make that happen. Perfect for 3rd party scripts you have no control over like Google Analytics, Segment, Intercom, etc.

All you need is this little class:

export default class ScriptLoader {
  constructor (options) {
    const { src, global, protocol = document.location.protocol } = options
    this.src = src
    this.global = global
    this.protocol = protocol
    this.isLoaded = false
  }

  loadScript () {
    return new Promise((resolve, reject) => {
      // Create script element and set attributes
      const script = document.createElement('script')
      script.type = 'text/javascript'
      script.async = true
      script.src = `${this.protocol}//${this.src}`

      // Append the script to the DOM
      const el = document.getElementsByTagName('script')[0]
      el.parentNode.insertBefore(script, el)

      // Resolve the promise once the script is loaded
      script.addEventListener('load', () => {
        this.isLoaded = true
        resolve(script)
      })

      // Catch any errors while loading the script
      script.addEventListener('error', () => {
        reject(new Error(`${this.src} failed to load.`))
      })
    })
  }

  load () {
    return new Promise(async (resolve, reject) => {
      if (!this.isLoaded) {
        try {
          await this.loadScript()
          resolve(window[this.global])
        } catch (e) {
          reject(e)
        }
      } else {
        resolve(window[this.global])
      }
    })
  }
}

Example Usage

const loader = new Loader({
    src: 'cdn.segment.com/analytics.js',
    global: 'Segment',
})

// scriptToLoad will now be a reference to `window.Segment`
const scriptToLoad = await loader.load()

Timber is Hiring a Developer Advocate

Zach Sherman — Thu, 12 Apr 2018 13:27:42 +0000

More about you

Our ideal candidate for this role should have a background in software development, moderate proficiency in at least two programming languages, and a respectable understanding of the practical challenges teams face during implementation. You get excited about new technologies, keep up with the latest trends in development, contribute to open source, participate in community events, and love meeting with software teams to help them grow and solve problems. You appreciate a well thought out API, clear and concise documentation, and generally have a high bar for what a great development experience should be.

Highlights of the role

As an early member of the team, you’ll have a profound impact on what we build, how we grow and interact with developers, and how we solve important problems for teams around the world. You’ll work closely with our product and engineering team to help us excite, involve and delight customers. You’ll run community engagement events such as hackathons, talks, giveaways, startup programs and more. You’ll get to build meaningful, long-term relationships with our customers and help them grow. You’ll get to travel and attend your favorite conferences and events, speak, and get to explore and experiment with cutting edge technologies.

Perks & benefits

Competitive salary and equity
Medical, dental and vision insurance
Flexible vacation policy
Commuter benefits
Unlimited education budget
Beautiful, spacious office in Brooklyn
Gym reimbursements

Key responsibilities

Educate and assist developers who want to build applications with Timber through technical content, sample applications, presentations and blog posts
Serve as a technical ally to developers in helping them monitor their applications
Participate and run technical conferences, events and hackathons
Build relationships with individuals and organizations who support and inspire developer communities
Communicate new features and launches to our community and customers
Improve developer onboarding (documentation, walkthroughs, CLIs)
Help run our content strategy (podcasts, articles, tutorials and more)
Contribute to our open source initiatives and packages
Identify growing opportunities in the developer space where Timber can solve problems

Requirements

2-3 years development experience with at least two programming languages
Experience deploying applications with AWS services, Azure services, Google Cloud platform or other cloud services
History of participation in open source developer communities, through contributions or other leadership activities
Technical writing skills demonstrated by past blogging, or other written work
Presentation skills as demonstrated by past speaking experience
Passion interacting with the developer community, through their networks and at events
Excellent writing and communication skills

Apply Here: https://jobs.timber.io/o/developer-advocate

Elixir Logger and the Power of Metadata

David Antaramian — Thu, 31 Aug 2017 14:38:00 +0000

Originally published on timber.io

In this post, we look at the ways in which Elixir's Logger can produce incredibly useful logs by adding contextual metadata.

Logger is one of the OTP applications distributed with the Elixir core. It is a powerful but elegant logging framework that is able to withstand the pressure of a production environment with little overhead to the processes producing logs. It is also incredibly extensible, providing ways to manipulate and enhance logs prior to their final presentation.

When an Elixir process produces a log, Logger handles transporting the log data asynchronously to a list of backends. A backend is responsible for writing out the data. Elixir projects will use the :console backend by default which prints the log messages to STDOUT.

Logging Metadata

Unlike many default logging packages that ship with languages, Logger treats log metadata as a first class concern. In its most basic usage, metadata is useful for grouping log lines about the same subject together. For example, you can set the request ID of an HTTP request as metadata, and all the log lines about that HTTP request would show that request ID.

Logger metadata is a keyword list stored in the process dictionary, so each process has an isolated metadata store. This means that setting metadata in one process won't affect the metadata in another process. The keyword list is copied whenever you log a message and sent alongside the message to every Logger backend. The backend can choose how it handles the metadata. We'll use the default :console backend in the examples below since it's so easy to get started with.

Configuring `:console`

The :console backend makes it easy to get up-and-running with metadata. The default configuration is to print metadata immediately before the log level. You just need to whitelist the metadata keys you will use in your config/config.exs:

config :logger, :console,
  metadata: [:request_id]

In the snippet above, the :console is configured to print out the :request_id metadata; any other metadata will be ignored. In Elixir 1.5, the special setting :all was introduced which lets the :console backend print out all metadata:

config :logger, :console,
  metadata: :all

Metadata whitelisting is unique to the :console backend. Other backends won't require it.

Adding Metadata for Later

There are two ways to add metadata to your logs. The first is to use the Logger.metadata/1 function. This function takes a keyword list and merges it into the existing metadata. This works great when you generate a piece of data you know will be useful later on, like the HTTP request ID described above. In fact, the Plug.RequestId module does this when setting the request ID:

defp set_request_id({conn, request_id}), header) do
  Logger.metadata(request_id: request_id)
  Conn.put_resp_header(conn, header, request_id)
end

Wehn Logger.metadata/1 is called in the function above, every log statement in the same process will have that request ID as part of the metadata! Let's assume we're handling an HTTP request with request ID bcc1fe and the plug has already handled setting up the metadata. Now, we log the following message during our business logic:

Logger.info("Updated profile for user ##{user_id}")

The final output on the console will look like this:

22:57:14.199 request_id=bcc1fe [info]  Updated profile for user #32

Adding Metadata for a Single Line

The second option is to add metadata at the point of logging. Using this option, the metadata is not persisted, so it is only used for that log line. This can be really useful when you have something that's only pertitent to the line itself, like a payment confirmation code:

defp handle_payment_success(p = %Payment{confirmation_code: c}) do
  Logger.info("Successfully processed payment", confirmation_code: c)
  p
end

Here, the log message indicates that the payment was successful and gives the confirmation code as metadata. Your resulting log line will be similar to the following:

22:58:43.040 confirmation_code=9184210 [info]  Successfully processed payment

Or, if you had set the HTTP request ID in the previous step:

22:58:43.040 confirmation_code=9184210 request_id=bcc1fe [info]  Successfully processed payment

It's brilliantly simple, yet offers an extremely powerful way to enhance your logs.

Metadata in Practice

Metadata is an extremely powerful way to enhance your logs, but using it effectively can be difficult. If you add too much metadata, you will start generating too much noise that you don't pay attention to, especially when you have a large amount of logs. Don't forget, in addition to the logs you are already generating, you also have logs generated by third-party libraries you rely on. They'll pick up on the metadata you've set and possibly add their own. Coalescing all these logs into usable information requires a game plan.

Start by thinking about your data across time. When fulfilling a request, what relevant information is always going to be the same across all log lines? This is the type of metadata you will want to set ahead of time as metadata using Logger.metadata/1. As your application progresses, each common data point acts like a string tying log lines together.

Consider the initial example we used when setting up setting up metadata. The HTTP request ID is useful when you receive a request and when you respond to it, but it's also useful for all the steps in between. It links together the log lines that deal with fulfilling this request.

If you deal with a system that is user-focused, you might also introduce the user's ID as metadata. This becomes another point that links log lines together, and instead of linking them across a single request, it will link them together across time. In the same vein, imagine you start a Task to asynchronously update the image for a product in your ecommerce catalog. You can use the product ID as metadata to associate all the log lines related to that update task (as well as any other actions related to that product ID).

Not all information is relevant all the time, though. Some information is only relevant to a specific step, and this is the metadata you should set at the same time you produce the log statement.

This is the case with the payment confirmation code example. The confirmation code is only useful for the payment processing step (in most cases), therefore, it makes sense that you only add it to the log statement it relates to. Likewise, the time it took to handle a database query could also be added as metadata. The same is true with the status code of an HTTP response. These values are most relevant to their individual log statements, and by setting them as metadata, they become easily surfaceable when browsing your logs.

How relevant data is across multiple log lines is dependent on your business needs. You may find that for your business it's appropriate to set the customer's ID as metadata that associates log lines. Another business might only record the customer ID on specific actions, like checkout. If you start thinking about why you are looking at logs and how you determine that a log statement is relevant, the data points become evident without much work.

Experimenting with metadata is the easiest way to get started, and there is very little work involved. You can improve and refine your metadata usage gradually over time until it meets your needs.

In a future post, we'll look at how Timber builds on metadata and why Elixir Logger's built-in support allows you (and us) to do amazing things once the metadata hits the backend.

Building a Blog with Next.js

Zach Sherman — Wed, 09 Aug 2017 21:26:08 +0000

Originally published on timber.io

Next.js is an awesome new framework for building universal React applications. In simple terms, that means you can use React to render templates on the server, as well frontend components the way you're most likely used to. The advantages of this are numerous (shared components, faster rendering, great tooling), but getting it all to work properly is generally a pain.

Next.js makes this process easy, and with the release of V3 I thought I'd whip up a blog to learn and demonstrate how it works. In this tutorial we'll be using the following:

next (3.X)
styled-components (phenomenal css-in-js solution)
next-routes (middleware for expressive routes in next)
express (for serving our pages, although you can also do static exports)

I'd highly recommend you follow along with the github repo here https://github.com/timberio/next-go/, as some components are left out for the sake of brevity.

1. Getting Started

Getting set up is simple, you can follow along in the docs but the gist of it is to install next, react, and react-dom, add a simple build script and create your index file.

yarn add next@beta react react-dom --save

Add the following scripts to your package.json

{
  "scripts": {
    "dev": "next",
    "build": "next build",
    "start": "next start"
  }
}

Then create an index.js file inside of a pages folder in the root

// ./pages/index.js

export default () => (
  <div>Welcome to next.js!</div>
)

Then you can just run yarn dev and you should be up and running on localhost:3000. Hot reloading is baked in by default, which you can peek at if you inspect the .next folder in your root directory.

2. Adding Some Style

Next up we'll configure styled-components to style our blog.

First run yarn add styled-components.

Then create a custom _document.js file in the root with the following:

import Document, { Head, Main, NextScript } from 'next/document'
import { ServerStyleSheet } from 'styled-components'
import 'styles/global-styles';

export default class SiteDocument extends Document {
  render () {
    const sheet = new ServerStyleSheet()
    const main = sheet.collectStyles(<Main />)
    const styleTags = sheet.getStyleElement()
    return (
      <html>
        <Head>
          <meta charSet="utf-8" />
          <meta name="viewport" content="initial-scale=1.0, width=device-width" />
          {styleTags}
        </Head>
        <body>
          <div className="root">
            {main}
          </div>
          <NextScript />
        </body>
      </html>
    )
  }
}

The custom _document.js allows us to override the default page layout and inject our own styles and markup surrouding our react app.

3. Creating a Layout

Now let's create a main layout which all of our blog views will use, put the following in layouts/Main.js:

/* layouts/Main.js */

import Head from 'next/head'
import Wrapper from './Wrapper'
import Nav from 'components/Nav'
import Footer from 'components/Footer'

export default ({ children, title = 'This is the default title' }) => (
  <Wrapper>
    <Head>
      <title>{ title }</title>
    </Head>
    <header>
      <Nav />
    </header>

    <main>
      { children }
    </main>

    <Footer>
      Footer
    </Footer>
  </Wrapper>
)

We'll use this layout to wrap our pages, which can override the <Head> tags and render content into the { children } block.

4. Rendering Posts

Now that we have our layout set up, let's modify our index.js page to take advantage of it, and also render some posts.

Update pages/index.js with the following:

import React from 'react'
import Layout from 'layouts/Main';
import { getPosts } from 'api/posts'
import { Link } from 'routes'

import Post from 'components/Post'

const IndexPage = ({ posts }) => (
  <Layout>
    <ul>
      {posts.map(p => (
        <Post key={p.title} post={p} />
      ))}
    </ul>
  </Layout>
)

IndexPage.getInitialProps = async ({ req }) => {
  const res = await getPosts()
  const json = await res.json()
  return { posts: json }
}

export default IndexPage

The key here is the getInitialProps on our IndexPage component, which fetches all of the required data needed for this page to render. When this page is accessed directly at localhost:3000, Next will take care of fetching the data before the page is rendered. If we're navigating to this page from another one, no additional page reloads will happen, Next's clientside routing will take over and fetch the data for us before rendering the component thanks to the Link component. You can even add the prefetch property to tell Next to prefetch that page for blazing fast page loads.

Now we'll use some sample json and place the api in api/posts/index.js:

import fetch from 'isomorphic-fetch'

export function getPosts() {
  return fetch('https://jsonplaceholder.typicode.com/posts')
}

export function getPost(slug) {
  return fetch(`https://jsonplaceholder.typicode.com/posts?title=${slug}`)
}

And add our Post component in components/Post/index.js:

import React from 'react'
import { Link } from 'routes'
import Wrapper from './Wrapper'

const PostItem = ({ post }) => (
  <Wrapper>
    <Link route='post' params={{ slug: post.title }}>
      <a>
        <h3>{post.title}</h3>
        <p>{post.body}</p>
      </a>
    </Link>
  </Wrapper>
)

export default PostItem

When you reload the page you should see a list of posts getting rendered by our index page like so (you can see the styles in the github repo https://github.com/timberio/next-go/).

5. Post Pages

Now that we have a list of posts, lets add a route to view each individual post. Create a new page in pages/Post.js like so:

import React from 'react'
import Link from 'next/link'
import styled from 'styled-components'
import Layout from 'layouts/Main';
import { getPost } from 'api/posts'

const PostPage = ({ post }) => (
  <Layout>
    <h1>{post.title}</h1>
    <p>{post.body}</p>
  </Layout>
)

PostPage.getInitialProps = async ({ query }) => {
  const res = await getPost(query.slug)
  const json = await res.json()
  return { post: json[0] }
}

export default PostPage

This page is reponsible for fetching and rendering individual posts, so let's add a route to show it. We'll be using next-routes for some nice expressive route definitions, so simply run:

yarn add next-routes

and add a routes.js folder in the root with the following:

const nextRoutes = require('next-routes')
const routes = module.exports = nextRoutes()

routes.add('index', '/')
routes.add('post', '/blog/:slug')

Then make sure to add this middleware in ./server.js

const express = require('express')
const next = require('next')
const routes = require('./routes')

const dev = process.env.NODE_ENV !== 'production'
const app = next({ dev })
const handle = app.getRequestHandler()
const handler = routes.getRequestHandler(app)

app.prepare()
.then(() => {
  const server = express()
  server.use(handler)

  server.get('*', (req, res) => {
    return handle(req, res)
  })

  server.listen(3000, (err) => {
    if (err) throw err
    console.log('> Ready on http://localhost:3000')
  })
})

Now our <Link route='post' params={{ slug: post.title }}> components in pages/index.js will map to this page with the proper params and if you click on one you should see something like this:

That's it! You can easily sub in your own endpoints in api/posts/index.js to fetch from your API or CMS of choice.

You can see a working demo at https://next-go.now.sh/ and view the code at https://github.com/timberio/next-go. You can also learn more about Next at https://learnnextjs.com/.

DEV Community: Timber

Vector - A High-Performance Logs & Metrics Router Written In Rust

vectordotdev / vector

A high-performance observability data pipeline.

What is Vector?

Principles

PromQL For Humans

Basics

Instant Vectors

It's easy to filter by label.

You can check a substring using regex matching.

Range Vectors

Important functions

For Range Vectors

Increase of http_requests_total averaged over the last 5 minutes.

Irate

HTTP Requests in the last hour.

For Instant Vectors

Broken by Status Code

Sum by Status Code

Offset

Operators

Vector Matching

API 5xxs are 10% of HTTP Requests

Many-to-One

Examples

CPU Usage by Instance

Memory Usage

Disk Space

HTTP Error Rates as a % of Traffic

Alerts Firing in the last 24 hours

3 Pillars of Observability

A Gentle Introduction to GraphQL with Elixir and Phoenix

Getting a working GraphQL application!

Latest Versions

Assumptions

About this guide

Moving on to the application

Creating a new project

Setting up our app

Adding in Absinthe

Creating our first GraphQL query

Designing the shape of our data

Getting our database tables running in Ecto

Building support for our first query in GraphQL

Sidebar: GraphQL Definitions

Back to our implementation

One final step before we can start testing

And finally, testing it out

Before You Go

Summary

Decorators in Python: What you need to know

How it works

Writing your own decorator

Wrapping functions correctly

Adding variables to the decorator

Logging to the Cloud

Multiprocessing vs. Multithreading in Python: What you need to know.

What is Threading? Why might you want it?

The Lock

Multithreading is not always the perfect solution

What is Multiprocessing? How is it different than Threading?

Let's Get Started:

What should you use?

The Pythonic Guide To Logging

print('Why not just use the print statement?')

logging.info("Hello World to Logging")

Levels

Formatting the Logs

Adding Context

Performance

Wrap Expensive Calls in a Level Check

Avoid Logging in the Hot Path

Storing & Accessing These Logs

Logging to a File

Rotating Logs

The Drawbacks

Wrapping Up

Logging to the Cloud

An Intro to Web Scraping with lxml and Python

Increase of `http_requests_total` averaged over the last 5 minutes.

Configuring `:console`