DEV Community: Sebastian G. Vinci

Python decorators in a nutshell.

Sebastian G. Vinci — Wed, 06 Feb 2019 02:35:58 +0000

What's a decorator anyway?

Decorators are a way to extend the behavior of pieces of software without actually having to modify them.

In Python, in particular, they are functions that can change the behavior of other functions without changing their code. They provide a transparent way of adding satelital functionalities to already defined functions.

For example, imagine we want to measure the time it takes to execute a couple functions in our program. We could code a timer around their behavior by modifying them all, or we can create a decorator, centralizing the logic of timing a function within one place and applying it to several functions.

A couple concepts we need to know first...

In Python, functions are first-class objects.

This means that functions can be assigned to variables and parameters, and even be returned by other functions.

Functions that return other functions, or that receive other functions as arguments, are called higher order functions.

def prepare_hello(name: str):
    def say_hello():
        print("Hello, {}!".format(name))
    return say_hello

hello_brian = prepare_hello("Brian")
hello_brian()

On the above snippet, prepare_hello is a higher order function, as it returns another function. Also, we can see how we can assign functions to variables, as hello_brian is a variable that holds the function returned by prepare_hello.

A couple known higher order functions in Python are map, filter, sort and reduce. These are functions that receive a collection and a function as argument, to let us process the collection using the given function in different ways.

`*args` and `**kwargs`.

You may already know these guys, they are pretty famous around Python. Both *args and **kwargs allow you to pass a variable number of arguments to a function. Your function is basically a blank check if the signature contains one or both of these guys.

def blank_check(*args, **kwargs):
    for arg in args:
        print("Arg: {}".format(arg))
    for keyword, arg in kwargs.items():
        print("Kw: {}, Arg: {}".format(keyword, arg))

blank_check(1, 2, 3, hello="Hello", word="World")

Here's the output of the above function:

Arg: 1
Arg: 2
Arg: 3
Kw: hello, Arg: Hello
Kw: word, Arg: World

As you can see, arguments passed without a keyword are handled by *args, and arguments passed with a keyword are handled by **kwargs.

This allows us to write functions that do not care at all about the arguments they receive, which will make a hell of a lot easier to write versatile decorators.

A thing to notice is that the important part of *args and **kwargs are the aesterisks. The name can be anything, as any other variable, args and kwargs are used by convention, as this practice is dark-ish and needs to be spotted right away.

Now, decorators!

Decorators are, basically, functions that wrap other functions. In Python it really is as easy as that.

def a_decorator(wrapped_function):
    def wrapper(*args, **kwargs):
        print('I am the decorator!')
        return wrapped_function(*args, **kwargs)
    return wrapper

That's the definition of a decorator, right there. You can see two interesting things there:

The decorator receives the function it is supposed to wrap as an argument, and returns a wrapper function. Therefore, a decorator is a higher order function.
The decorator, by using *args and **kwargs, does not care at all what are the arguments of the wrapped function, it just does its thing and calls the wrapped function with all the given arguments.

Now, we can use it wherever we want:

@a_decorator
def sum_numbers(num_a, num_b):
    return num_a + num_b

print(sum_numbers(1, 2))

The output will be:

I am the decorator!
3

It is really transparent, see? And although it is pretty magical, it is really explicit. You go to the definition of the function and you see right there, in your face, what decorators are being executed.

This implementation, though, has a subtle issue:

print(sum_numbers.__name__)

That will print out wrapper, as that's the name of the wrapper function the decorator returns. Now, when debugging, that thing right there will be a pain in the butt. But we can fix it! And we can fix it with the standard library! :D

There is a module in python called functools that provides a lot of awesome things to work with functions. I really encourage you to go and see what it has to offer, but today we are just going to use one decorator it provides.

The decorator functools.wraps receives a function as argument and basically overrides the wrapped function properties with the ones of the given function.

Let's change our decorator using functools.wraps to see this in action.

from functools import wraps

def a_decorator(wrapped_function):
    @wraps(wrapped_function)
    def wrapper(*args, **kwargs):
        print('I am the decorator!')
        return wrapped_function(*args, **kwargs)
    return wrapper

@a_decorator
def sum_numbers(num_a, num_b):
    return num_a + num_b

print(sum_numbers.__name__)

That will print sum_numbers again, fixing our issue elegantly. Now, how come wraps receives parameters? How do I do that? You'll need an intermediate function that receives those parameters and returns the wrapper function.

Your decorator will not actually be a decorator, it will be a decorator builder, which receives a couple of arguments and returns a decorator.

def a_decorator(name):
    def actual_decorator(wrapped_function):
        @wraps(wrapped_function)
        def wrapper(*args, **kwargs):
            print('I am the decorator, {}!'.format(name))
            return wrapped_function(*args, **kwargs)
        return wrapper
    return actual_decorator

@a_decorator("Sum Numbers")
def sum_numbers(num_a, num_b):
    return num_a + num_b

print(sum_numbers(1, 2))

The actual problem we wanted to solve

Let's say, we have a function we want to time:

def fibonacci(n, a = 0, b = 1):
    if n == 0: 
        return a 
    if n == 1: 
        return b 
    return fibonacci(n - 1, b, a + b)

Now, let's just drop the idea of adding timing functionality directly to it, as it would pollute its code, making it more complex to follow, just because we wanted to add non-critical behavior to it.

So, let's write a decorator called timer that receives a cool operation name (to make it more readable in the logs) and add @timer("Fibonacci calculator") to the original function.

from datetime import datetime
from functools import wraps

def timer(operation_name):
        def timer_decorator(wrapped_function):
                @wraps(wrapped_function)
                def wrapper(*args, **kwargs):
                        start = datetime.now()
                        result = wrapped_function(*args, **kwargs)
                        end = datetime.now()
                        print("{} took {}.".format(operation_name, end - start))
                        return result
                return wrapper
        return timer_decorator

@timer("Fibonacci calculator")
def fibonacci(n, a = 0, b = 1):
    if n == 0:
        return a
    if n == 1:
        return b
    return fibonacci(n - 1, b, a + b)

fibonacci(30)

Now, in the output we'll see how much time each execution of fibonacci took, taking into account it is a recursive function and for n=30 it will execute itself 30 times. So we'll see in the output Fibonacci calculator took <time> thirty times.

This experiment also makes it very clear that Python doesn't do tail recursion optimization, so keep that one in mind :P.

Conclusion

Python makes decorators fairly easy, and they are really awesome for this kind of features we all need in production environments.

Caching, unit testing, routing, these are all problems that can be elegantly solved using decorators, and I think, particulary in Python, they are really beginner friendly.

Intersection, union and difference of Sets in Python.

Sebastian G. Vinci — Mon, 04 Feb 2019 20:47:50 +0000

I wanted to talk about sets, and four great operations they provide:

Intersection: Elements two sets have in common.
Union: All the elements from both sets.
Difference: Elements present on one set, but not on the other.
Symmetric Difference: Elements from both sets, that are not present on the other.

We are going to do this using Python (3.7), because it rules!

So, let's dive right into it.

Some basic understanding of sets.

Sets are data structures that, based on specific rules they define, provide certain cool operations and guarantees to make a lot of things easier when we use them.

Two basic rules we need to know are these:

Items in a set are unique. We can not have a set that contains two items that are equal.
Items in a set are not ordered. Depending on the implementation, items may be sorted using the hash of the value stored, but when you use sets you need to assume items are ordered in a random manner.

And a cool guarantee they provide us: Checking if an element is present on a set has a time complexity constant (O(1)), always. This means, checking if a set contains an element is super fast.

The most popular implementation of a set used to achieve this is the hashed set. Oversimplified, it basically stores the items in a key-value store, where the key is a hash of the value stored.

So, to create a set in Python we use the constructor set which receives any iterable: my_pets = set(['dog', 'cat', 'parrot']) (no, I don't have a parrot, nor a cat, dog person here).

Curly braces may be used too, which will avoid creating an intermediate list, but as they are also used for dictionaries, you can not create an empty set with curly braces. Also, someone can get confused, so I avoid them and stick to the constructor.

Sets are collections, therefore they are iterable:

my_pets = set(['dog', 'cat', 'parrot'])
for pet in my_pets:
    print('Hello, {}!'.format(pet))

The above program serves as a demonstration of how elements in a set are ordered, below is the output of the program where you can see the order I defined was not respected by the interpreter:

Hello, dog!
Hello, parrot!
Hello, cat!

So, with this information, let's just dive into these cool operations.

Intersection

The intersection between two sets, results in a third set that contains the elements present in both.

For example, if I calculate the intersection between {1, 2, 3} and {2, 3, 4}, the output will be {2, 3}.

If this helps you understand, here is a naive implementation in Python:

a = set([1, 2, 3])
b = set([2, 3, 4])
result = {element for element in a if element in b}
print(result)

Man, is Python awesome or what? Of course, this will print {2, 3} as expected. Anyway, that was irrelevant because Python provides a function to do this as part of its standard library. The same can be achieved with the following:

a = set([1, 2, 3])
b = set([2, 3, 4])
result = a.intersection(b)
print(result)

Notice that, because of the nature of the operation, a.intersection(b) and b.intersection(a) is the same. It is a commutative operation.

Union

The union between two sets, results in a third set with all the elements from both sets.

For example, if I calculate the union between {1, 2, 3} and {2, 3, 4}, the output will be {1, 2, 3, 4}. Kind of like a concatenation, but having in mind that sets can not have repeated elements.

Again, let's see a naive implementation in Python:

a = set([1, 2, 3])
b = set([2, 3, 4])
c = set()

for element in a: 
    c.add(element)

for element in b: 
    c.add(element)

print(c)

As expected, the output is {1, 2, 3, 4}. Again, though, Python does provide a native function to do this:

a = set([1, 2, 3])
b = set([2, 3, 4])
c = a.union(b)
print(c)

Notice that, because of the nature of the operation, a.union(b) and b.union(a) is the same. It is a commutative operation.

Difference

The difference between two sets results in a third set with the element from the first, that are not present on the second. Right now I'm going to tell you, this operation is not commutative.

For example, if I have a set a = {1, 2, 3} and a set b = {2, 3, 4}, the difference between a and b is {1} and the difference between b and a is {4}.

Here's a naive implementation:

a = set([1, 2, 3])
b = set([2, 3, 4])
c = {element for element in a if element not in b}
print(c)

This outputs {1} as expected, and will print {4} if we invert a and b. Again, an implementation using Python's standard lib:

a = set([1, 2, 3])
b = set([2, 3, 4])
c = a.difference(b)
print(c)

Symmetric difference

The symmetric difference between two sets results in a third set with the elements, from both sets, that are not present on the other.

For example, if I calculate the symmetric difference between {1, 2, 3} and {2, 3, 4}, the output will be {1, 4}. This operation is commutative.

Here's a naive implementation:

a = set([1, 2, 3])
b = set([2, 3, 4])
c = set()

for element in a:
    if element not in b:
        c.add(element)

for element in b:
    if element not in a:
        c.add(element)

print(c)

The above implementation will print {1, 4}, as expected. Of course, Python has this on its standard libary too: symmetric_difference.

a = set([1, 2, 3])
b = set([2, 3, 4])
c = a.symmetric_difference(b)
print(c)

Conclusion

Sets are really useful, and actually pretty fun. Give them a try. I lived a lot of time constrained to lists, and when I finally sat down and learned how to work with sets started finding better, more elegant solutions to a lot of problems.

Scala REST API documented with Swagger (Part 2 of 2)

Sebastian G. Vinci — Wed, 19 Apr 2017 19:09:48 +0000

This is the continuation of a post you can find here.

Where we left off...

So, we have an API that has a status endpoint and the code in place to document it with swagger.

Starting the service, we'll see that our documentation is available at http://localhost:8080/docs, and we have a status endpoint at http://localhost:8080/professionals-api/status.

So, let's keep going!

Professionals API

Let's create our model at a package called com.svinci.professionals.api.domain.professional:

First, professionals have jobs where they excel, right? And they may have more than one, so we'll need a case class for a job:

case class Job(name: String, description: String)

Now, contact information will be displayed or not according whether the client is authenticated or not, so we'll need a case class for the contact information:

case class ContactInformation(phone: String, eMail: String, address: String)

And then, we just need a professional. It will have an id, a name, an age, a list of jobs and, optionally, contact information:

case class Professional(id: String, name: String, age: Int, jobs: List[Job], contactInformation: Option[ContactInformation])

Also, when a client creates a professional, the id should be generated by the API, and contact information should be mandatory, so we'll need a create request, and it would be nice if it knew how to transform itself to a full professional:

case class ProfessionalCreationRequest(name: String, age: Int, jobs: List[Job], contactInformation: ContactInformation) {

  def toProfessional: Professional = Professional(
    id = UUID.randomUUID().toString,
    name = name,
    age = age,
    jobs = jobs,
    contactInformation = Option(contactInformation)
  )

}

That should do it, let's move on to the repository. Create a file called ProfessionalRepository.scala with the following content, the code is explained with comments:

package com.svinci.professionals.api.domain.professional

import scala.collection.concurrent.TrieMap

/**
  * We are using cake pattern to solve dependency injection without using any library. You can find a really good explanation of this pattern at http://www.cakesolutions.net/teamblogs/2011/12/19/cake-pattern-in-depth.
  *
  * This component holds the interface for professionals repository.
  */
trait ProfessionalRepositoryComponent {

  /**
    * Professional repository instance definition.
    */
  def professionalRepositoryInstance: ProfessionalRepository

  /**
    * Professional repository interface.
    */
  trait ProfessionalRepository {

    /**
      * Receives a creation request, transforms it to a Professional and stores it. Returns the created professional.
      */
    def create(professionalCreationRequest: ProfessionalCreationRequest): Professional

    /**
      * Removes the professional that has the given id. Returns the removed professional if any.
      */
    def remove(professionalId: String): Option[Professional]

    /**
      * Returns the professional that has the given id if any.
      */
    def findById(professionalId: String, displayContactInformation: Boolean): Option[Professional]

    /**
      * Returns a list with all the stored professionals.
      */
    def all(displayContactInformation: Boolean): List[Professional]

  }

}

/**
  * This component defines an implementation of ProfessionalRepository which holds the data in memory.
  */
trait InMemoryProfessionalRepositoryComponent extends ProfessionalRepositoryComponent {

  /**
    * In memory professional repository instantiation.
    */
  override def professionalRepositoryInstance: ProfessionalRepository = new InMemoryProfessionalRepository

  /**
    * In memory implementation of a professional repository.
    */
  class InMemoryProfessionalRepository extends ProfessionalRepository {

    /**
      * This is the data structure chosen to store the professionals.
      * In this Map you'll find professionals indexed by their ids.
      * TrieMap was chosen to make this class thread safe.
      */
    private[this] val storage: TrieMap[String, Professional] = TrieMap()

    /**
      * @inheritdoc
      */
    override def create(professionalCreationRequest: ProfessionalCreationRequest): Professional = {
      val professional = professionalCreationRequest.toProfessional
      storage.put(professional.id, professional)
      professional
    }

    /**
      * @inheritdoc
      */
    override def remove(professionalId: String): Option[Professional] = storage.remove(professionalId)

    /**
      * @inheritdoc
      */
    override def findById(professionalId: String, displayContactInformation: Boolean): Option[Professional] = {
      val maybeProfessional = storage.get(professionalId)

      if (displayContactInformation) {
        return maybeProfessional
      }

      maybeProfessional.map(professional => professional.copy(contactInformation = None))
    }

    /**
      * @inheritdoc
      */
    override def all(displayContactInformation: Boolean): List[Professional] = {
      val allProfessionals = storage.values.toList

      if (displayContactInformation) {
        return allProfessionals
      }

      allProfessionals.map(professional => professional.copy(contactInformation = None))
    }

  }

}

Notice that the read operations receive a flag called displayContactInformation, if it's true, they just return what they have, but if it's false they copy the professionals omitting the contact information.

I wouldn't normally put that logic in the repository, I would have a service layer or something, but as this is a really small API, I prefer keeping it simple.

Now we just need our servlet, we'll need a file called ProfessionalServlet.scala with the following content:

package com.svinci.professionals.api.domain.professional

import com.svinci.professionals.api.infrastructure.ServletSupport

/**
  * We are using cake pattern to solve dependency injection without using any library. You can find a really good explanation of this pattern at http://www.cakesolutions.net/teamblogs/2011/12/19/cake-pattern-in-depth.
  *
  * As this is an entry point to our application, there is no need to create an interface (it's a servlet after all, so there are no functions exposed).
  */
trait ProfessionalServletComponent {

  /**
    * As defined by cake pattern, with self type annotations we are defining that any class that extends this trait, needs to extend ProfessionalRepositoryComponent too.
    * This makes the interface and instance defined by ProfessionalRepositoryComponent available in this trait.
    */
  this: ProfessionalRepositoryComponent =>

  /**
    * Professional servlet instantiation.
    */
  def professionalServletInstance: ProfessionalServlet = new ProfessionalServlet(professionalRepositoryInstance)

  /**
    * This is the scalatra servlet that will serve our professionals management endpoints.
    */
  class ProfessionalServlet(val repository: ProfessionalRepository) extends ServletSupport {

    /**
      * This value defines the documentation for this endpoint. We are giving the endpoint a name, the return type, a description/summary and the schema for the expected boy.
      */
    private[this] val createProfessional = apiOperation[Professional]("create") summary "Insert a professional in the system" parameter bodyParam[ProfessionalCreationRequest]
    /**
      * We are routing our creation endpoint to the root of this servlet, passing the api operation for swagger to document.
      */
    post("/", operation(createProfessional)) {
      val creationRequest = parsedBody.extract[ProfessionalCreationRequest]
      repository.create(creationRequest)
    }

    /**
      * This value defines the documentation for this endpoint. We are giving the endpoint a name, the return type, a description/summary and the specifications for the id path param.
      */
    private[this] val removeProfessional = apiOperation[Option[Professional]]("remove") summary "Remove a professional by its id." parameter pathParam[String]("Id of the professional to remove")
    /**
      * We are routing our removal endpoint to the root of this servlet with the id as a path param, passing the api operation for swagger to document.
      */
    delete("/:id", operation(removeProfessional)) {
      val professionalId = params('id)
      repository.remove(professionalId)
    }

    /**
      * This value defines the documentation for this endpoint. We are giving the endpoint a name, the return type, a description/summary and the specifications for the id path parameter.
      */
    private[this] val findById = apiOperation[Option[Professional]]("findById") summary "Find a professional by its id." parameter pathParam[String]("id of the professional you are looking for.")
    /**
      * We are routing our findById endpoint to the root of this servlet with the id as a path param, passing the api operation for swagger to document.
      */
    get("/:id", operation(findById)) {
      val professionalId = params('id)
      val authenticated = request.getAttribute("authenticated").asInstanceOf[Boolean]
      repository.findById(professionalId, authenticated)
    }

    /**
      * This value defines the documentation for this endpoint. We are giving the endpoint a name, the return type and a description/summary.
      */
    private[this] val all = apiOperation[List[Professional]]("allProfessionals") summary "Retrieve all professionals."
    /**
      * We are routing our all endpoint to the root of this servlet, passing the api operation for swagger to document.
      */
    get("/", operation(all)) {
      val authenticated = request.getAttribute("authenticated").asInstanceOf[Boolean]
      repository.all(authenticated)
    }

    /**
      * This is the description of this servlet, requested by swagger.
      */
    override protected def applicationDescription: String = "Professionals management API."

  }

}

/**
  * This is the default instance of ProfessionalServletComponent. Here we define that the ProfessionalServletComponent will use the InMemoryProfessionalRepositoryComponent.
  */
object DefaultProfessionalServletComponent extends ProfessionalServletComponent with InMemoryProfessionalRepositoryComponent

One thing to notice about this servlet is that we are retrieving an attribute from the request to check if the client is authenticated: request.getAttribute("authenticated").asInstanceOf[Boolean]. This attribute should be populated by a filter or a before handler. Let's add it to the ServletSupport trait, in the before execution.

   /**
    * From the servlet request, check if the client is authenticated.
    *
    * This function can be exposed as protected as is, and let the servlets that extend ServletSupport use it,
    * but that wouldn't be convenient in a real situation, as checking if an authentication token is actually valid
    * would have performance implications (query to a database, REST API call, etc.).
    */
  private[this] def isAuthenticated()(implicit request: HttpServletRequest): Boolean = {
    val authenticationToken: Option[String] = Option(request.getHeader("X-PROFESSIONALS-API-TOKEN"))

    // Here you may perform token validation calling an authentication service or something.
    // We'll keep it simple in this example, and we'll assume the client is authenticated if the header is present.
    authenticationToken.isDefined
  }

  /**
   * Before every request made to a servlet that extends this trait, the function passed to `before()` will be executed.
   * We are using this to :
   *   - Set the Content-Type header for every request, as we are always going to return JSON.
   *   - Set the date to the request, so we can calculate spent time afterwards.
   *   - Generate a transaction identifier, an add it to the MDC, so we know which lines of logs were triggered by which request.
   *   - Check if the client is authenticated and add the flag to the request attributes.
   *   - Log that a request arrived.
   */
  before() {

    contentType = "application/json"
    request.setAttribute("startTime", new Date().getTime)
    MDC.put("tid", UUID.randomUUID().toString.substring(0, 8))

    val authenticated = isAuthenticated()
    request.setAttribute("authenticated", authenticated)

    logger.info(s"Received request ${request.getMethod} at ${request.getRequestURI}")

  }

There, we are creating the function isAuthenticated and calling it from the function we pass to before() that was already written from the previous post.

Now, we need to add the professional servlet to our Module object.

  /**
    * Default instance of ProfessionalServlet
    */
  def professionalServlet: DefaultProfessionalServletComponent.ProfessionalServlet = DefaultProfessionalServletComponent.professionalServletInstance

And now we can mount it in our ScalatraBootstrap class.

    context.mount(Module.professionalServlet, "/professionals-api/professionals", "professionals")

It's time to test it, let's boot the sbt console running sbt at the root of the project and then running jetty:start.

Let's create a professional:

$ curl -v localhost:8080/professionals-api/professionals -X POST -H "Content-Type: application/json" -d '{"name": "Mario", "age": 45, "jobs": [{"name": "Plumber", "description": "Fixing your pipes broh!"}], "contactInformation": {"phone": "555-1234", "eMail": "mario@gmail.com", "address": "Check your pipes ;)"}}'

{"id":"f967b053-68c1-41af-a8b9-27135516c0fc","name":"Mario","age":45,"jobs":[{"name":"Plumber","description":"Fixing your pipes broh!"}],"contactInformation":{"phone":"555-1234","eMail":"mario@gmail.com","address":"Check your pipes ;)"}}

As you see there, the API received the request, and returned the created Mario. Let's retrieve all professionals now, to check the API is actually storing stuff.

$ curl localhost:8080/professionals-api/professionals

[{"id":"f967b053-68c1-41af-a8b9-27135516c0fc","name":"Mario","age":45,"jobs":[{"name":"Plumber","description":"Fixing your pipes broh!"}]}]

There it is! You can see Mario there, yet it hasn't contact information, shall we check that the authentication works?

$ curl localhost:8080/professionals-api/professionals -H "X-PROFESSIONALS-API-TOKEN: 1234"

[{"id":"f967b053-68c1-41af-a8b9-27135516c0fc","name":"Mario","age":45,"jobs":[{"name":"Plumber","description":"Fixing your pipes broh!"}],"contactInformation":{"phone":"555-1234","eMail":"mario@gmail.com","address":"Check your pipes ;)"}}]

Awesome! We can see his mail and phone now. Let's retrieve Mario by his id f967b053-68c1-41af-a8b9-27135516c0fc.

$ curl localhost:8080/professionals-api/professionals/f967b053-68c1-41af-a8b9-27135516c0fc -H "X-PROFESSIONALS-API-TOKEN: 1234"

{"id":"f967b053-68c1-41af-a8b9-27135516c0fc","name":"Mario","age":45,"jobs":[{"name":"Plumber","description":"Fixing your pipes broh!"}],"contactInformation":{"phone":"555-1234","eMail":"mario@gmail.com","address":"Check your pipes ;)"}}

This is great, let's remove Mario now, and move on to check the documentation.

$ curl localhost:8080/professionals-api/professionals/f967b053-68c1-41af-a8b9-27135516c0fc -H "X-PROFESSIONALS-API-TOKEN: 1234" -X DELETE

{"id":"f967b053-68c1-41af-a8b9-27135516c0fc","name":"Mario","age":45,"jobs":[{"name":"Plumber","description":"Fixing your pipes broh!"}],"contactInformation":{"phone":"555-1234","eMail":"mario@gmail.com","address":"Check your pipes ;)"}}

$ curl localhost:8080/professionals-api/professionals -H "X-PROFESSIONALS-API-TOKEN: 1234"
[]

Great, we removed it and then the API is no longer retrieving it. Now, it's time to see how the documentation is working, but I don't want to keep reading JSON, I mean, JSON is a good format, and it's easy to read... but wouldn't a UI be great for this? It shouldn't be that hard to make...

Swagger UI

Well it isn't, and they actually did it. You'll need docker to keep following this example btw. Installation instructions can be found here.

Execute the following command one you have docker installed:

docker run -p 8081:8080 --name swagger-ui-instance -d swaggerapi/swagger-ui:v2.2.9

This will boot a docker container that serves swagger ui with an NGINX. It doesn't matter anyway. You just visit http://localhost:8081 in your browser, find the text input in the top navigation bad, type http://localhost:8080/docs there and hit Explore.

There you have it! An interactive graphic interface to see your API documentation.

See the version tag v2.2.9 we are setting when we run the docker instance? That's important. As I said on the previous post, scalatra swagger provides Swagger V1 JSONS, which are deprecated, and v2.2.9 of swagger UI is the latest version that support Swagger V1.

Packaging a standalone JAR

This is a quick step, we just need to add two lines to the build.sbt file, as most of the setup was already done when we started the project:

mainClass in Compile := Some("com.svinci.professionals.api.JettyLauncher")
mainClass in assembly := Some("com.svinci.professionals.api.JettyLauncher")

This way, we are telling assembly which is our main class. That JettyLauncher object was written in the previous post and it's an object that's capable of starting the JettyServer as the scalatra plugin does.

Now we'll run some commands:

First, to assembly the JAR we run sbt assembly. There are going to be a couple WARN lines of log, but they aren't important.
Second, we go to the directory where the JAR is located doing cd target/scala-2.11.
There you'll find the JAR called professionals-api.jar. By doing java -jar professionals-api.jar you can run the JAR with the default configuration.
The configuration present in application.conf can be overridden wit java options as in java -Dapplication.port=8082 -jar professionals-api.jar <- this command will run the server but it will listen in the port 8082.

Conclusion

Scala, SBT, Scalatra and Swagger are awesome. Yes, I'd like to move to Swagger V2, but it isn't that problematic. APIs written with these technologies work just fine and perform very well.

You can find the code to this post series here.

Thanks a lot for reading!

See you in the comments!

Scala REST API documented with Swagger (Part 1 of 2)

Sebastian G. Vinci — Mon, 17 Apr 2017 01:40:41 +0000

In this post series we are going to build a REST API using SBT, Scala and Scalatra. Something I like a lot about Scalatra is that it has a library to document your API using Swagger with very little efort.

Our API will be very simple. It will manage a catalog of professionals, but contact information will only be provided if the client is authenticated.

DOWNSIDE: Scalatra swagger only supports swagger v1, which is deprecated now. There are tools to convert swagger v1 JSONs to swagger v2, and even to RAML, so this doesn't really bother me.

What we'll build on this part

Part 1 will only consist of the set up of the project, a status endpoint and the implementation of a servlet to retrieve swagger documentation.

The endpoints to manage professionals, the authentication filter, the standalone build (jar deployment) and a UI to see the API documentation on an interactive manner will be approached on the second part.

Preconditions

I'll assume you are familiar with the Scala language, SBT and REST.

To work through this tutorial you'll need to install JDK 8 (open jdk is fine) and SBT.

All the commands I'll provide to install, run and do stuff were tested on Linux Mint 18. They'll probably work on Mac OS without any issue. If you're working on windows I'm really sorry.

Set Up

I'll show how to create this project by hand, but there are project templates and stuff you can use. I think SBT has a command to generate empty projects, but also lighbend activator is a fine tool to start from a template.

First of all, let's create our project directory running mkdir scala-scalatra-swagger-example.

Now, this are the directories you'll need to have:

project
src/main/resources
src/main/scala
src/main/webapp/WEB-INF

Under project directory, you need to create a file called build.properties with the following content:

sbt.version=0.13.13

We are defining our project to be built using sbt 0.13.13.

Also, under project directory, you'll need a file called plugins.sbt with the following content:

addSbtPlugin("org.scalatra.sbt" % "scalatra-sbt" % "0.5.1")

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.4")

addSbtPlugin("org.scalariform" % "sbt-scalariform" % "1.6.0")

Here we are importing scalatra SBT plugin, needed to work with scalatra (it will provide some tools to work with Jetty and stuff). Then, we import assembly, which we'll use to build our standalone jar. Finally, we are adding scalariform, so our code gets formated on compile time.

Now, under src/main/webapp/WEB-INF we'll need a file called web.xml with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
      version="3.0">

  <!--
    This listener loads a class in the default package called ScalatraBootstrap.
    That class should implement org.scalatra.LifeCycle.  Your app can be
    configured in Scala code there.
  -->
  <listener>
    <listener-class>org.scalatra.servlet.ScalatraListener</listener-class>
  </listener>
</web-app>

This basically tells jetty that our only listener is the one provided by Scalatra.

Now, under src/main/resources we'll need our application configuration file, called application.conf.

application.port=8080

Yes, it only contains the port where the API will listen, but I wanted to show how configuration will work on an application like this, as it is a real need in production.

Also, logging is a necessity in production, so, under src/main/resources too, we'll need our logback.xml.

<configuration>

  <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>[%d] [%level] [tid: %X{tid:-none}] [%t] [%logger{39}] : %m%n%rEx{10}</pattern>
      <charset>utf8</charset>
    </encoder>
  </appender>

  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
  </root>

</configuration>

This only defines a console appender (sends logs to the standard output), with a really standard pattern (the only weird thing is the tid thingy, we'll talk about it later).

Now we'll glue everything together with a build.sbt file.

import com.typesafe.sbt.SbtScalariform
import com.typesafe.sbt.SbtScalariform.ScalariformKeys
import org.scalatra.sbt._

import scalariform.formatter.preferences._

val ScalatraVersion = "2.5.0"

ScalatraPlugin.scalatraSettings

organization := "com.svinci.professionals"

name := "api"

version := "0.0.1-SNAPSHOT"

scalaVersion := "2.11.8"

resolvers += Classpaths.typesafeReleases

SbtScalariform.scalariformSettings

ScalariformKeys.preferences := ScalariformKeys.preferences.value
  .setPreference(AlignSingleLineCaseStatements, true)
  .setPreference(DoubleIndentClassDeclaration, true)
  .setPreference(AlignParameters, true)
  .setPreference(AlignArguments, true)

libraryDependencies ++= Seq(
  "org.scalatra"        %%  "scalatra"          % ScalatraVersion,
  "org.scalatra"        %%  "scalatra-json"     % ScalatraVersion,
  "org.scalatra"        %%  "scalatra-swagger"  % ScalatraVersion,
  "org.json4s"          %%  "json4s-native"     % "3.5.0",
  "org.json4s"          %%  "json4s-jackson"    % "3.5.0",
  "com.typesafe"        %   "config"            % "1.3.1",
  "ch.qos.logback"      %   "logback-classic"   % "1.1.5"             % "runtime",
  "org.eclipse.jetty"   %   "jetty-webapp"      % "9.2.15.v20160210"  % "container;compile",
  "javax.servlet"       %   "javax.servlet-api" % "3.1.0"             % "provided"
)

assemblyJarName in assembly := "professionals-api.jar"

enablePlugins(JettyPlugin)

What's this build.sbt doing?

At the top of the file we are importing our plugins and the things we need to set up the build of our project.
Then we define a value containing scalatra version (2.5.0) and applying ScalatraPlugin.scalatraSettings.
After that, we are defining some artifact information (organization, name, version, scala version), adding typesafe repository (lightbend used to be called typesafe) to our resolvers and then we configure scalariform.
Then, our dependencies are being defined. Notice scalatra dependencies, json4s, config, logback and jetty.
Then, we are configuring assembly, so everything is packaged under a jar called professionals-api.jar
Finally, we are enabling jetty plugin (it comes with the scalatra plugin).

At this point, we can run sbt on the root of the project to see that it's loaded correctly, and even compile it (although there is no code to compile).

Now it would be the time to open this project with an IDE (yes, I've been using vim up to this point).

Utility objects and traits

We will create now some infrastructure for our code. First, let's create our package com.svinci.professionals.api.infrastructure. Under this package, let's write the following (the code is explained with comments):

A file called Configuration.scala with the following content:

package com.svinci.professionals.api.infrastructure

import com.typesafe.config.ConfigFactory

/**
  * This will provide a singleton instance of our configuration.
  * Also, it will encapsulate typesafe.config, so the rest of the application doesn't need to know about the configuration library we are using.
  */
object Configuration {

  /**
    * This is our configuration instance. Private and immutable.
    */
  private[this] val configuration = ConfigFactory.load()

  /**
    * Methods like this one should be defined to access any type of configuration from its key.
    * The reason we do it is to define an interface that makes sense for our application, and make the rest of the code
    * agnostic to what library we are using. Just a good practice.
    * @param key Configuration key.
    * @return The configured Int.
    */
  def getInt(key: String): Int = configuration.getInt(key)

}

A file called ApiInformation.scala with the following content:

package com.svinci.professionals.api.infrastructure

import org.scalatra.swagger.{ApiInfo, Swagger}

/**
  * Information of our API as a whole.
  */
object ProfessionalsApiInfo extends ApiInfo(
  title = "professionals-api",
  description = "Professionals CRUD operations.",
  termsOfServiceUrl = "some terms of service URL",
  contact = "some contact information",
  license = "MIT",
  licenseUrl = "http://opensource.org/licenses/MIT"
)

/**
  * Swagger instance for our API. It's defined  as an object so we have only one instance for all our resources.
  */
object ProfessionalsApiSwagger extends Swagger(swaggerVersion = Swagger.SpecVersion, apiVersion = "1.0.0", apiInfo = ProfessionalsApiInfo)

A file called ServletSupport.scala with the following content:

package com.svinci.professionals.api.infrastructure

import java.util.{Date, UUID}

import org.json4s.{DefaultFormats, Formats}
import org.scalatra._
import org.scalatra.json._
import org.scalatra.swagger.SwaggerSupport
import org.slf4j.{LoggerFactory, MDC}

/**
  * We'll have a couple servlets probably (a status endpoint, the CRUD servlet for our professionals, and if we deploy this to production we'll probably add some more),
  * so it's convenient to have the things every servlet will need to define in one trait to extend it.
  * 
  * This trait extends ScalatraServlet and adds json and swagger support.
  */
trait ServletSupport extends ScalatraServlet with JacksonJsonSupport with SwaggerSupport {

  /**
    * As we are going to document every endpoint of our API, we'll need our swagger instance in everyone of our servlets.
    */
  override protected implicit def swagger = ProfessionalsApiSwagger

  /**
    * This is a logger... to log stuff.
    */
  private[this] val logger = LoggerFactory.getLogger(getClass)
  /**
    * Scalatra requires us to define an implicit Formats instance for it to know how we want JSONs to be serialized/deserialized.
    * It provides a DefaultFormats that fill all our needs today, so we'll use it. 
    */
  protected implicit lazy val jsonFormats: Formats = DefaultFormats

  /**
    * Before every request made to a servlet that extends this trait, the function passed to `before()` will be executed.
    * We are using this to :
    *   - Set the Content-Type header for every request, as we are always going to return JSON.
    *   - Set the date to the request, so we can calculate spent time afterwards.
    *   - Generate a transaction identifier, an add it to the MDC, so we know which lines of logs were triggered by which request.
    *   - Log that a request arrived.
    */
  before() {

    contentType = "application/json"
    request.setAttribute("startTime", new Date().getTime)
    MDC.put("tid", UUID.randomUUID().toString.substring(0, 8))

    logger.info(s"Received request ${request.getMethod} at ${request.getRequestURI}")

  }

  /**
    * NotFound handler. We just want to set the status code, and avoid the huge stack traces scalatra returns in the body.
    */
  notFound {

    response.setStatus(404)

  }

  /**
    * After every request made to a servlet that extends this trait, the function passed to `after()` will be executed.
    * We are using this to:
    *   - Retrieve the start time added in the `before()` handler an calculate how much time the API took to respond.
    *   - Log that the request handling finished, with how much time it took.
    */
  after() {

    val startTime: Long = request.getAttribute("startTime").asInstanceOf[Long]
    val spentTime: Long = new Date().getTime - startTime
    logger.info(s"Request ${request.getMethod} at ${request.getRequestURI} took ${spentTime}ms")

  }

}

Status Endpoint

We'll code now a status endpoint that will always return a JSON with the following content:

{
  "healthy": true
}

If you had a database, or an API you depend on, you could add their statuses there. First of all, let's create our package com.svinci.professionals.api.domain.status and, under this package, write the following (the code is explained with comments):

A file called Status.scala with the following content:

package com.svinci.professionals.api.domain.status

/**
  * This is the object our status endpoint will convert to JSON and return.
  */
case class Status(healthy: Boolean)

A file called StatusService.scala with the following content:

package com.svinci.professionals.api.domain.status

/**
  * We are using cake pattern to solve dependency injection without using any library. You can find a really good explanation of this pattern at http://www.cakesolutions.net/teamblogs/2011/12/19/cake-pattern-in-depth.
  * 
  * This is the component that defines a StatusService interface (trait, actually), and names an instance of it.
  */
trait StatusServiceComponent {

  /**
    * This is the definition of the instance of StatusService an implementation of this component will hold.
    */
  def statusServiceInstance: StatusService

  /**
    * StatusService interface definition.
    */
  trait StatusService {

    /**
      * Retrieve the application status.
      * @return The application status.
      */
    def status: Status

  }

}

/**
  * Default StatusServiceComponent implementation.
  */
trait DefaultStatusServiceComponent extends StatusServiceComponent {

  /**
    * Here, we define how the DefaultStatusService is created.
    */
  override def statusServiceInstance: StatusService = new DefaultStatusService

  /**
    * Default StatusService implementation.
    */
  class DefaultStatusService extends StatusService {

    /**
      * @inheritdoc
      */
    override def status: Status = Status(healthy = true)

  }

}

A file called StatusServlet.scala with the following content:

package com.svinci.professionals.api.domain.status

import com.svinci.professionals.api.infrastructure.ServletSupport

/**
  * We are using cake pattern to solve dependency injection without using any library. You can find a really good explanation of this pattern at http://www.cakesolutions.net/teamblogs/2011/12/19/cake-pattern-in-depth.
  *
  * As this is an entry point to our application, there is no need to create an interface (it's a servlet after all, so there are no functions exposed).
  */
trait StatusServletComponent {

  /**
    * As defined by cake pattern, with self type annotations we are defining that any class that extends this trait, needs to extend StatusServiceComponent too.
    * This makes the interface and instance defined by StatusServiceComponent available in this trait.
    */
  this: StatusServiceComponent =>

  /**
    * This is the StatusServlet instance held by this component. Notice that we are instantiating StatusServlet passing the statusServiceInstance provided by StatusServiceComponent.
    */
  def statusServletInstance: StatusServlet = new StatusServlet(statusService = statusServiceInstance)

  /**
    * This is the scalatra servlet that will serve our status endpoint.
    */
  class StatusServlet(val statusService: StatusService) extends ServletSupport {

    /**
      * This value defines the documentation for this endpoint. We are giving the endpoint a name, the return type and a description/summary.
      */
    private[this] val getStatus = apiOperation[Status]("status") summary "Retrieve API status."

    /**
      * We are routing our status endpoint to the root of this servlet, and passing to scalatra our apiOperation.
      */
    get("/", operation(getStatus)) {
      statusService.status
    }

    /**
      * This is the description of this servlet, requested by swagger.
      */
    override protected def applicationDescription: String = "API Status."

  }

}

/**
  * This is the default instance of StatusServletComponent. Here we define that the StatusServletComponent will use the DefaultStatusServiceComponent.
  */
object DefaultStatusServletComponent extends StatusServletComponent with DefaultStatusServiceComponent

Now we go back to the package called com.svinci.professionals.api.infrastructure and create a file called Module.scala with the following content:

package com.svinci.professionals.api.infrastructure

import com.svinci.professionals.api.domain.status.DefaultStatusServletComponent

/**
  * We are using cake pattern to solve dependency injection without using any library. You can find a really good explanation of this pattern at http://www.cakesolutions.net/teamblogs/2011/12/19/cake-pattern-in-depth.
  *
  * In this object we'll hold all the instances required by our application.
  */
object Module {

  /**
    * Default instance of StatusServlet.
    */
  def statusServlet: DefaultStatusServletComponent.StatusServlet = DefaultStatusServletComponent.statusServletInstance

}

Now, for our API to run we'll need an application to run, just as any scala application. Under a package called com.svinci.professionals.api we'll write a file called JettyLauncher.scala with the following content:

package com.svinci.professionals.api

import com.svinci.professionals.api.infrastructure.Configuration
import org.eclipse.jetty.server.Server
import org.eclipse.jetty.servlet.DefaultServlet
import org.eclipse.jetty.webapp.WebAppContext
import org.scalatra.servlet.ScalatraListener

/**
  * This is the MainClass of our application.
  */
object JettyLauncher extends App {

  /**
    * Server instantiation. We are retrieving the port we need to listen at from our Configuration object.
    */
  val server = new Server(Configuration.getInt("application.port"))

  /**
    * Web application context instantiation.
    * This class will hold the context path, the resource base, the event listener and one servlet.
    */
  val context = new WebAppContext()

  context setContextPath "/"
  context.setResourceBase("src/main/webapp")
  context.addEventListener(new ScalatraListener)  // We use scalatra listener as event listener.
  context.addServlet(classOf[DefaultServlet], "/")  // We don't need to add our servlets here, we're going to add them using the scalatra life cycle.

  server.setHandler(context)  // We add the WebAppContext to the server.

  server.start()  // Start the server.
  server.join()  // Join the server's thread pool so the application doesn't quit.

}

And now we need to add our servlets to the scalatra lyfe cycle, so we write a file called ScalatraBootstrap.scala under the src/main/scala directory (outside any package) with the following content:

package com.svinci.professionals.api

import javax.servlet.ServletContext

import com.svinci.professionals.api.infrastructure.Module
import org.scalatra._
import org.slf4j.LoggerFactory

/**
  * This class in looked up by scalatra and automatically instantiated. Here we need to code all the start up code of our API.
  */
class ScalatraBootstrap extends LifeCycle {

  private[this] val logger = LoggerFactory.getLogger(getClass)

  /**
    * In the method we need to mount our servlets to the ServletContext provided as parameter.
    * Any additional startup code should be wrote here too (warm up, scheduled tasks initialization, etc.).
    */
  override def init(context: ServletContext) {

    logger.info(context.getContextPath)

    logger.info("Mounting Changas API servlets.")
    context.mount(Module.statusServlet, "/professionals-api/status", "status")
    logger.info(s"API started.")

  }

}

As you can see there, we are mounting our status servlet to the path /professionals-api/status, so all the routing we do inside the servlet will be relative to that path. The third parameter we pass to the mount method is a name we are assigning to that servlet.

This class needs to be in that location so scalatra can find it. If you place it anywhere else you'll see an assertion error: java.lang.AssertionError: assertion failed: No lifecycle class found!.

Now it's time to test our server and our status endpoint. Boot the sbt console by running sbt on the root of the project, and once inside run jetty:start. You'll have control over sbt console after executing that command, and to stop the API you can run jetty:stop. Of course, CTRL + C will work too, but that will close the sbt console too.

You can test the API now:

$ curl http://localhost:8080/professionals-api/status
{"healthy":true}

Great, stop the server now and we'll get back to coding.

Swagger Documentation

Now we'll create the servlet that will return our endpoints documentation.

First, we need to create a package called com.svinci.professionals.api.domain.docs, and under that package we'll write a file called DocsServlet.scala with the following content:

package com.svinci.professionals.api.domain.docs

import com.svinci.professionals.api.infrastructure.ProfessionalsApiSwagger
import org.scalatra.ScalatraServlet
import org.scalatra.swagger.NativeSwaggerBase

/**
  * This servlet, as is, will be able to return swagger v1 JSONs. This is the entry point to our documentation.
  */
class DocsServlet extends ScalatraServlet with NativeSwaggerBase {

  /**
    * Application swagger global instance.
    */
  override protected implicit def swagger = ProfessionalsApiSwagger

}

This is a really simple servlet, so I didn't feel like doing cake pattern here, it didn't make sense. Now, in the Module.scala object we'll need to add a new function to retrieve an instance of this servlet:

  /**
    * Swagger documentation servlet instance.
    */
  def docsServlet: DocsServlet = new DocsServlet

Once we have everything in place, we mount the new servlet to the ServletContext on ScalatraBootstrap.

context.mount(Module.docsServlet, "/docs", "docs")

We can now start our server again and test:


$ curl http://localhost:8080/docs
{"apiVersion":"1.0.0","swaggerVersion":"1.2","apis":[{"path":"/professionals-api/status","description":"API Status."}],"authorizations":{},"info":{}}

$ curl http://localhost:8080/docs/professionals-api/status
{"apiVersion":"1.0.0","swaggerVersion":"1.2","resourcePath":"/professionals-api/status","produces":["application/json"],"consumes":["application/json"],"protocols":["http"],"apis":[{"path":"/professionals-api/status/","operations":[{"method":"GET","summary":"Retrieve API status.","position":0,"notes":"","deprecated":false,"nickname":"status","parameters":[],"type":"Status"}]}],"models":{"Status":{"id":"Status","name":"Status","qualifiedType":"com.svinci.professionals.api.domain.status.Status","required":["healthy"],"properties":{"healthy":{"position":0,"type":"boolean"}}}},"basePath":"http://localhost:8080"}

Notice that general information of our API is found at /docs, and then at /docs/professionals-api/status you'll find documentation for the servlet that was mounted on /professionals-api/status.

Conclusion

That's it in this part of the post series. We have an API working, with a status endpoint and swagger documentation.

Right now the second part hasn't been written, but in a couple days I'll have it done and I will put the link here.

The code to this example can be found here.

See you in the comments!

Hi, I'm Sebastian Vinci

Sebastian G. Vinci — Sun, 16 Apr 2017 06:57:24 +0000

I have been coding for 8 years.

You can find me on GitHub as svinci.

I live in Buenos Aires, Argentina.

I mostly program in these languages: Java, Scala, Python and JavaScript.

I am currently learning more about infrastructure as a code.

Nice to meet you.

Deploy using docker swarm

Sebastian G. Vinci — Sun, 16 Apr 2017 06:42:39 +0000

I just finished this tutorial that shows how to build a web site using ReactJS and Redux. For testing purposes I also showed how to build a REST API using nodeJS.

Now, let's deploy those two using docker swarm.

Installation

First of all, you'll need to clone the React tutorial repository from here.

Now, you'll need to install Docker. Installation instructions can be found here.

Once you have docker running, you need to restart the docker daemon to make it run in swarm mode. This can be achieved by running the following command:

$ docker swarm init

Now you're good to go.

Docker Images

Now we'll need to write the docker images for our API and our UI.

API Image

Let's start with the API. Here's the Dockerfile for our NodeJS API.

FROM node:boron

RUN mkdir -p /usr/src/app/config
WORKDIR /usr/src/app

COPY package.json /usr/src/app/
RUN npm install

COPY config/default.json /usr/src/app/config/default.json
COPY index.js /usr/src/app/index.js

EXPOSE 8100

CMD ["npm", "run", "start"]

We are building our image from node:boron (NodeJS 6). Then we're creating a directory in /usr/src/app to deploy the API there, copying the package.json there and installing dependencies.

Then we copy the configuration and our index.js file. We expose port 8100 and start the server.

Now we build the image running docker build -t svinci/todo-api .. Feel free to change the namespace.

UI Image

Now, the UI will be a little more tricky. The container has to deploy an NGINX to work as a web server of the UI, and it has to have the rules to acces the API passing through it.

Let's start by making a directory called conf that will hold NGINX configuration. In it we'll write a file called nginx.conf with the following content.

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections 1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '[$remote_addr] [$time_local] [$request] '
                      'status code: $status, response size: $body_bytes_sent, referer: "$http_referer", '
                      'user agent: "$http_user_agent", $request_time seconds';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    gzip on;
    gzip_disable "msie6";

    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/vnd.ms-fontobject application/x-font-ttf font/opentype image/svg+xml image/x-icon;

    include /etc/nginx/conf.d/*.conf;
}

This basically sets the queue size, mime types, access log format and gzip. Now, in a subdirectory of conf called conf.d, we'll write the following in a file called default.conf.

server {
    listen       80;
    server_name  localhost;

    location /todos {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://api:8100;
    }

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

}

This sets the listening port to 80, configures a proxy pass to the API and sets the site entry point.

Now, with this done, here's the Dockerfile.

FROM nginx

COPY conf /etc/nginx

COPY index.html /usr/share/nginx/html
COPY bundle.js /usr/share/nginx/html

EXPOSE 80

We are building our image from nginx, copying our NGINX configuration, and copying our bundle and our index.html to the default static content location for the web server. Then we expose the port 80.

Now, before actually building the docker image, we have to do a tiny modification. This example was meant to be run locally, so we need to modify the file under todo-site/src/client.js to change http://localhost:8100/todos by just /todos so the UI executes the requests through our NGINX. It should look like this:

import * as superagent from "superagent";

export function get() {

    return new Promise((resolve, reject) => {
        superagent.get("/todos")
            .end((error, result) => {
                error ? reject(error) : resolve(result.body);
            });
    });

}

export function add(text) {

    return new Promise((resolve, reject) => {
        superagent.post("/todos")
            .send({'text': text})
            .end((error, result) => {
                error ? reject(error) : resolve(result.body);
            });
    });

}

export function toggle(id) {

    return new Promise((resolve, reject) => {
        superagent.patch("/todos/" + id)
            .end((error, result) => {
                error ? reject(error) : resolve(result.body);
            });
    });

}

Now we build the image running npm install && npm run build && docker build -t svinci/todo-site ..

Compose

We now have to write our compose file. This file defines all the services we need running, with its dependencies, replication factor, networks and stuff.

We need to write the below content into a file called compose.yml on the root of the repository.

version: "3"

services:
  api:
    image: svinci/todo-api
    networks:
      - todo_network
    ports:
      - 8100
    deploy:
      replicas: 1
      update_config:
        parallelism: 1
      restart_policy:
        condition: on-failure
  site:
    image: svinci/todo-site
    ports:
      - 80:80
    networks:
      - todo_network
    depends_on:
      - api
    deploy:
      replicas: 1
      update_config:
        parallelism: 1
      restart_policy:
        condition: on-failure

networks:
  todo_network:

We are defining two services here:

api: The To Do REST API.
site: Our NGINX with the static content.

As you see, we are setting the images it needs to use, and the network they belong to.

We are also defining that the site depends on the API to work.

Deployment configuration is being provided too. Notice the replicas under deploy for each service, we could have redundancy here out of the box.

Deploy

Now, let's deploy this. To deploy we need to run the following command:

docker stack deploy --compose-file=compose.yml todo_stack

If you visit http://localhost/ in your browser after a couple seconds (NGINX takes like 15 seconds to start), you'll see the site fully working.

Conclusion

It's fairly easy to dockerize stuff, and I really prefer to work this way as I don't need to install too much software on my computer (NGINX for example). Also, if you deploy like this in production, your local environment will ressemble production a lot, which is awesome.

You can find the source code of this example here on a branch called docker-swarm.

See you in the comments!

React JS Web Site Example (Almost like real life).

Sebastian G. Vinci — Sun, 16 Apr 2017 04:44:58 +0000

I've been trying to use react on my personal projects for a couple weeks now, but I found out that there isn't one example on the internet (that I could find) that ressembles what I want in a real life scenario.

Asynchronous HTTP requests, loading animations, error pages, etc. None of those things are covered by one concise example that can be found on the first two pages of google.

Having said that, I took one example that took me far enough, and started researching and building on top of it.

What are we going to do?

We are going to build a simple To Do List web application.

To do this, we are going to build a very simple REST API in Node.js using rest-api-starter, and a web site based on React.JS, Redux and Bootstrap.

What am I going to need to follow this tutorial?

First, a Node.js 6 installation, an IDE and a browser (which you probably have already, as you are reading this). Instructions on how to install Node.js can be found here.

Second, a Python 2.7 installation. If you are on a Mac OS or an Ubuntu based system, you already have it. Instructions on how to install Python can be found here.

All the commands I'll provide to install, run and do stuff were tested on Linux Mint 18. They'll probably work on Mac OS without any issue. If you're working on windows I'm really sorry.

Can we start coding already?

Allright, first of all, let's make our directories.

$ mkdir todo-api
$ mkdir todo-site

API project

Now, let's start with the API. We are going to cd to the API directory, and run npm init.

$ cd todo-api
$ npm init

You can leave all the defaults.

Now we have a node project there, we are going to install rest-api-starter and uuid (for id generation and stuff).

$ npm install --save rest-api-starter uuid

Now, rest-api-starter requires a tiny configuration file on a subdirectory called config.

$ mkdir config
$ cd config && touch default.json

The config/default.json file should look exactly like the one below:

{
  "app": {
    "http": {
      "port": 8100,
      "host": "0.0.0.0",
      "queue": 10,
      "secret": "",
      "transactionHeader": "X-REST-TRANSACTION"
    },
    "log": {
      "level": "info",
      "transports": [
        {
          "type": "console"
        }
      ]
    }
  }
}

Now, let's code our rest API. We need CORS support to be able to easily develop on our local environment and three handlers:

POST /todos: Create an item.
GET /todos: Retrieve all items.
PATCH /todos/:id: Mark an item as done or undone.

Also, an OPTIONS handler for each path should be implemented for CORS support. So, our index.js file will look like this:

const uuid = require('uuid');
const serveBuilder = require('rest-api-starter').server;
const todos = [];

const router = (app) => {

    app.use(function(req, res, next) {
        res.header("Access-Control-Allow-Origin", "*");
        res.header("Access-Control-Allow-Methods", "GET, POST, PATCH, OPTIONS");
        res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
        next();
    });

    app.options('/todos', (request, response) => response.status(200).send());

    app.post('/todos', (request, response) => {
        const todo = {
            'id': uuid.v4(),
            'isDone': false,
            'text': request.body.text
        };
        todos.push(todo);
        response.send(todo);
    });

    app.get('/todos', (request, response) => {
        response.send(todos);
    });

    app.options('/todos/:id', (request, response) => response.status(200).send());

    app.patch('/todos/:id', (request, response) => {
        let result = null;
        todos.forEach((todo) => {
            if (todo.id === request.params.id) {
                todo.isDone = !todo.isDone;
                result = todo;
            }
        });

        if (!result) {
            response.status(404).send({'msg': 'todo not found'});
        } else {
            response.send(result);
        }
    });

};

serveBuilder(router);

Now, add "start": "node index.js" to the scripts section of your package.json file to start the server. By running npm run start on the root of the API project, you'll have your server listening on http://localhost:8100.

Site project

Now we're going to cd to the site project and run an npm init there. Defaults are fine here too.

$ cd todo-site
$ npm init

And now, we install the dependencies we need:

$ npm install --save babel-core babel-loader babel-preset-es2015 babel-preset-react bootstrap jquery superagent webpack react react-dom react-redux redux redux-thunk style-loader css-loader

Webpack

We'll be using webpack to transpile and unify all the code into una file called bundle.js, so it will be convenient to add "build": "webpack --debug" and "serve": "npm run build && python -m SimpleHTTPServer 8080" to the scripts section in our package.json.

Now we'll need a webpack.config.js.

const webpack = require('webpack');

module.exports = {
    entry: {
        main: './src/app.js'
    },
    output: {
        path: __dirname,
        filename: 'bundle.js'
    },
    module: {
        loaders: [
            {
                test: /\.js$/,
                exclude: /node_modules/,
                loader: 'babel-loader',
                query: { presets: [ 'es2015', 'react' ] }
            },
            {
                test: /\.css$/,
                loader: "style-loader!css-loader"
            },
            {
                test: /\.(png|jpg|gif|ttf|svg|woff|woff2|eot)$/,
                loader: "url-loader"
            }
        ]
    },
    plugins: [
        new webpack.ProvidePlugin({
            $: "jquery",
            jQuery: "jquery",
            bootstrap: "bootstrap"
        })
    ]
};

This webpack configuration transpiles all the javascript files that are using ES6 and JSX, and then puts them together, with all their dependencies, in one big file called bundle.js.

If any stylesheet is required from src/app.js, it will import it and add it to the bundle (following any imports made from the stylesheets) and the generated bundle script will add a <style> tag to the HTML.

It also uses the ProvidePlugin to expose JQuery and bootstrap, so we can forget about importing them.

Stylesheets

Now, let's start with some structure. Let's create a directory called css in the root of the project and add the following app.css.

@import "../node_modules/bootstrap/dist/css/bootstrap.min.css";

That stylesheet just imports bootstrap, but you can add custom style and import any stylesheet you want there. That should be the entrypoint for all the stylesheets in the project.

HTML. Site entrypoint.

Then, we create our index.html in the project.

<!DOCTYPE html>
<html>
    <head>
        <title>Todo List</title>

        <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    </head>
    <body>
        <div id="app"></div>

        <script src="bundle.js"></script>
    </body>
</html>

This is a pretty simple HTML file. It has a title, the viewport recommended by bootstrap, a div with the id app and the import of our bundle.

That div called app will be our application container. We'll tell react to render its components there.

React Components

Let's write our React.js components. A React component is an independent piece of code that receives some props and renders HTML from that props. It should JUST be React, a component's code should know nothing about Redux. Just presentation. (I can't stress this enough).

Create a directory called src on the root of the project, and write the code below to a file named components.js.

import React from 'react';

function Todo(props) {
    const { todo } = props;
    if (todo.isDone) {
        return <del>{todo.text}</del>
    } else {
        return <span>{todo.text}</span>
    }
}

function TodoList(props) {

    const { todos, toggleTodo, addTodo } = props;

    const onSubmit = (event) => {
        event.preventDefault();

        const textInput = document.getElementById('todo-input');

        const text = textInput.value;

        if (text && text.length > 0) {
            addTodo(text);
        }

        textInput.value = '';
    };

    const toggleClick = id => event => toggleTodo(id);

    return (
        <div className='todo-list-container'>
            <div className="panel panel-default">
                <div className="panel-body">
                    <form onSubmit={onSubmit}>
                        <div className="form-group">
                            <label>To Do Text: </label>
                            <input id="todo-input" type='text'
                                   className='todo-input form-control'
                                   placeholder='Add todo' />
                        </div>
                        <button type="submit" className="btn btn-default">Submit</button>
                    </form>
                </div>
            </div>
            {
                todos.length > 0 ?
                    <div className='todo-list list-group'>
                        {todos.map(t => (
                            <a key={t.id}
                                className='todo-list-item list-group-item'
                                onClick={toggleClick(t.id)}>
                                <Todo todo={t} />
                            </a>
                        ))}
                    </div> :
                    <div className="alert alert-info" role="alert">ToDo list is empty.</div>
            }
        </div>
    );
}

function Layout(props) {
    return (
        <div className='container'>
            <div className='row'>
                <div className='col-lg-6 col-lg-offset-3'>
                    <div className='page-header'>
                        <h1>To Do List <small>Keep it organized.</small></h1>
                    </div>
                    {props.children}
                </div>
            </div>
        </div>
    )
}

function ProgressBar(props) {
    const { completed } = props;

    const style = { 'width': completed + '%'};

    return (
        <div className="progress">
            <div className="progress-bar progress-bar-striped active" role="progressbar" aria-valuenow={completed} aria-valuemin='0' aria-valuemax='100' style={style}>
                <span className="sr-only">{completed}% Complete</span>
            </div>
        </div>
    )
}

export function TodoPage(props) {

    const {state, toggleTodo, addTodo, retrieveTodos } = props;

    if (state.error) {
        return (
            <Layout>
                <div className="alert alert-danger" role="alert">{state.error.toString()}</div>
                <input className='retry-button btn btn-default' type='button' value='Retry' onClick={retrieveTodos}/>
            </Layout>
        );
    } else if (state.initialized) {
        return (
            <Layout>
                <TodoList todos={state.todos} toggleTodo={toggleTodo} addTodo={addTodo} />
            </Layout>
        )
    } else {
        retrieveTodos();
        return (
            <Layout>
                <ProgressBar completed="45"/>
            </Layout>
        );
    }

}

That's our presentation layer. We export one function, called TodoPage, which uses some components only available inside the module.

These components receive the application's state, and three actions: toggleTodo, addTodo, retrieveTodos. The components don't know what they do, they just know how to invoke them, and they don't even care about a return value.

Notice that the components receive the state and the actions, and just cares about how the state is displayed, and how are those actions mapped to HTML events.

API Client

Now, let's write our API client using superagent and ES6 promises. under a directory called src created on the root of our project write the following code on a file called client.js.

import * as superagent from "superagent";

export function get() {

    return new Promise((resolve, reject) => {
        superagent.get("http://localhost:8100/todos")
            .end((error, result) => {
                error ? reject(error) : resolve(result.body);
            });
    });

}

export function add(text) {

    return new Promise((resolve, reject) => {
        superagent.post("http://localhost:8100/todos")
            .send({'text': text})
            .end((error, result) => {
                error ? reject(error) : resolve(result.body);
            });
    });

}

export function toggle(id) {

    return new Promise((resolve, reject) => {
        superagent.patch("http://localhost:8100/todos/" + id)
            .end((error, result) => {
                error ? reject(error) : resolve(result.body);
            });
    });

}

That module exports three functions:

get: Executes a GET request to /todos in our API to retrieve all To Do items.
add: Executes a POST request to /todos in our API to add a To Do item.
toggle: Executes a PATCH request to /todos/:id to change the isDone flag of that item.

Redux Actions

Let's talk about actions...

Actions, in Redux, are pieces of information that get sent to the store. These payloads trigger modifications on the state of the application.

Actions are basically Redux's way of saying "Hey! This happened!".

WARNING: Not actual modifications, the state of the application shoud be treated as an immutable object. You should never modify the state, but copy it, change the copy and keep going. More on it further down.

Actions are generated via action builders. These builders are functions that get invoked with some information and return the action, which is sent to the store via a dispatch function provided by Redux.

An interesting concept, necessary for real world applications, are asynchronous actions. These aren't actually just a piece of information, but another function that receives the dispatch function as parameters and, after some asynchronous operations, dispatches another action. Let's explain it with some code.

Write the following code on a file called actions.js under the src directory.

import { get, add, toggle } from './client';

export function addTodo(text) {
    return (dispatch) => {
        add(text)
            .then(get)
            .then((todos) => dispatch(receiveTodos(todos)))
            .catch((err) => dispatch(error(err)));
    };
}

export function toggleTodo(id) {
    return (dispatch) => {
        toggle(id)
            .then(get)
            .then((todos) => dispatch(receiveTodos(todos)))
            .catch((err) => dispatch(error(err)));
    };
}

export function retrieveTodos() {
    return (dispatch) => get()
        .then((todos) => dispatch(receiveTodos(todos)))
        .catch((err) => dispatch(error(err)))
}

function receiveTodos(todos) {
    return {
        type: 'RECEIVE_TODOS',
        payload: todos
    }
}

function error(err) {
    return {
        type: 'ERROR',
        payload: err
    };
}

We here are defining all the behavior of our application.

Our application has to retrieve To Do items from the API, toggle them and create them. These actions are asynchronows.

The addTodo action builder returns an asynchronous action that, after posting a new To Do item to the API, and retrieving all the To Do items again, dispatches the receiveTodos action. On error, it dispatches the error action.
The toggleTodo action builder returns an asynchronous action that, after toggling the To Do item on the API, and retrieving all the items again, dispatches the receiveTodos action. On error, it dispatches the error action.
The retrieveTodos action builder returns an asynchronous action that, after retrieving all the To Do items from the API, dispatches the receiveTodos action. On error, it dispatches the error action.

Notice that these (not as they are defined here, we'll see how) are the actions that are used by our components to handle HTML events.

The other two actions are ordinary actions, that receive some data and returns a payload.

The receiveTodos action builder returns an action of type RECEIVE_TODOS with the retrieved todos as payload.
The error action builder returns an action of type ERROR with the received error as payload.

This might sound confusing. I think Redux is not an easy to understand state manager, its concepts are pretty hard to understand, but if you put this in practice and read the code you'll end up liking it a lot.

Redux Reducer

This takes us to the reducers. A reducer is a function that receives the current state of the application and an action. As stated before, an action is a way of saying that something happened, and a reducer grabs that event/information and does what it need to do to the state to impact that event on it.

Basically, they receive the current state of the application and an action that was performed (an event or something, like a user click for example) and return the new state of the application.

Let's see more code. Write the following code on a file called reducer.js under the src directory.


const init = {'todos': [], 'error': false};

export default function(state=init, action) {
    switch(action.type) {
        case 'RECEIVE_TODOS':
            return {'todos': action.payload, 'error': false, 'initialized': true};
        case 'ERROR':
            return {'todos': [], 'error': action.payload, 'initialized': true};
        default:
            return state;
    }
}

This reducer is defining the initial state of the application and taking care of handling the actions it receives.

If the action it received is of type RECEIVE_TODOS, it returns the new state, ensuring that error is false, initialized is true and todos contains the received todos.

If the action it received is of type ERROR, it returns the new state, ensuring that error contains the ocurred error, initialized is true and todos is an empty array.

If the action it received has no handler, it just passes through the current state of the application as no changes are to be applied.

Sorry I repeat myself so much, but this concept took me a while: React components receive Redux's action builders, and ivoke them on HTML events. These events get dispatched to Redux's reducers to do what they have to do to the state based on the information provided by the action.

Container Components

Another new concept: containers. Containers are a type of component, they are called Container Components. They do the connection between React components (which are just presentational components and know nothing about redux), and redux's actions and state.

They basically wrap the react component, and grabs the state and the actions and maps them to props.

Let's see the code. Write the following code in a file called containers.js under the src directory.

import { connect } from 'react-redux';
import * as components from './components';
import { addTodo, toggleTodo, retrieveTodos } from './actions';

export const TodoPage = connect(
    function mapStateToProps(state) {
        return { state: state };
    },
    function mapDispatchToProps(dispatch) {
        return {
            addTodo: text => dispatch(addTodo(text)),
            toggleTodo: id => dispatch(toggleTodo(id)),
            retrieveTodos: () => dispatch(retrieveTodos())
        };
    }
)(components.TodoPage);

It grabs our TodoPage, our actions and the state, and puts them into props, for our component to see. This is where everything is glued together.

Web Application Start Up

Let's go to our application entry point now. Write the following code in a file called app.js under src.

import '../css/app.css';

import React from 'react';
import { render } from 'react-dom';
import { createStore, applyMiddleware } from 'redux';
import thunk from 'redux-thunk';
import { Provider } from 'react-redux';
import reducer from './reducer';
import { TodoPage } from './containers';

const store = createStore(reducer, applyMiddleware(thunk));

document.addEventListener("DOMContentLoaded", function() {

    render(
        <Provider store={store}>
            <TodoPage />
        </Provider>,
        document.getElementById('app')
    );

});

This file is importing our css entry point file, our reducer and the TodoPage container (not the component, the container).

Then, it creates the Redux store (basically, where te state lives). You might have noticed that our reducer is not handling any of our asynchronous actions, thats why we are passing that applyMiddleware(thunk) to createStore. redux-thunk takes care of handling asynchronous actions just like that.

We now wait for the DOM to be loaded, and then calling React's render function. This function receives a component and the container HTML element (that's our div#app from index.html).

The component we are passing to the render function is a Provider tag, with only one child (this is important, it can not have more than one child), which is our TodoPage container component. We are passing our store to the Provider tag by the way.

You're ready to go

We now can run npm run serve in the root of the site project, and npm run start in the root of the API project. Now we can visit http://localhost:8080/ and use our To Do list.

Conclusion

I find this pair (React, Redux) to have a pretty complex ramp up, but once you get the hang of it, applications are written quickly and the code looks great too. Yeah, it's a lot of boiler plate sometimes, but it looks nice and it actually performs pretty well too.

I come from the JQuery world, then moved on to Angular.JS, and now I moved to React.JS and Redux and I actually like it.

You can find the code to this example here.

See you in the comments!

DEV Community: Sebastian G. Vinci

Python decorators in a nutshell.

What's a decorator anyway?

A couple concepts we need to know first...

In Python, functions are first-class objects.

*args and **kwargs.

Now, decorators!

The actual problem we wanted to solve

Conclusion

Intersection, union and difference of Sets in Python.

Some basic understanding of sets.

Intersection

Union

Difference

Symmetric difference

Conclusion

Scala REST API documented with Swagger (Part 2 of 2)

Where we left off...

Professionals API

Swagger UI

Packaging a standalone JAR

Conclusion

Scala REST API documented with Swagger (Part 1 of 2)

What we'll build on this part

Preconditions

Set Up

Utility objects and traits

Status Endpoint

Swagger Documentation

Conclusion

Hi, I'm Sebastian Vinci

Deploy using docker swarm

Installation

Docker Images

API Image

UI Image

Compose

Deploy

Conclusion

React JS Web Site Example (Almost like real life).

What are we going to do?

What am I going to need to follow this tutorial?

Can we start coding already?

API project

Site project

Webpack

Stylesheets

HTML. Site entrypoint.

React Components

API Client

Redux Actions

Redux Reducer

Container Components

Web Application Start Up

You're ready to go

Conclusion

`*args` and `**kwargs`.