DEV Community: Yev

Getting a development domain and TLS certificates for free

Yev — Wed, 28 Jul 2021 00:00:00 +0000

Recently I have been experimenting with Http/3 protocol which uses TLS by default. Having self-signed TLS certificates makes browsers unhappy about the traffic, that’s why I needed a way to get a trusted certificate. While it took me more time than expected, I decided to write down the taken steps.

Getting the domain and certificates
What has just happened
Summary

Getting the domain and certificates

This section walks through the process of getting a domain name and TLS certificates.

To start with, we need to have access to a DNS provider and a domain. I chose DuckDNS because it is absolutely free and allows creating TXT DNS records, that are used to pass the ACME DNS challenge. At DuckDNS we create our own domain and copy the token that looks like UUIDv4 string. This token is needed to update the DNS records via the DuckDNS api.
Next we pass the ACME challenge and get the TLS certificates utilising Let’s Encrypt and CertBot. Let’s Encrypt is the certificate authority that issues certificates that are valid in all the browsers.CertBot is a tool that is usually used to pass ACME challenges on a wide variety of server platforms. Fortunately for us, there is a docker container dedicated to getting certificates at Let’s Encrypt for DuckDNS domains. The usage is as easy as running this command in the terminal:

docker run -it --rm \
-e DUCKDNS_TOKEN="396db3e9-5ecc-4d0c-a708-70a926b1389c" \
-e DUCKDNS_DOMAIN="melgenek.duckdns.org" \
-v `pwd`/melgenek_certs:/etc/letsencrypt \
maksimstojkovic/letsencrypt:latest

In the docker run we specified the token that DuckDNS provides for accessing their API and my domain. As a result of running this command the ACME challenge is succeeds, and the volume attached to the container contains the certificates. The private key and the public certificate can be found in the files ./melgenek_certs/live/melgenek.duckdns.org/privkey.pem and ./melgenek_certs/live/melgenek.duckdns.org/cert.pem respectively.

What has just happened

The docker container that we used in the previous section did the needed magic to validate the DNS record and get the certificates. In order to get the understanding of what the ACME flow looks like, let’s generate one more certificate using the plain CertBot. After passing the challenge, Let’s Encrypt is sure that the domain is ours and gives us a certificate.

The first step is to start the CertBot flow. The CertBot is installed as a standalone binary and can be run from terminal. The command specifies for which domain we want to pass the DNS challenge and where to store the certificates.

certbot certonly --manual \
   --preferred-challenges dns \
   --register-unsafely-without-email \
   -d "melgenek.duckdns.org" \
   --work-dir "./melgenek_certs/work_dir" \
   --logs-dir "./melgenek_certs/logs_dir" \
   --config-dir "./melgenek_certs/config_dir" \
   --agree-tos

As a result of the running the command CertBot generates a secret value and asks us to put it into the DNS record.

Please deploy a DNS TXT record under the name
_acme-challenge.melgenek.duckdns.org with the following value:

NaM0ODwHZfL6b1pjM_rrgfCSwVcy_CALKkxR2YzyE7A

Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

CertBot proposes to set the TXT record at the _acme-challenge.melgenek.duckdns.org subdomain. The validation of such a record would result in issuing a wildcard certificate for *.melgenek.duckdns.org. We are setting the TXT record for the domain itself so that the certificate is valid only for the melgenek.duckdns.org domain. The next cUrl call creates the TXT record in DuckDns.

curl "https://www.duckdns.org/update?domains=melgenek.duckdns.org&token=396db3e9-5ecc-4d0c-a708-70a926b1389c&txt=NaM0ODwHZfL6b1pjM_rrgfCSwVcy_CALKkxR2YzyE7A"

Before proceeding with the ACME challenge, we can check that the DNS record is actually set using dig. In the lookup result we see that the TXT record is exactly what we expect it to be.

dig melgenek.duckdns.org TXT

;; ANSWER SECTION:
melgenek.duckdns.org. 59 IN TXT "NaM0ODwHZfL6b1pjM_rrgfCSwVcy_CALKkxR2YzyE7A"

Clicking “Enter” in the terminal where CertBot is waiting for input would trigger Let’s Encrypt validation of the TXT record. Passing this validation successfully produces certificates the specified folder ./melgenek_certs/config_dir/live/melgenek.duckdns.org

The challenge secret is a not reused in the future, so it makes sense to clean up the TXT record. This is done by adding the clear=true query param to the previous cUrl query.

curl "https://www.duckdns.org/update?domains=melgenek.duckdns.org&token=396db3e9-5ecc-4d0c-a708-70a926b1389c&txt=NaM0ODwHZfL6b1pjM_rrgfCSwVcy_CALKkxR2YzyE7A&clear=true"

Summary

In this article we described a simple way of getting perfectly valid TLS certificates as wells as a domain that can be used for development purposes. Hopefully, this information helps build exciting software and learn new technologies.

Enhancing DynamoDb client with Scala 3

Yev — Thu, 31 Dec 2020 00:00:00 +0000

After 8 years of development, Dotty is going to become Scala 3 soon. It’s the right time to try out Scala 3 and its new features. In this article, I am going to show a practical example of making DynamoDb more type-safe and convenient using macros,type class derivations, extensions methods, and a handful of implicits.

Introduction
Encoding attributes. Implicits
Obtaining class field names. Macros
Avoiding the map construction. Type class derivation
Convenience operators. Extension methods
Conclusion

Introduction

To start with, let’s look at how to put and get an item from DynamoDb utilizing the plain aws-sdk client. The examples assume that there is a case class:

case class NewYear(year: Int, wish: String) {
  def gift: String = "A fairy pony"
}

val year = NewYear(2020, "I wish Scala 3 was released soon")

The case class is defined in the file NewYear.scala. The instance of the case class val year=... as well as the case class itself are at the top-level as well. This is the first notable feature of Scala 3: the package objects were removed from the language. The values and methods don’t need to be defined inside an object or class anymore.

The instances of NewYear are going to be stored in the dynamo table new-years. I am omitting the code of creating the dynamo table, the whole example can be found in the github repository.

Now we know the entities that are going to be stored in the database. We can take a look at the put/get code:

ddb.putItem(
  PutItemRequest.builder()
    .tableName(TableName)
    .item(Map(
      "year" -> AttributeValue.builder().n(year.year.toString).build(),
      "wish" -> AttributeValue.builder().s(year.wish).build()
    ).asJava)
    .build()
)

val item = ddb.getItem(
  GetItemRequest.builder()
    .tableName(TableName)
    .key(Map(
      "year" -> AttributeValue.builder().n("2020").build()
    ).asJava)
    .build()
)

What problems can we see here?

Firstly, the AttributeValues are being built explicitly, the year of type Int is transformed into a String in order to be set as the value of the attribute. Not only we can make a mistake of choosing the DynamoDb type by confusing the .n and .s value setters on the builder, but also the code looks ugly and verbose. It’s a great use case for type classes to provide a unified way of converting scala types to the AttributeValue.

Secondly, the attribute names are passed as strings. The key of our table is year. A typo in the key of the get request can lead to no results being returned. You can argue that having constants for the attribute names is enough. However, keeping the same name of the key in the item map and the case class field is going to be crucial to write cases classes in the .item without an explicit construction of the map.

Here comes the last issue that we’ll try solving in this article. The .item ignores the instance of the class that we defined and expects a map. This case class has only 2 fields, so building a map is quite an easy task. Having 20 fields will definitely be error-prone because an additional field in the case class requires an additional line inside the .item map.

After making some improvements employing the features of Scala 3, we can get a neat code like this

ddb.putItem(
  PutItemRequest.builder()
    .tableName(TableName)
    .item(year)
    .build()
)

val item = ddb.getItem(
  GetItemRequest.builder()
    .tableName(TableName)
    .key[NewYear](_.year, 2021)
    .build()
)

In the following sections, I’ll explain how to create the utilities for the code above to work.

Encoding attributes. Implicits

I’m going to introduce the improvements step by step. To start with, we’ll make the attribute building more pleasant. In order to do this, we’ll have a type class for the conversion of a scala type to the AttributeValue and back. Every type that can be converted to the AttributeValue will have an implementation of the AttributeCodec trait:

trait AttributeCodec[A] {
  def encode(a: A): AttributeValue
  def decode(a: AttributeValue): A
}

given AttributeCodec[String] with {
  def encode(a: String): AttributeValue = AttributeValue.builder().s(a).build()
  def decode(a: AttributeValue): String = a.s()
}

implicit val intCodec: AttributeCodec[Int] = new AttributeCodec[Int] {
  def encode(a: Int): AttributeValue = AttributeValue.builder().n(a.toString).build()
  def decode(a: AttributeValue): Int = a.n().toInt
}

You can see that I defined two instances:

the instance for the String type is defined with Scala 3 syntax
the instance for the Int uses the old Scala 2 syntax

The Scala 3 syntax is slightly nicer, the instance names can be omitted. They are never used after all, because the instances are found in the implicit scope via summoning. In the new Scala, the method implicitlyis renamed to be summon:

summon[AttributeCodec[Int]].encode(2021) // AttributeValue.builder().n(2021.toString).build()

We can make the encoding look even better by adding a so-called “summoner” method to the AttributeCodec. There is a new keyword using that marks the implicit function parameters. It replaces the old implicit:

object AttributeCodec {
  def apply[A](using codec: AttributeCodec[A]): AttributeCodec[A] = codec

  // The old syntax
  // def apply[A](implicit codec: AttributeCodec[A]): AttributeCodec[A] = codec
}

After this first round of enhancements our application is in the following state:

ddb.putItem(
  PutItemRequest.builder()
    .tableName(TableName)
    .item(Map(
      "year" -> AttributeCodec[Int].encode(year.year),
      "wish" -> AttributeCodec[String].encode(year.wish)
    ).asJava)
    .build()
)

val item = ddb.getItem(
  GetItemRequest.builder()
    .tableName(TableName)
    .key(Map(
      "year" -> AttributeCodec[Int].encode(2021)
    ).asJava)
    .build()
)

Obtaining class field names. Macros

Our next step is the derivation of the attribute names based on the case class field names. When we complete implementing this macro, the code will use fields instead of strings.

ddb.putItem(
  PutItemRequest.builder()
    .tableName(TableName)
    .item(Map(
      FieldName[NewYear](_.year) -> AttributeCodec[Int].encode(year.year),
      FieldName[NewYear](_.wish) -> AttributeCodec[String].encode(year.wish)
    ).asJava)
    .build()
)

val item = ddb.getItem(
  GetItemRequest.builder()
    .tableName(TableName)
    .key(Map(
      FieldName[NewYear](_.year) -> AttributeCodec[Int].encode(2021)
    ).asJava)
    .build()
)

As you can see the attribute name is defined via the accessor of the field FieldName[NewYear](_.year). The automatic acquisition of the field name is performed with a macro:

inline def apply[T](inline f: T => Any): String = ${getName('f)}

This is an apply method that is defined with the modifier inline and calls the macro implementation getName. The methods that are implemented via macros are always required to be defined with the inline modifier. The second inline modifier on the parameter is optional. Let’s look at a simple example to understand why it’s needed in this situation. The following code prints the parameter that is passed to the method:

import scala.quoted._
object InlineFunctions {
  inline def showExpr(expr: Any): String = ${showExprImpl('expr)}

  inline def showExprInlined(inline expr: Any): String = ${showExprImpl('expr)}

  private def showExprImpl(expr: Expr[Any])(using Quotes): Expr[String] =
    '{ ${Expr(expr.show)} + " = " + $expr }
}

The implementation of the macro is defined in the method showExprImpl. The first parameter has the type Expr. This type represents the abstract syntax tree for all the constructs that compose our code. For example, it’s subtype Literal represents a single value, and the subtype Block contains multiple statements. The ${expr} is the same as $expr. It’s called “splicing” and calculates the value of an expression. For example, the ${Expr("hello")} is just the string hello. The transformation can be reversed with the use of quotes '{expr} which is the same as 'expr. Thus '{"hello"} is equal to Expr("hello"). In essence the showExprImpl prints the string representation of the expression and its value using the splices and quotes. The Quotes context parameter contains some low-level operations and is used implicitly by these operations. I defined 2 different functions: one with the inline parameter and another one without it. Let’s call them and see the output.

import InlineFunctions._

object InlineMain extends App {
  val a = 1
  val b = 2

  println(showExprInlined(a + b)) // demo.inline.InlineMain.a.+(demo.inline.InlineMain.b) = 3
  println(showExpr(a + b)) // expr$proxy2 = 3
}

The value of the sum operation is the same. The expressions that are passed to our macro are different though. The inline modifier preserves the original expression. That’s exactly what we need in order to get the field name from the function such as (w:NewYear) => w.year.

After we had a quick look at the inline modifier, splices, and quotes, it’s time to move on and implement the getName method.

private def getName[T](f: Expr[T => Any])(using Type[T], Quotes): Expr[String] = {
  import quotes.reflect._
  val acc = new TreeAccumulator[String] {
    def foldTree(names: String, tree: Tree)(owner: Symbol): String = tree match {
      case Select(_, name) => name
      case _ => foldOverTree(names, tree)(owner)
    }
  }
  val fieldName = acc.foldTree(null, f.asTerm)(Symbol.spliceOwner)
  Expr(fieldName)
}

In the implementation, we dived even deeper into the Scala magic by calling f.asTerm so as to get access to the AST that the compiler sees. This is so-called TASTy Reflect. It provides an even more comprehensive view of the structure of the code. The power comes with a cost. Using TASTy Reflect can break type correctness guarantees and may fail at macro expansion time. In this particular use case, we are safe because we are only interested in reading the syntax tree, not in its modification. The .asTerm call produces the Tree instance. Similarly to Expr, the Tree has multiple subclasses that together represent our code. For instance, the call FieldName[NewYear](_.year) is expanded to

Inlined(EmptyTree,List(),Block(List(DefDef($anonfun,List(),List(List(ValDef(_$1,TypeTree[TypeRef(ThisType(TypeRef(NoPrefix,module class demo)),class NewYear)],EmptyTree))),TypeTree[TypeRef(TermRef(ThisType(TypeRef(NoPrefix,module class <root>)),module scala),Any)],Select(Ident(_$1),year))),Closure(List(),Ident($anonfun),EmptyTree)))

This AST has quite many nesting levels. That’s why we use the TreeAccumulator that traverses this tree for us. When the traversal reaches the desired Select instance, it returns the name of the field.

The NewYear has a method defined outside the constructor and in the current implementation the call FieldName[NewYear](_.gift) is perfectly valid. It returns the string gift even though the field is not defined in the primary constructor. In order to prevent any fields and methods to be passed into the getName method, we define a compile-time validation that issues a compilation error when the field is not a part of the primary constructor. Here is the final implementation of the macro including the validation:

import scala.quoted.Expr.{ofTuple, summon}
import scala.quoted._

object FieldName {
  inline def apply[T](inline f: T => Any): String = ${getName('f)}

  private def getName[T](f: Expr[T => Any])(using Type[T], Quotes): Expr[String] = {
    import quotes.reflect._
    val acc = new TreeAccumulator[String] {
      def foldTree(names: String, tree: Tree)(owner: Symbol): String = tree match {
        case Select(_, name) => name
        case _ => foldOverTree(names, tree)(owner)
      }
    }
    val fieldName = acc.foldTree(null, f.asTerm)(Symbol.spliceOwner)
    val primaryConstructorFields = TypeTree.of[T].symbol.caseFields.map(_.name)
    if(!primaryConstructorFields.contains(fieldName))
      report.error(s"The field '$fieldName' is not one of the primary constructor parameter.", f)
    Expr(fieldName)
  }
}

Avoiding map construction. Type class derivation

Earlier in this article, I mentioned that it is crucial to know how to derive the field name and omit to have the attribute names as strings. The reason is that we will be generating the map based on the case class.

trait ItemCodec[T] {
  def encode(t: T): Map[String, AttributeValue]
}

There will be an instance of the ItemCodec trait created for any case class. Unlike AttributeCodec, which had the explicitly defined instances, the type ItemCodec instances are derived automatically. In Scala 2 you would use libraries like Magnolia in order to construct the macros and generate these instances. Scala 3 introduces some convenience utilities in the language itself. One of them is the trait Mirror. The language provides an instance of Mirror.Product for every case class. For our NewYear the implementation of this trait looks like this:

class NewYearMirror extends Mirror {
  type MirroredMonoType = NewYear
  type MirroredLabel = "NewYear"
  type MirroredElemLabels = ("year", "wish")
  type MirroredElemTypes = (Int, String)
}

What we need to do is go field by field, and encode every field into the format that the DynamoDb client understands:

for every field get the AttributeCodec. For example, for the field year we need to summon an instance AttributeCodec[Int]
set the encoded value in the map with the key year

NewYearMirror provides enough information to write such a type class derivation, because we have both field names and field types.

private inline def getAttributeNamesAndCodecs[N <: Tuple, T <: Tuple]: List[(String, AttributeCodec[Any])] =
  inline (erasedValue[N], erasedValue[T]) match {
    case (_: EmptyTuple, _: EmptyTuple) => Nil
    case (_: (nameHead *: nameTail), _: (typeHead *: typeTail)) =>
      val attributeLabel = constValue[nameHead].toString
      val attributeCodec = summonInline[AttributeCodec[typeHead]].asInstanceOf[AttributeCodec[Any]]
      (attributeLabel, attributeCodec) :: getAttributeNamesAndCodecs[nameTail, typeTail]
  }

inline given derived[T <: Product](using m: Mirror.ProductOf[T]): ItemCodec[T] = {
  val namesAndCodecs = getAttributeNamesAndCodecs[m.MirroredElemLabels, m.MirroredElemTypes]
  new ItemCodec[T] {
    override def encode(t: T): Map[String, AttributeValue] = {
      namesAndCodecs.zip(t.productIterator)
        .map { case ((name, codec), value) =>
          name -> codec.encode(value)
        }
        .toMap
    }
  }
}

The most interesting parts of these two methods are how to go over fields one by one, and how to transform the field type to be a value.

Let’s understand the traversal first. We have a type type MirroredElemLabels = ("year", "wish") which is a tuple. In Scala 3 there is an extractor for the tuple type *:. It works the same way as for sequences so that there are a head and a tail element. In order to pattern match the tuple, we need to have its value. The erasedValue pretends to give us the value. In fact, it would always raise a NotImplementedError exception when called. That’s why we only pattern match the types and don’t use the values of erasedValue.

The second puzzle is how it’s possible to have the type of "year" instead of a type String, and how to transform the type "year" to the value year. In Scala 3 there are singleton types that’s why the type "year" is valid. These types have only one instance, in this case, year. In order to acquire this value we call the function constValue.

Convenience operators. Extension methods

All the planned functional improvements have been implemented. We can only add some operators to the GetItemRequest and PutItemRequest as if they were natively scala. Scala 2 approach is to define the implicit classes that wrap the objects and expose additional methods on them. Scala 3 has a dedicated keyword for adding such operators. It’s called extension methods.

extension[T] (b: GetItemRequest.Builder) {
  inline def key: GetItemRequestBuilderExtension[T] =
    new GetItemRequestBuilderExtension[T](b)
}

class GetItemRequestBuilderExtension[T](b: GetItemRequest.Builder) {
  inline def apply[A: AttributeCodec](inline k: T => A, v: A): GetItemRequest.Builder =
    b.key(Map(
      FieldName[T](k) -> AttributeCodec[A].encode(v)
    ).asJava)
}

extension[T: ItemCodec] (b: PutItemRequest.Builder) {
  def item(t: T): PutItemRequest.Builder =
    b.item(ItemCodec[T].encode(t).asJava)
}

Here is the resulting code that uses this syntactic sugar.

ddb.putItem(
  PutItemRequest.builder()
    .tableName(TableName)
    .item(year)
    .build()
)

val item = ddb.getItem(
  GetItemRequest.builder()
    .tableName(TableName)
    .key[NewYear](_.year, 2021)
    .build()
)

Conclusion

In this article, we got acquainted with some Scala 3 features based on a real world example of using the DynamoDb client. We also had a gentle introduction to the macros and type class derivations. If you need more information around this topic, then take a look at this arctile and the macro tutorial.

The result of the exercise that is described in this article is in github.

Serverless Tapir

Yev — Wed, 29 Jul 2020 00:00:00 +0000

This blog post could start with a drawing of a tapir that has a pair of tiny angel wings and sits on a cloud. Unfortunately, I cannot draw. But I can code. So in this article will tell how to run a Tapir program with AWS Lambda and API Gateway.

Introduction
Implementing the interpreter
Packaging the Lambda
Deployment
Summary

Introduction

Tapir is an amazing scala framework that allows defining http endpoints as scala values. Let’s look at the example that I borrowed from the official documentation:

import sttp.tapir._
import sttp.tapir.json.circe._
import io.circe.generic.auto._

type Limit = Int
type AuthToken = String
case class BooksFromYear(genre: String, year: Int)
case class Book(title: String)

val booksListing: Endpoint[(BooksFromYear, Limit, AuthToken), String, List[Book], Nothing] = 
  endpoint
    .get
    .in(("books" / path[String]("genre") / path[Int]("year")).mapTo(BooksFromYear))
    .in(query[Limit]("limit").description("Maximum number of books to retrieve"))
    .in(header[AuthToken]("X-Auth-Token"))
    .errorOut(stringBody)
    .out(jsonBody[List[Book]])

This is a representation of a single endpoint with a noticeable feature that it does not have any logic attached. The endpoint is solely a description of input parameters and outputs like response bodies.

In order to make this definition runnable, we have to use an interpreter. Here is an example implementation in akka-http:

import sttp.tapir.server.akkahttp._
import akka.http.scaladsl.server.Route
import scala.concurrent.Future

val booksListingRoute: Route =
    booksListing.toRoute { case (bfy: BooksFromYear, limit: Limit, at: AuthToken) =>
      Future.successful(Right(List(Book("The Sorrows of Young Werther"))))
    }

The logic of the route is generic, there is no explicit use of the akka-http response codes or marshallers.There are in fact many options of server interpreters such as play, http4s, finatra and vert.x.However, these frameworks help only with writing the code. The next steps are most likely the docker packaging,the definition of the kubernetes manifests and deployment scripts.

Deploying an application takes a signification amount of effort. I want to have the code up and running quickly and easily, preferably with a single bash command. Tapir, AWS Lambda and AWS CDK will help me with this venture. The complete code can be found in the github repo.

Implementing the interpreter

We start with an interpreter that transform Tapir code in an AWS Lambda. We will follow these steps:

define the logic to transform a Tapir request into the Lambda-specific request.
then Tapir parses this request and gives me the inputs like query and path params, an input body and headers.
the next step is to execute the user-defined logic using these inputs as arguments.
finally, the response from the user-defined logic is transformed back into the response that Lambda understands.

AWs Lambda itself does not know anything about http. It is just an execution environment. We utilize the API Gateway Http API. It parses the request and invokes our lambda as soon as the request is ready to be processed. Http API passes the parsed request in a predefined format. I use the "com.amazonaws" % "aws-lambda-java-events" to have the predefined java classes APIGatewayV2HTTPEvent and APIGatewayV2HTTPResponsethat correspond to the Http API request and response models.

As mentioned above the implementation starts mapping the APIGatewayV2HTTPEvent into Tapir’s ServerRequest.

class HttpApiServerRequest(event: APIGatewayV2HTTPEvent) extends ServerRequest {
  def method: Method = Method(event.getRequestContext.getHttp.getMethod.toUpperCase)

  def protocol: String = event.getRequestContext.getHttp.getProtocol

  def uri: URI =
    new URI(s"https://${event.getRequestContext.getDomainName}${event.getRawPath}?${event.getRawQueryString}")

  def connectionInfo: ConnectionInfo = ConnectionInfo(
    local = None,
    remote = Some(InetSocketAddress.createUnresolved(event.getRequestContext.getHttp.getSourceIp, 0)),
    secure = Some(true)
  )

  lazy val headers: Seq[(String, String)] = event.getHeaders.asScala.toList

  def header(name: String): Option[String] = event.getHeaders.getIgnoreCase(name)
}

A couple of moments are worth clarification here.

First of all, Http API passes all the headers with lowercased names, so the header lookup is done ignoring the case.

Secondly, in the ConnectionInfo we have only the remote client, there is no port given to the lambda. The communication to Http API is always secure.

The next step after defining the server request is the DecodeInputsContext. As the name suggests, this class is used by Tapir during the inputs extraction.

class HttpApiDecodeInputsContext(event: APIGatewayV2HTTPEvent, pathConsumed: Int = 0) extends DecodeInputsContext {
  def method: Method = Method(event.getRequestContext.getHttp.getMethod.toUpperCase)

  def nextPathSegment: (Option[String], DecodeInputsContext) = {
    val path = event.getRawPath.drop(pathConsumed)
    val nextStart = path.dropWhile(_ == '/')
    val segment = nextStart.split("/", 2) match {
      case Array("") => None
      case Array(s) => Some(s)
      case Array(s, _) => Some(s)
    }
    val charactersConsumed = segment.map(_.length).getOrElse(0) + (path.length - nextStart.length)

    (segment, new HttpApiDecodeInputsContext(event, pathConsumed + charactersConsumed))
  }

  def header(name: String): List[String] = 
    event.getHeaders.getIgnoreCase(name).flatMap(_.split(",").toList).toList

  def headers: Seq[(String, String)] = event.getHeaders.asScala.toList

  def queryParameter(name: String): Seq[String] =
    event.getQueryStringParameters.getIgnoreCase(name).flatMap(_.split(",").toList).toList

  def queryParameters: QueryParams = QueryParams.fromMap(event.getQueryStringParameters.asScala.toMap)

  def bodyStream: Any =
    throw new UnsupportedOperationException("Trying to read streaming body from a non-streaming request")

  def serverRequest: ServerRequest = new HttpApiServerRequest(event)
}

Multivalued headers and query params are combined with commas in Http API. So we split them back into a List.

The body is always passed into the lambda as a string so the whole incoming request is consumed before the lambda invocation. That’s why the body cannot be streamed, and the bodyStream has no implementation.

Now that we know how to map the lambda input into the input that Tapir understands, let’s implement the routes. The same way akka http has akka.http.scaladsl.server.Route, or http4s has org.http4s.HttpRoutes, we will also have a Route type:

type Route = PartialFunction[APIGatewayV2HTTPEvent, APIGatewayV2HTTPResponse]

Tapir also requires a MonadError implementation. For akka http there is an instance of a MonadError based on Future.That’s why the akka http routes return Futures as results. A lambda function that runs on top of the Lambda Java runtime requires functions to be simple blocking functions with a signature def onEvent(event: APIGatewayV2HTTPEvent): APIGatewayV2HTTPResponse.I chose Try to be the result type of the Lambda logic. The example conversion of a function with some business logic into a Lambda route would be:

import scala.util.{Success, Try}
import sttp.tapir.server.httpapi._  

def logic(bfy: BooksFromYear, limit: Limit, at: AuthToken): Try[Either[String, List[Book]]] = {
  Success(Right(List(Book("The Sorrows of Young Werther"))))
}
val serverEndpoint = bookListing.serverLogic((logic _).tupled)
val booksListingRoute: Route = serverEndpoint.toRoute

The Route is a partial function. If the http request matches the Tapir endpoint definition then we execute the logic for this endpoint. If no then we try another route. This check is expressed via the isDefinedAt method of the partial function.We use the sttp.tapir.server.internal.DecodeInputs, pass the context and let Tapir parse the request for us.

def isDefinedAt(event: APIGatewayV2HTTPEvent): Boolean = {
  DecodeInputs(e.input, new HttpApiDecodeInputsContext(event)) match {
    case _: DecodeInputsResult.Values => true
    case _: DecodeInputsResult.Failure => false
  }
}

When the http request matches our endpoint, it’s time to run the actual application logic. The result of the DecodeInputs is just a sequence of values. Thus, we map these values into scala classes, so that we get a tuple of case classes, like (BooksFromYear, Limit, AuthToken) in the example above. This conversion is the responsibility of sttp.tapir.server.internal.InputValues.

def apply(event: APIGatewayV2HTTPEvent): APIGatewayV2HTTPResponse = {
  DecodeInputs(e.input, new HttpApiDecodeInputsContext(event)) match {
    case values: DecodeInputsResult.Values =>
      InputValues(e.input, values) match {
        case InputValuesResult.Value(params, _) => valueToResponse(params.asAny)
        case InputValuesResult.Failure(input, failure) => handleDecodeFailure(input, failure)
      }
    case DecodeInputsResult.Failure(input, failure) => handleDecodeFailure(input, failure)
  }
}

The next step, after the inputs are successfully parsed, is to pass these inputs into the user-defined logic.

def valueToResponse(value: Any): APIGatewayV2HTTPResponse = {
  endpoint.logic(TryMonadError)(value.asInstanceOf[I]) match {
    case Success(Right(result)) => OutputToHttpApiResponse(ServerDefaults.StatusCodes.success, endpoint.output, result)
    case Success(Left(err)) => OutputToHttpApiResponse(ServerDefaults.StatusCodes.error, endpoint.errorOutput, err)
    case Failure(e) => OutputToHttpApiResponse(StatusCode.InternalServerError, e.getMessage)
  }
}

The only thing left is to map the Tapir’s output to the response type that Http API understands. This is what OutputToHttpApiResponse does in the valueToResponse method. The implementation requires some lines of code and it can be found here.

We put the functionality that is described in the toRoute function. The Route is a partial function, so we can compose multiple routes into a single one via the orElse method. If none of the routes matches the request, we use the predefined EmptyRoute that answers with the 404 response code.

implicit class RichHttpApiServerEndpoint[I, E, O](endpoint: ServerEndpoint[I, E, O, Nothing, Try]) {
  def toRoute: Route = {
    // the partial function that is described above
  }
}

implicit class RichHttpApiServerEndpoints[I, E, O](serverEndpoints: List[ServerEndpoint[_, _, _, Nothing, Try]]) {
  def toRoutes: Route = {
    serverEndpoints
      .map(_.toRoute)
      .foldRight(EmptyRoute)(_ orElse _)
  }
}

Let’s also provide an interface that has to be implemented in order to have a complete lambda. This interface requires a list of ServerEndpoints and then uses these endpoints to process a request.

trait HttpApiFunction {
  val serverEndpoints: List[ServerEndpoint[_, _, _, Nothing, Try]]
  def onEvent(event: APIGatewayV2HTTPEvent): APIGatewayV2HTTPResponse = {
    serverEndpoints.toRoutes(event)
  }
}

object BookFunction extends HttpApiFunction {
  override val serverEndpoints = List(booksListingRoute)
}

Lambdas will have the handler that points to the your.package.BookFunction::onEvent. The class will be instantiated by the Java runtime and the onEvent function will be invoked for every http request.

Packaging the Lambda

I am not using lambda layers, so the Lambda function has to contain all of the dependencies. The fat jar is packaged using the sbt-assembly plugin. The only interesting part of this process is the merge conflict resolution. The reference configurations are concatenated and some files with conflict names are discarded.

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", _@_*) => MergeStrategy.discard
  case PathList(ps@_*) if ps.last endsWith "reference-overrides.conf" => MergeStrategy.concat
  case PathList(ps@_*) if ps.last endsWith "module-info.class" => MergeStrategy.discard
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

Deployment

The interpreter is ready, it can be used to compose Tapir endpoints together into a single Route. The Lambda function is also bundled into a fat jar. Everything is ready for uploading and running this jar. We can go into the AWS UI, click a couple of buttons, create the lambda and upload the code. However, this is a manual process. The whole idea of the experiment, that is described in this article, is to have an easy and automatic process of deploying applications.

In order to automate the lambda creation process, we will use AWS Cloud Development Kit. This is a tool that allows defining the AWS infrastructure in Java and transforming this definition into CloudFormation stacks.CloudFormation then makes sure that the infrastructure is created in AWS.

Let’s create a function that receives an HttpApiFunction and the name of the Lambda function. This function creates a so-called App, that contains all the resources. Then it synthesizes the Cloudformation template based on the code.

def deploy(httpApiFunction: HttpApiFunction, title: String): Unit = {
  val app = new awscdk.core.App()
  ...
  app.synth()
}

All the resources have scopes. The top-level scope is the App. Each app is a set of Stacks. Every Stack corresponds to a Cloudformationstack. For us, it is enough to create a single Stack.

val stack = new Stack(app, s"$title-stack")

The first meaningful resource that we create is the lambda function. It is a lambda that is based on the Java 8 runtime. It sets the handler method to be the def onEvent(event: APIGatewayV2HTTPEvent): APIGatewayV2HTTPResponse of our lambda class. Additionally, we set the local path of the fat jar file that we have bundled. This jar is automatically uploaded by CDK into an s3 bucket. Then this s3 object is used as the source for the lambda function.

val function = lambda.Function.Builder.create(stack, s"$title-lambda")
  .memorySize(192)
  .timeout(Duration.seconds(30))
  .functionName(title)
  .runtime(lambda.Runtime.JAVA_8)
  .handler(s"${httpApiFunction.getClass.getName.replace("$", "")}::onEvent")
  .code(Code.fromAsset("lambda/target/scala-2.13/assembly.jar"))
  .build()

After the lambda is present, we create the API Gateway Http API.

val api = HttpApi.Builder.create(stack, s"$title-api")
  .apiName(title)
  .build()

After both lambda and API are set up, we need to bind them together so that Http API passes the requests into lambda. For this to happen, we create a Lambda integration.

val integration = LambdaProxyIntegration.Builder.create()
  .handler(function)
  .payloadFormatVersion(PayloadFormatVersion.VERSION_2_0)
  .build()

The last step is to define all the http routes and point them to the same lambda function. Tapir’s ability to transform endpoints into Open API helps us construct the routes. For every path in the Open API specification, we create a route in the Http API.

import sttp.tapir.docs.openapi._
private def apiRoutes(httpApiFunction: HttpApiFunction, integration: LambdaProxyIntegration): List[AddRoutesOptions] = {
  val openAPI = httpApiFunction.serverEndpoints.map(_.endpoint).toOpenAPI("any name", "v1")
  openAPI.paths
    .map { case (path, pathItem) => 
    val methods =
      pathItem.get.map(_ => HttpMethod.GET) ++
        pathItem.post.map(_ => HttpMethod.POST) ++
        pathItem.delete.map(_ => HttpMethod.DELETE)
    AddRoutesOptions.builder()
      .methods(methods.toList.asJava)
      .path(path)
      .integration(integration)
      .build()
    }
    .toList
}

That’s it. Now we pass our lambda class into the deployment function, run the cdk deploy command in the terminal, and after a short time, our code is up and running in AWS.

object Main extends App {
  HttpApiCdkServer.deploy(BookFunction, "my-bookshop")
}

Cdk also allows adding outputs into stacks. One of such outputs can be the url of the Http API that we deployed.

CfnOutput.Builder.create(stack, "api-url").exportName("url").value(api.getUrl).build()

This output is shown after the cdk deploy succeeds.

✅ my-bookshop-stack

Outputs:
my-bookshop-stack.apiurl = https://2qpnchfg11.execute-api.us-east-1.amazonaws.com/

Calling this url results in the request being handled by our lambda that is described above.

curl -v -H "X-Auth-Token: Bearer token" https://2qpnchfg11.execute-api.us-east-1.amazonaws.com/books/novel/2020?limit=10

Result:
[
  {
    "title": "The Sorrows of Young Werther"
  }
]

Summary

In this article, I showed how to run the Tapir application in Lambda and automatically deploy the code with CDK. I believe that the use of CDK brings a lot of benefit into scala applications because it keeps the infrastructure close to the code. The infrastructure has the same version as the code does. This helps us to be sure that the required infrastructure is created at the deployment time. In addition, CDK gives a way to deploy with a command.

The code for this article can be found in the github repo.

DEV Community: Yev

Getting a development domain and TLS certificates for free

Table of Contents

Getting the domain and certificates

What has just happened

Summary

Enhancing DynamoDb client with Scala 3

Table of Contents

Introduction

Encoding attributes. Implicits

Obtaining class field names. Macros

Avoiding map construction. Type class derivation

Convenience operators. Extension methods

Conclusion

Serverless Tapir

Table of Contents

Introduction

Implementing the interpreter

Packaging the Lambda

Deployment

Summary