DEV Community: Jairo andres

Serverless takeaways

Jairo andres — Tue, 13 Dec 2022 03:26:33 +0000

I don't start to writing because I know all the answers, I start to writing because I know that I need to figure out the answers - Ilya Grigorik

Serverless is a trend that has been growing in the last two years. Datadog shared state serverless study in 2020, this study showed the growth of serverless across different cloud providers, Show Figure 1.

Figure 1 Datadog study about serverless Got from https://www.datadoghq.com/state-of-serverless/

Real definition of Serverless

Serverless has different definitions, for instance AWS has its definition and also Martin flower has another definition for serverless, these definitions are so accurate.

However, I prefer the definition described in the book called “Serverless an architecture on AWS”, where the author remarks the thought that serverless is more about “management less” using all the benefits of public cloud and cloud native.

Serverless first strategy?

Serverless is a complete universe of different technologies, which means a lot of services classify in this category. A big mistake it is trying to compare serverless only to FaaS (function as a services) Show Figure 2. Therefore, it is important to answer the next question, how serverless technologies could be divided?

Figure 2: Tweet about FAAS and Serverless got from https://twitter.com/tjholowaychuk/status/902999008674594816?lang=en

We can analyze serverless in two main levels

Code , that is related with the business logic, libraries, and frameworks, in fact the code doesn't change a lot, in some cases the code need to add some lines of code to support different invocations
Architecture , that focuses in the components and their interactions, the architecture level is the key when we use serverless, you can see some types of serverless technologies in the Figure 3

Figure 3: Classification of serveless technologies based in the book serverless architecture on AWS

Over the last years I had to make different decisions about what technology o architecture I should implement, however decisions don't have to be either-or, with the correct strategies I could change the questions, and dilemmas to figure out better solutions.

Below I will share some takeaways using serverless architecture which could help you to understand if serverless architecture is the correct choice for you

Mindset

Serverless is a new way to build software, therefore coming with new strategies and new technologies, nevertheless the idea of abstraction should be kept, because at the end of the day serverless is one of many possible solutions.

In architecture, we talk about the big picture. This 'big picture' must be easy to understand for every architect, even if they do not know anything about the cloud provider. Having said that, the diagrams about the architecture should be draw in different levels, following the c4model. I always try to consider it in three levels.

Solution level, it is only about abstractions (frequently I use c4model o another type of diagrams)
Implementation level, it is when the cloud providers apear, Show Figure 3 in this level cloud provider names icons are used.
Code level (structure of project), it is when the structure of the code is defined, the project structure should be related with the architecture.

Figure 4: Serverless architecture using AWS got from https://aws.amazon.com/serverless/

A big advantage of drawing diagrams in different levels is that it become the final solution more understandable, because you can see different patterns that are implemented in each solution

Quickly is different to easy

With serverless You can jump in and prototype something quickly, but you’re gonna need to think about what does that mean to build a production application - Mike Roberts

Less and complexity, are two words to remark, because these words make the base of serverless. For instance in AWS you have available interfaces to create a QUEUE connect with a lambda in minutes Show Figure 5.

Figure 5: Interface to create a SQS in AWS

That is like magic, as you can see in the Figure 5 you don't need to struggle with complex configuration o technologies, nevertheless, starting with services only because seems easy to use it is not the better approach.

In order to avoid create a Frankenstein with a lot of services sticks around this I would like to give some questions that I try to answer before to decide which service to use

is it the implementation necessary? Or is it only for trending?
What is the price model?
What other options could work well when the technology is not enough? Is it difficult to migrate to another technology?
Are there libraries to support easy integration?
Who is behind the stack? What patterns are implementing it?
Trying to imagine some complex use case, is it possible to resolve it with serverless technology?
What about documentation? Serverless is relatively new, and the documentation could be a challenging to find, it is basically about trying and learning from the errors.

Serverless partners

Serverless was not created to work only with another serverless services, In the real world the better solution is the hybrid architecture. For instance, a common use case it is tried to split a monolithic into microservices, Show Figure 6.

Figure 6: MindTouch serverless architecture got from here

The last is MindTouch serverless architecture, which is an fascinating architecture because of its legacy component, which is connected with a new serverless architecture.

Figure 7: Transactional architecture + event driven architecture

As you can see in the Figure 3 there is another example of the hybrid architecture (transactional+event driven). Therefore, using serverless architecture to potentiate a legacy system and develop new features around it, could be a good choice

Tools

As software developers, our work is to put things together and try to avoid reinventing the wheel. There are some tools o libraries that I consider useful if you are working with serverless:

Serverless Framework: this is a complete framework to develop serverless architecture, serverless framework allows you to integrate with AWS and whole its ecosystem. Serverless framework use a powerful CLI, basing its configuration on YAML files. In addition, serverless abstract complexity allowing developers to be more productive.
CDK: This is a framework which allows you to write code as infrastructure the main different with common frameworks it doesn't use YAML instead it uses languages such as python, go, and typescript indeed you can write lambdas functions and different serverless architectures
AWS Lambda Powertools: These are a couple of libraries that could help you to write lambdas functions in python, for instance the middleware factory is a library which can execute code in different moments: before the lambdas is invoked, after the lambda is invoked and when some exceptions are generated
MiddyJS: This is a simple but incredible library that allows you to create middleware using the express style. Re-use of code is a significant part of best practices in software development.
typedi: This is a light tool to work with dependency injection in TS. With it, you can build well-structured and easily testable serverless applications.

LocalStack: This is a framework to simulate different AWS services using Docker, it contains a useful CLI, which allow you running test without execute any deploys. There are two versions, standard and pro, the pro contains some advanced features but in general the standard is enough.

Observability

Observability is specially important in serverless because there are a lot of services working together at the same time, and it is probably that something fails. Effective monitoring is not only about logs, it is a complete strategy, and I would like to remark three points.

Metrics
Logs
Distributed tracing

Each cloud provider has its own tools or services to implement observability, however in some cases that couldn't be enough. Underneath, I'll show some tools that have been useful for me.

Datadog:Datadog is a complete platform for monitoring cloud services and serverless applications, in addition Datadog come with an intuitive platform which you can create different dashboards
Lumigo: Lumigo is similar to Datadog, but the main different is that lumigo focuses on traceability. The more incredible feature it is the graphs and the following to the transaction to the time of live.
ELK: ELK stack are three main technologies Elastisearch, Kibana, and Logstash they are powerful tools to implement an observability solution, the only inconvenient could be that ELK is a general solution, if you want to use in serverless you need to adapt the solution
Grafana + Prometheus: That stack is similar to ELK, Grafana is a tool to create dashboard and Promethus have the responsibility to recollect logs and expose information using a query language called PromQL, in the same way that ELK that stack is not focuses on serverless, but that can be adapted

Software engineering is always a game of tradeoffs, hence the correct tool always depends on the context which could be related with money, libraries, experience, etc. In my perspective the tool is not the more essential thing, the deal is that whatever solution you decided should allow you centralized logs, metrics, traceability, and dashboards.

Mentions

Thanks to Jeisson Rangel for reading drafts and give me feedback of this

Remote Work

Jairo andres — Sun, 06 Nov 2022 18:10:47 +0000

Remote work processes have freed the organization from space and time. The Async Mindset is the philosophical underpinning required to cultivate and facilitate that freedom. - Running Remote

In February 2020, 17 percent of U.S. employees worked from home 5 days or more per week. By March of the same year, more than 40% employees of the US employees moved to remote working 5 days per week (Show Figure 1). Companies looking for talent in different countries made the concept of remote work quite popular a long time ago among developers.

Figure 1: Stats about remote work took from https://www.statista.com/statistics/1122987/change-in-remote-work-trends-after-covid-in-usa/

It is important to clarify the different between work from home and remote work :

Work from home : consists in making your home your place of work. In most cases it uses the same practices as work in place (some companies strictly that others)
Remote work : It is a new mindset, in which people can work from wherever location that they want. Indeed, it comes with new challenges, such as insurance policy, taxes models and asynchronous management

Having said that, this article is focuses in some learnings made from working remote , these are summarized in the next five points

Over communication

One of the most important things in remote work is effective communication, a strong reason for that could be that communication it's one of the four rules principles to survive in time of crisis. However, in remote work, communication would be a challenge, making small talk in the workplace with colleagues and coworkers is lost. Furthermore, a lot of companies only work as an autonomous team, which means each team in the company can take decision for themselves. But, with a great power comes with a great responsibility. As a developer part of your responsibility is keeping the context in the more quantity of features, bugs, and situations as possible, to reach these goals it is necessary to communicate effectively those.

There are some points that can be help you deal with communication problems.

Meetings don’t have to be toxic: Since I read about the pyramid communication show Figure 2 I realized the importance to escalate ways of communication, they should always do similar work, starting with the more asynchronous tools (deep understanding, message in slack, email, discussions) and escalate to the more synchronous tool (google meet, zoom, Microsoft Teams). Nevertheless, be careful with spending a huge time in deep work, trying to guess how to work something, “know your limits”, when you can't understand something, you need to look for the expert in the topic and clarify your doubts.
Inform when you do know about something or when you have blockage, it is common that your coworkers are busy, try to ask in general channel for help, tools such as pr in GitHub or Bitbucket can be a good place to discussion and resolve questions.
Divide and conqueror: technology is no easy, because it has huge branches, before sending a long question, divide your problem in small problems and try to ask shorter question, it can be more effective, and it can help you find a better solution

Figure 2: The Pyramid of communication took from https://michael.team/pyramid/

Focus

Let your mind become a lens, thanks to the converging rays of attention; let your soul be all intent on whatever it is that is established in your mind as a dominant, wholly absorbing idea. - Deep Work book

We live in a time where focusing our attention on only one task is a big challenge, what could be the reason for it? The short answer is because of notifications. Many people have the necessity to consume short videos and short content. The book deep work mentions the importance of working on only one thing at the time, for instance if you need to resolve a problem with math you can't resolve the problem and keep a conversation in social media. For that reason, it is better to break down tasks, assigns priorities, and take them one by one. Focus is the more important thing in the productivity, and it is a big advantage about working in office. I struggled developing this ability, however there are some ideas that helped me.

Pomodoro : it is a classic tool, Pomodoro is a basic strategy that can be appropriate for yourself depending on your necessities, for me pomodoro works better using the 45 minutes to work and 5 minutes to rest rule
Turn off notifications: Social media can consume all your time in the blink of an eye , when you start working, you must turn off your phone notifications, only leave the essential notifications like slack.
Booked your time to work on your calendar: That is a good strategy to notified in what time you are more efficient

Focus times is no about hard work, it is about efficient work, Focus is only useful if you can drive your efforts in the correct way.

Learn for fun

When was the last time that you learned something because you like it? And not because you think in business? For me there are three types of learning.

Leaning to complement your career: it happens when you learned something that you intend to apply immediately in your work/ profession
Learning by hobby: Do you have a personal project? Starting a personal a project comes with different challenges which help you to learn about different topics and get more experience.
Learning by fun: That is a beautiful way to learn, because it hasn't relation with your major. You can deeply learn about history, about poetry, fantasy, whatever thing that you want. A good strategy could be to reduce your time of procrastination and invest this time in this type of learning.

Be sponge

Be sponge that is a beautiful sentence, the meaning is “learning a lot”, take a breath a look around you, what can you see? I am 100% sure that you can see incredible people, so you must work collaboratively and learn a lot from your team. One manner to learn fast is being an observer, if you can't understand something, ask about it, if you are in doubt ask bout it, if you don't agree, raise your hand. I would like shared three situations in which raise your hand should be a rule.

Starting in new project: this is a common situation , by nature people communicate as playing a round of the telephone game, you must do always double-check, if you do not understand 100% idea, you can try to paraphrase the problem or solution, and validate that you and your team are in the same level of understanding
Unexpected behavior about functionality (?): Some projects have poor documentation, that is the main reason why the users don’t know how to use the applications, i.e. a client report an error message, when trying to enter a URL that requires login, you can spend a lot of time concluding that the behavior is correct, or ask directly to the person and clarify the situation.
Situations converting in too formal: it is easy to detected, however it is not too easy to management, many companies that move to remotely have no idea how to deal with “over formalized” situations, send emails work well for clients, but inside the company, it’s suggested to implement applications which helps with asynchronous management i.e. Slack, Discord, JIRA.

Transparency = Empower

One beautiful thing the co-author Liam Martin said of the book running remotely said it was that running remote is about numbers, and the importance to share these with your team, how it is growing the company? What are the goals in Q3? Q4? These number are relevant, because they help to employees to take the correct decision in critical situations.

In management, transparency plays an important role too, such as estimation of the tasks, quality of code, security/performance and quality attributes. Perhaps, transparency is a powerful tool to help in the success of the projects because it creates more confident teams and allows the clients to understand the reality about software development.

Mentions

Thanks to Jeisson Rangel for reading drafts and give me feedback of this

Management dependencies in python

Jairo andres — Mon, 17 Oct 2022 07:19:33 +0000

Putting ideas into words is certainly no guarantee that they'll be right. Far from it. But though it's not a sufficient condition, it is a necessary one. -Paul Graham

If you are a developer in python, what would be the answer to the next question: How to manage dependencies in python?. Probably the answer include pip, virtualenv and requirements.txt. These tools are the stack that python developers have been used for a long of time in different projects. However, due to the concept 'modern python', new interesting features can be used as asyn/await, f string, pattern matching, type hints, walrus operator etc. Having said that, modern python come with a new specification to management dependencies called pyproject.toml

What is pyproject.toml?

The pyproject.toml is a file written in human-readable syntax, it is located in the root directory of the project and contains all the metadata, dependencies libraries and configuration of additional tools like: black, mypy and pytest. The next configuration is a basic example.

[project]
name = "spam"
version = "2020.0.0"
description = "Lovely Spam! Wonderful Spam!"
readme = "README.rst"
requires-python = ">=3.8"
license = {file = "LICENSE.txt"}
keywords = ["egg", "bacon", "sausage", "tomatoes", "Lobster Thermidor"]
authors = [
  {email = "hi@pradyunsg.me"},
  {name = "Tzu-Ping Chung"}
]
maintainers = [
  {name = "Brett Cannon", email = "brett@python.org"}
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python"
]

dependencies = [
  "httpx",
  "gidgethub[httpx]>4.0.0",
  "django>2.1; os_name != 'nt'",
  "django>2.0; os_name == 'nt'"
]

[project.optional-dependencies]
test = [
  "pytest < 5.0.0",
  "pytest-cov[all]"
]

[project.urls]
homepage = "https://example.com"
documentation = "https://readthedocs.org"
repository = "https://github.com"
changelog = "https://github.com/me/spam/blob/master/CHANGELOG.md"

[project.scripts]
spam-cli = "spam:main_cli"

[project.gui-scripts]
spam-gui = "spam:main_gui"

[project.entry-points."spam.magical"]
tomatoes = "spam:main_tomatoes"

Example took from PEP 621

Tools

Different tools use the pyproject.toml specification to automate the management dependencies process. There are a lot of tools s hown in Figure 1 , but the most popular are:

Poetry

Poetry is the most popular library for packing and managing dependencies. In additional, it has a CLI which developers can use easily interact with the pyproject.toml file, the next code is an example of pyproject.toml

[tool.poetry]
name = "poetry-intro"
version = "0.1.0"
description = ""
authors = ["jairo <me@jairoandres.com>"]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Poetry includes an incredible CLI to deal with versioning problems which is common in projects that have different environments (DEV, PROD and QA). Developers who use poetry can install dependencies by environment and create group of dependencies, for example define dependencies only for documentation.

Figure 2: CLI poetry. Took from poetry docs.

poetry has been growing in popularity, that is the reason why famous libraries like pendulum uses poetry

Hatch

Hatch was recently added to PYPA repositories, that is because hatch is probably the tool that implement PEP standards in the most meticulous way, the next code is an example from pyproject.toml

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test"
description = ''
readme = "README.md"
requires-python = ">=3.7"
license = "MIT"
keywords = []
authors = [
  { name = "Jairo Castaneda", email = "me@jairoandres.com" },
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python",
  "Programming Language :: Python :: 3.7",
  "Programming Language :: Python :: 3.8",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: Implementation :: CPython",
  "Programming Language :: Python :: Implementation :: PyPy",
]
dependencies = [
]
dynamic = ["version"]

[project.urls]
Documentation = "https://github.com/unknown/test#readme"
Issues = "https://github.com/unknown/test/issues"
Source = "https://github.com/unknown/test"

[tool.hatch.version]
path = "test/ __about__.py"

[tool.hatch.envs.default]
dependencies = [
  "pytest",
  "pytest-cov",
]
[tool.hatch.envs.default.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=test --cov=tests"
no-cov = "cov --no-cov"

[[tool.hatch.envs.test.matrix]]
python = ["37", "38", "39", "310", "311"]

[tool.coverage.run]
branch = true
parallel = true
omit = [
  "test/ __about__.py",
]

[tool.coverage.report]
exclude_lines = [
  "no cov",
  "if __name__ == . __main__.:",
  "if TYPE_CHECKING:",
]

In comparison with other tools, The CLI hatch is faster. This has a huge amount of configurations references to personalized pyproject.toml.

Another good feature of Hatch, this allows execution scripts which is similar to NMP package.json scripts

[tool.hatch.envs.default.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=test --cov=tests"
no-cov = "cov --no-cov"

The last example allows executing scripts of test coverage and defining them by environment. Managing dependencies in file pyproject.toml is less automated, but that is great at the same time because it avoids adding unnecessary data. Recently the popular framework FastAPI has changed changed flit to hatch

PDM

PDM, as described, is a modern Python package and dependency manager supporting the latest PEP standards. One thing to remark is that PDM doesn't use virtualenv, That is because of PEP 582the next code is an example from pyproject.toml

[project]
name = "pdm-project"
version = "0.1.0"
description = ""
authors = [
    {name = "Jairo Castaneda", email = "me@jairo.com"},
]
dependencies = [
    "fastapi>=0.85.0",
]
requires-python = ">=3.10"
readme = "README.md"
license = {text = "MIT"}
[project.optional-dependencies]

[build-system]
requires = ["pdm-pep517>=1.0.0"]
build-backend = "pdm.pep517.api"

[tool]
[tool.pdm]
[tool.pdm.dev-dependencies]
dev = [
    "pytest>=7.1.3",
]

PDM has an incredible console

Figure 3: CLI PDM

Figure 4: Folder pypackages which contain libraries installed by PDM

Other tools

Flit is other popular tool, but this tool is only for package management. In general, it is used to create and publish packages but need other tool to complement its management dependencies.

Bento is other tool, like Flit, bento is used for publish and packaging. Bento has a special syntax in a file called beto.info

What tool I should be used?

I have forgotten how many times I read the tittle “modern python package and dependency manager”. Python has a lot of tools for resolver problem with packaging management. But none of these tools are compatible which each other, for instance: the format in lock file (which is very important because of hashes, pins, and tree of dependencies) is totally different in each tool.

In my opinion, pyproject.toml is the future to management dependencies in python, i.e., NodeJS uses a file as an entry point to managing dependencies. Choosing the better tool should be a task which takes time to analyze what requirements the project has, and you should have three points in mind.

Review repository, starts, issues bugs compatibility with your CI/CD tools even some PaaS services don't support pyproject.toml
Review standards, what standard follows? What is the status of the PEP?, is it finalized? Is it draft?
Review who is behind the project?

Mentions

Thanks to Jeisson Rangel for reading drafts and give me feedback of this