DEV Community: Patryk Woziński

Working with databases at a scale

Patryk Woziński — Tue, 02 Aug 2022 14:09:34 +0000

Hey! This time I will cover the topic of unsafe migrations in more significant projects. If you have a relatively small project, then there’s nothing wrong with making some changes. Scale changes everything in working with databases. But let’s start from scratch - feel free to go to the topic you are interested in.

What is a database transaction?

Let’s assume we are playing a game with boxes. You can buy a package from me, but you can’t see what’s inside, okay? I am preparing everything for these boxes to be grouped, and only when the box is ready will you see it in the game. If it’s during the packing process - you don’t know about that box. After that, if the package matches your expectations (the size, the color, etc.), you should buy it. At this moment, I’m passing that box to you, and you can tell me if it’s okay or not. If, for instance, the box contains the same t-shirt you already have, you can return the package, but not that one box element. Everything or nothing - that’s the rule!

And… well, now let’s compare that shitty analogy to databases. The box is a transaction that may include a lot of different elements (SQL statements like INSERT, DELETE or UPDATE operations). So buying the box is an analogy of committing (saving, persisting) transactions into the database. These transactions are ACID (atomicity, consistency, isolation, and durability), so you can agree on committing everything or nothing. If you don’t agree with some box elements → you can discard the whole package - the same as the database is doing a rollback of your operations.

What is database migration?

Okay, so it's time to explain a database migration if you know what a database transaction is. When working with applications, we need to introduce changes to our databases. As Heraclitus (a Greek philosopher) said: "change is the only constant in life" - the same is in our applications. If you're creating any professional software, you should use migrations to avoid manual changes in your databases and keep the history of changes in a good way. Most (if not all) frameworks have tools built-in for managing database structure from the code. Thanks to these tools, we can write every structure change as a line of SQL queries or, as in many technologies - use their DSLs provided to describe what's changing. For instance, these transactions might look like this PHP Doctrine example:

<?php

declare(strict_types=1);

namespace DoctrineMigrations;

use Doctrine\DBAL\Schema\Schema;
use Doctrine\Migrations\AbstractMigration;

/**
 * Auto-generated Migration: Please modify to your needs!
 */
final class Version20220530204833 extends AbstractMigration
{
    public function getDescription(): string
    {
        return '';
    }

    public function up(Schema $schema): void
    {
        $this->addSql('ALTER TABLE blog.posts ADD identifier CHAR(10)');
    }

    public function down(Schema $schema): void
    {
        $this->addSql('ALTER TABLE blog.posts DROP identifier');
    }
}

Or like this Elixir Ecto example:

defmodule PostRepo.Migrations.AddIdentifierToPostsTable do
  use Ecto.Migration

  def up do
    alter table("posts") do
        add :identifier, :string, length: 10
    end
  end

  def down do
    alter table("posts") do
        remove :identifier
    end
  end
end

It looks much different, but that’s about using direct SQL queries (as in PHP Doctrine) or DSL provided by the framework (Elixir’s Ecto). As you can see, both contain sections up and down - the first is used when we’re deploying our changes, and the second is for rolling back changes when something happens and we have to revert changes. These migrations might be auto-generated by the framework or manually created, but the effect is the same - database changes described in code and kept in the same git repository.

What is a deployment unit?

Okay, so if we are on the same page, what transactions and migrations are - then we can cover the last important topic necessary to understand problems with migrations at a bigger scale. This topic is called a deployment unit. When deploying our changes into the production environment, they are usually closed together as one Docker image. This image may contain migrations code with the database changes you plan to introduce. The deployment might be composed of a minimum of one pull request merged. Sometimes, there are more pull requests - especially if you're working on a much bigger project or you are not doing continuous delivery, but you are doing releases on-demand. What is worth adding is that - the migrations come first into production; after that, your business code changes are available to the users. That has some drawbacks - that you'll see in the next part of this post.

Mistakes

Right now, I’ll cover some possible mistakes when you are working on a bigger project with more significant traffic. I will split errors into a few groups like common mistakes - no matter which database system you’re using - that’s a problem and PostgreSQL mistakes. So let’s move on and cover all tricky things.

Common mistakes

Adding a column with a default value

If you are using PostgreSQL version below 11, MariaDB below 10.3, or MySQL below 8.0.12, you should be careful when adding a new column with a default value. The reason is that doing an ALTER TABLE operation and adding another column will rewrite the whole table. This is because all changes will be executed on a table copy. Then after introducing all changes, it will apply them to the original one. So this operation might hurt for more extensive tables because we block incoming queries. However, an INSTANT algorithm in newer database engine versions will not stop incoming SQL queries, so it's not that bad.
Modifying or removing columns in the same deployment unit with your code change

As you already know from that point about deployment units - migrations are executed before deploying the code from a particular deployment. Because of that, you may encounter downtime if you are deploying a migration that will do some database structure changes and the code that is using the new version of the structure because - after executing migrations, the old code will still be looking for the old design, ouch!
Changing the type of a column

This problem is so widespread that it might be a prom queen. Well, you might ask - why? The situation is more difficult because - at first, we are deploying migrations (as you already might know from previous paragraphs), so all the old code will try to use the ancient structure (pre-migration). The second thing is... this ALTER will force the entire table to be recreated (copied into memory, changed, applied) in database memory. This will block all reads and writes operations in engines like PostgreSQL, and for MySQL and MariaDB, we'll stop all writes. That's horrible, isn't it? So, for instance, if you have a large database table with millions of records, changing column type will be so dangerous for you in terms of extended downtime. Remember: reliability is the essential feature of every application because not working application is just useless. So, instead of changing column type - consider adding a new column with the desired column type, write data to both columns, backfill old data from old records and prevent writing to the old column. Then you can drop the column. It sounds like a journey, but that's not a sandbox if you take care of an extensive database table. Do that slower but without any downtimes; it's a win!
Renaming database columns, tables

That's similar to the previous one with changing a column's type. However, renames may cause a lot of errors in your system, so it's much better to follow the last instruction to create a new table/column, write to both spaces, and iteratively get rid of the old one with an invalid name.

PostgreSQL mistakes

Not using concurrently built indexes.

PostgreSQL has the functionality to add indexes concurrently. Remember - by default, migrations are transactional, so when adding an index to an existing table, it's better to disable a transaction for a particular migration and run concurrently. Thanks to that, PostgreSQL will not lock mutations (INSERT, UPDATE, DELETE) as in non-concurrent CREATE INDEX operation. However, it's more expensive because the engine needs to execute two table scans, etc. More info.
Don’t set NOT NULL on an existing column.

If you are working on a more extensive database with higher throughput, adding NOT NULL to your column might not be the best option in PostgreSQL. Why? Because this operation blocks, reads, and writes to your table until each row will be checked if NOT NULL is possible to apply. Instead, it's better to add a check constraint that will ensure you're ready to set NOT NULL. Of course, this check shouldn't be validated yet. Instead, you can validate this constraint in the subsequent migration (the best deployment unit) and apply classically NOT NULL. This is a much safer way.

It all matters when it comes to the scale

Yep, all these rules and tips apply when struggling with a bigger scale. For small applications with small databases and not so much traffic - it doesn’t matter if you’ll go by the book or just by a shortcut. But when you are doing everything possible to keep the application working, you have a massive load on your system - you should consider these mistakes and try to avoid such situations. Of course, you’ll spend more time making database-related changes, but thanks to that, you’ll save money and your client's trust. That’s important - keep your system reliable, and you’ll sleep better. I’m not Matthew Walker (the author of the phenomenal book “Why we sleep”) to talk about the nature of sleep; just trust me - it’s better to make changes incrementally if we’re talking about databases.

Golang - test doubles, where to keep?

Patryk Woziński — Mon, 27 Jun 2022 15:17:43 +0000

Hey! I am wondering where should I put my test doubles like Spy/Stub/Dummy implementations of specific interfaces. What do you think?

I'm of course asking about something that's aligned with go standard layout (internal, pkg directories etc).

In other programming languages I'm keeping these implementations in /test directory with all tests, object-mothers, object-builders... but in Golang we're often keeping tests close to the production code.

What do you think? Have you're dealing with it? :)

Have a great week!

:strong_migrations for Elixir - catch unsafe migrations easily

Patryk Woziński — Fri, 29 Oct 2021 17:56:20 +0000

Hey, hey! I just published my next small open-source library (actually, it's a "port" from Ruby). Thanks to :strong_migrations you can catch all unsafe things that happening in your application's migrations like:

remove column
rename column
drop table
add index non-concurrently
add index concurrently in a transaction
drop index concurrently in a transaction

... and soon much more like migrations that are adding NOT NULL, etc. There are also a few configuration possibilities, so you can prepare everything to meet your application's requirements.

Any feedback or feedback is welcome. That is my first time when I was working on the AST - that's also why I've opened an issue with the "help wanted" label. It's related to refactoring possibilities of the AST parser.

https://github.com/patrykwozinski/strong_migrations

Find refactoring candidates with Churn

Patryk Woziński — Tue, 18 May 2021 09:37:01 +0000

👋 Some time ago I've written a blog post about the problem related to working with classes/modules/functions/methods with both high complexity level and high frequency of changes. It's a problem because if you're doing changes in files that are complex and are often edited - you can easily introduce an error into your application. That's quite dangerous!

Refactoring in itself has no sense - but when it comes to solving problems with more complicated pieces of code that have a chance to destroy your critical business path - then refactoring is worth doing.

How to find refactoring candidates? Use churn! I've created this tool (it's still under development) for detecting which modules need your attention. That's ported Ruby/PHP library. :) I'm still learning Elixir and this project is helping me a lot.

Every improvement related to other things than bottlenecks is muda. When you're using Churn then you'll be focused only on real problematic places in your code.

👉 https://github.com/patrykwozinski/churn

Feel free to contribute, there are many things to do in this project (cache, better parallelism, and so on).

Happy refactoring! 👨‍💻

Essentialism and Software Engineering

Patryk Woziński — Thu, 29 Apr 2021 07:32:04 +0000

Hi after a long time! This time I'd like to present to you my short story about creating a side-project with my friends. Yep, I know that the title is a little click-baiting, but I want you to read this post.

Let's start from the beginning - as I've mentioned I'm working on a side-project that is a SaaS application for car rental companies. I know, I know - it's not rocket science, but we're trying to do the best experience both for our customers and... for us. We want to have an amazing developer experience level and ease of introducing new engineers to the project in the future. I'm nearly sure you have a graveyard of side-projects in your GitHub repositories - you're not alone. Most programmers can't finish projects because of over-engineering, over-thinking, over-...doing fancy stuff like tactical DDD, BDD, TDD, XDD instead of GETTING F***ING SHIT DONE. I was the same - I was doing an amazing architecture, clear, beauty code... but! The projects went away because I was exhausted from doing everything in a purist way. This time is different, seriously!

We've started the project wanting to deliver an amazing product and also still learning some things meanwhile. It wasn't that easy because every step in the wrong direction may cause over-engineering. Okay, I've blamed DDD for some fails and yes, I keep this opinion but it's related to tactical DDD - not strategic. The second one is brilliant, amazing, and my lovely approach to software engineering. So... yes, we did deep modeling of our domain, and we just wanted to understand how the business should work in the best possible way. Probably you get me now like an idiot that ignores amazing and powerful tools... but I am not. I love them but not when I need to deliver a new startup-like system.

Ah, yea - let's go back to the main topic. We did an event storming, domain modeling, etc. just to understand as much as we can the business model of our system. We didn't write any line of code yet, we just talked and moved some stickers on a virtual board. Then after all we've discovered our domains - core, supporting, and domain. If you've ever heard about DD you probably know something about them, but we did it in a practice with a goal. The goal was to understand which modules should be created with an overengineering back of the head. If something is not our core - then it's not recommended to waste the time doing too much detailed application architecture or something in this place.

Let's meet the prom queen of our application. It's so hot topic nowadays at the conferences - to be honest, also I did two or three presentations about it some time ago. The queen is a modular monolith. This time the most important are boundaries between the modules. We've established one rule: you can not use anything between the modules - the walls are like a Gandalf saying "YOU SHALL NOT PASS!". And yes, we did it. Our modular monolith has solid walls between the business blocks. Every block represents our domain language. Vehicle Catalog, Fleet Management, RentalUnits, Reservations, Vehicle Maintenance, Rentals, and so on. Do you feel the climate of a car rental company? I hope so!

Another rule that we've established is that - we're not doing cross-context queries and commands. Everything should be event-driven. Sounds like a complex solution but to be honest it's not. If you need some information - just subscribe to a specific domain event and for example, denormalize information. There's nothing wrong when many parts of the application know something about... let's say a vehicle. In a Vehicle Catalog that's the set of specific make, model, variant, etc but in Fleet Management, it's related to car's plates, mileage. In an availability context (yes, we have it xd) it's just a subject that you can check if in the specific period it's free to take.

Okay, but let's talk about the trade-offs. In a pure Domain-Driven Design solution using tactical tools - you should create a pure model in the Domain layer that is not extending nor using anything from the outside. This model should be used only via an application layer that's not leaking any details to the higher levels like a user interface. Bla, bla - ye that's the way how we do it in the core-domain modules but we don't care about it in generic/supporting ones. Let's meet the Vehicle Catalog module - it's our business context that represents all the existing makes, models, variants, and their data - that's like EURO TAX. As you may know - it's just supporting one and... to be honest we can handle importing from the CSV file, inserting it into the database in one single file, and... we're okay with this solution! Why? Because we won't make money on inserting and publishing information about the cars, it's not our business model. The same we are doing with the UI layers. Yes, in a clear way we should only dispatch commands and queries from this place but... once again - we don't care if it's only a supporting domain. This project is created using API Platform - that's the tool for bootstrapping fully operational API-first applications and let us create full features using just a few notations. THAT'S OKAY, don't worry. And yes - our entities contain infrastructure annotations like ORM mapping or API Platform definitions and one more time - that is fine. Of course - my software craftsmanship's heart bleeding when I see it but... I want to GET SHIT DONE not to create another dead shrine to clean code.

If you're still here - then there are two possibilities. The first one - you got me as an asshole after a weekly Bootcamp in a Pcim (that's a small city in Poland) who doesn't know about software development OR... you got me as a crazy man. Anyway, let's keep talking about software engineering and my side-project. The next big thing: events. Domain events are the most important thing in the whole project. Why? Because they hide implementation, hide my ignorance, hide low-quality of non-important modules. Let's go back to the Vehicle Catalog context. As you may know - we're doing a data import here, and also we're exposing a few endpoints just for the clients (front applications). There is one more thing - we're doing a translation of this piece of shitty code into a nice, simple domain event. What is a full entity of a vehicle? The aggregate contains make, model, and variant like: Porsche 911 964 Targa. That's one, specific vehicle. If we've imported this one - we're emitting an event: "Vehicle Created" that contains only basic information about this car, no more. Then the event is consumed by many different contexts - for example, Fleet Management consumes this event and creates duplication of needed data - thanks to that we can use FleetManagement endpoints for example to present available cars in the fleet without calling Vehicle Catalog for details like name of the car's model. I think that's a huge power gain! We don't have any coupling between modules and we're so proud about that!

What if we would like to do a refactoring of some CRUD-like module that is currently implemented as a small ball of mud? That's easy because we're doing a refactor inside the module and still keeps communication without any change - we'll use the same events as before. That's beautiful.

Okay, now - think about your side-project. Don't you think doing a DDD, BDD, TDD, Hype-Driven Development is not just a muda in many places of your system? Wouldn't it be easier to create solid walls between business boundaries and hide the implementation behind them? Think about Pareto Principle and about lean. You don't need to over-think the project to create something good that gives your clients true value. Create a model of your business - what will earn money and give a value and then - build the best possible model here but not better. Perfectionism is not your goal - your goal is to bring an amazing product to the people who need it.

Software Engineering is not about creating shrines of clean code that would make Uncle Bob happier. For me - the best engineer should be described as a Product Engineer; not PHP Engineer nor C# engineer and even not a WordPress 5.7.1 Developer (xd).
Take shortcuts where the road is flat (supporting domain) and climb with the best tools ever when you're fighting about your business (core domain). Zero waste software development? Yep, I'd name it like that. Essentialism is crucial.

Big thank you to Filip, Przemek, Daniel, and Darek with whom I'm creating this side-project. It's a great journey!

I'm often posting my thoughts and interesting links that I found on Twitter.

Less cognitive load when working with git and pull requests

Patryk Woziński — Mon, 15 Mar 2021 19:26:26 +0000

Hi! After a long time, I'd like to share with you one of my thoughts. Pull Requests, Code Reviews, version control, and git - that's buzzwords with whom we are so familiar on daily software engineering work. That's of course not hard to commit some changes, push to the git repository and then open a pull request, true. After all, we're merging these changes to the main branch like master or development, etc. - of course, it depends on your project's git-flow. Nothing interesting, that's the thing that we're teaching interns and juniors, yep.

But... I think it's not that easy as we think, there is one interesting thing that I've considered in the last few days. What's your approach to creating commits and merging? Have you ever thought about it? What's the logic behind your commits and decision about how to merge -> using squash or not? That's the topic that I'd like to cover today! :)

At first, let's talk about the commits - how to split them? I'll show you in my opinion the best way to present changes in your code. If you're starting with a task -> let's say you want to cover the task like "New feature request: adding an opinion about the doctor". It's huge to do on this topic, that's sure. Let's stick for a moment in this place. It's worth dividing this issue into a few smaller and treat the topic as epic. After some time we could discover subtasks there like:

create an API of this action
prepare the mock implementation of higher-level components
prepare implementation details like all the side-effected elements (database, queues, etc)
As you see I'm a fan of the top-to-the-bottom approach in software engineering.

Let's make the new branch from the main development (master or develop - up to your company's process) for a whole epic and name it like add-doctor-opinion. Every subtask of this issue should be committed separately. If it's a bigger thing to do - just create a branch from your task branch then treat branches as a possibility to hiding implementation details. It's like an encapsulation of the knowledge in programming. If your sub-branch is ready to merge - every commit should represent something important at a specific level of abstraction. In our example those commits could look like:

API tests to cover business needs
Lightweight implementation of the API
Fix XYZ in ABC

Thanks to this people who are reviewing your code can do it step by step going deeper by commits in implementation details and check your thinking way. If everything is ready and your team approves the code - merge using squash. I love squashing commits because it shows changes without knowledge leaks for the future coders. Of course - if they would need to know more - they can open your archived pull request and check all the commit-steps in your thinking process. Less cognitive load => happier coders #devExperience!

Okay, so you've merged all the changes on the specific level of abstraction into your main task branch and now it's time to go to the production. That's also an amazing time to merge using squash because you still need to hide implementation details from the future code-readers. Every change you did is important but especially for you and code reviewers at this moment. If people from the future need more information about your changes they should check your Pull Request and what was happened there on the specific level of abstraction. In this situation, they also might go deeper into the sub-pull-requests of your subtasks.

[Feature] Add opinion #main-pullrequest
- Create an API of adding opinions #creating-api-pull-request
- Api tests (failing) #commit
- Stub implementation #commit
- Fix for parameter #commit
- Dummy implementation of adding API #dummy-implementation-pull-request
- Implement the most simple version of API #commit
- Fix XYZ #commit
- Implement database adapters for API #db-adapters-pull-request
- Install Hibernate library to use database #commit
- Integration tests of adapters #commit
- Hibernate implementation of entities #commit

This approach to working with git could maximize the productivity of your team, future code readers, and also it's a huge help for code reviewers so thanks to this the work can be much more enjoyable. Remember: your code is an implementation detail - your commit names can show higher-level decisions. That's a huge gain for Developer Experience in my opinion!

I'm still practicing this approach and learning how to decrease cognitive load when working with git. I hope you enjoyed this post, have a great week!

I'm often posting my thoughts and interesting links that I found on Twitter.

Everything is about filter, map and reduce

Patryk Woziński — Mon, 07 Dec 2020 23:09:19 +0000

I'm after my first quarter worked as Elixir engineer. Does it was easy for me? No. Changing language from PHP to Java? Why not! PHP to C#? Let's go for it! But learning functional concepts is something more than changing your daily toolset from the wood hammer into the metal hammer. It's about how are you looking at the problems from a higher level.

In object-oriented languages, we think about the data encapsulated in the objects. They are connected by relations that describe the world around us. Sounds reasonable I hope.

In functional programming, there is no more data in the heart of the problem. There you need to think about the problem as to how to transform something into a different thing.

Let's assume you’re working on an application where the main problem is to prepare a salad for your family. You have a few vegetables like tomato, cucumber, and red pepper. At first, you need to peel the cucumber. So... How to do that? You need to filter vegetables (you don't want to peel the pepper, trust me) to find cucumber, map it to take off the skin. Viola! The cucumber has been peeled! The code could look like below:

vegetables |> filter(cucumber) |> map(&peel/1)

Okay, so now you have to slice all the vegetables like in the example, just without filtering because you want to do that for all the vegetables.
After everything, you would like to mix all these things into a tasty salad. How to do that functionally?

prepared_vegetables |> reduce(&mix/2)

So... As you see the most important is to understand operations like filter, map, and reduce. Look - we’ve never worried about the state or something - the only significant operations are input transformations. Of course, functional programming also has tons of academic terms but you don’t need them to start your journey with FP!

I wrote this post as a short note for myself from the future. Most of my posts (like this one) were written on the iPhone in a bed. Please forgive me for typos and my poor English, lol. I hope I’ll find more motivation to write something more technical in the near time. I'm often posting my thoughts and interesting links that I found on Twitter.

The best technical book you’ve ever read

Patryk Woziński — Wed, 04 Nov 2020 23:33:35 +0000

Hi Geeks! 📚

What's the best technical book you’ve ever read? One, just one book.

From my side, it is ”Philosophy of Software Design” by John Ousterhout. I've learned so much about code design and also about software architecture. Thanks to this book I’ve learned about modularity in software engineering and how significant it is! It has changed my point of view on the whole software engineering world. Highly recommended! :)

If you have some space on a shelf - pick this book and you’ll not regret it.

And you? What’s your strongest recommendation? Share it, maybe thanks to this someone else’s career will be just better! ✌🏻

The importance of code editors

Patryk Woziński — Thu, 22 Oct 2020 19:04:47 +0000

The post was originally published on my website patryk.it.

Elixir coding tools

Hi! I was trying to write something about Elixir for the last week, but… you know what will happen if you found a new IDE and trying to customize everything to be so useful for yourself. So! I have wasted about 6 hours configuring my code editor. I am simply a master of procrastination. Thanks to this - I can give you this story. Enjoy reading!

I love being an IDE's power user

In the last years, I was working with amazing IDE's like PHPStorm, IntelliJ, or GoLand - they were so powerful, ergonomic, and they increased my productivity with every new keyboard shortcut! I was on very interesting workshops organized by DDD-WAW (the group focused on Domain-Driven Design topics) and Michał Michaluk from Bottega-IT was presenting some practices that give excellent results in refactoring legacy code. He was using tons of IntelliJ's shortcuts. I just was amazed by this. :D You can watch the previous version of this presentation in Polish here: 4Developers 2019: Refactoring live session toward tactical Domain-Driven Design, Michał Michaluk - YouTube.

As you maybe know I am working with Elixir at this moment. I was testing Visual Studio Code, IntelliJ, Atom - every one of them had a lot of features that has frustrated me, seriously. They are so slow (in my case?), had not the best autocomplete suggestions, etc. I gave them three weeks. After that, in simple words: I was disappointed.

Finally!

On the last Friday evening, I was trying to write some articles in VIM (GOYO + Limelight rocks!) - and then I became enlightened: WHY NOT TO USE A VIM FOR CODING IN ELIXIR?! After a few hours of configuring the tool and everything was finished. It works amazingly and extremely fast. I've created a repository with my configuration (requires VIM 8+) - you can find it here.
Configuring vim-elixir and alchemist.vim gave to me super support in coding. You also need deoplete.vim plugin. Just read about it on the Internet, I'm not a VIM expert YET. I don't feel as productive as when I was using PHPStorm for PHP or GoLand for Go neither IntelliJ for Java, but it's pretty good and I'm so happy at this moment.

What I am missing?

So, I have a great tool, but it's not perfect. To be the perfect one I need better moving in the codebase skills, but I think it's because I don't know yet the project as well as I want to. I'm still looking for something that will help me with smooth switching between the modules. It's not that fast as was while I was using JetBrains tools. Ah! JetBrains - please, give us some alchemist tool!

Hope for the best

I hope in the next days my productivity will be even better than it's today. If you have any interesting plugins to recommend to me - comment on this post. Any support is appreciated!
At this moment, you can call me a real hipster. Vim and Elixir - that's the combo! Better give me a cup of Starbucks coffee because I deserve it!

Keep fingers crossed and maybe I can write something more technical in the near future!

Elixir application performance monitoring tools

Patryk Woziński — Tue, 13 Oct 2020 14:01:01 +0000

Hi guys!
What do you use to monitor web application performance in Elixir? I think about implementing some mechanisms but still don't know what the best fit for the Elixir language.

AppSignal
DataDog
NewRelic

Which one gives the most useful metrics and information? I want to have there also logs from the slowest database queries, etc.
Do every one of them needs to tag EVERY function in the whole application to track them?

I love NewRelic from my previous job, but still, I'm not sure if it works correctly with Elixir apps.

What's your opinion?

My first week as an Elixir Engineer

Patryk Woziński — Tue, 06 Oct 2020 12:22:52 +0000

The post was originally published on my blog patryk.it. It's a part of "From PHP to Elixir" series.

Hi guys!
Finally, I am working as an Elixir Engineer at Zenloop. That’s a company focused on the customer experience where Elixir is a primary programming language. After my first week of work, I can say one important thing: the elixir ecosystem is not mature. Many times you’ll not find packages or libraries that can pass your expectations and you will be forced to create them from the scratch.

I’m still while onboarding process, but thanks to my buddy I have the first real Elixir task. A few days ago I got a task to do research of libraries that supports parsing CSV and XLSX documents. That's so generic thing to do in mainstream programming languages like Java, Python, or PHP but in this case, it was not obvious. I found a few packages on the Hexdocks (that's like packaging for PHP, but it's even better) - few of them were not looking stable for me or had not the best support from the creators. That's important when you're looking for a tool for professional usage - not for a home project.

In my case, I’ve needed tools that help me to define the encoding of the given file and something that could guess which delimiter for the CSV file was used. There is no package, library, or tool that meets my requirements exists in the Elixir ecosystem and I’ve needed to create them from the scratch. I feel sad.

Besides missing tools that have forced me to create my own is that - the Elixir is perfect for processing sets of the data. That’s not a great discovery - but for me, it was pretty cool to see how a functional approach helps to deal with file processing in a real project. The next things that are so fancy for me are protocols and behaviours. I’m interpreting them as a type of IoC (Inversion of Control), protocols are for data and the behaviours for modules. Do you think it’s a good topic to cover in the next post?

Working with CSV in Elixir

Patryk Woziński — Thu, 01 Oct 2020 13:17:59 +0000

Hi guys!
I'm not sure if asking here is okay, but let's try. I have a question about working with CSV files uploading, etc. How are you dealing with different encodings in Elixir? I was looking for some packages to process csv but unfortunately, I found only a few (2?) libraries that are not too weak. And... the problems is that - these libraries expect from me to define encoding/charset of an uploaded file. It's okay if you're working on some-static files, but I want to give my user's ability to upload a file with any encoding and my application should detect which of encoding it is (German characters, Polish, and so on). I wasted time because the same solution in Python would take 10% of the time with production-ready solution. :( Python/Java/C++ have a library like UnviersalDetector, but for Elixir I can not find anything similar to this.

Do you have any tips/tricks for a new guy like me in this technology?