DEV Community: RomanMizulin

Choose functional stack for next project

RomanMizulin — Sat, 07 Dec 2024 14:59:35 +0000

Top Open Source functional programming technologies

RomanMizulin ・ Oct 29

#scala #functional #haskell #elixir

Top Open Source functional programming technologies

RomanMizulin — Tue, 29 Oct 2024 18:45:35 +0000

Author: Roman Matveev. LinkedIn x.com

Purpose

The goals of this article is to research the most popular FP based technologies to prove FP soundness and versatility and to demostrate that FP has been already with us for a long time.
Methodology involved analyzing GitHub for the most popular projects, blog posts and academic papers.

A little bit of historical context

Engaging talk with a lot of references to historical moments emphasizing point that:

"Good" languages are differed from "bad" languages by theoretical base (specifically math logic brought to computer science). Not just engineered.

"Propositions as Types" by Philip Wadler - YouTube

LISP

Ancestor of almost all functional languages.
Originally was designed to solve 1 specific problem of symbolic differentiation, for which FORTRAN was very inconvenient. And eventually became general purpose-purpose language. In that sense (growing from 1 specif problem) it is similar to Erlang.
With time passed became purer language thanks to academic influence.
It had inspired a lot of other languages including the whole family of so-called LISP languages.
The second-oldest high-level programming language still in common use, after FORTRAN.
It brought to the table such essential concepts without which we can not imagine our industry today as :
- Conditionals not limited to goto (if-then-else)
- First-class functions
- Recursion
- Garbage collection
- Dynamic typing
- Interactive programming environment (REPL)
- Full language available at load time, compile time, and run time

Haskell

Appeared in 1990 with stable release at 2010.
If LISP started all funtional obsession in XX century then Haskell summarized ideas of languages appeared in the next 30 years. The main idea was not that academics should not reinvent same features in their languages, but to make common ground for benefits of all. Fun part is that writing I/O related code in Haskell was so annoying, so Monadic I/O was ultimately invented that has groundbraking influence. Here stonk monad plot showing Haskell m-wording the world.

Other

There are plenty languages worth to mention and their role in history, but let me focus on some of them relevant today.

OCaml

Direct descendant of ML which Appeared as dissatisfaction with LISP writing proofers. The idea was to have types which lower logical errors. (LISP -> ML -> OCaml)
OCaml was written in 1996 in France. It was inspired by a long line of research into ML, starting in the 1960s, and continues to have deep links to the academic community.
Less pure comparing to Haskell but more focused on Industry.
So if you want to participate in making tooling for whole ecosystem for future generations - it is a good opportunity to learn and make huge impact.

Erlang

Appered from need for faster develompent to reliable run on telephony switches, because in 80s managers realized that they spend more on developing and maintaning software than on hardware itself. It was designed to be concurent, distributed and high available from beginning long time ago before async fashion in JS and Python. In contrast, OTP and Erlang almost 30 years(open-sourced) old battle tested technology. Interesing, that originally it was propriate closed language of Ericsson's company, which is powering world networking now.

Top projects grouped by language

Elixir

Appeared at 2012. Built on top of BEAM to make Erlang\OTP runtime more accessible, because Erlang is not user friendly really; and shares the same abstractions and OTP framework for building distributed, fault-tolerant applications. Though initially designed with dynamic typing, Elixir is actively developing static typing capabilities cnrs. As Joe Armstrong said about types "no good type system save you from node failure" — that is why Erlang language was dynamically typed — it was not focus. And it is one of the two languages in this list available on Leetcode, but I would recommend to use https://exercism.org/ for practice firstly. You can think about this language as more modern and easy to use language for BEAM runtime than original Erlang language.
Why does it matter? Because Erlang really powers high load. You can find out more here about real world use cases of Erlang its VM called BEAM. Whats-app cloud handles 2 millions connection per a node for instance.
It is simple language nevertheless a powerful one. Maybe Elixir has not really good theoretical fundament, but it is still good language and its author José Valim actively works on bringing type system with synergy of academic world.

Picture from talk by creator of the lang José about different levels of abstractions Keynote: Gang of None? Design Patterns in Elixir - José Valim | ElixirConf EU 2024 - YouTube

Phoenix: Top from stack overflow surveys

Stars: 21k
One of the most if not the most popular FP frameworks that has won "the most admired web framework" at stackoverflow research for several years in a row.
Stack Overflow Developer Survey 2023
Fullstack, Ruby inspired. Connects high load with easy of use.
There are big companies using this framework check related block at Phoenix Framework

analytics

Stars: 19.1k
Simple, open source, lightweight (< 1 KB) and privacy-friendly web analytics alternative to Google Analytics. Serious niche and good moral idea.
All we can do - is to embrace such kind of projects.

We are dedicated to making web analytics more privacy-friendly.

supabase/realtime

Stars: 6.6k
One of the projects of famous supabase project which leverages Phoenix from the first place. It makes use from BEAM through Elixir.

This is a server built with Elixir using the that enables the following functionality:

Broadcast: Send ephemeral messages from client to clients with low latency.

Presence: Track and synchronize shared state between clients.

Postgres Changes: Listen to Postgres database changes and send them to authorized clients.

Erlang/OTP

Not just language but framework too running on BEAM. Erlang is not user friendly or looks familiar to popular languages so Elixir was created to make this incredebly concurrent platform accesable to regular web developers. .
Joe Armstrong remarked in an interview with Rackspace in 2013:

"If Java is 'write once, run anywhere', then Erlang is 'write once, run forever'."

CouchDB

Stars: 6.2k

If there’s one word to describe CouchDB, it is relax

Document based reliable scalable database with nicely designed HTTP/JSON interface.
With accompanient of Pouchdb can be the best choice for offline-first applications with low effort data syncronisation.

RabbitMQ

Stars: 12.2k

RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine. It is currently used by millions worldwide.

Famious publish/consumer message queue with a lot of scenarios to implement.

emqx

Stars: 14k

The most scalable open-source MQTT broker for IoT, IIoT, and connected vehicles

Gleam

Appeared at 2016. V1 was released at 4 Mar 2024, so it does not have popular projects yet, but it grows fast. Running on Erlang VM as Elixir, but statically typed in opposite of dynamically typed Elixir and Erlang. Inspired by many languages and as author says "I stole the best ideas for this language." © Louis Pilfold .
And I have to admit that coding in Gleam is pleasant experience. It is really easy to learn and fast LS gives instant feedback on type coherence.
If you can choose among Erlang, Elixir and Gleam. You should deferentially look at Gleam, but if you like Ruby look at Elixir of course.
Maybe it is true opensource sponsored language, in this meaning it is a good example of growing contrasting to Elm.
Try it at The Gleam Language Tour

P.S. Remember Golang mascot? yeah... Now look at this piece of art.

Front end in Gleam

Lustre allows to write fullstack applications in Gleam reusing same types!
Basically it follows the same MVC architecture as Elm.

Clojure

Clojure is a compiled language, yet remains completely dynamic. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection... is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system.

Appeared at 2007. From quote we can conclude that one of the main factor becoming popular is running on JVM. Concurent of Scala.
Used at Walmart Labs
“Our Clojure system just handled its first Walmart black Friday and came out without a scratch.”
For me dynamic nature is red flag nowdays, especially when there such beatiful languages such Elm, Ocaml, Haskell and maybe Gleam if it woud not have so many braces.
Here are really titans of Clojure world in terms of numbers of stars.

metabase

Stars: 37.4k
Very popular tool for work with data for non-technical people. It let to get and visualize simple data without SQL, but for more complex queries SQL is required. One technical person can make views and all non-technical part of a team can use it.

logseq

Stars: 31.1k
A privacy-first, open-source platform for knowledge management and collaboration.
Notion and Obsidian like alternative.

penpot

Stars: 29.5k
The open-source tool for design and code collaboration.
Figma alternative.
By number of stars we can assume that IT world at least admired this tool.

Elm

Appeared at 2012. Extremely efficient and deligtful to use to build browser UI. Not popular now - in its afterhype period of time, but definitaly impactful. Can be seen as more simple and compact child of Haskell and OCaml. Has elegant type system. Can be learn in a weekend.
There are elm-ui, elm-review popular libraries. And there is not much activity on official site actually, but languages is alive. In talk Evan told that he made compiler for servers, but he really struggles to make sustainable financial model. Also seems like Evan does not like online marketing his language still loving conferences. Personnaly I agree with him and I think much people prefer real events to virtual ones but it is important lesson for language designers: online presense matters.
Check Elmcraft and oficial forum for latest news and inspiration.

Haskell

The most academic language in the list. From the beginning made with good type design comparing to the most other popular languages where types were introduced on the go.
Highly admired by creator of ROC language - Richard Feldman. He also praises Elm by the way.
This language has a lot of concepts, so it is a good idea to learn language to explore FP ideas in practice.
And I would say that language is excellent for well understood problem that can be something formally described to leverage its strong type system.
Some researches say that there are better language than Haskell like Agda or Idris. Ok, but for many of FP languages Haskell is a progenitor.
The most elegant and concise language in the list, nevertheless it can be unpleasant to use in production due to various ways to do the same thing.

ShellCheck

Stars: 35.6k
A static analysis tool for shell scripts adopted by:
Travis CI, Codacy, Code Climate, Code Factor, CircleCI via the ShellCheck Orb
GitHub (only Linux), Trunk Check (universal linter; allows you to explicitly version your shellcheck install) via the shellcheck plugin
And you can use it for too if you work with bash/sh scripts.

pandoc

Stars: 33.3k
Universal markup converter. Supports numerous format from/to. Practical and helpful.

PostgREST

Stars: 22.7k
REST API with automatically generated OpenAPI for any PostgreSQL database. Expose database schema with filters and make PostgreSQL views for complex API endpoints.
If you often write CRUD API for database tables manually, it will save you some time.

And more less popular projects

Like hadolint: Dockerfile linter, validate inline bash, written in Haskell; Purescript, Unison, Idris languages implemented in Haskell.
I recommend to check ideas behinds those projects. Especially Unison's mind-blowing something naive idea about infinite computational resources.

Scala

Brings FP to Java land. Good path for whom looking for functional language to code and the best job opportunities. But not elegant language at all in my humble opinion.

spark

Stars: 40k

Apache Spark™ is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.

Created in 2009 at UC Berkeley's AMPLab, open-sourced in 2010 - quite some time. De facto standard for big data processing making use of Scala's functionality. Adopted by Netflix, Uber, Pinterest, Alibaba.

Enso

Stars: 7k
Interesting analytical tool and language with one-to-one visual counterpart. Its IDE has some bugs, and requiring login for some reason, but it is worth checking.
And can be used for "Detecting Credit Card Fraud with Enso" as stated in their blog post.

Python with returns lib

Python is not functional language itself. And as I can relate does not want to be: look at lambda for instance. But this package allows to write more robust applications.
Link to lib: GitHub - dry-python/returns: Make your functions return something meaningful, typed, and safe!

In closing

It is safe to say that functional language can be used in almost any domain excluding maybe system engineering, if not counting Rust as functional language.
You may wonder after all why then functional programming languages are not dominating everywhere if it is so good? So Richard gave a good answer to it: Why Isn't Functional Programming the Norm? – Richard Feldman - YouTube. Let it brew.

Do you want full stack type safe scalable app?

Look closer at Gleam or Phoenix. It has good balance of simplicity and FP feature along side with good development experience. Maybe add CouchDB since it runs on BEAM too.
Great article on using Gleam for a long time.

Do you want a queue?

RabbitMQ for general use, EMQX for IoT.

Do you want analytics?

Give Enso or MetaBase a shot.

Do you want complex reliable backend for WEB or for CLI?

Look at Haskell or OCaml. Which one specifically? There is a good article comparing from an experienced developer in both languages.

Do you want frontend?

Elm or Elm inspired frameworks. Fortunately there are infinitely many Elm inspired frameworks like mentioned Lustre or something esoteric like Unison lang library.

Do you want to try FP in TS or Python?

fp-ts for TypeScript, returns for Python.

Do you want insane Data/ML stuff?

Spark

P.S. Radar version of this article would be good.

References

John McCarthy, 1979, History of Lisp
Hudak, Paul; Hughes, John; Peyton Jones, Simon; Wadler, Philip (2007). "A history of Haskell"
Beating the Averages
Functional Programming in 40 Minutes • Russ Olsen • GOTO 2018 - YouTube
Why algebraic data types are important - Bartosz Milewski

Rate my python async snippet

RomanMizulin — Sat, 27 Apr 2024 22:20:11 +0000

Simple tips for predictable code

RomanMizulin — Mon, 22 Apr 2024 10:25:35 +0000

Functional programming is a complex topic in Python, but here are simple rules, that anyone can stick to and write more reliable and readable functions from my 4y+ experience:

Write type annotations.
Do not mutate input data.
Use less classes.

That's it!

P.S. Yes, there are standard and 3rd party libraries which can enhance productivity, but see tips above as a general rule and common ground.

Platform engineering

RomanMizulin — Sun, 07 Apr 2024 19:24:47 +0000

Definition

Platform engineering - is a process of crafting of an internal portal and related DevOps tools for developers. The goal is to encompass infrastructure technologies in a way to reduce developer's cognitive load via meeting developer's needs.
How does it differ from DevOps? While DevOps is related more to approach or philosophy of making and managing products in collaborative manner with shared responsibility between developers and operations, platform engineering is focused on solving specific problems of internal developer teams by providing set of tools. So platform engineering contributes to DevOps.

Who need and when

When there are emerging different tools and approaches for common problems from different teams, several problems and intuitive intentions come:

Unifying development tools for
- easy onboarding
- cross team workforce exchange
- to have common ground for tools and processes that will be build on top of these tools
Finding "golden path" that enables developers to work efficiently with pleasant development experience (DevExp)
Managing services, i.e. keep track of ownership, technical requirements and other meta information.

What platform engineer should know

This field necessitates a dual expertise: infrastructure knowledge(mostly cloud) and programming prowess. Also you should love DevExp. If you use ancient tools and have no intentions to make it better even for personal use then maybe platform engineering is not for you really.

Technologies and tools

Of course, you can build everything from the scratch, but there are already popular technologies in this field.

One of the them which is must-know is Backstage - framework offering basic necessity, a lot of community driven plugins such an integration with Gitlab/Github and a basement for company specific needs by developing own plugins.
That said, after gathering development needs, develop section inside backstage in backstage via React, Typescript and choose plugin architecture pattern one of 3.

Another quite widespread technology is Argo CD - a declarative, GitOps continuous delivery tool for Kubernetes. Using Git repositories as the source of truth for defining the desired application state. You define tracking strategy once or when need, you push, Argo updates target environment.
Overall, complete solution can look complicated. Be sure to evaluate what you really need depends on company size and already used instruments. Check out
catalog of platform tools Platform Tooling!
And of course, try to avoid companies with old/boring technologies at all cost.

Backstage showcase

Backstage Demo
How adoptation of the service can look like.

What reports say

Security is number 1 consideration
Product manager of internal platform matters
It requires both time and effort before an Internal Developer Portal (IDP) starts to generate tangible value for developers. # Certification
Simple introduction into Backstage Introduction to Backstage: Developer Portals Made Easy (LFS142x) - Linux Foundation - Training
Any certificates for developing kubernetes applications
Any cloud certificates # What looks like typical vacancy From requirement we can see that tech stack can be anything, but some kind of Cloud, Container orchestrator, Programming language

AWS / Azure ADO, Terraform, KMS and Cognito, Grafana, AWS Cloudwatch & Cloudtrail, MS SQL/SSIS / Aurora Postgres, ASP .NET, Hangfire, Entity Framework, Dapper, Redis, Amazon ElastiCache, Amazon EC2, Rabbit MQ, AWS QuickSight, Azure Active Directory, Amazon CloudWatch, Azure Active Directory, SCCS: Git, Argo CD. Programming Languages: React, C# .NET, Python.

Author social links

Feel free to write a comments or remarks.

linkedin.com/in/r-matveev/

Roman (@mizulinshina) • Instagram photos and videos

Profile / X

Communities

Slack - big and active space [Join Platform Engineering on Slack | Slack](https://join.slack.com/t/platformengin-b0m7058/shared_invite/zt-2fbb2ci12-uURYT_JMYQ~_gQ~IKNYd0Q

References and futher reading list

Practices and tools for a startup

RomanMizulin — Thu, 23 Mar 2023 01:20:09 +0000

Approaches needed for a start-up are absolutely not suitable for large companies like Yahoo, Google or Amazon.

The purpose of the article is to give a general idea of the "right" approaches when creating a startup.

What approaches are needed for a startup?

Making a feature fastly is more important than following "best practice".
You can choose unstable libraries, but more convenient and functional which help for rapid evolution.
Bet on people who can make a feature here and now rather than on those who will do it right, but for longer.

These approaches work well in the short term, but after a relatively short time, problems may begin slowing down the pace of development:

Coupling and cohesion of the code
- You can't just figure out what's going on in the code section without making 10 deep jumps. The speed of development gradually decreases.
- It is impossible to split the code, to separate it into separate entities or modules or services because technical and business entities are closely intertwined with each other or because the module takes over too much.
- In a couple of years, it will be easier to rewrite the entire project from scratch than to try to refresh and refactor the current code base. Examples: Photoshop, khan academy. You can name a few by yourself.
- It may be necessary to rebuild the application a little differently for a new hypothesis. Again, high connectivity will prevent this. Instead of building the necessary application from the components brick by brick with improvements, you will have to copy-paste and make small changes in a lot of code sections. And God willing, you will have time to write tests before that.
Bad DevExp
- Increasing cognitive load.
- Semi-automated deployment -- every time you need to keep in mind some technical elements such as environment variables in order to perform a step-by-step algorithm that could be generally speaking automated at the level of a Bash script or CI\CD.
- Partly manual formatting of the code. There are auto-formatters for almost all languages used in production.
Insufficient elaboration of the task before its transfer to development It is fraught with repeatedly redoing the features, or the programmer will be forced to think about the possible states and transitions between them himself, taking into account the business domain nuances, which can take a lot of time, not to mention the increased cognitive load. Maybe this is a programmer in a startup?

The recipe for overcoming problems is quite simple:

If possible, write isolated code - low-coupling with high cohesion.
Automate processes. The ideal is a ready-to-use development platform in which most technical issues are solved, which allows developers to focus on business-specific tasks. For small teams it is a luxury, for big teams it is crucial. In a startup, start with strict naming conventions\paths\approaches - do not forget to document, or even better at the level of checks in the CI\CD, put into auto-formatters, pyinfra and look at gitops, when the transition to Kubernetes is ripe. Ideally, automate some checks at the pre-commit stage of a project based on agreements and force devs to use pre-commit.
Detail tasks before transferring to development or lay down time for study in development.

A balance between long-term and short-term goals. Naturally, if possible, because most startups are constantly on the verge of collapse. There are things that, if not laid down at the very beginning, then they will shoot at an inconvenient moment - they will significantly slow down development if they do not force them to fix bugs most of the time.

We start a project from business entities. Our task is to clearly reflect business entities in technical models and implement business logic tolerably from the point of view of speed. You can rent or buy impressive capacities for relatively little money, and it is unlikely that it will be cheap to write a product from scratch. Therefore, you can choose not the fastest language like Python or Javascript and related abstract magic technologies, but you will fork out for a powerful server. It is the best way to build MVP for checking hypotheses than writing incredibly fast bulletproof code for most cases.

Selection of tools

Use what the team is best able to work with, or lay down time for transition and adaptation.
Get acquainted with new related solutions that can solve some problems and speed up the development of a type of auto-generation of elements for the frontend based on design, or tools for creating internal tools, specific CI\CD for mobile applications.
You can use tools that are currently in beta, nevertheless, these tools should already have a living ecosystem and an active contributor.

Monolith or microservices?

Of course, firstly it is better to make a good monolith with clear interfaces for the libraries\modules than full-fledged microservices for all the best of the best practices. Maybe your product will not live up to the need for microservices, and it most likely will not live, but if it does, then you have low-coupling code and, possibly, autotests, which will make it easy to cut the monolith piece by piece into microservices.

Environment

There is no more painful thing than repetitive necessary time-consuming steps to run or build an application. The environment should not make you suffer by doing monotonous actions to launch the service and test a new feature or catch a bug. Ideally, this is one command - and written in a README file - for each environment without a long wait, when the service code is checked for quality, assembled and rolled out to dev stand.
pyinfra -- take a closer look to automate processes at the initial stage.

ORM or NOT for ORM?

There are a lot of ORM libraries for various especially high-level languages. It is okay for a really simple CRUD application. For something at least somewhat complex (2+ JOINs) and/or especially performance - only raw SQL queries I would suggest.
There is an additional level of abstraction between the technology and your code, which may limit you or, most unpleasant of all, it will put an unobvious bug after updating the library at the most inconvenient moment.
ORM dialect can be updated to the major version and, surprise, all your ORM code has become a legacy.

GraphQL or REST API?

Selection criterias:

Developer's experience.
Configuration of backs\fronts. Where do you want to shift the work of getting data from the backup?

IMHO, it is better REST in most cases, because the frontend focuses on designing, and endpoints can be written around business entities, thereby creating a clear visual structure.

Database

Most likely, you will have enough of SQLITE to test the hypothesis. If you are creating a mobile application, then take a closer look at firebase or PocketBase.

PostgreSQL is a wonderful database, but take into account the overhead costs of hosting, monitoring, configurations and dumps -- administration.

Containers

Docker - must-have. It has a lot of documentation and a large community, any issue can be solved very quickly.
Docker-compose - when containers become more than 1.
Kubernetes + rancher -- when you have money for a DevOps engineer.

Low code, no code

There are a lot of solutions on the market for websites, mobile applications, frontends (locofy), backends.
Keep in mind that powerful and low-code multifunctional solutions without code sometimes require a long study. And, again, this is an additional level of abstraction, which means less control and restrictions.

Internal tools

For what? To automate internal business and technical processes.

budibase low-level open-source code platform for creating internal tools, workflows and admin panels in minutes. Supports PostgreSQL, MongoDB, Rest API, Docker, K8s and more.

re-tool is not open source, but also good.

Monitoring and metrics

uptime-kuma -- a minimalistic multifunctional thing to understand that your service is alive. There are integrations for alerts.
prometheus + grafana -- the earlier, the better. It will give an idea of how your service is used by technical and business metrics.

Files

If you just have files, then any s3-like bucket will suit you. If you need to store media files and especially do transformations with them, then take a closer look at download.io and https://cloudinary.com/

PWA

Maybe PWA is suitable for the first version instead of a mobile application? You will save a lot of time, and you are unlikely to get to the top of apps store quickly.
In addition, some solutions also allow you to build a native application from the PWA.

Uber starter pack for MVP

ruff for Python, rome/tools for JS
sqlite / pocketbase / firebase
piinfra
pre-commit
Docker
Gitlab git flow