DEV Community: Typeable

What do types mean for programmers?

Catherine Galkina — Fri, 25 Feb 2022 19:07:17 +0000

Author: Ville Tirronen

Why is our company named Typeable? What does it even mean?

Types are one of the most essential concepts for software developers. They can also be one of the most confusing and multifaceted things in our field. And they are an endless source of arguments on the Internet. Thus, calling a company Typeable might seem like taking a stance. I believe many of the arguments around this central concept are based on different notions of the term and not so much on true disagreement. So, let's discuss a bit what the types even mean (to us anyway) and why software developers talk (and disagree) so much about them. And why Typeable is a great name after all.

Basic meaning of the word "Type"

When computer scientists or programmers use the word "type", what do they mean by it? In the ordinary sense, the word "type" is used in sentences like "he is not my type" or "there are different types of skis". Its synonyms include words like "kind", "sort", "class", "category", "family" etc. (funnily, all these are also keywords in some programming languages). We use this word to express that some things are unlike some other things, usually in some fundamental way. And that, I believe, is also exactly the origin of this term in software development.

We use the word 'type' to differentiate one thing from the other

Software engineers talk about "types" a lot more in their day to day work than they do outside of their work. And it seems that the concept of a "type" is somehow much more essential to programming than to other trades. While civil engineers might occasionally be talking about different types of cement, programmers seem to find daily uses for the word. Why is that?

Why do software developers in particular talk about types?

As you have probably heard, or discovered, everything that is processed with a computer is represented with only sequences of zeroes or ones. While this is not the whole truth, it gives the basic idea of what is going on inside a computer. To give a concrete example, the number 1 is often represented with the bit sequence 00000001 while the number 2 is 00000010 and 3 is 00000011 and so on. Similarly, the letter a is often encoded as 01100001 and b as 01100010.

So, to do proper processing, one has to know how to interpret these sequences of bits. If you are careless and read 01100001 as a number, you will get 97 instead of an a. They are different types of things.

However, after punch-cards went out of fashion (mostly because they were insane pain in the nether regions to use) computer programmers have not really worked directly with bits. Instead, we jumped straight to talking about different kinds of numbers, memory addresses, and strings of characters (all automatically converted to the bits by, and for, the computer). Thinking about things in the memory of a computer as different types of things, like numbers or characters, is much more efficient than thinking about bits. If nothing else, types allow us to have discussions about how to program: you can tell the person next to you: "This routine operates with numbers and you cannot use it with texts".

A deck of punched cards comprising a computer program. Written in PL/1 for the IBM 360. ArnoldReinhold, CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0, via Wikimedia Commons

Further, even if we conceptualize computers working with basic types like numbers and characters etc., we'll find that we might have, for example, different kinds of numbers. One number might be about money on the client account, while the other number represents an hourly fee for some service. These two are of different kinds of things and it does not do to mix them up. Thinking about "types" of things is an inescapable fact in building programs.

Types and Type Checking

Anyone who has tried programming has probably found out that it is really easy to make mistakes when writing programs. While there are lots of ways for programs to go wrong, one big class of mistakes arises from confusing a thing of one type with a thing of another type. Instead of subtracting a value from a number representing a customer's balance, you might subtract the number from the memory address where the customer balance is stored.

At some point someone decided that it would be better to have computers check that this particular mistake would never happen, that is, they enabled computers to do type checking to verify that the most elementary mistakes can't happen. This turned out to be a great idea, and it is pretty hard to find a programming toolkit that doesn't check types in one form or another.

How do we make types checkable? There are two mainstream approaches for computer-based type checking. These are often called dynamic and static typing. We'll give a very brief description of both, not going into details, as a lot has already been said about static and dynamic typing.

A dynamic view

Dynamic typing arises from the idea that it is a bit silly to have the same sequence of bits mean two different things. Conceptually, we could encode an a as 00 01100001 and the number 97 as 01 01100001 instead of representing both with the sequence 01100001 and leaving it ambiguous which one we mean. By doing so, we enable the computer to check the first two bits before executing any routine. If the bits are 00 then we have a letter and if it says 01 then the thing is a number. And in case of a mixup, the computer can safely stop the program from causing a disaster by terminating the processing.

This makes programming safer. If a software developer makes a programming mistake, the programs can now indicate that they have failed to compute a sensible result instead of returning a plausible looking, but the wrong result. The signal indicating that two different things have been mixed up is called a type error.

A static view

Another way to check types is based on inspecting the program code. Instead of tagging values in the memory of a computer with information about their types we attempt to analyze the program code. That is, we look at what the developer has written and attempt to determine whether executing that code would lead to a type error. This is certainly harder to do than doing the same thing with runtime checks, but it offers a seeming advantage of knowing about the errors before the software is even executed.

The inherent problem with the static approach is that if the code can be arbitrarily complex, the process of deciding whether there is an error is very hard or even impossible (see Rice's theorem). Thus, the programmer usually needs to help the process by annotating the code, explicitly naming the types, or by following harder constraints on what the code can be like. Usually, one tends to give up on things like storing both texts and numbers in the same variable or list.

The pros and cons of dynamic typing

From the get-go it seems that static type checking has a great advantage over the dynamic. It can report errors before the program is even executed! That is, you don't need to wait for a user to try invoicing a customer to see if your invoicing routine works at all.
The basic mitigation is, of course, having the developers run their programs before they are given to the actual clients. That is, to try and see if programs do stop on error instead of computing the desired result.

Testing programs well is a hard task. Even simple things can have billions of input value combinations and rarely anything can be fully tested. That is not to say that there aren't occasional serious issues that arise from unexpected changes in types.

However, dynamic typing also offers advantages. Most obviously, it can be much more flexible than the static approach. Static checking, due to being a harder problem to solve, constraints the programmer to write only such programs that can be shown correct by whatever routine is checking them. Dynamic typing has no such limitation: anything you can program is fair game if it does not end up mixing types badly. Further, it is common for dynamic systems to be more resilient to certain changes. Many, if not all, dynamic languages follow a principle of "duck typing": you are free to change which types of things you pass around in the program if those said types happen to support the same routines as the previous types you used.

The pros and cons of static typing

There is only one significant con about static type checking. It is difficult. Surely, routines for doing it have existed for decades, but still, it commonly needs the software developer to pay attention to it while writing programs. There are often all kinds of (type) annotations, like "this variable holds a Number" or "this is a list containing Letters". Also, some things can be either hard or impossible to express. For example, a process that stores either a letter or a number in the same memory address depending on user input is usually not worth the bother of expressing.

Dynamic typing with testing seems nearly equivalent in safety and also perhaps superior in ergonomics, doesn't it? But what are the advantages of static typing then? Without delving into advanced uses of static types, there are some. For one, they are better at guaranteeing some simple conditions than dynamic types. If the program typechecks, we know that some things hold for all inputs without needing to devise tests. For another, there seems to be some cognitive advantage in forcing programmers to explicitly write what types of things they are talking about.

But which is better?

We are now in the argument territory! Both static and dynamic systems have keen proponents who seemingly love to fight about types on the internet. "It is unethical not to use static typing" or "Static typing is for weak minds" they say. "Why would you wait for the program to run to know it is wrong?", "Why would you spend ages convincing the type checker that the program is correct?". But we're not here to argue.

We're not arguing here, but reasoning about types!

Many people have tried to empirically prove one approach to be better than the other, but thus far there seem to be no convincing results for either system being better. Looking at the results of these studies suggests that if one way is better than the other, it is not very much better.

But there is something different about static types that makes some people pursue them more than seems empirically practical.

The naive connection between static and dynamic types

Up to this point, we have discussed types such as numbers and pieces of text, and their only use has been to allow us to name sets of things and alert us on their inconsistent use.

However, most sensible programming languages, whether they are dynamically or statically typed, have facilities for the developer to create their own types for modeling whatever domain they are working with. Like, we aren't thinking about bits when we use numbers, we aren't thinking so much about numbers and letters when we model things like Customer or Order. We can tell the computer that a Customer is represented with two pieces of text and a number and then pass that around as a unit. Like we do when we talk about what our program does: we don't go into technical implementation details, but try to be more declarative.

Likewise, we can often define types that represent the concept of "one of the following things". For example, a customer invoice may be either "valid", "outdated", or "voided (on some date)". A single type represents all these alternatives. Similarly, a routine may return an "error as text" or a "result as a number" and we use a single type to model this. For such types, there exists some run-time metadata that allows the computer to determine which particular case they represent. Types that are "either this or that" are sometimes called sum-types.

As it happens, the sum types provide an interesting insight into the connection between static and dynamic types. Essentially, they allow statically typed systems to represent dynamically typed behavior! For example, if we want to create a routine that can accept either a type Customer or type Address we can simply create a type for containing either of those and have our routine accept that new type. We have no need for dynamic typing for this.

There is a thought experiment in which we use a static language and create a single sum type for representing all of the things found in some dynamic language. Then we can build the same routines as we could in the dynamic language with our static language!

From point of view of static programming languages, dynamically typed languages have only one type: a huge sum of all possible kinds of things in the language. There are no dynamically typed languages, there are only languages that happen to have one static type.

The corollary is of course that statically typed programming languages protect us from errors only up to a point we have applied the type checking to. It is possible to effectively bypass type checking entirely or in parts, and for both good and bad outcomes.

This point of view is perhaps taken to absurdity, but it points out that in one way, there is something more to static types than there is to dynamic types and that in some sense, dynamic typing and static typing talk of subtly different things. Kind of the same but not exactly the same.

But what are static types?

Up to this far, we talked about the type as an attribute of data. That is, there is some sequence of zeroes and ones that is the "thing with the type". This 01100001 is a Number. But then, when we started talking about static type checks we suddenly said that we do type checking without running the program and without even having any data. If types are property of data, what are we checking here? Certainly not types, since if there is no data, there can be no type!

We could be seen to talk about types in sense of "the list customers which, when the program is executed, will contain data with type Customer". But that is a bit of mental gymnastics and certainly not how we seem to be talking about type checking. Instead, we straight out say that customers is a list containing Customer values. We instead talk about the symbols on the screen: "See that expression over here, that is of type Int".

So, the things, in static lingo, that have types are actually not data, but pieces of program code. The expression 1 + 1 is what has the type Int and not as much the of the bit sequence 0000010 stored in the computer's memory. In fact, there are things with types that are never stored in the memory of the computer. For example, constant expressions like 1 + 1 are often eliminated before the program is executed. Moreover, given a sufficiently rich system of types, we can define types like Void (in Haskell sense), which have no run-time representation at all but still help us to express some things about our code.

So, in one very real sense, dynamic typing and static typing, talk of subtly or not so subtly different things.

Mathematics and static types

Software developers are not the only people who talk about types. Mathematicians and modern statically typed languages have both independently grown towards the mathematical meaning of types as well as have drawn great inspiration from them.

Mathematicians precessed computer scientists with a concept of (a static) type for some hundred years. Around the rise of the 20th-century mathematicians started to pursue theories that could serve as solid foundations for mathematics. And during those pursuits, we got probably the first mention of a type that matches the software developers' view of the thing we have in static typing.

Bertrand Russell

There is a nice anecdote about the invention also. Bertrand Russell, one of the mathematicians in pursuit of foundations for their field, accidentally discovered a flaw or paradox in one of their attempts at foundational theory. A Mathematical Set is a thing that either contains or does not contain some other things. You can have a set of basic colors ({red, green, blue}) or even a set of all sets. And by accident, you might also invent a set of all sets that don't contain themselves. Except, that if this set contains itself, it shouldn't and vice versa. A Paradox!

To word this out in non-mathematical terms, consider an island where there is a barber and villagers. Barber shaves everyone who doesn't shave themselves. Does the barber shave himself? Yes if no and no if yes. Or is this just a nonsensical wordplay? Yes, but if you are developing the foundations for all mathematics, being able to write out such nonsense with the said foundations is probably not a very good thing.

Russell's solution to this was to essentially say, "Look, barbers and villagers are different types of things and what you said is nonsense". Similarly, he invented a system of types for sets, where each set had a rank. Each set could only contain sets of smaller rank or the formula was non-sense.

Observe the similarity to static types. Here look at simply what is being said, instead of what it means. We don't need to know the truth or falsity of the statement to determine sensibility. Likewise, software developers look at the written code instead of data and results of program execution to determine whether it is well typed and worth executing.

Alonzo Church @ University of Berkeley

Russell's types have little to do with programming, but there is another concept of types invented in the mid-century, which does. Logician Alonzo Church accidentally discovered a minimal computational system that was seemingly able to express all imaginable computations. This system also had similar paradoxes making it unsuitable as a mathematical foundation (software developers today would call those just "infinite loops" or infinite recursion instead). The solution was the same: define a type for each expression in the formula and then inspect it (textually) and from that, determine whether the system is "sensible".

What is interesting about Church's system was that it eventually was turned into an actual, executable by the computer, programming language Lisp. Occurring around 1958 this made Lisp the second (or third) oldest programming language, still in use nowadays. Lisp is a dynamically typed language, but computer scientists must have had an inkling of the mathematical versions of the concept of "type" from very early on even if they did not appear directly in programming languages for some time.

Type theory is an ongoing mathematical pursuit with some deep and interesting results. Some of which occasionally end up in programming languages (or are independently discovered in the context of programming).

Conclusion

When considering the dynamic vs. static typing discussions on the internet, it occasionally seems that one has stepped into the territory of absurdness. One party is talking about types as properties of some expressions in their text editor and the other party is talking about types as properties of data. Some are doing math. No wonder there are arguments! And these arguments are made worse with the reasonable doubt that static and dynamic type checking are too different for rational comparison. For example, I can write Python code with dynamic type checking without significant problems. But I doubt I would be able to write any Haskell code without static type checking. It makes no sense to compare dynamic and static languages and hope to understand if static or dynamic typing is a better choice (for some purpose).

And what is discussed here is certainly not all when it comes to types in programming languages! In F# there are "type providers" that go beyond inspecting just the program code and draw-type information from external sources like a database. For example, you can effectively say: "Hey compiler, go look up what are the possible values for the database field X and call that set type T". There are Gradual types, which aim to build a system where dynamic and static types form a smooth continuum where developers can alternate between one or another, depending on their needs. There are languages like Isabel where you program to prove mathematical theories and languages like Idris where you do proofs to program. Types get drawn into all kinds of unexpected shapes and are used everywhere.

So, why is Typeable a good name? Simply because it alludes to a great many things and a source of interest that is central to software development.

Do You Know Where Haskell Is Used?

Catherine Galkina — Mon, 13 Dec 2021 14:08:35 +0000

Introduction

Hi there! We’re getting on with our series of posts on "where all those strange programming languages of yours are used". Last time we told you about Lisp, and now we’re going to pick up the talk about functional languages and turn our eyes to Haskell. It’s the main development language at Typeable and, surely, we couldn’t ignore this topic.

Haskell is a general-purpose purely functional programming language. Its distinctive features include strict static typing, lazy evaluations, algebraic data types, and a serious theoretical background. This language is relatively young, it emerged in 1990 but has already produced a big impact on other languages and programming language theory on the whole.

One of our recent posts described useful Haskell-based utilities, but all of them are intended to be used by tech-savvy people. Today we’re going to give some examples of practical use in various industries.

Where and how is Haskell used?

Probably, the most googled and well-known example of Haskell application in major companies is the ~~Facebook~~ Meta spam filters. Messages containing spam, links to potential malware and phishing attacks are filtered by Sigma, the system that was rewritten in Haskell in 2015. The choice of language was based, among other things, on performance requirements, the need for pure functions and static typing, as well as the possibility of interactive development. The Haxl framework is used under the bonnet, which the company has open-sourced. You can find more details about the development process and technical aspects in the company’s blog post.
Eaton is a manufacturer of electrical and hydraulic equipment, as well as components for the aviation and automotive sectors. The company is using Haskell for day-to-day tasks such as scripting, hardware simulation, remote control tools for vehicle systems, etc. However, the most interesting thing is that they have entrusted hydraulic elements to the control of code written in Atom DSL, which is also implemented in Haskell. Atom is used to develop hard real-time systems and allows describing declaratively the system state transition rules. During compilation, the tasks are scheduled, which is why the resulting code has a deterministic execution time and constant memory consumption. This makes verification of the obtained code much easier and generally increases the system security, which is, of course, very important in this subject domain. You can read about all this in more detail on the slides and in the Atom repository.
Now let’s jump to an entirely different sphere, that is, the world of music. The service called Chordify allows converting music from YouTube, SoundCloud, etc. into chords so that you could learn to play your favorite songs. Haskell is mostly used during the tracing process; to this end, developers use the HarmTrace library that analyses the sequences of musical harmonics.

Kaspersky Lab is developing its own secure operating system KasperskyOS for IoT systems. Surely, this OS sets forth specific requirements for reliability and cyber security. The KasperskyOS team is actively using Haskell in the security system compiler and to create auxiliary development tools. The security configuration is described in a special DSL which is then compiled to C. As you can see, Haskell again demonstrates its usefulness when it comes to security.
Some of in-house tools for automotive development in Tesla are written in Haskell, which can be seen from the regularly published vacancies. As far as I could understand from the vacancy descriptions, Haskell is used for the intermediate high-level representation of internal systems which is used to generate C code, documentation, and probably something more.

BIOCAD is one of the major biotech companies based in Russia. The company carries out the complete cycle of medicine development all the way through to mass production and marketing. In the course of medicine development BIOCAD develops its own internal services for data processing and storage. The backend of these services is written in Haskell; it uses the graph database Neo4j, for which the company has developed and published the driver called Hasbolt.
Haskell has filled its niche in the development of blockchains and cryptocurrency. It seems to me, the majority of Haskell-related vacancies I’ve ever seen were in this field. Cardano should be mentioned as one of the most famous. The main cause of Haskell’s popularity in this sphere is security again. For more details see the post by FPComplete.
If you find blockchains to be too dull, serious or "inhuman", I can tell you about Co-Star, an astrological application for personalized horoscopes. Their backend is written in Haskell, and the developers explained their choice of language on the application website.

The backend of Wire messenger, with the server code publicly available in the company’s repository and detailed documentation, also written in Haskell.
The American Museum of Natural History uses Haskell to study the phylogenetic graphs reflecting the evolutionary relationships between different species. The code of this project is also available on GitHub.

Conclusion

The list of examples for this post was compiled to demonstrate the variety of applications and, surely, many of them have remained offscreen. Here you can find more details on the companies and industries using Haskell:

Here at Typeable we are using Haskell to develop projects for health care, tourism, fintech and many other sectors.

Legacy: alarming symptoms and problems

Catherine Galkina — Mon, 29 Nov 2021 14:21:08 +0000

Author: Victoriia Zaripova

Introduction

The purpose of this post is to give non-IT specialists a basic insight into the hazards of obsolete computer systems and code, and the way they may harm the business.

You might have heard the word "legacy" – usually with a negative connotation – from the technical staff. This term is used to denote the methods, technologies, and computer systems or application software that are declared to be obsolete due to some reason. However, does this legacy always have negative implications for the business, is it mandatory to get rid of it and how can you understand that it really creates problems?

As a rule, the developers, analysts, testers, and support staff are those who face the legacy issues most often, which is evidenced by the broad experience of our own team. For end-users, these issues usually remain under the bonnet, and for CEOs they are concealed behind report figures without showing any logic behind them.

Basically, the situation, on the whole, reminds of the permanent tuning of outside appearance and interior in a car where the engine is at its last gasp but the passengers suspect nothing. Sooner or later, this automotive marvel is bound to die somewhere between point A and point B, but until that time the car will make a positive impression, especially if the passenger compartment is extremely comfortable and the suspension – or, speaking straight out, the frontline – allows driving over potholes almost smoothly.

However, there is also no point in replacing your tried and tested tractor with a trendy smart right on the spot. Everything depends on the objectives and current state of the issue

How can you understand that legacy is a hindrance?

Here is the list of alarming symptoms and issues you can notice even without being a professional:

Lack of well-defined and clear documentation for the supported business processes

Perhaps, now the documentation seems to be of no importance since each of your employees is a real pro who can explain the entire business process from A to Z at any moment and knows the information system they are using inside out. However, the real-life experience usually shows that this is far from being the case. Moreover, even the terms used by the personnel in different departments differ significantly. The most basic example is the attempt to build the business process for supply agreements approval. Lawyers treat the term “agreement” quite differently from accountants, and both departments focus on entirely different stages of the agreement approval.

Today, businesses are more willing to recognize the value of transparent and well-organized business processes and you’ll have little difficulty in finding a post on the advantages of this approach. Here is an example: Benefits of BPM | 11 Massive Advantages of Business Process Management. Besides, this article is also notable for the studies it’s based on, in particular, the study of 2020 stating that the outbreak of COVID made businesses invest more in the business process automation and altered their assessment of how important it is to change the obsolete processes.

Rigidity in improvements

Due to some reasons you are not able to integrate easily and quickly with the leading services, timely scale up or flexibly modify the software and, consequently, always miss the boat of market opportunities.

Just believe, this is not a chain of tragic accidents, but a disturbing symptom.

This symptom is especially clearly seen in governmental institutions where rigidity is embedded in the work format, and where all changes require long-term approval. For instance, excess expenses of the US government for the IT infrastructure in 2019 (over 70 billion dollars) triggered the internal investigation showing that obsolete software is the key issue.

Nevertheless, commercial companies whose way of operation implies flexibility and openness to new technologies also suffer from the legacy. Airline companies, banks, insurance companies, and retailers sooner or later turn out to be unable to overcome the rigidity of obsolete software. As a result, their operation becomes unstable, they are not able to bring new products and services to market or even stably support the existing ones. For example, in August 2016 Delta Air Lines faced an unexpected failure of the booking, check-in and boarding systems which always seemed to be reliable, which resulted in an hours-long downtime and cancellation of more than 2000 flights. This and other cases are described in the post "Legacy systems are problems for boardrooms not computer geeks", Financial Times, Jan. 31, 2017.

Low spirits in the IT team and loss of users’ loyalty

One would think that the legacy cannot affect the team morale.

There is no secret, as any modification of legacy systems resembles a patch on the A-bomb – most probably it won’t go off but you wouldn’t like to touch it again if you can help it. Specialists in obsolete languages and technologies don’t come cheap – one has only to think about the Y2K problem. Novices, on the other hand, try to patch the old environment with something new, so with time, all this becomes a monstrous zoo of technologies where not every specialist would dare to plunge. As the result, the users complain more and more about the problems and the product development is hindered.

Moreover, the personnel are more likely to leave as they’ll be demotivated and not willing to dig through the problems of obsolete monstrous code with documentation missing or passed over as verbal folklore.

Of course, no one is indispensable, but you should remember that selection and adaptation of a new staff member come at a price and the more complex and intricate the project is, the more time it will take the new employee to become up to speed (while being paid for work with the old code!). Meanwhile, the rest of the team will have to solve business-critical problems.

And now imagine that dramatic market changes (and they’ve always been dramatic recently) require an immediate response. It’s not necessary to imagine for yourself, you can just read what situation the businesses are facing today: The coronavirus (COVID-19): business impact.

Standard solutions

Thus, we’ve found out that legacy is not just a problem for the IT department but also a potential threat to the company stability and growth. Let’s consider standard solutions for this situation:

Ignore – this path can be costly if you already see the problems listed above but it can be optimal in some cases. In the long run, if the business pathway is unchangeable, it’s better to rely on the trusted comrade-in-arms with whom you went through fire and water;
Replace it with completely new software – sometimes this path is the only possible solution because the legacy, like a disease, can infect the environment very quickly.
Replace gradually/component-wise – this way is usually the revolution from below, when the IT department systematically performs refactoring of the home system, but it usually requires almost boundless tenacity and consistency;
Reengineering of business processes and the software – this path is perhaps the most radical, practical, and effective but it’s labour-intensive and needs the deliberate commitment of the management.

Conclusion

As you can see, legacy can do harm to the business, not only to the programmers’ nerves. You can deal with the legacy in different ways, and different ways will work for specific cases.
In our next post, we’ll describe in more detail when and what way should be selected and will also tell how technical tools can help you to work with the legacy.

How Does Functional Programming Contribute to Modern Languages?

Catherine Galkina — Mon, 15 Nov 2021 14:30:19 +0000

Author: Vladislav Lagunov

Modern programming languages have a large set of various tools and useful features, which makes it possible to write quite a different code in the same language for the same task.
Programming paradigm is a way of thinking in the first place – the way the programmer thinks about the data representation and processing. In other words, programming paradigm exists in the programmer’s mind and is not a part of the language. Different languages can support a specific paradigm to a varying degree. If we open Wikipedia and start reading about the most popular programming languages, we’ll see that many of them are described as “multi-paradigm”: they allow programming in different styles though some of them will be more convenient to use.

I ♥ FP by https://impurepics.com/

In our recent post, we spoke about the practical applications of Lisp and mentioned, without going into details, that it strongly influenced the development of other programming languages. Now it’s high time to explore this topic in more detail and figure out what contribution functional programming in general (not just Lisp!) has made to the development of other languages. Since we use Haskell as the main programming language, we couldn’t help touching upon this topic.

In this post, we’re going to consider several mechanisms which either emerged in the functional programming languages or were most widely used and popularized by these languages, eventually turning up in originally non-functional languages.

First-class functions

The distinctive feature of FP style, in general, is the wide use of functions which become one of the main development tools. Let’s quickly look through the main definitions describing the differences between the functions and procedures and other similar constructs of non-functional languages.

Higher-order function is the function that either takes another function as an argument or returns some function as the result. They are also called functionals. This behaviour can be implemented even in pure C using function pointers:

void update_user_balance(int user_id,
                         double (*update_fn)(double))
{
  // ...
  user->balance = update_fn(user->balance);
  // ...
}

First-class functions are the ones which you can manipulate in the same way as any other values: pass as arguments, return as results, assign to variables and structure fields.

Lambda function is the anonymous function. Besides the absence of a name, support of lambda functions removes other language restrictions on function declarations (in some languages, i.e. in C99 standard, function declarations can occur only at the higher level). Support of lambda functions implies that the function can be declared in any location where other expressions are valid. Lambda functions are mostly used in higher-order functions; when used in combination, they provide convenience and significant shortening of the code.

// Example of a lambda function used to print out
// the contents of std::vector
int main() {
  std::vector<int> v;
  v.push_back(1);
  v.push_back(2);
  std::for_each(v.begin(), v.end(), [] (int x) { 
    std::cout << x << '\n'; 
  });
}

Closure is the function that can capture some variables from the context it was declared in, without letting the garbage collector erase the data which can be used in this function for as long as the application has the reference to the function itself. An example in TypeScript:

function createClosures() {
  // The variable can be seen in the functions below
  let counter = 0;
  // The values of fields inc and print are closures
  return {
    inc: () => { counter++; },
    print: () => console.log('counter value: ' + counter),
  };
}

const c = createClosures();
c.inc();
c.inc();
c.print(); // >>> "counter value: 2"

List comprehension

List comprehension allows writing down concisely the processing or generation of lists using existing ones. Miranda was one of the first languages to use such syntax which was adopted by Haskell; later on, similar constructs appeared in “less functional” languages such as Python, C#, Ruby.

As an example, let's consider the code in Python and Haskell that creates word combinations using adjectives and nouns. These two fragments look very much alike and only have minor syntactical differences, don’t they?

# Example in Python
nouns = ["waterfall", "moon", "silence"]
adjectives = ["late", "divine", "blue"]
phrases = [a + " " + n for n in nouns for a in adjectives]
# >>> ['late waterfall', 'divine waterfall',
#   'blue waterfall', 'late moon', 'divine moon',
#   'blue moon', 'late silence', 'divine silence',
#   'blue silence']

-- The same in Haskell
nouns = ["waterfall", "moon", "silence"]
adjectives = ["late", "divine", "blue"]
phrases = [a ++ " " ++ n | n <- nouns, a <- adjectives]
-- >>> ['late waterfall', 'divine waterfall',
--  'blue waterfall', 'late moon', 'divine moon',
--  'blue moon', 'late silence', 'divine silence',
--  'blue silence']

Algebraic data types

These types can also be called ADTs, sum types, discriminatory unions, disjunctive unions, coproducts, and, probably, some other clever words. Maybe you have the idea of such types under different names but, in a few words, this is a compound type containing the discriminant field (which can be called a tag) together with the associated data. The following is the Haskell example of such compound type describing the user’s possible actions in a hypothetical application implementation TodoMVC. Some actions involve the “payload” (a string or an element ID).

data UserAction
  = TextInput String
  | EnterPressed
  | EscPressed
  | CheckTodoItem ItemId Bool
  | EditTodoItem ItemId String
  | DeleteTodoItem ItemId
  | ApplyFilter TodoFilter

Despite its simplicity and usefulness for modelling domain objects, ADTs are rarely supported on a full scale in popular languages and databases. Here are some examples implementing similar types: Enum in Rust, Sealed Classes in Kotlin, std::variant in C++

Pattern Matching

Pattern Matching is a syntactic construct giving access to the data of the structure consisting of one or several alternatives with different sets of fields (the very ADT, algebraic type sum, enum, std::variant, etc. we discussed earlier). Pattern Matching resembles the switch-case operator you all know from imperative languages. However, its main advantage is that the compiler checks the access to alternative fields statically by using the information about the expression type, while the switch-case doesn’t prevent errors related to incorrect access to the fields, missing cases or redundant checks.

Pattern Matching is another technique popularized in functional languages where it turned out to be useful. Now it is adopted and integrated – in various forms – in Python, Java, C#, Rust, and other popular languages.

-- Example of state update function in a hypothetical
-- TodoMVC written in the Elm architecture style.
-- Pattern Matching is used to analyze the
-- user-generated event.
updateTodoList :: UserAction -> TodoState -> TodoState
updateTodoList action oldState = case action of
  TextInput newTitle -> oldState {title = newTitle}
  EnterPressed -> appendNewItem oldState
  EscPressed -> stopEditing oldState
  CheckTodoItem itemId itemChecked ->
    updateItemById itemId (#checked .~ itemChecked)
  EditTodoItem itemId itemTitle ->
    updateItemById itemId (#title .~ itemTitle)
  DeleteTodoItem itemId -> deleteItembyId itemId oldState
  ApplyFilter newFilter -> oldState {filter = newFilter}

Lazy evaluations

In most programming languages evaluation is performed when the value is assigned to the variable; all arguments are evaluated before the function call (strict evaluations). The alternative approach is “lazy” implying that the evaluation is postponed until the value is actually used. Lazy evaluation allows working with infinite data structures, writing declarative code with definitions arranged in the order convenient for reading instead of the order of their evaluation. If you use the DSL approach, lazy evaluations will help you to easily implement such constructs as if-then-else (the values will be evaluated only in the branch you need).

The history of the term can be traced back to lambda calculus, forming part of the theoretical basics of functional programming, so no wonder that it’s used in FP languages. For instance, in Haskell everything is evaluated lazily by default.

Some elements of “laziness” can be found in other languages, even in pure C the operators && and || are lazy as they don’t evaluate their second argument if the first one has been evaluated as 0 or 1, correspondingly. In higher-level languages the more common term is “deferred evaluations” which are implemented using generator functions and the “yield” keyword. Such generators can be found, for example, in Python or Java.

Continuations

Continuation is the “remaining evaluation”, i.e. the description of what is to be done with the result of expression for each subexpression in the program. An expression gets the continuation in the form of an additional argument and when the result is obtained, the current function calls the passed continuation with the evaluated value instead of returning the result directly. Such style of passing the result is called Continuation-Passing Style (CPS).

// Direct style of passing the result 
function getFoo(): Foo {..}

// CPS style
function getFooCPS<A>(cont: (foo: Foo) => A): А {..}

CPS is rarely found immediately in the software source code. Compilers are one of the main areas of its application – as the intermediate format before generating the machine code. Code conversion to CPS allows transforming recursive function calls to tail recursion that can be easily optimized to avoid stack growth during evaluations.

Continuations are a very powerful tool that can be used to implement control structures such as early function exit, explicit call for tail recursion, imperative cycles, etc. For more details on the use of continuations see the example for the Scheme language here.

Futures and promises

Futures, Promises, or Deferred, are a construct containing the evaluation of asynchronous value. They emerged in functional programming as a tool used to simplify parallel evaluations and execute queries in distributed systems.

const getJson = url => fetch(url).then(resp =>
    resp.json());

// Sending 2 serial queries
const getUserInfo = getJson('/current-user-id').then(
  userId => getJson(`/user-info/${userId}`).then(
    userInfo => console.log(
      `Hi ${userInfo.firstName}, your id is ${userId}`)
  )
);

Promises were popularized mainly because of their adaptation and extensive use in the browser. JavaScript execution in the browser is limited by only one execution thread, so blocking while waiting for an HTTP response, as is the case with most platforms, would have led to the page hangup and annoy users. This is why callback functions are used to process responses to HTTP queries in the browser. At the same time, it’s not very convenient to combine such requests and the term “Callback Hell” came up to describe the code that has become illegible due to a large number of callbacks. Promises have allowed partially solving the issue of sending parallel queries and serial query chains:

// Sending 3 parallel queries
const fetchInParralel = Promise.all([
  getJson('/current-user-info'),
  getJson('/shopping-cart'),
  getJson('/recently-viewed-items'),
  ]).then(([userInfo, cart, viewedItems]) => {
    // Display the page using the data sent from the server
    // ...
  })

In many popular languages (such as C#, Java, JavaScript), promises have become the main tool for asynchronous programming.

Monadic interface

The names of many constructs and programming techniques in Haskell were borrowed from category theory and other branches of mathematics. One of such terms – the Monad – has become the object of many memes and jokes about functional programming. There are lots of posts on the Web explaining what “Monads” are in functional languages and how to use them.

To put it simply, “monad” is just an interface with two methods that allow combining evaluations into a chain in the same way it’s done in the promise chain example. The promises themselves also illustrate the monadic interface implementation.

// Example of a monad used to generate pseudo-random values;
// parameter А is the type of generated value
class Random<A> {
  // Creation of Random using a random valueе
  static of<A>(value: A): Random<A> {...}
  // Method used to implement the call chain
  chain<A, B>(this: Random<A>, then: (a: A) => Random<B>):
    Random<B> {...}
}

declare function randomNumber(min: number, max: number):
    Random<number>;
declare const randomString: Random<string>;

// Example of using the monad chain
const randomUser: Random<User> = randomString.chain(
  userName => randomNumber(12, 90).chain(
    userAge => Random.of({ name: userName, age: userAge })
  )
);

Among other applications of monads in purely functional languages, such as Haskell, is the side effect encapsulation. As it’s not possible to make a database query, read a file or even print a line in the standard output in these languages by calling ordinary functions, monads are used to perform these actions. At the same time, their application goes beyond side effects – the monadic interface is universal and allows writing generic, laconic, high-level code, which is why the monads are used everywhere in Haskell. Beyond Haskell, monads themselves are not so widely used, but their influence can be seen primarily in programming using promises and in the async-await construct, which we’ll talk about below.

Async

Getting back to the examples of code with promises, you can notice that despite all advantages of the promises, the call chain looks no better than callbacks. The async-await syntactic structure allows taking it a step further and improving the code with the promise chain, turning it into something much like the code with blocking calls.

const getJson = url => fetch(url).then(resp => resp.json());

// Sending 2 serial queries
async function getUserInfo() {
  const userId = await getJson('/current-user-id');
  const userInfo = await getJson(`/user-info/${userId}`);
  console.log(`Hi ${userInfo.firstName}, your id is ${userId}`);
};

The emergence of async-await can be traced back to the researches devoted to functional programming in Haskell and ML which gave rise to the async workflows in F# (2007) and then C# (2011).

Conclusions

Programming languages move on and develop actively, accumulating new, ever more advanced and convenient tools. As you can see, such popular languages as Python or С++ are acquiring a growing number of features, originating from functional programming. More recent languages, such as Scala or Kotlin, support functional tools from the very beginning.

It turns out that functional programming is much closer than might appear at first sight even if you develop in C++ or Java!

We’d be happy to learn about your experience with these or any other functional tricks in your day-to-day development.

Alternative Approach to Dynamic Environments: Using Octopod

Catherine Galkina — Mon, 01 Nov 2021 15:30:57 +0000

Author: Sergey Shurlakov

In this post, I’d like to tell you how we use dynamic environments (review or preview environments) in our work here at Typeable, what issues we’ve managed to solve, and how and why we use our Octopod solution for these purposes instead of GitLab Dynamic Environments. In case you don’t know what the dynamic environment is, I recommend reading the post by Flant where the author gives a detailed account of the types of dynamic environments, their purpose and applications. The author also looks into this topic using GitLab as an example and provides detailed cases and descriptions. As for us, we use an alternative approach, somewhat different in terms of ideology, and work with review environments in Octopod. Previously we related the history of Octopod creation and the causes that motivated us to create it. We won’t repeat ourselves but will focus on the differences of our approach and the issues we’ve fixed.

To begin with, I need to tell you that Octopod is a universal tool not tied to any specific package management method in Kubernetes. Nevertheless, as the de facto standard in the world of Kubernetes, it’s primarily meant to simplify the deployment of Helm charts. At Typeable, we use Helm, so starting from version 1.4 the standard Octopod pack already includes all you need to work with this utility.

Main differences from GitLab

Probably the most important thing is that we started working with review environments before GitLab had an interface for dynamic environments. However, there are some other reasons, including the ideological difference of our approach to the implementation and use of review environments. But let’s start from the beginning.

We don’t store the environment configuration in the code. Why? We intentionally untie the environment from the code for several reasons.
a. Not all team members have access to the code. Our processes are shaped in such a way that analysts, testers, project managers, and other team members who don’t write the code usually have read-only access to the repository or even no access at all. Such an approach allows us to better control the code and reduces the number of risks and potential issues. However, they need to be able to create review environments independently without involving a DevOps engineer or developer.
b. In GitLab, in order to manage the dynamic environment, you need to create the file .gitlab-ci.yml first and then modify it, as appropriate. Thus, each code branch will have its own file with the required environment settings. The risk exists that the settings might “leak” to the main branch and the new environments will have invalid parameters. This has to be fixed somehow, which only increases the gitflow complexity by potentially increasing the number of conflicts and, consequently, the number of merges or rebases.
c. We often have to make changes in the review environment configuration. In Octopod, it’s easy to change the environment parameters and variables and, most importantly, to switch between various endpoints in the services we are integrated with. We have lots of integrations with external systems and it’s not always possible to test the application functionality by connecting to the test API. It’s often necessary to interact with the production API.
So the settings of the review environment are stored in Octopod and managed through the Web interface or Octopod’s CLI (octo CLI) and are fully isolated from the code.
In the GitLab environment, variables are global for the project. A lot of review environment parameters are presented by the environment variables. GitLab provides an interface used to set the environment variables without making changes in .gitlab-ci.yml, but these variables have the global scope, i.e. they apply to all dynamic environments of the project. It’s only possible to write the environment variables for a specific review environment by inserting them in .gitlab-ci.yml in the appropriate branch, which contradicts item 1.

In Octopod we’ve solved this problem using the Application Configuration and Deployment Configuration. Octopod generates the list of key-value parameters that allows viewing and managing the available chart settings in UI. You only need to select the required key and write the value. It’s also possible to provide custom keys that are missing from the list. We’ve provided two types of configuration settings: Application Configuration and Deployment Configuration. The Application Configuration is the configuration (Helm values) that is passed to the application. For example, this can be a database connection string or environment variables. The Deployment Configuration is used only during the environment creation phase. The key values are passed to Helm and allow redefining the default values thus directly influencing the deployment process. Here a chart version or URL of the Helm repository can be used as an example.

This is what the staging configuration looks like in Octopod:
Resource constraints. There can be a lot of review environments, as well as the resources they consume. In the long run, you run out of resources and have to solve this issue somehow. This is why we archive the environment when the Jira ticket moves to the column Done (we’ve cross-integrated Jira, Octopod and GitHub). Archiving can also be done manually. Archiving implies the “scale-to-zero” approach where Pods are released while all other resources (e.g. Persistent Volumes) are saved. This allows extracting the environment from the archive as necessary and restoring its operating condition very quickly. The archived environments which are more than 14 days old are deleted completely, i.e. a cleanup is carried out involving the deletion of absolutely all resources, including PVC, certificates etc. We set up the automation of this process using octo CLI, a utility in the Octopod command line that has an extended functionality as compared with the Web interface. This approach can also be implemented in GitLab but the task is not trivial and requires complex logic in .gitlab-ci.yml.
The state of the review environment has to be tracked accurately. One of Helm’s disadvantages is that it’s not possible to track the state of the environment after it’s been deployed. After Helm has finished its work, you have to check whether the environment keeps functioning all by yourself. During the work, something may go wrong. The review environment is not working already but we don’t know about this until we need it. The guys of Flant have fixed this issue using the kubedog which is built into the werf. Octopod keeps track of the environment state in a somewhat different way. We write all control scripts for Octopod in Rust and use the kube-rs client to check the statuses of all ReplicaSets. The check is carried out every 5 seconds, which is why the status of the review environment condition in Octopod is always up-to-date.
Writing a .gitlab-ci.yml script can be a challenge. In Octopod we’ve solved this issue by providing ready-made Helm scripts that allow deploying any valid Helm chart. In many cases, this makes it absolutely unnecessary to write any scripts and minimizes the involvement of a DevOps engineer. At the end of the post, we’ll move to a hands-on exercise and deploy an instance of WordPress in Octopod using the Helm chart by bitnami.
The need to create complex interdependent environments. If you need to create an environment that depends on infrastructure services such as PostgreSQL or Redis, these services can be deployed as separate review environments using their Helm charts while their connection parameters can be passed through the Application Overrides. In this way, it’s possible to use, for example, one instance of PostgreSQL or one authentication service for several review environments.
A closer and more reliable integration with Kubernetes cluster is required. Octopod works in the same Kubernetes cluster where it deploys all review environments. The issue of cluster access and transmission of Secrets has been solved fundamentally; Octopod works through the Service Account. In GitLab you can configure seamless integration with the Kubernetes cluster, which solves the problem of secrets transmission. However, if the integration doesn’t work due to any reason, there arise some difficulties.

Practice. Installing Octopod

There are two main ways to install Octopod:

For industrial application. Octopod is installed in the Kubernetes cluster using the official Helm chart.
For individual use and to get acquainted with the main features of Octopod, installation is carried out locally and it’s fully automated.

Now we’re going to install Octopod locally. Installation is currently supported for Linux and MacOS. Installation in Windows requires Windows Subsystem for Linux 2 (Docker Desktop is installed in Windows and integrated into WSL 2; other components are installed in WSL based on the Linux instructions). First of all, you’ll need to install the following infrastructure components:

Then we run the script that will install Octopod:

/bin/bash -c "$(curl -fsSL
    https://raw.githubusercontent.com/typeable/octopod/master/octopod_local_install.sh)"

An important point about Octopod installation in Windows. WSL 2 may have issues with SSL connection to the resources due to the settings of antivirus software functioning as the firewal. The issue manifests itself as time out when you try to download data from repositories.

When the installation is completed, we open the browser and type in http://octopod.lvh.me in the address bar and see

Perhaps, it should be noted here that lhv.me is a simple service returning the IP-address of local host 127.0.0.1 to any request. It’s convenient to use it as you don’t have to make changes in /etc/hosts every time.
Now we need to create a new deployment. Octopod is delivered with predefined parameters for bitnami Helm charts and WordPress is used as an example.
Click on the NEW DEPLOYMENT button. A window will appear on the screen where you need to set the environment parameters. In the simplest case, it’s enough to type in the name. For example, wordpress. However, we’ll add two more App overrides (to this end click on ADD AN OVERRIDE):

key	value
wordpressUsername	admin
wordpressPassword	P@ssw0rd

We add these two variables only for the period of the review environment creation when the variables will be used to initialize WordPress. After that, you can delete them. It goes without saying that here we show the user name and the password explicitly and in plain text only for the sake of simplification and for demonstration purposes in the local Octopod version. In real life, we use appropriate tools such as Hashicorp Vault.

Click on the SAVE button.
After some time, the review environment will be created.

With that, the creation of a review environment is completed. Now you can open WordPress using the link provided in the Links column. To open the admin panel, add /admin to the URL (http://wordpress.lvh.me/admin). Type in the user name and password provided during the environment creation.

If everything has gone well, you can delete the variables wordpressUsername and wordpressPassword and change the password, as necessary, in the WordPress admin panel.

Conclusion

In our opinion, not only the developers and testers but also other people with limited technical skills can make use of review environments. This is why we’ve done our best to make the entry barrier for Octopod as low as possible, create a user-friendly and clear interface and ensure intuitive and predictable user interaction. These are the reasons behind the path we’ve chosen to develop Octopod and solve the problems described in this post. Surely, Octopod differs greatly from GitLab Dynamic Environments in terms of ideology and uses different approaches to solving similar issues. However, it performs its mission without a hitch – DevOps engineers are now free from the mundane tasks of creating and maintaining review environments and it’s well worth it.

In our turn, we’d be happy to get feedback from Octopod users. We are looking forward to your suggestions, wishes and PRs on GitHub!

7 Useful Tools Written in Haskell

Catherine Galkina — Mon, 18 Oct 2021 14:18:22 +0000

Author: Nikolay Rulev

Introduction

Haskell is a pure functional general-purpose programming language with static typing and lazy evaluation. It was created in 1990 and was named after Haskell Curry. At present, the GHC compiler is its main implementation.

Here at Typeable we like this language very much and use it as our main language for software development. Though Haskell is already quite widely used in the industry, it’s still sometimes believed to be an "academical" language used exclusively for scientific purposes. In this post, I’d like to tell you about several rather popular Haskell tools which may be of use to you.

Useful tools written in Haskell

1. Xmonad

Xmonad is a tile window manager for X Window System, written in Haskell. The distinctive feature of this tile manager is that it’s configured in Haskell, too, which allows us to use all features of the language. Thus, while configuring the window manager, you can add weather forecast data, measure the system condition, etc.

Here is an example of configured system (by erikbackman):

A config example (only a part is shown, not the entire config):

main :: IO ()
main = xmonad . dynamicProjects (projects apps) . ewmh . docks . cfg =<< polybar
  where
    cfg dbus =
      def
        { manageHook = App.manageApps <+> manageSpawn,
          logHook = dynamicLogWithPP (polybarHook dbus),
          startupHook = myStartupHook,
          terminal = fromMaybe "xterm" $ App.terminal apps,
          modMask = mod4Mask,
          borderWidth = 1,
          keys = keybinds . KeybindConfig apps,
          handleEventHook = handleEventHook def <+>
            fullscreenEventHook,
          layoutHook = myLayouts,
          focusedBorderColor = "#bd93f9",
          normalBorderColor = "#434C5E",
          workspaces = myWS
        }

    apps =
      defaultAppConfig
        { App.terminal = Just "kitty",
          App.launcher = Just "rofi -matching fuzzy -show drun -modi drun,run -show-icons",
          App.mailClient = Just "claws-mail"
        }

    myStartupHook :: X ()
    myStartupHook = do
      setDefaultCursor xC_left_ptr

You can find further information on the xmonad website.

2. ShellCheck

ShellCheck is a utility used to analyze bash scripts. It allows novices to fish out common errors when they just start learning this scripting language and helps more advanced users not to miss small bugs in large and complex scripts. The project is being constantly expanded and its large community is adding new rules. You can view the ShellCheck and try it out at its home page.

Let’s use the following script by way of example (try to find the errors in it without looking into the ShellCheck result):

#!/bin/bash
## Example: ShellCheck can detect some
## higher level semantic problems

while getopts "nf:" param
do
    case "$param" in
        f) file="$OPTARG" ;;
        v) set -x ;;
    esac
done

case "$file" in
    *.gz) gzip -d "$file" ;;
    *.zip) unzip "$file" ;;
    *.tar.gz) tar xzf "$file" ;;
    *) echo "Unknown filetype" ;;
esac

if [[ "$$(uname)" == "Linux" ]]
then
    echo "Using Linux"
fi

It’s very easy to check the file: shellcheck example.bash.

As the result, we’ll be informed about all potential errors in the script:

3. PureScript

Haskell was used to develop several popular compiled functional languages for the frontend development which are eventually transpiled into JavaScript. Today we’re going to talk about PureScript. This language has a strict type system supporting algebraic data types, HKT (Higher-Kinded Types), and type classes with functional dependencies. It also supports several necessary and handy features such as FFI (Foreign Function Interface) with JS, and the resulting JS looks very much readable. The language has already acquired its own ecosystem: the package manager spago and IDE based on its own language server.

Below you can find the example of a simple button component written in Halogen:

module Example.Basic.Button (component) where

import Prelude

import Halogen as H
import Halogen.HTML as HH
import Halogen.HTML.Events as HE
import Halogen.HTML.Properties as HP

type State = { enabled :: Boolean }

data Action = Toggle

component :: forall q i o m. H.Component q i o m
component =
H.mkComponent
    { initialState
    , render
    , eval: H.mkEval $ H.defaultEval
        { handleAction = handleAction }
    }

initialState :: forall i. i -> State
initialState _ = { enabled: false }

render :: forall m. State -> H.ComponentHTML Action () m
render state =
let
    label = if state.enabled then "On" else "Off"
in
    HH.button
    [ HP.title label
    , HE.onClick \_ -> Toggle
    ]
    [ HH.text label ]

handleAction :: forall o m. Action
  -> H.HalogenM State Action () o m Unit
handleAction = case _ of
Toggle ->
    H.modify_ \st -> st { enabled = not st.enabled }

For further study, you can proceed to the official website.

4. Agda

Haskell works really well when it comes to developing programming languages! Agda is another example. This is a language with a strong theoretical background and dependent types whose syntax looks much like that of Haskell. Moreover, this is also an automated theorem proving system: logical assertions are expressed as types, and successful type checking is the proof of program correctness.

What's this all for? First of all, this allows avoiding a vast number of run-time errors and to some extent prevents the programmer from writing incorrect code, as utopian as it may sound. Let’s consider a classic example: we’d like to rewrite the head function that takes the list head but we want to make sure that we don’t pass an empty list to this function anywhere in our program.


-- Natural numbers in the form of
-- Peano numbers and their addition
data Nat : Set where
  zero : Nat
  suc  : Nat -> Nat

_+_ : Nat -> Nat -> Nat
 zero  + m = m
 suc n + m = suc (n + m)

-- Vector, dependent type whose length
-- is stored in the type
data Vec (A : Set) : Nat -> Set where
  []   : Vec A zero
  _::_ : {n : Nat} -> A -> Vec A n -> Vec A (suc n)

-- head that cannot be applied to an empty vector
head : {A : Set}{n : Nat} -> Vec A (suc n) -> A
head (x :: xs) = x

5. PostgREST

PostgREST is an independent web server that will turn your database schema directly into RESTful API. This approach saves you the trouble of writing the boilerplate for CRUD and allows you to focus on the important details of business logic. The advantages include the use of PostgreSQL binary protocol and serialization of JSON directly to SQL to improve performance.

Let’s take a simple city and country scheme as an example:

As the result, we get the following API for this scheme:

Documentation is here.

6. Octopod

Octopod is a service we developed in Typeable that allows our team to use the entire Helm’s potential without the need for technical knowledge. Our solution makes it easier to manage dynamic environments for the whole team: DevOps, developers, testers, and project managers. All you need is to indicate the URL to the Helm repository and Octopod will automatically get the required configurations.

We wrote about this service earlier, and you can find more information on Octopod home page.

7. Pandoc

Pandoc is a multitool that allows converting data in one text format into a totally different format. The utility understands a lot of useful syntax extensions, including the document metadata: author, heading, dates, footnotes, tables, highlighting of code snippets, LaTex inserts with formulas, etc. Supported document formats include Markdown, epub, docs, RTF, ODT, HTML, LaTeX, various formats of wiki markdown and bibliography.

You can download the tool and view all supported formats here.

I didn’t list all supported conversion options as it would’ve taken all the post. Just look at the graph showing what and how can be converted:
a big picture!

Conclusion

Here I’ve listed the open-source projects which I believe to be the most interesting and useful for a wide circle of users. This list can be used as an answer to the question “Is there anything useful written in Haskell beyond the science?”

Surely, the list is not exhaustive and can be extended. If you think that it should include a certain project or just want to share a cool Haskell tool – let us know and leave a comment!

Do You Know Where Lisp Is Used Nowadays?

Catherine Galkina — Mon, 04 Oct 2021 14:22:05 +0000

Examples of software from different domains written in Lisp dialects

Introduction

Lisp is the second-oldest high-level programming language (after Fortran) and the first functional language. It was developed in 1958 and has changed since that time giving rise to lots of dialects and producing a significant effect on the development of other languages. At present, the best-known dialects are Common Lisp, Scheme, Racket and Clojure.

Left: A LISP machine at the MIT Museum. Right: Symbolics 3640 lisp machine, photo by Michael L. Umbricht and Carl R. Friend (Retro-Computing Society of RI)

Lisp became the "pathfinder" for many ideas which found application in the modern programming languages: tree-like structures, dynamic typing, higher-order functions and many others. In this post, we won’t dwell on Lisp’s contribution to theoretical computer science but will focus on the practical benefit.

Originally, Lisp was designed for artificial intelligence research, in particular, as a presentation of the mathematical notation for symbolic computations. But how common are the Lisp dialects nowadays and where are they applied?

Where and how is Lisp used?

During my university years, I used to deal with Lisp dialects quite often. While I was doing some initial research for this post, I was surprised and happy to find mentions of the code written in some Lisp dialect in the applications I’m using myself. I believe you’ll also find familiar names in this list.

GNU Emacs is the text editor developed by Richard Stallman in 1984, it’s the first program of the GNU project and a participant in the everlasting fight for “The Best Text Editor” title. It’s written mostly in its own Lisp dialect, Emacs Lisp, which is also used to write Emacs configs and extensions. Though this dialect can be used as a general-purpose scripting language, it is still tailored to the development of a text editor. For instance, Emacs Lisp has a rich library to work with text files.
Grammarly is an online service used to process texts in English. The service makes use of artificial intelligence to analyze the text and generate recommendations on its improvement. In addition to the grammar and spelling, the text brevity, vocabulary and tone are checked. The development started in 2009; today the service has 30 million active users and is regularly listed in various rankings. Personally, I use Grammarly as a browser extension and I was surprised to find that all their backend, all the text processing mechanism is written in Common Lisp. In their technical blog, the developers explained why they gave preference to CL and what it’s like to use Lisp in production.
Boeing 747 and 777 use Allegro NFS Server written in Common Lisp. Further on the subject of aviation: Boeing and Airbus use Piano – a software package in Common Lisp for aircraft design development and analysis. You can learn more about the low-level programming in Common Lisp from this talk.
ITA Software is the once independent company that developed the system for air tickets search and pricing. Now it forms part of Google’s travel industry department. In most cases, the company develops its software using Allegro Common Lisp.
Another application in the transport sector is the London Tube that uses the software developed by Portuguese Siscog where Common Lisp is the main programming language. The company develops decision support systems designed for resource planning, allocation, and management in the transportation sector.
Circle CI is one of the largest and, probably, most well-known platforms for CI/CD with millions of builds every month and other impressive statistics that can be found on their website. The major part is written in Clojure; ClosureScript is used for frontend development. In general, developers are so fond of Clojure that in 2021 it ranked second among the most loved technologies of Stack Overflow users according to their annual survey.
Computer-aided design systems AutoCAD use the AutoLISP dialect as the internal application development language. Though AutoLISP is the oldest of internal programming languages in AutoCAD, it is actively supported and is available for development in other CAD systems (Bricscad, IntelliCAD, DraftSight).
Apache Storm is a distributed tool for real-time processing of large data volumes. The project is based on Clojure and Java and was open-sourced under Apache License 2.0 after it was purchased by Twitter. Interoperability with Java is the main feature of Clojure that allows integrating the Clojure code into any project already written in Java.
Riemann is a tool for distributed system monitoring. It aggregates events from user servers and applications, combines them into a stream and transmits them for further processing or storage. Greater flexibility and fault-tolerance make Riemann different from other similar systems. Moreover, it’s written in Clojure almost completely. The code is available on GitHub and is distributed under Eclipse Public License 1.0.
The news site Hacker News was developed as a project of Y Combinator and is written in Arc. This is a Lisp dialect implemented in Racket. The site author, Paul Graham, also took part in the development of this language and, as you can see, has found a practical use for it. One of the goals was to create a community similar to that formed on Reddit in its early days (surprisingly, at that time it was written in Common Lisp as well until it was migrated to Python in 2005).

Conclusion

I’ve tried to compile this list by including examples from different domains, development tools, off-the-shelf applications for non-tech users, and systems contained in the products regularly used by a good many people without much thought.

Of course, the list is not complete. It highlights the most interesting Lisp applications – in my subjective opinion – in modern software. More complete lists of libraries, applications and companies using Lisp dialects can be found on the following resources:

Clojure:
- https://clojure.org/community/success_stories
- https://github.com/razum2um/awesome-clojure
Common Lisp
- https://github.com/azzamsa/awesome-cl-software
- https://common-lisp.net/lisp-companies
Scheme/Racket
- https://github.com/caocao485/awesome-racket-and-scheme
- https://github.com/avelino/awesome-racket

The confusion around your API

Catherine Galkina — Tue, 07 Sep 2021 15:10:30 +0000

Author: Ilya Kostyuchenko

How to prevent your APIs from scattering.

In this post, I’d like to tell you what OpenAPI is and why you may need it.

Your Pokemons

Your status quo: you’ve got two developers. One is developing the backend of your product, the other is working on the frontend.

You’ve got an idea for a new supercool application and you want to develop it as soon as possible, so you call an experienced architect to design well in advance the ideal API for which the frontend and the backend can be developed simultaneously.

The architect develops the ideal API, describes it in one big document and issues it to the developers.

Each of the developers takes the API description, reads it carefully and implements the described API.

Measure thrice and cut once

In the long run, we’d like to get the frontend and the backend, each of which will deal with the same requests (the frontend will send and the backend will process them). Ideally, these requests should also comply with what the architect has described (although it’s not so important).

Now let’s see what must happen to make the frontend and the backend requests coincide:

The backend developer hasn’t made any mistakes while implementing the API
The backend developer has read the architect’s API description correctly
The frontend developer hasn’t made any mistakes while implementing the API
The frontend developer has read the architect’s API description correctly

If anyone makes a mistake even in one of the items, your entire project will fail. And that’s assuming that the architect doesn’t make any errors (spoiler: they all do).

It should be specially noted that heavy responsibility falls on human understanding and human communication of all technical details. The human understanding is generally rather hard to debug and test.

People are a weak link in the chain

Given the situation, the desire to minimize the human factor in the API development seems quite natural. You’d like to exclude the human factor from the moment when the architect described API in the document. (As a matter of fact, it would be great to avoid the architect’s errors, too, but the technologies haven’t advanced thus far yet).

Obviously, to make this possible, the architect should deliver not a human-readable document but a computer-readable document, i.e., some sort of formal specification for a specific API. If we have such API description, we can at least try to allot the subsequent tasks to automation.

For each programming language, there are usually not so many ways to make a “canonical” implementation of any HTTP API when using a specific framework. Frameworks usually don’t have so many ways to make a request with a JSON object in the body, nor so many ways to read an integer in the request path.

OpenAPI

It would be great to be able to describe your HTTP API once and get coinciding frameworks from this description to develop the backend and the frontend, which, considering that there is no human factor in the chain, will more likely coincide. (Provided that the frameworks have been generated without errors, which is in fact a simpler task).

And behold! This has been invented already! This is OpenAPI!

OpenAPI allows describing HTTP API formally as YAML files. A rather extensive example can be found on editor.swagger.io. There you can simultaneously view the source YAML and the human-readable HTML-page.

If the architect describes the original specification in the form of an OpenAPI specification, the interaction between the backend and the frontend can be generated automatically and will always coincide. Thus, we eliminate the human factor from the chain altogether!

More than just code generation

Code generation is just one of the OpenAPI applications. OpenAPI is an open format of HTTP API description not tied to any specific ecosystem. It allows exchanging the accurate API description between the systems which would otherwise require manual API “synchronization”.

There are a lot of tools based on OpenAPI specifications. Openapi.tools is a good source of such projects. There you can find such projects as GUI specification editors, test server generators based on specifications, search for vulnerabilities by specifications and many other things!

Using Open API allows you not only to improve the accuracy of your API descriptions, but also to get access to a larger number of tools which might be useful for the project development.

The 7 assumptions about strings you probably have

Catherine Galkina — Fri, 20 Aug 2021 16:47:49 +0000

Author: Ville Tirronen

How Unicode erases most of our assumptions on How Strings Actually Work

We programmers mostly fly by the seat of our pants when it comes to writing simple stuff. For simple things, we have a strong set of assumptions instead of specific knowledge of how things work. These are assumptions, such as knowing that if b = a + 1, then b is greater than a or that if we malloc some buffer, we now have the requested amount of memory we can write on. We don't go and look at the specifications for each and everything small thing we do.

We do this because checking everything would slow us down. But, if we did check, we'd find out that we're usually wrong in our assumptions. There are numeric overflows and then a + 1 might be a lot less than a. Sometimes malloc will give us a null instead of a buffer and were hosed.

We usually have to be bitten by these issues before we update our assumptions even a little bit. And even then, we usually correct them in broad strokes. After having a nasty overflow bug, we might correct our assumptions on integers to "a + 1 is greater than a unless there's a chance that a is a very big number". And we work based on that instead of having any precise rules how overflows work in our minds.

Adjusted assumptions are called experience. They make you faster and correct more often. However, we might relocate some stuff, like proper handling of malloc, entirely from our internal category of 'easy stuff' to our internal category of 'complex stuff'. And then we might actually go and look up how it works.

About Strings

For beginners, Strings are the archetypal example of 'easy stuff'. Most likely, we learned letters and numbers as children and they feel very familiar to us. Secondly, when learning to program most of us have done lot's of programming exercises using Strings, because they are about the only interesting pre-built data type in most languages. We feel quite confident how Strings work when programming with them. Thirdly, we might have a good number of assumptions related to functioning of some simple character set, like ASCII or ISO-8859-1. Either because we're that old, or because our teachers were that old. Those were character sets of the simpler times!

Origin:https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/UNIVAC_1050-II.jpg/1280px-UNIVAC_1050-II.jpg

Now, in the real, Strings are a very complicated thing. Contrast them to, for example, your usual, found in any language, Int. We know and understand the representation (64 bits, two's complement) (or we can spend 15 minutes in Wikipedia to learn it) and we understand its semantics (behaves like a number, except if too large or too small). For Strings, we used to know the representation (one byte per character, check the ASCII table for what character it is), but we almost never know the semantics. Our String could contain our customers name. It could contain a number, bit of JSON or even an SQL statement.

Strings are the ultimate Any-type and chances are that if there is no ready made representation for some item in a program, it will be stored and operated as a String. Regardless of whether you have dynamic or static types, this throws all type safety to the wind. And, to compound, many of the things we use Strings for are bloody dangerous, like SQL or HTML. And for that reason, SQL injections and cross site scripting lead the vulnerability top lists year after year.

But, at least we understand how Strings work, as you know, Strings? We know how to concatenate, change case and so on, right?

Unicode

Understanding Strings is lot harder now than it was in around year 2000. We have been transitioning to Unicode for few decades now and its already been few years since I've heard anyone complain why their characters aren't displayed right. Printing them is another matter. I hope that it will be solved in 22th century.

While being otherwise awesome, Unicode effectively erases most of our 'useful' assumptions on how Strings actually work, but we haven't been very verbal on that happening. And unfortunately, many of us are probably still working with outdated assumptions on how Strings work. And, to make it worse, many of us no longer understand the memory representation of Strings either. Admittedly, I don't, really.

Broken assumptions

Next, lets go through some of my old assumptions that I needed to throw out with ISO-8859-1 character set. Surely, this is not an exhaustive list, but hopefully it is enough to kick the (Unicode) Strings out of your mental compartment of 'simple things'.

A character is representable by single byte

In the olden days of ASCII, each character fitted it seven bytes, making it easy to size buffers and scan memory. With Unicode this is an terrible assumption. Let's walk through one arbitrary example to show why.

At some point, Wordpress devs were fighting to stop SQL-injections from happening. The one example issue they were trying to fix was someone adding unwanted single quotes in the user input and messing their database with it.
Something like this imaginary example:

select 1 from accounts 
where user = '%s' 
    and password = '%s'

↓↓ (User supplies "whocares' or true -- " as password)

select 1 from accounts 
where user = 'Avery' 
    and password = 'whocares' or true -- '
-- And now everyone can log in as Avery!

Now, the simplest imaginable way to solve this is to properly encode the single quote in the user input. (But, that is simple in imagination only. Don't). That is, each single quote ' must be encoded as \', or backslash-single-quote.

PHP devs then wrote addslashes function and everything was well for a while. The only problem was that they did the escaping byte by byte and not character by character. The devs were also blind to the problem as they only worked with single byte Unicode characters (mostly old ASCII). Then, someone figured out that if you fed the system a String like "뼧 or true -- " you'd get the SQL injection again.

To understand why lets look up how these characters are represented in Unicode:

code	character
`0xbf27`	`뼧`
`0xbf5c`	`뽜`
`0x27`	`'`
`0x5c`	`\`

What the addslashes actually did was to replace all the value 27 -bytes with bytes 5c 27. So, "뼧 or true -- " turned into "뽜' or true -- " and again, there were injections.

It is not hard to imagine other similar disasters.

String lengths are somewhat stable

In ASCII, the many of the common String processing operations were invariant regards to the length of the Strings. This is not so with Unicode. And though this property is probably relevant only if you're manually allocating buffers, or trying to size up graphics, let's look at few cases where String lengths change unexpectedly.

Firstly, to pick a common String operation as an example, does length(x) = length(toUpper(x)) hold for Unicode x? No, since Unicode has, among other things, ligature characters such as ﬁ, which expand 2 fold to FI.

Second example concerns normalization. Since there are multiple code points for the same character, Unicode forces you to do normalization so that two users don't, for example, end up with identical screen names. One would guess that normalization, or the process of picking up a canonical representation for some set of characters would not affect the number of normalized characters, but it indeed does: single character ﷺ expands 18 fold into صلى الله عليه وسلم.

So, it is probably better not to assume anything about lengths of Strings after any operation.

Upper and lowercase are somehow linked

We, who lived with variants of ASCII tend make lot of use of upper and lower casing operations. Besides of them now being able to change the lengths of the Strings, there are some additional sharp edges. Most importantly, the old assumption that upper and lower case letters are in unique correspondence is lost.

With Unicode, converting string to uppercase can lose more information than just what case the characters were in. For example if you lowercase the Kelvin symbol K, you get an ordinary lowercase k back, with no way of converting it back. This has surprisingly lot of relevance when doing case insensitive comparisons, since toLower('K') == toLower('k') but toUpper('K') != toUpper('k').

Origin: https://upload.wikimedia.org/wikipedia/commons/thumb/1/1a/Upper_case_and_lower_case_types.jpg/800px-Upper_case_and_lower_case_types.jpg

Space is 0x20

This assumption is still true. The byte 0x20 represents space in Unicode. But so do U+2000, U+2001, U+2002 and many others, including a zero width space character U+FEFF. Whitespace is special. We can't allow screen names like "TheAlex" and "TheAlex " at the same time because HTML will not show that whitespace and other users couldn't tell the difference. So we must remove leading and trailing whitespace before processing.

And now, Unicode makes it possible to screw up royally here. All it takes is one spot in the code where someone forgets about multitude of whitespace and we end up with unnormalized data in our database. And things start to go fail here and there.

Characters look different

Unlike ASCII, Unicode has multiple code points for the same character and multiple characters that look nearly, or completely, identical without being the same character. As a concrete example, paste "tyрeablе" == "typeable" to your favourite REPL. repl.it is handy if you have none at the hand.

Got False? That is because the p is not a p but a Russian character for 'er' sound.

As to why this is a problem, let's take this bit of our database schema as an example:

"uniq_address" UNIQUE CONSTRAINT, btree (country, city, address)
"uniq_name" UNIQUE CONSTRAINT, btree (name)

I would posit that in Unicode era, these constraints make no sense at all.
Being user input the user is free to mimic whatever address or name they want. This allows the user to attempt all kinds of heists by, say having same screen name as someone else. Also, things like addresses don't stay
digital. Sooner or later, it's going to be read or printed and then the difference, which the database was keen to notice, will be gone. Is there anything analog in your process that could be exploited by pretending to be an another user?

This problem certainly preceded Unicode, especially in some character sets like ISO-8859-5, but Unicode makes this much worse and more widely applicable. Getting down to it, you can't assume almost anything about how the string is going to l̷o̵o̷k̵ ̶l̴i̴k̵e̷.

Text goes from left to right

Quickly, what happens if I'd paste this to my terminal?

‮rm -rf your_home_directory # dlrow olleh ohce

I dare you to try yourself. You can use any reasonable dumb thing to paste this in instead of your terminal if you care about your home directory.

Some languages are not written from left to right, and to accommodate them, Unicode has these 'flip the direction of writing' -codes. The actual text is the same even though it is written from right to left, so your terminal probably would try to wipe your files if you had tried my example.

Besides messing with my colleagues on Teams with this, this bidirectional writing has been used for quite a many hoaxes, the longtime favourite being flipping long URLs backwards so they look innocent.

Strings have the same decoding

One of the things we happily assumed with ASCII (and variants) was that the decoding was trivial and unlikely to go wrong. Some of my University colleagues can read ASCII fluently from hex dumps! This meant that the only problem when transmitting data as Strings was to correctly parse the contents of the String.

Unicode, being a multibyte encoding adds another step. You must first parse the String, before you can get started on the content.

Now, parsing is one of the problem areas that is known to cause security issues. One of the key problems is that the same String may get parsed differently in one program than in another. A good contemporary example of this is having and html sanitizer (thing that stops XSS) speak bit different dialect of HTML than the browser that the user is using. If these bits disagree on the interpretation of some String, the sanitizer might judge it to be free of scripts and other malicious items, while the browser could interpret things slightly differently and start executing bits of the input as scripts. Using the same channel for control and content must be worth more than the billion dollar mistake of including null in programming languages!

Now this is exacerbated by Unicode, since not all Unicode parsers agree on all sets of bytes. Mostly, it is the illegal Unicode sequences that get handled differently. For example "e3 80 22" is an invalid Unicode sequence and one Unicode parser might judge it to be one illegal character while another could be more lax and interpret it as three: ã, \x80 and ". Now, to put this into web context, the last of the three could be a problem since
it would allow XSS through attribute values.

Concluding thoughts

As a software engineer, Unicode puts a lot of complexity on my table and much of that I really wouldn't need. The individual gotchas listed above are not so hard to handle by themselves, but the effect their presence has on the whole system can be significant. Now you need to decide what kind of strings you allow in your system, you need figure out how to properly normalize them, how to eliminate homoglyphs and strip and trim whitespace.

The problem with this is that all such things must happen uniformly. If you normalize a String in a certain way in one bit of your program and some other bit does it differently, you have an inconsistency, or a security issue at the worst. You also have to take this into account, because, well mistakes happen and try to record precisely what has been done to each String so you can take that into account when using them.

And, unfortunately, no, you cannot just 'fix your strings' at every use point. Some string operations are only safe to do once or you lose information or worse. You need to know and track the semantics of Strings to know what steps you need, and what steps you can't take in the context you are working on.

Addendum: I'm bit of a sloppy writer, so, I feel I need to re-iterate the original point, so I don't come off as some ASCII-fan.

Some people read this as argument against Unicode, which it is not. I don't want to come back to ISO-8859-1, because that sucks. Also, I'm ready to deal with lot of complexity to allow people write their names properly. What I tried to argue here was that working with Unicode is necessarily more complex than working with ASCII was. And that I see people going about with lots of beliefs about string processing that belong to ASCII era and which do not work with Unicode.

Some of the examples are low level, some are dated, but some, like homoglyphs are ubiquitous. Whether they are relevant for you all depends on what kind of work you do and which language you use.

(Also, in first example, consider UTF-16 and PHP not having null terminated strings)

Property-based testing with QuickCheck

Catherine Galkina — Mon, 09 Aug 2021 14:25:12 +0000

Author: Volodya Kalnitsky

What Is Property-Based Testing?

Property-based testing (PBT) is the approach to software testing that implies an automatic check of the function properties (predicates) specified by the tester. Checking, i.e. search for counter-examples is carried out using the automatically generated input data. PBT allows developers to increase the test coverage significantly and spend their time efficiently saving them the trouble of inventing the input data for tests on their own. Normally, the values generated during property-based testing is not limited by anything, which is why the check can be carried out using the values the developer may have forgotten or neglected to include in the unit tests (surely, you won’t brute force all the values of input parameters).

The PBT approach was popularized by the QuickCheck library written in Haskell, and in this post, we’ll show how to use this tool effectively.

As a rule, PBT libraries consist of two parts:

Runners, which are in charge of running the tests and checking the validity of the predicate.
Arbitraries that take care of pseudo-random data generation while enabling shrinking, i.e. the way to “simplify” the found counter-example.

In my opinion, the skill of PBT consists in the ability to create a fast and effective data generator that allows obtaining potentially “problematic” values. To that end, you need both the knowledge of the subject domain and the skill of using the tools provided by the QuickCheck library.

In this post, I’ll show how the real code can be tested using QuickCheck.

QuickCheck library

The type responsible for data generation is the wrapper around a function that accepts the pseudo-random generator and also an integer parameter setting restrictions on the generated data size.

newtype Gen a = MkGen { unGen :: QCGen -> Int -> a }

The size constraint is needed for inductive types, e.g. for the tree depth (so that the tree wouldn't be infinite).

The generator size can be changed using the functions resize and scale, and the current generator parameter can be obtained using the sized function:

resize :: Int -> Gen a -> Gen a
scale :: (Int -> Int) -> Gen a -> Gen a
sized :: (Int -> Gen a) -> Gen a

Implementation of the arbitrary values generation and shrinking for a specific type is an instance of the Arbitrary class that has two methods: arbitrary, i.e. the value generator itself, and shrink, which is the function used to get the list of “shrunken” values.

class Arbitrary a where
  arbitrary :: Gen a
  shrink :: a -> [a]

Properties are set using the type Property. We won’t go into the details of its implementation because the library provides combinators to create various properties of the functions which allow, among other things, checking the truth or the falsehood of predicates.

The properties check is started by the function that goes by the same name, quickCheck.

In addition to the above said, QuickCheck allows displaying information about the test data values distribution analysis.

Further, we’ll describe all these features of the library using a small example.

QuickCheck usage example

By way of example, let's consider a naive parser and serializer for a JSON subset where there are no Boolean or Null types, and where spaces that are optional in JSON are prohibited.

The data type is declared in the following way:

data Json
  = Object [(String, Json)]
  | Array [Json]
  | String String
  | Number Double
  deriving (Show, Eq, Generic)

To set the limit for the tree size, we should avoid creating new branches if the size parameter is equal to zero and enable the parameter reduction in recursive calls.

Let's write a correct instance for the Arbitrary class:

instance Arbitrary Json where
  arbitrary = sized arbitrary'
    where
      arbitrary' 0 = pure $ Array []
      arbitrary' n =
        oneof [ Object <$> resize (n `div` 2) arbitrary
              , Array <$> resize (n `div` 2) arbitrary
              , String <$> arbitrary
              , Number <$> arbitrary
              ]

It should be noted that here we’re dividing the size by 2 and don’t decrease by one. The instance Arbitrary for the list will produce a list of the length not exceeding the size. In this way, we can create a logarithmic, not exponential, dependency of a medium-size tree on the size. In practice, we don’t need a linear dependency, we only need to avoid the discrepancy occurring because each of the constructors Array or Object generates an infinite tree (this is because the exit from recursion is statistically rarer than the generation of new constructors). Here the constant “2” is selected randomly.

Let’s test our generator:

 V1> generate (arbitrary  :: Gen Json)
Object [("%\1003607*SF\STX\166973ti\59844B",Array [Number 3.575309217492902]),("",String "4\USO\DLE\1065483e\STX\FS}\146478"),("\DEL\59467AU\1020011\997210I\57595\EM\fDm\DEL",Object []),("sX%8\1083528D-r\146202{S",Array [Array [],Array [Object [],Array [],Array []]]),("",Number 4.890220146762664),("\158800m\1047365\&07",Array [String "\5524\1069330"])]

 V1> generate (arbitrary  :: Gen Json)
String "\ACK[Q\1038205\64353EFz|\159918\77959\&0\1013613-\12406\1042312"

 V1> generate (arbitrary  :: Gen Json)
Number (-6.706967599855459)

As we can see, the Arbitrary instance for the String type generates “potentially problematic” lines to try and trigger typical errors in the applications using the data. The programmer should keep in mind special symbols, empty lines, whitespaces, etc.

Now let’s implement the serializer and the parser for our data type.

Serialization is implemented rather bluntly:

serialize :: Json -> String
serialize (Object props) =
  "{" ++ intercalate "," (map toKeyValue props) ++ "}"
  where
    toKeyValue (key, value) = serializeString key ++ ":" ++ serialize value
serialize (Array entries) =
  "[" ++ intercalate "," (map serialize entries) ++ "]"
serialize (String str) = show str
serialize (Number n) = show n

We’ll carry out parsing using the standard approach implemented in the libraries of parser combinators – all parsing functions will be of the type String -> Maybe (a, String), where a is the type we want to obtain as the result, and the second component of the pair (of the String type) is the line part that was not taken up by the parser.

The selected format is convenient because we can see from the first symbol what type we’re dealing with, which makes backtracking unnecessary.

Here we won’t describe the code itself but will move on to testing.

decode :: String -> Maybe (Json, String)
decode ('{' : rest)                = first Object <$> decodeProps rest
decode ('[' : rest)                = first Array <$> decodeArray rest
decode ('"' : rest)                = first String <$> decodeString rest
decode (c : rest)
  | isDigit c = first Number <$> decodeNumber (c:rest)
decode _ = Nothing

decodeProps :: String -> Maybe ([(String, Json)], String)
decodeProps ('}' : rest) = Just ([], rest)
decodeProps (',' : rest) = decodeProps rest
decodeProps ('"' : input) = do
  (key, ':' : input') <- decodeString input
  (value, input'') <- decode input'
  (restProps, input''') <- decodeProps input''
  return ((key, value) : restProps, input''')

decodeList :: String -> Maybe ([Json], String)
decodeList (']' : rest) = Just ([], rest)
decodeList (',' : rest) = decodeList rest
decodeList input = do
  (entry, rest) <- decode input
  first (entry :) <$> decodeList rest

decodeString :: String -> Maybe (String, String)
decodeString ('\\' : '"' : rest) = first ('"' :) <$> decodeString rest
decodeString ('"' : rest)        = Just ("", rest)
decodeString (c : rest)          = first (c :) <$> decodeString rest
decodeString _                   = Nothing

decodeNumber :: String -> Maybe (Double, String)
decodeNumber = listToMaybe . reads

Finally, the parse will let us get the result only if the line has been successfully parsed using decode:

parse :: String -> Maybe Json
parse input = case decode input of
  Just (json, "") -> Just json
  _               -> Nothing

Let’s formulate the property we’d like to test (parsing is the inverse function for serialization, i.e. the parse function applied to a serialized value produces the initial value).

prop_serialize_parse :: Json -> Property
prop_serialize_parse json = parse (serialize json) === Just json

Now let’s run the check of this property:

 V1> quickCheck prop_serialize_parse
*** Failed! Falsified (after 4 tests and 2 shrinks):
Array [String "\n"]

Obviously, we've forgotten about the part of escape sequences used in the instance Show String.

For the sake of simplicity, let’s give up our homebrewed implementation of the decodeString and reuse reads from the Prelude.

It is assumed that the opening quotation mark was taken up by the calling function, so it’s necessary to get it back:

decodeString :: String -> Maybe (String, String)
decodeString = listToMaybe . reads . ('"':)

We also find out that the numbers can be negative:

*> verboseCheck prop_serialize_parse
Failed:
*** Failed! Falsified (after 5 tests and 5 shrinks):
Array [Object [("",Number (-1.0))]]

Which can be also easily accounted for in our parser:

decode (c : rest)
  | isDigit c || c == '-' = first Number <$> decodeNumber (c:rest)

> quickCheck prop_serialize_parse
*** Failed! Falsified (after 4 tests and 7 shrinks):
Object [("",Object [("",String "\n")])]

Let's try and test our implementation of the JSON subset with regard to the existing one (the library aeson). We want to make sure that serialization returns a valid JSON:

prop_serialize_returns_json :: Json -> Property
prop_serialize_returns_json json = Aeson.decode @Aeson.Value (BS.pack $ serialize json) =/= Nothing

The result:

*** Failed! Falsified (after 4 tests):
String "\ETB\171675^\153309mX"
Nothing == Nothing

Of course, we’re working with the escape sequences incorrectly. The instance Show for the String doesn’t process them in the same way as the aeson decoder.

Surely, we should have implemented the serialization correctly but since this post is devoted to QuickCheck, it would be more interesting to show how to make QuickCheck omit the values we are definitely not interested in.

Let’s assume that we’ve decided on limiting the lines only to printed characters with the code range 32-126.

To do this, we can use the function suchThat :: Gen a -> (a -> Bool) -> Gen a that allows generating the values satisfying the specified predicate (in our case, this is the code range constraint):

instance Arbitrary Json where
  arbitrary = sized arbitrary'
    where
      arbitraryString =
        arbitrary `suchThat` all ((\code -> code >= 32 && code <= 126) . ord)
      arbitrary' 0 = pure $ Array []
      arbitrary' n =
        oneof [ Object <$> listOf
                ((,) <$> arbitraryString <*> resize (n `div` 2) arbitrary)
              , Array <$> resize (n `div` 2) arbitrary
              , String <$> arbitraryString
              , Number <$> arbitrary
              ]

After running such a test, we’ll notice that the time of its operation has increased significantly because now we’re rejecting the lines containing even one symbol from the unspecified interval.

The following code lets us know that we use approximately 6% of generated examples. We use the function classify that prints out the percent of the test cases which were useful for us:

 > quickCheck (\s -> classify (all ((\code -> code >= 32 && code <= 126) . ord) s) "useful" (s === s))
+++ OK, passed 100 tests (6% useful).

6% is not much, that’s why this method of lines generation is not suitable. To make the tests fast, it’s necessary to try and generate the data meeting the desired invariants at once instead of using the function suchThat or similar functions.

The situation becomes somewhat better if you place the suchThat inside the listOf (the String in Haskell is the list of symbols ([Char])):

arbitraryString =
  listOf (arbitrary `suchThat` ((\code -> code >= 32 && code <= 126) . ord))

However, generating a symbol from the specified interval is even quicker:

arbitraryString = listOf (chr <$> chooseInt (32, 126))

Unfortunately, it’s not often easy to write a generator producing only the values meeting a specific predicate, especially if the predicate requires any constraints, which are interrelated in some way, for various parts of the structure.

Shrinking

Shrinking is a way to “reduce” the found example to the minimum possible. The function shrink :: Arbitrary a => a -> [a] comes in after the counter-example has been found.

shrink must return the finite (and probably empty) list of all possible “simplifications” of the value with the type a. An empty list will mean that the minimal counter-example has already been found.

You can view the result of the shrink operation by running verboseCheck. Suppose that we want to check a rather strange assertion that no lines contain exactly two 'a' symbols. Obviously, here the line "aa" is the minimum counter-example. After finding the first counter-example we see how shrink is trying, again and again, to reduce the line to find this minimum counter-example:

> verboseCheck (\str -> 2 /= length (filter (== 'a') str))

  ...

Failed:
"a8aL"

Passed:
""

Passed:
"aL"

Passed:
"a8"

Passed:
"8aL"

Failed:
"aaL"

Passed:
""

Passed:
"aL"

Passed:
"aL"

Failed:
"aa"

Passed:
""

Passed:
"a"

Passed:
"a"

*** Failed! Falsified (after 69 tests and 10 shrinks):
"aa"

This search algorithm was implemented in the function shrinkList:

shrinkList :: (a -> [a]) -> [a] -> [[a]]
shrinkList shr xs = concat [ removes k n xs | k <- takeWhile (>0) (iterate (`div`2) n) ]
                 ++ shrinkOne xs
 where
  n = length xs

  shrinkOne []     = []
  shrinkOne (x:xs) = [ x':xs | x'  <- shr x ]
                  ++ [ x:xs' | xs' <- shrinkOne xs ]

  removes k n xs
    | k > n     = []
    | null xs2  = [[]]
    | otherwise = xs2 : map (xs1 ++) (removes k (n-k) xs2)
   where
    xs1 = take k xs
    xs2 = drop k xs

The shrinkList is trying to:

Delete one half of the list, one-quarter of the list, one-eighth part and so on starting from the end and from the beginning
Apply shrink to one of the list elements

Let’s write shrink for JSON (we’ll just reuse shrink implementations for pair, list, line, and number):

shrink (Object props)  = Object <$> shrink props
shrink (Array entries) = Array <$> shrink entries
shrink (String str)    = String <$> shrink str
shrink (Number n)      = Number <$> shrink n

Using generic-random

In the example described above, we implemented the arbitrary method manually mostly for illustrative purposes. However, in a real production code where dozens or hundreds of data types are declared this may become a tiring process. The [generic-random] library (https://hackage.haskell.org/package/generic-random) allows obtaining the instance of Arbitrary automatically.

Its operation is based on the generic programming of data types (datatype-generic programming). To describe this approach briefly, we can represent the data types in a general way as a sum type (an alternative of several constructors, as in our example with Json) or as a product type (a structure with several fields).

If it’s known how to obtain an arbitrary value for each of the alternatives in this sum type, we can use these values to get an arbitrary value of the sum type itself. Moreover, each constructor in the alternative can have a certain weight so that it would occur more often or less often than the rest. For a product type (more simply, a structure), if it’s known how to obtain an arbitrary value of each field, we can use them to create an arbitrary value of the structure as a whole.

For recursive types, we can indicate the value for the recursion base during generation when the structure size reaches zero. In our “manual” implementation of the Arbitrary instance for the Json type, it was the value Array [].

Let’s use generic-random to write a new definition of arbitrary for Json:

instance Arbitrary Json where
  arbitrary = genericArbitraryRec uniform `withBaseCase` return (Array [])

genericArbitraryRec reduces the size of generated structure at each recursive call; uniform sets a uniform distribution for the constructors in the alternative; withBaseCase indicates what generator should be called for a zero-size structure.

Conclusion

Thus, we’ve seen in this tutorial how the QuickCheck library is used for property-based testing and in particular, learned how to:

Describe the test data generators and adjust them to improve the efficiency and reduce the test execution time;
Describe simple properties of the functions and run tests to check them;
Analyze the examples of data used for testing;
Search for the minimum counter-example for which the tests fail;
Generate the required instances automatically.

Thank you for your attention!

When it Pays to Choose Microservices

Catherine Galkina — Mon, 19 Jul 2021 14:53:31 +0000

Author: Victoriia Zaripova

Hi everybody! My name is Victoria. Here at Typeable, I deal with the issues of application architecture, so I couldn't help asking the perennial question: To be or not to be? Specifically, is it worthwhile switching our solutions to microservices or not? To understand this, I've done small research of potential pros and cons. Here are my findings.

Microservices started gaining popularity in 2011-2014, smoothly replacing heavyweight SOA and monolithic solutions, where the architecture obstructed access to the rapidly growing market of cloud applications.

The approach itself evolved at the intersection of technologies out of the competitive need to bring the business to the next level instantaneously. Because of this, the solutions developed avalanche-like and quickly acquired add-ons, patterns, and CI/CD accessories. These reasons are still relevant for the business, and the interest in microservices has not declined over the last decade. At the same time, developing a microservice-based solution is a creative intellectual task for an IT team. It allows trying out state-of-the-art approaches and pinning down the conservatism dragons of previous solutions. That is, the challenge is quite noble.

However, the benefit of giving in to this magic is highly questionable.

Like any other fancy solution, microservices aren't always beneficial. Neither do they give a plaster for all sores.

Nevertheless, let's look into the matter.

History of the solution and microservices

The evolution of a typical IT solution can take the following path:

An MVP startup is a solution mainly aimed to try the market and decide whether this concept works altogether. Complex technologies, advanced interfaces and high-load services are not needed here. Cost minimization and time to market are valued most of all at this stage. At times, MVP fails as early as at the stage of the focus group interview. Does it make sense to start building a microservice-based solution right away? My answer is no, don't give it a second thought. Time will be lost while you're thinking over the architecture and setting up the infrastructure. And this is the most valuable resource for a startup. Besides, it would be difficult to use the key profit factor of microservices – parallel development – with the startup team.
Newborn monolithic solution. The application architecture is not fully verified, all internal and external services that will be used are not defined. The goals and objectives of end-users are not quite known. The data structure and functionality development strategy is not clear. Definitely, here it also doesn't make sense to start heading toward microservices, unless your team consists of profs for whom microservices are as easy as a pie.
An old commercially available and stable monolithic solution with a thoroughly verified functionality; any arising issues are solved by skilful refactoring; the development is expected to be smooth, without any competition between functional blocks. Does it make sense to switch to microservices? Probably, this needs to be thought if the aim is to provide integration with external services, increase the load on the existing functional blocks, ensure their independent development and market launch or if you want to cut corners on the technology stack or improve the solution characteristics by reusing components. In this case, you should be ready to reinforce your team with necessary specialists and spend time to break the monolith down into independent or very loosely coupled functional blocks and then bring these blocks out from the monolith as separate services. This will definitely require additional resources, as well as time to train the team and search for suitable technologies.
Multilevel monolith in DDD concept. The functionality is split into logically grouped isolated or loosely coupled blocks, which are still located within a monolithic structure; the logic is separated from the infrastructure. This is a good starting point for the migration to microservices.
Distributed monolith. The development team has almost succeeded in breaking down the monolith, but something has obviously gone wrong. The services are too tightly connected to each other; explicit and implicit dependencies exist; the business is suffering because it sets forth requirements for one module, but it turns out that several modules need to be reworked; it's difficult to trace the effect of changes; testing and deployment require individual non-trivial approaches. Stop here and just think about it. Do you need microservices? Are there really any problems that cannot be solved by an initially monolithic solution? If you do need it, you should rebuild this pyramid but in the correct order.
SOA solution. You shouldn't take microservices as the next step of the SOA evolution. It must be kept in mind that the main difference between the bus and the message exchange channel between services is that it is the bus that carries a significant portion of data conversion and orchestration logic. The message exchange channel between microservices, in its turn, must be fully reliable and straightforward. Its aim is to transmit a message. Implementing microservices in such solutions can be very costly because the supposedly "independent" modules can be closely coupled through the bus logic, and a simple replacement of this logic with aggregating patterns won't be sufficient. Moreover, the existing SOA solutions mostly use heterogeneous technologies and constructs which are difficult to adapt. Does it make sense to think about such migration at all? Usually, it doesn't. Most probably, the choice of SOA was driven by the complexity and diversity of the applications to be integrated into, the application scale, and the need to provide data consistency and the complex logic of data conversion. In this case, the choice is right. The microservices as an alternative should be considered only if the main part of the solution has become obsolete; when it's no more necessary to integrate with heavyweight application; if removing legacy code and migrating to simpler web solutions are on the agenda; if there is no need to maintain the high level of abstraction and data consistency but the speed of developing individual modules is still critical or it is necessary to enable independent work of separate teams.
Monstrous monolith. The monolith full of legacy code and built on patches. The monolith developed so long ago that it's turned into a huge structure resistant to changes. The monolith consisting almost completely of spaghetti code. This is probably the only case when it makes sense to raze everything to the ground, sweep up the debris, and build a new solution on the ruins. Probably, based on microservices.

As a rule, the development team starts thinking about the microservices at the stage of startup or monstrous monolith. And maybe, this thought just jumps to the minds on the back of the microservice boom.

Now, when to choose microservices?

If:

the aim is to develop a medium-sized non-trivial web application consisting of a set of loosely coupled or completely isolated modules;
there are critical requirements for the application's resistance to loads and/or support of integration with external services (payment systems, banks, external storages etc.);
the business requires significant acceleration of the development right now, plans to launch the changes on the market in all areas at once, and is not ready to wait for the sequential implementation of key changes in each area;
it's necessary to use a heterogeneous technology stack (for the purposes of renovation, adaptation to market conditions, acceleration of internal processes, etc.);
it's possible to distinguish the modules that allow reuse and support calls by various channels (authorization and authentication services, search engines, audit, etc.);
the business sets forth requirements for the system blocks at different rates; the importance of the quick release of the individual block also varies;
there's a commercial need to make frequent changes in an individual block in the future (to follow a trend or the marketing strategy);
strategic business objectives require or will require a point-like scaling or different rates of changes in various points of the application;
tactic business objectives require making multiple micro-changes in different modules of the system on the fly without disrupting the application as a whole (24/7 access and high probability of bugs due to the system complexity);

then you probably need to think about developing a solution using microservices.

Note that almost in every case it's not the development team who drives the decision-making but the business, and this is important. If microservices don't solve the business tasks, this is a waste of time and money. If the development team or the business itself has no idea of the current and strategic paradigm of the product, this is also a waste of time and money.

For example, interesting findings are provided in the research conducted by Camunda in 2018 among 354 companies in different countries and industries. Though the research revealed that 63% of enterprises support the adoption of microservices or are already adopting them, only 45% explicitly document the business processes. It creates a certain problem for evaluating the influence of microservice architecture on the implementation of these processes. At the same time, companies report that the top reasons for adopting a microservices architecture are: improved scalability of applications (64%), shortened development cycle (60%), support of digital transformation trends and integration with next-generation applications (54%), greater autonomy for development teams (54%); improved application resilience (50%).

However, based on the data provided by O'Reilly that ran a similar survey in 2020 among 1052 companies, 77% of respondents are using microservices and about one-third of respondents have been using them for the last three years. Of course, these two pieces of research cannot be compared, but the increasing popularity of microservices is obvious. Here similar issues were also found: incorrect decomposition and complexity of both the solution itself and the microservices management. Nevertheless, the surveys figuratively show that corporate culture takes the center stage. However, when it comes to the adoption of microservices, it's also an inhibitive factor.

Besides, there are some constraints you should take into account:

You have a large team that has nothing to do :) It's a joke but it has a grain of truth. The minimum pool for one microservice is a team of six to nine persons, including developers, testers, and, advisably, an analyst. These people must not be occupied with anything else than their microservice or, as the last resort, two microservices. You can say, where there are two, there are three, and where there are three, there are four and so on. But this is a wrong path. Where there are two, there are no more than two. Period.
Your DevOps architecture is configured to support independent development or you are ready to allocate resources and time for this. To start with, you do have DevOps. Make sure that you have.
You are ready to organize test environments suitable for independent testing of microservices, as well as for testing of the microservices interaction, and you are ready to migrate from the concept of pure end-to-end testing to the structure consisting of modular, integration, component and end-to-end tests including placeholders development and contract publication.
You are ready to spend time on failures or you've already fallen into all traps while developing your monolith and know exactly into which isolated services you can break it down without impairing the data quality, its processing speed and application reliability in general. At the very least, you clearly understand the data structure in each domain, know the business needs and can identify loosely coupled or completely independent domains. In no case should you link the microservice structure to the enterprise structure. Hierarchical schemes look pretty on paper, but in real life they often hide the pitfalls of poorly organized business processes.
And, finally, make sure that the development doesn't involve legacy applications that must be integrated with, and that the data transactionality support is not critical in the domains you've identified (unless you know and are ready to use SAGA, the distributed transactions pattern).

Microservices are always associated with a degree of complexity, so if the business has no issues which could be resolved by microservices, don't add them, as the business will not appreciate this.

I've already selected the microservices, what's gone wrong?

Alas, if the time to release hasn't been reduced, everything has gone wrong.

Most probably, you need to assess once again the potential sources of the problem:

Too many microservices. Probably, it makes sense to replace the seven microservices overburdening five teams with only five? Or even two. In fact, the opinion that there must be many microservices is erroneous. In most cases, the company is neither Amazon nor Netflix.
Poor analysis of business domains. There exist implicit dependencies, transactionality is required, there's no common solution for the architecture.
Preliminary agreements on API development, testing, CI/CD cycle setup are missing/violated.
The teams are either not autonomous or too autonomous, and the practice of experience exchange and retrospective is missing.
The team's life activities are not managed and supported properly. There are conflicts over resources, there is no clear vision of the business purposes of both the application as a whole and individual services.

However, a situation may occur when it seems that something is going wrong but you're not quite sure of this.

To save you the trouble of reading another long text, I'm just placing a picture here with a number of examples.

Conclusions

Here let me finish my post and make brief conclusions on this subject:

There's no use destroying what is working well just for the sake of fashion.
If you've decided on destroying something, consult the property (business) owner first and explain all potential consequences. First of all, it's necessary to analyze whether this will produce a positive effect on the business processes and whether there are any needs that cannot be satisfied by an existing solution. Don't draw a nice model over the ruins. First of all, this will have negative consequences for the development teams.
If the business approached you with this praiseworthy initiative, also explain the consequences. Business is business, they don't have to know about all pitfalls.
Before you start, make sure that you have all resources and clearly understand the path.
Don't miss the warning signs indicating that you've taken the wrong turn.

And finally. Most teams who have succeeded in using microservices had to rebuild their architecture on multiple occasions and followed the path of sequential monolith breakdown. So keep your head up.

Credits and additional materials

As a final point, I would like to recommend several articles on this topic:

Сhaos engineering make disciplined microservices by Shamik Mitra, Jun. 24, 21 · Microservices Zone
When to use and not to use microservices by Hardik Shah, Dec. 24, 20 · Container Journal
Microservices vs Nanoservices: Weighing Framework Options by Sudip Sengupta, Mar.10, 21 · BMC blogs
Camunda Whitepaper Building Scalable Business Automation with Microservices by Sandy Kemsley, Dec. 2019 · Camunda

I found these materials interesting and worth noticing.

Creating a Haskell Application Using Reflex. Part 4

Catherine Galkina — Tue, 22 Jun 2021 15:11:15 +0000

Author: Nikita Anisimov

Part 1

Part 2

Part 3

Hi there! In our new post, we’ll take a look at how we use JSFFI.

JSFFI

Let’s make it possible to set a deadline date in our application. Suppose that we need to make not just the text input, but a dropdown datepicker. Of course, we can write our own datepicker in Reflex but there are lots of various JS libraries we could use. If there exists an off-the-shelf JS code which is, for instance, too long to be rewritten using GHCJS, it’s possible to call it using JSFFI (JavaScript Foreign Function Interface). In our case, we’ll use flatpickr.

Let’s create a new JSFFI module and immediately add its import to the Main. We insert the following code in the created file:

{-# LANGUAGE MonoLocalBinds #-}
module JSFFI where

import Control.Monad.IO.Class
import Reflex.Dom

foreign import javascript unsafe
  "(function() { \
  \ flatpickr($1, { \
  \   enableTime: false, \
  \   dateFormat: \"Y-m-d\" \
  \  }); \
  \})()"
  addDatePicker_js :: RawInputElement GhcjsDomSpace -> IO ()

addDatePicker :: MonadWidget t m => InputElement er GhcjsDomSpace t -> m ()
addDatePicker = liftIO . addDatePicker_js . _inputElement_raw

Let’s not forget to add the required script and styles to the head element too:

  elAttr "link"
    (  "rel" =: "stylesheet"
    <> "href" =: "https://cdn.jsdelivr.net/npm/flatpickr/dist/flatpickr.min.css" )
    blank
  elAttr "script"
    (  "src" =: "https://cdn.jsdelivr.net/npm/flatpickr")
    blank

Now we try to compile the same way as before and get the following error:

src/JSFFI.hs:(9,1)-(16,60): error:
    • The `javascript' calling convention is unsupported on this platform
    • When checking declaration:
        foreign import javascript unsafe "(function() {    flatpickr($1, {      enableTime: false,      dateFormat: \"Y-m-d\"    });   })()" addDatePicker_js
          :: RawInputElement GhcjsDomSpace -> IO ()
  |
9 | foreign import javascript unsafe
  |

Indeed, now we’re building our application using GHC that has no idea of what JSFFI is. Recall that now the server is being launched that sends the updated DOM when this is needed using web sockets, and the JavaScript code is alien to it. The conclusion suggests itself that our datepicker just doesn’t work when you build using GHC. Nevertheless, GHC won’t be used to build the client’s application production version; we’ll compile to JS using GHCJS and embed the JS code we’ve obtained into our page. ghcid doesn’t support GHCJS, which is why it makes no sense to run nix shell; we’ll use nix immediately for our build:

nix-build . -A ghcjs.todo-client -o todo-client-bin

The directory todo-client-bin with the following structure will appear in the root directory of the application:

todo-client-bin
└── bin
    ├── todo-client-bin
    └── todo-client-bin.jsexe
        ├── all.js
        ├── all.js.externs
        ├── index.html
        ├── lib.js
        ├── manifest.webapp
        ├── out.frefs.js
        ├── out.frefs.json
        ├── out.js
        ├── out.stats
        ├── rts.js
        └── runmain.js

After opening the index.html in the browser we’ll see our application. Though we’ve built our project using GHCJS, it’s more convenient to carry out development using GHC together with ghcid, which is why we’ll modify the JSFFI module in the following way:

{-# LANGUAGE CPP #-}
{-# LANGUAGE MonoLocalBinds #-}

module JSFFI where

import Reflex.Dom

#ifdef ghcjs_HOST_OS

import Control.Monad.IO.Class

foreign import javascript unsafe
  "(function() {\
    flatpickr($1, {\
      enableTime: false,\
      dateFormat: \"Y-m-d\"\
    }); \
  })()"
  addDatePicker_js :: RawInputElement GhcjsDomSpace -> IO ()

addDatePicker :: MonadWidget t m => InputElement er GhcjsDomSpace t -> m ()
addDatePicker = liftIO . addDatePicker_js . _inputElement_raw

#else

addDatePicker :: MonadWidget t m => InputElement er GhcjsDomSpace t -> m ()
addDatePicker _ = pure ()

#endif

We’ve added conditional compilation: depending on the platform, we’ll use either the JS function call or a stub.

Now we need to change the input form for new tasks by adding the date selection field there:

newTodoForm :: (EventWriter t (Endo Todos) m, MonadWidget t m) => m ()
newTodoForm = rowWrapper $ el "form" $ divClass "input-group" $ mdo
  iEl <- inputElement $ def
    & initialAttributes .~
      (  "type" =: "text"
      <> "class" =: "form-control"
      <> "placeholder" =: "Todo" )
    & inputElementConfig_setValue .~ ("" <$ btnEv)
  dEl <- inputElement $ def
    & initialAttributes .~
      (  "type" =: "text"
      <> "class" =: "form-control"
      <> "placeholder" =: "Deadline"
      <> "style" =: "max-width: 150px" )
  addDatePicker dEl
  let
    addNewTodo = \todo -> Endo $ \todos ->
      insert (nextKey todos) (newTodo todo) todos
    newTodoDyn = addNewTodo <$> value iEl
    btnAttr = "class" =: "btn btn-outline-secondary"
      <> "type" =: "button"
  (btnEl, _) <- divClass "input-group-append" $
    elAttr' "button" btnAttr $ text "Add new entry"
  let btnEv = domEvent Click btnEl
  tellEvent $ tagPromptlyDyn newTodoDyn $ domEvent Click btnEl

We compile our application, try to run it and still see nothing. If we have a look at the developer console in the browser, we’ll see the following error:

uncaught exception in Haskell main thread: ReferenceError: flatpickr is not defined
rts.js:5902 ReferenceError: flatpickr is not defined
    at out.js:43493
    at h$$abX (out.js:43495)
    at h$runThreadSlice (rts.js:6847)
    at h$runThreadSliceCatch (rts.js:6814)
    at h$mainLoop (rts.js:6809)
    at rts.js:2190
    at runIfPresent (rts.js:2204)
    at onGlobalMessage (rts.js:2240)

Here we notice that the function we need is not defined. This is because the element script with the link, as well as every single element of the page are created dynamically. That’s why when we call the flatpickr function, the script containing the library with this function might be not loaded yet. Obviously, we need to set the loading order.
Let’s solve this issue using the package reflex-dom-contrib. This package includes many functions useful for development. Adding this dependency is non-trivial. The thing is that Hackage offers an obsolete version of this package, which is why we have to take it directly from GitHub. Let’s update default.nix in the following way.

{ reflex-platform ? ((import <nixpkgs> {}).fetchFromGitHub {
    owner = "reflex-frp";
    repo = "reflex-platform";
    rev = "efc6d923c633207d18bd4d8cae3e20110a377864";
    sha256 = "121rmnkx8nwiy96ipfyyv6vrgysv0zpr2br46y70zf4d0y1h1lz5";
    })
}:
(import reflex-platform {}).project ({ pkgs, ... }:
let
  reflexDomContribSrc = builtins.fetchGit {
    url = "https://github.com/reflex-frp/reflex-dom-contrib.git";
    rev = "11db20865fd275362be9ea099ef88ded425789e7";
  };

  override = self: pkg: with pkgs.haskell.lib;
  doJailbreak (pkg.overrideAttrs
  (old: {
    buildInputs = old.buildInputs ++ [ self.doctest self.cabal-doctest ];
  }));

in {
  useWarp = true;

  overrides = self: super: with pkgs.haskell.lib; rec {
    reflex-dom-contrib = dontHaddock (override self
      (self.callCabal2nix "reflex-dom-contrib" reflexDomContribSrc { }));
  };

  packages = {
    todo-common = ./todo-common;
    todo-server = ./todo-server;
    todo-client = ./todo-client;
  };

  shells = {
    ghc = ["todo-common" "todo-server" "todo-client"];
    ghcjs = ["todo-common" "todo-client"];
  };
})

We add the import of module import Reflex.Dom.Contrib.Widgets.ScriptDependent and make changes in the form:

newTodoForm :: MonadWidget t m => m (Event t (Endo Todos))
newTodoForm = rowWrapper $ el "form" $ divClass "input-group" $ mdo
  iEl <- inputElement $ def
    & initialAttributes .~
      (  "type" =: "text"
      <> "class" =: "form-control"
      <> "placeholder" =: "Todo" )
    & inputElementConfig_setValue .~ ("" <$ btnEv)
  dEl <- inputElement $ def
    & initialAttributes .~
      (  "type" =: "text"
      <> "class" =: "form-control"
      <> "placeholder" =: "Deadline"
      <> "style" =: "max-width: 150px" )
  pb <- getPostBuild
  widgetHoldUntilDefined "flatpickr"
    (pb $> "https://cdn.jsdelivr.net/npm/flatpickr")
    blank
    (addDatePicker dEl)
  let
    addNewTodo = \todo -> Endo $ \todos ->
      insert (nextKey todos) (newTodo todo) todos
    newTodoDyn = addNewTodo <$> value iEl
    btnAttr = "class" =: "btn btn-outline-secondary"
      <> "type" =: "button"
  (btnEl, _) <- divClass "input-group-append" $
    elAttr' "button" btnAttr $ text "Add new entry"
  let btnEv = domEvent Click btnEl
  pure $ tagPromptlyDyn newTodoDyn $ domEvent Click btnEl

We’ve used the new function widgetHoldUntilDefined that will build the element passed to it in the last parameter only when the defined script has already been downloaded.
Now, if we open our page created using GHCJS we’ll see the datepicker we use.

However, we’ve not used this field in any way. Let’s change the type Todo making sure that we’ve added the import of Data.Time:

data Todo = Todo
  { todoText     :: Text
  , todoDeadline :: Day
  , todoState    :: TodoState }
  deriving (Generic, Eq, Show)

newTodo :: Text -> Day -> Todo
newTodo todoText todoDeadline = Todo {todoState = TodoActive False, ..}

Now we change the function with the form for a new task:

...
  today <- utctDay <$> liftIO getCurrentTime
  let
    dateStrDyn = value dEl
    dateDyn = fromMaybe today . parseTimeM True
      defaultTimeLocale "%Y-%m-%d" . unpack <$> dateStrDyn
    addNewTodo = \todo date -> Endo $ \todos ->
      insert (nextKey todos) (newTodo todo date) todos
    newTodoDyn = addNewTodo <$> value iEl <*> dateDyn
    btnAttr = "class" =: "btn btn-outline-secondary"
      <> "type" =: "button"
...

And add the date to the list item widget:

todoActive
  :: (EventWriter t (Endo Todos) m, MonadWidget t m)
  => Int -> Text -> Day -> m ()
todoActive ix todoText deadline = divClass "d-flex border-bottom" $ do
  elClass "p" "p-2 flex-grow-1 my-auto" $ do
    text todoText
    elClass "span" "badge badge-secondary px-2" $
      text $ pack $ formatTime defaultTimeLocale "%F" deadline
  divClass "p-2 btn-group" $ do
  ...

As usual, the result we’ve got can be found in our repository.

In the next part, we’ll see how to use routing in a Reflex-based application.