My Architecture Failures and Solutions

Prehistory

I have been working as a front-end developer for one year. On my first project there was an “enemy” backend. This is not a big problem when communication is established.

But in our case it was not the case.

We developed code that relied on the fact that the backend sends us certain data of a certain structure and content. While the backend was considered normal to change the contents of the responses - without warning. As a result, we spent hours trying to find out why a certain part of the site stopped working.

We realized that we needed to check what the backend returns before relying on the data it sent us. We created a task to research the validation of data from the frontend.

This study was entrusted to me.

I made a list of what I want to be in the tool that I would like to use to validate the data.

The most important selection points were the following items:

• a declarative description (scheme) of validation, which is transformed into a function-validator, which returns true / false (valid, not valid)
• low entry threshold;
• similarity of validated data with a description of validation;
• ease of integration of custom validations;
• ease of integration of custom error messages.

As a result, I found many validation libraries by looking at the TOP-5 (ajv, joi, roi ...). They are all very good. But it seemed to me that for the solution of 5% of complex cases - they doomed 95% of the most frequent cases to be rather wordy and cumbersome.

So I thought: why not develop something that would suit me?
Four months later, the seventh version of my validation library quartet was released.
It was a stable version, fully tested, 11k downloads on npm. We used it on three projects in a campaign for three months.

These three months have played a very useful role. quartet showed all its advantages. Now there is no any problem with the data from the backend. Every time they changed the response - we immediately threw an error. The time spent finding the causes of bugs was reduced dramatically. There are practically no data bugs left.

But there were also disadvantages.

Therefore, I decided to analyze them and release a new version with corrections of all errors that were made during development.
On these architectural errors and their solutions will discuss below.

Architectural rake

"String"-typed schema language

I will give an example of the old version of the scheme for the person object.

const personSchema = {
  name: 'string',
  age: 'number',
  linkedin: ['string', 'null']
}

This scheme validates an object with three properties: name - must be a string, age - must be a number, link to account on LinkedIn - must either be null (if there is no account) or string (if there is an account).

This scheme meets my requirements for readability, similarity with validated data, and I think the entry threshold for learning to write such schemes is not high. Moreover, such a scheme can be easily written from the typescript type definition:

type Person = {
  name: string
  age: number
  linkedin: string | null
}

(As we see, the changes are more cosmetic)

When deciding what should be used for the most frequent validations (for example, those used above). I chose to use strings, like validator names.

But the problem with strings is that they are not available to the compiler or error analyzer. The ‘number’ string for them is not much different from ‘numder’.

Solution

I decided to remove from the quartet 8.0.0 the use of strings as names of validators inside the schema.

The scheme now looks like this:

const personSchema = {
  name: v.string
  age: v.number,
  linkedin: [v.string, null]
}

This change has two big advantages:

• compilers or static analyzers of code - will be able to detect that the name of the method is written with an error.
• Strings are no more used as an element of the scheme. This means that for them it is possible to allocate a new functional in the library, which will be described below.

TypeScript Support

In general, the first seven versions were developed in pure JavaScript. When switching to a Typescript project, it became necessary to somehow adapt the library for it. Therefore, type declaration files for the library were written.

But this was a minus - when adding functionality, or when changing some elements of the library, it was always easy to forget to update the type declarations.

There were also just minor inconveniences of this kind:

const checkPerson = v(personSchema) // (0)

// ...

const person: any = await axios.get('https://myapi.com/person/42')
if (!checkPerson(person)) { // (1)
  throw new TypeError('Invalid person response')
}
console.log(person.name) // (2)

When we created an object validator on the line (0). We would like to see after checking the real response from the backend on line (1) and handling the error. On line (2) for person to have type Person. But it did not happen. Unfortunately, such a check was not a type guard.

Solution

I made a decision to rewrite the entire quartet library into Typescript so that the compiler would check the library for its type consistency. Along the way, we add to the function that returns the compiled validator - a type parameter that would determine what type of type guard is the validator.

An example looks like this:

const checkPerson = v<Person>(personSchema) // (0)

// ...

const person: any = await axios.get('https://myapi.com/person/42')
if (!checkPerson(person)) {// (1)
  throw new TypeError('Invalid person response')
}
console.log(person.name) // (2)

Now on line (2) the person is of typePerson.

Readability

There were also two cases where the code was poorly read: checking for compliance with a specific set of values(checking enum) and checking the other properties of the object.

a) Enum check
Initially there was an idea, in my opinion a good one. We will demonstrate it by adding the field "sex" to our object.
The old version of the scheme looked like this:

const personSchema = {
  name: 'string',
  age: 'number',
  linkedin: ['null', 'string'],
  sex: v.enum('male', 'female')
}

The option is very readable. But as usual, everything went a little out of plan.
Having the enum announced in the program, for example:

enum Sex {
  Male = 'male',
  Female = 'female'
}

Naturally you want to use it inside the scheme. So that if one of the values ​​changes (for example, ‘male’ -> ‘m’, ‘female’ -> ‘f’), the validation scheme also changes.

Therefore, enum validation is almost always recorded as:

const personSchema = {
  name: 'string',
  age: 'number',
  linkedin: ['null', 'string'],
  sex: v.enum(...Object.values(Sex))
}

That looks rather cumbersome.

b) Validation of rest properties of the object

Suppose we add another characteristic to our object — it may have additional fields, but all of them must be links to social networks — that means they must be either null or be a string.

The old scheme would look like this:

const personSchema = {
  name: 'string',
  age: 'number',
  linkedin: ['null', 'string'],
  sex: v.enum(...Object.values(Sex)),
  ... v.rest(['null', 'string']) // Rest props are string | null
}

Such way of description outlines the remaining properties - from those already listed. Using the spread-operator - rather confuses the person who wants to understand this scheme.

Solution

As described above, strings are no longer part of validation schemes. The validation scheme has only three types of Javascript values. Object - to describe the object validation scheme. Array to describe - several options for validity. Function (generated by the library or custom) - for all other validation options.

This provision made it possible to add functionality that allowed us to increase the readability of the scheme.

In fact, what if we want to compare the value with the string ‘male’. Do we really need to know anything other than the value itself and the ‘male’ string.

Therefore, it was decided to add values ​​of primitive types as an element of the schema. So, where you meet the primitive value in the scheme, this means that this is the valid value that the validator created by this scheme should check. Let me give you an example:

If we need to check the number for equality 42, then we write it like this:

const check42 = v(42)

check42(42) // => true
check42(41) // => false
check42(43) // => false
check42('42 ') // => false

Let's see how this affects the person's scheme (without taking into account additional properties):

const personSchema = {
  name: v.string,
  age: v.number,
  linkedin: [null, v.string], // null is primitive value
  sex: ['male', 'female'] // 'male', 'female' are primitive values
}

Using pre-defined enums, we can rewrite it like this:

const personSchema = {
  name: v.string,
  age: v.number,
  linkedin: [null, v.string],
  sex: Object.values(Sex) // same as ['male', 'female']
}

In this case, the extra ceremoniality was removed in the form of using the enum method and using the spread-operator to insert valid values ​​from the object as parameters into this method.

What is considered a primitive value: numbers, strings, characters, true,false, null and undefined.

That is, if we need to compare the value with them - we simply use these values ​​themselves. And the validation library will create a validator that will strictly compare the value with those specified in the schema.

To validate the residual properties, it was chosen to use a special property for all other fields of the object:

const personSchema = {
  name: v.string,
  age: v.number,
  linkedin: [null, v.string],
  sex: Object.values(Sex),
  [v.rest]: [null, v.string]
}

Thus the scheme looks more readable. And more similar to definitions from Typescript.

Validator's binding to the function that created it.

In older versions, error explanations were not part of the validator. They are folded into an array inside the v function.

Previously, in order to get explanations of validation errors, it was necessary to have a validator with you (to perform a check) and a function v(to get explanations of invalidity). It all looked like this:

a) We add explanations to the schema.

const checkPerson = v({
  name: v('string', 'wrong name')
  age: v('number', 'wrong age'),
  linkedin: v(['null', 'string'], 'wrong linkedin'),
  sex: v(
    v.enum(...Object.values(Sex)),
    'wrong sex value'
  ),
  ... v.rest(
    v(
      ['null', 'string'],
      'wrong social networks link'
    )
  ) // Rest props are string | null
})

To any element of the schema, you can add an explanation of the error using the second argument of the compiler function v.

b) Clear the array of explanations.

Before validation, it was necessary to clear this global array in which all explanations were recorded during validation.

v.clearContext() // same as v.explanations = []

c) Validate

const isPersonValid = checkPerson(person)

During this check, if an invalidity was discovered, and at the stage of creating the scheme — it was given an explanation, then this explanation is placed in the global array v.explanation.

d) Error handling

if (!isPersonValid) {
  throw new TypeError('Invalid person response:' + v.explanation.join(';'))
} // ex. Throws ‘Invalid person response: wrong name; wrong age ’

As you can see there is a big problem.
Because if we want to use the validator
is not in the place of its creation, we will need to pass in the parameters not only it, but also the function that created it. Because it is in it that there is an array in which the explanations will be folded.

Solution

This problem was solved as follows: explanations have become part of the validation function itself. What can be understood from its type:

type Validator = (value: any, explanations?: any []) => boolean

Now if you need an explanation of the error, you are passing the array into which you want to add the explanations.

Thus, the validator becomes an independent unit. A method has also been added that can transform the validation function into a function that returns null if the value is valid and returns an array of explanations if the value is not valid.

Now the validation with explanations looks like this:

const checkPerson = v<Person>({
  name: v(v.string, 'wrong name'),
  age: v(v.number, 'wrong age'),
  linkedin: v([null, v.string], 'wrong linkedin')
  sex: v(Object.values(Sex), 'wrong sex')
  [v.rest]: v([null, v.string], 'wrong social network')
})

// ...
const explanations = []
if (!checkPerson(person, explanation)) {
  throw new TypeError('Wrong person:' + explanations.join(';'))
}

// OR

const getExplanation = v.explain(checkPerson)
const explanations = getExplanation(person)
if (explanations) {
  throw new TypeError('Wrong person:' + explanations.join(';'))
}

Afterword

I identified three prerequisites for which I had to rewrite everything:

• The hope that people are not mistaken when writing strings
• Using global variables (in this case, the v.explanation array)
• Check on small examples during development - did not show the problems that arise when used in real large cases.

But I am glad that I conducted an analysis of these problems, and the released version is already used in our project. And I hope it will be useful to us no less than the previous one.

Thank you all for reading, I hope my experience will be useful to you.