xy2xy - yandex inner technologies analog list

Kirill Sulim — Thu, 11 Feb 2021 08:13:32 +0000

TLDR: xy2xy is repository with list of yandex inner technologies and its analogs

xg2xg - a list of inner google technologies

Couple of years ago I found xg2xg repository on github.
I was fascinated how many projects there is inside google.

This summer because of COVID-19 I've had more free time than usual and decided to create similar list for Yandex inner technologies.
I've tried to find it, but there wasn't such list, so I've created xy2xy repository to fill it with Yandex inner technologies list.
I was interested why there is no list of yandex technologies.
I've checked articles from yandex official blog and repositories in yandex github page and collect all interesting technologies in this list.
And that was the second time to be fascinated.
Yandex have a lot of open-source projects I didn't hear of.

Markdown tables

At the start of my project I wanted to use .md syntax for tables as it is made such way in xg2xg.
But after I've added a dozen of technologies to my list I understand: my .md file is a mess because of a big amount of information in each cell.
I've tried to switch to HTML syntax for tables and this partially solve this issue.
My file was bigger, but I was able to understand what is the data in my table.
Then I've added commentary with column name to my <td> tag, and it resolves the issue.
Now I always understand what column I edit.

So I recommend to use HTML for all non-trivial markdown tables.
Those tables will be bigger, but it will also increase readability of source code.

For who is xy2xy?

There are 3 types of people that list can help:

1. Ex Yandex developers

If you've used some inner Yandex technology to solve some tasks, and then you have similar tasks outside Yandex you have to look for an alternative.
You can use open-source project or commercial solution instead of inner Yandex technology.
And this list can help you find alternatives.

2. Experienced developers who are starting to work at Yandex

You can use open-source technologies in Yandex of course, but in many cases inner technologies are better integrated in infrastructure.
So try to use inner technologies if you can.
xy2xy will help you find Yandex alternatives to technologies you have experience with.

3. Developers who want to work at Yandex

When I was starting my career I was interested in technologies of big companies.
I've read a lot about best practices of software development.
For example, I learned about continuous integration and continuous delivery and Jenkins as an example of open-source technology which provides it.
Also, I've read about Map-Reduce technology and what tasks you can solve with it.
In the log run I all helped me to understand what should I learn to be hired as a developer in big tech company.

If you learn open-source analogs of inner Yandex technologies you are interested in your knowledge will help you with your career in Yandex or any big tech company.

What about NDA?

There is a simple rule to check if inner technology is part of NDA or not.
If there is post which describe technology on official company blog or YouTube channel or technology is presented in official github repository
that means the company itself do not consider this technology as a part of NDA and you can add it to open-source list.

Contribution

If you are still interested please visit xy2xy page.
Feel free to add any inner Yandex technology which is not a part of NDA.
Please follow contribution guidelines of the xy2xy repository.

Strict YAML deserialization in Python with marshmallow

Kirill Sulim — Wed, 27 Jan 2021 19:19:14 +0000

The problem

I want to read some not so simple config from .yaml file.
I have config structure described as dataclasses.
I want to all type checks have been performed and in case of invalid data exception will be raised.

So basically I want something like

def strict_load_yaml(yaml: str, loaded_type: Type[Any]):
    """
    Here is some magic
    """
    pass

And then use it like this:

@dataclass
class MyConfig:
    """
    Here is object tree
    """
    pass

try:
    config = strict_load_yamp(open("config.yaml", "w").read(), MyConfig)
except Exception:
    logging.exception("Config is invalid")

Config classes

Here is my config.py file with example dataclasses:

from dataclasses import dataclass
from enum import Enum
from typing import Optional


class Color(Enum):
    RED = "red"
    GREEN = "green"
    BLUE = "blue"


@dataclass
class BattleStationConfig:
    @dataclass
    class Processor:
        core_count: int
        manufacturer: str

    processor: Processor
    memory_gb: int
    led_color: Optional[Color] = None

Solution that didn't work

This is a very common pattern, right?
It must be very easy.
Just import standard yaml library and problem solved?

So I imported PyYaml and call load method:

from pprint import pprint

from yaml import load, SafeLoader


yaml = """
processor:
  core_count: 8
  manufacturer: Intel
memory_gb: 8
led_color: red
"""

loaded = load(yaml, Loader=SafeLoader)
pprint(loaded)

and I have got:

{'led_color': 'red',
 'memory_gb': 8,
 'processor': {'core_count': 8, 'manufacturer': 'Intel'}}

Yaml loaded just fine, but it is a dict.
No problem, I can pass it as **args constructor:

parsed_config = BattleStationConfig(**loaded)
pprint(parsed_config)

and result will be:

BattleStationConfig(processor={'core_count': 8, 'manufacturer': 'Intel'}, memory_gb=8, led_color='red')

Wow! Easy! But... Wait. Is processor field a dict? Damn it.

Python don't perform type checking at constructor and do not parse Processor class.
Well, this is the time to go to stackowerflow.

Solution that required yaml tags and almost works

I've read stackowerflow answers and PyYaml documentation and have found out that you can mark your yaml doc with tags for types.
Your classes must be descendants of YAMLObject and so my config_with_tag.py will look like this:

from dataclasses import dataclass
from enum import Enum
from typing import Optional

from yaml import YAMLObject, SafeLoader


class Color(Enum):
    RED = "red"
    GREEN = "green"
    BLUE = "blue"


@dataclass
class BattleStationConfig(YAMLObject):
    yaml_tag = "!BattleStationConfig"
    yaml_loader = SafeLoader

    @dataclass
    class Processor(YAMLObject):
        yaml_tag = "!Processor"
        yaml_loader = SafeLoader

        core_count: int
        manufacturer: str

    processor: Processor
    memory_gb: int
    led_color: Optional[Color] = None

And loading code:

from pprint import pprint

from yaml import load, SafeLoader

from config_with_tag import BattleStationConfig


yaml = """
--- !BattleStationConfig
processor: !Processor
  core_count: 8
  manufacturer: Intel
memory_gb: 8
led_color: red
"""

a = BattleStationConfig

loaded = load(yaml, Loader=SafeLoader)
pprint(loaded)

And what I will get?

BattleStationConfig(processor=BattleStationConfig.Processor(core_count=8, manufacturer='Intel'), memory_gb=8, led_color='red')

Good. But my YAML is full of tags and lost its readability. And Color is still string. So I can just add YAMLObject to parent classes? Right? No.

class Color(Enum, YAMLObject):
    RED = "red"
    GREEN = "green"
    BLUE = "blue"

Will lead to:

TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

I didn't find a quick way to resolve it. And I did want to add tags to my yaml, so I've decided to keep looking for a solution.

Solution with marshmallow

I found a recommendation to use marshmallow to parse dict generated from JSON object.
I decided that these cases are the same as mine only uses JSON instead of YAML.
And so I tried to use class_schema generator for dataclass schema:

from pprint import pprint

from yaml import load, SafeLoader
from marshmallow_dataclass import class_schema

from config import BattleStationConfig


yaml = """
processor:
  core_count: 8
  manufacturer: Intel
memory_gb: 8
led_color: red
"""

loaded = load(yaml, Loader=SafeLoader)
pprint(loaded)

BattleStationConfigSchema = class_schema(BattleStationConfig)

result = BattleStationConfigSchema().load(loaded)
pprint(result)

And I get:

marshmallow.exceptions.ValidationError: {'led_color': ['Invalid enum member red']}

So, marshmallow wants enum name, not value. I can change my yaml to:

processor:
  core_count: 8
  manufacturer: Intel
memory_gb: 8
led_color: RED

And I will get my ideally deserialized object:

BattleStationConfig(processor=BattleStationConfig.Processor(core_count=8, manufacturer='Intel'), memory_gb=8, led_color=<Color.RED: 'red'>)

But I felt there was a way to use my original yaml.
So I've explored marshmallow documentation and found following lines:

Setting by_value=True. This will cause both dumping and loading to use the value of the enum.

Turn out, you can pass this configuration to metadata dictionary of field generator from dataclasses like this:

@dataclass
class BattleStationConfig:
    led_color: Optional[Color] = field(default=None, metadata={"by_value": True})

And I will get the object parsed from my original yaml.

Magic function

And after all I can collect my magic function:

def strict_load_yaml(yaml: str, loaded_type: Type[Any]):
    schema = class_schema(loaded_type)
    return schema().load(load(yaml, Loader=SafeLoader))

This function can require additional set up for dataclass but solve my problem and do not require tags in yaml.

Some words about ForwardRef

If you define your dataclasses with forward reference (string with class name) marshmallow can be confused and didn't parse your classes.

For example this configuration

from dataclasses import dataclass, field
from enum import Enum
from typing import Optional, ForwardRef


@dataclass
class BattleStationConfig:
    processor: ForwardRef("Processor")
    memory_gb: int
    led_color: Optional["Color"] = field(default=None, metadata={"by_value": True})

    @dataclass
    class Processor:
        core_count: int
        manufacturer: str


class Color(Enum):
    RED = "red"
    GREEN = "green"
    BLUE = "blue"

will lead to

marshmallow.exceptions.RegistryError: Class with name 'Processor' was not found. You may need to import the class.

And if we move Processor class upper marshmallow will lost Color with the same error.
So keep your classes without ForwardRef if possible.

Code

All code available on GitHub repository.

DEV Community: Kirill Sulim