DEV Community: Ivan G

Flatten Map Spark Python

Ivan G — Mon, 15 May 2023 15:38:20 +0000

Problem

We have a DataFrame with a map property, looking like this:

root
 |-- t: timestamp (nullable = true)
 |-- tags: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

We want to transform it to the following:

root
 |-- t: timestamp (nullable = true)
 |-- key1: string
 |-- key2: string
 ...
 |-- keyN: string

where key1...keyN are all possible combinations of keys in the map.

Solution

One of the challenges of working with map type data is that it does not have a fixed schema. Unlike other types, such as array or struct, that have a predefined number and name of columns, map can have different keys and values for each record. This means that we cannot easily predict how many columns will be generated when we try to flatten a map type column into multiple columns. For example, if we have a map column that stores the names and ages of different people, one record might have two keys (name and age), while another record might have three keys (name, age, and gender). This makes it difficult to process and analyze the data in a consistent way.

This approach has a limitation: it depends on the current state of the data. If the data changes in the future, the solution might not work anymore. However, this is not a problem for me because I only need to run it once. For those who use Delta Lake, this is also not an issue because Delta Lake can handle schema changes automatically.

Get Unique Keys

One of the first steps in data analysis is to identify the unique values or keys in a dataset. This can help us to understand the structure of the data, as well as to create the necessary flat schema.

The pyspark code for finding the keys in a dataframe is as follows:

map_keys = (df
            .select(f.map_keys("tags").alias("tag_keys"))
            .select(f.explode("tag_keys").alias("tag_keys"))
            .distinct())
map_keys = [x["tag_keys"] for x in map_keys.collect()]

This code operates on a DataFrame named df and performs the following operations:

The select function is used with the map_keys transformation from the pyspark.sql.functions module. It retrieves the keys from the "tags" map column in the DataFrame df and renames the resulting column as "tag_keys".
The explode function is applied to the "tag_keys" column, which transforms each map key into a separate row, effectively "exploding" the map into multiple rows.
The distinct function is used to remove any duplicate rows from the DataFrame.
The resulting DataFrame is assigned to the variable map_keys.
The collect function is called on the map_keys DataFrame, which triggers the execution of the Spark job and returns the results as a list of Row objects.
A list comprehension is used to extract the values of the "tag_keys" column from each Row object in the list, and this list of extracted values is assigned back to the variable map_keys.

In summary, this code extracts the distinct keys from a map column named "tags" in the DataFrame df and stores them as a list of strings in the map_keys variable. You should get a python array similar to this:

['url',
 'b_sys_name',
 'bi',
 'ua',
 'referrer',
 'out',
 'height',
 'b_name',
 'profile',
 'ppn',
 'width',
 'inactive_mins']

One step done.

Generate Flat Schema

To generate flat schema, i'll use python list comprehension again. Honestly, list comprehension is awesome - this is one of the reason I actually started using Python a long time ago. I'm getting of the rails. Ok, the last time:

"Discover the Magic of Python List Comprehension: Unlocking the Key to Efficiently Generating Flat Schemas. Dive Into the World of Simplified Data Processing in Python and Spark, and Uncover Why List Comprehension Has Been the Secret Ingredient That Made Me Fall in Love With Python Many Years Ago."

Back to the topic. Creating flat schema is as easy as this:

xdf = (df
       .select(*df_all.columns[:-1], *[f.col(f"tags.{x}") for x in map_keys]))

This code operates on a DataFrame named df_all and performs the following operations:

The DataFrame df is used as the starting point for the transformation. This is our original data.
The *df.columns[:-1] expression uses the unpacking operator * to unpack all columns except the last one ([:-1]) from the df DataFrame. This ensures that all columns of df except the last one are included in the selection, because the last column tags is the one we don't need to include in the result (we want to flatten it!).
The expression [f.col(f"tags.{x}") for x in map_keys] is a list comprehension that generates a list of column expressions dynamically. For each value x in the map_keys list, it creates a column expression using f.col(f"tags.{x}"). This expression accesses the tags map column in the df DataFrame and retrieves the value associated with the key x.
The resulting list of column expressions is unpacked using the * operator within the select function. This includes the dynamically generated column expressions in the selection alongside the columns from df.columns[:-1].
The resulting DataFrame is assigned to the variable xdf.

This will generate a very wide table (in my case) which looks similar to:

root
 |-- t: timestamp (nullable = true)
 |-- version: string (nullable = true)
 |-- iid: string (nullable = true)
 |-- ip: string (nullable = true)
 |-- event: string (nullable = true)
 |-- user_name: string (nullable = true)
 |-- windows_version: string (nullable = true)
 |-- domain: string (nullable = true)
...

Everything after t is taken from the map keys.

And now you can just execute the query and reap the benefits.

Appendix. Why Flattening can be Beneficial

Flattening nested data structures involves transforming a complex nested structure into a simple one-dimensional list or array. This process can be useful for several reasons:

Simplifying data access: Nested structures can make it challenging to access specific pieces of information. Flattening them makes it easier to access and manipulate individual data points.
Easier data processing: Flattening a nested structure can simplify data processing and analysis by eliminating the need for complex nested loops or recursion.
More efficient storage: Storing nested structures can be memory-intensive, and flattening can help reduce the storage space required by eliminating unnecessary levels of nesting.
Better data visualization: Data that is flattened can be more easily visualized in tables, charts, and graphs, which are often one-dimensional.
Improved performance: In some cases, flattening can improve the performance of an application by reducing the amount of time and resources required to access and process data.
Simplified data integration: When working with data from different sources, flattening nested structures can help in integrating and combining datasets more easily. It allows for a consistent format and facilitates merging data based on common attributes.
Smoother data transformation: Flattening nested structures can streamline the process of data transformation. It enables the application of operations and transformations uniformly across the entire dataset, without needing to account for complex nested hierarchies.
Enhanced compatibility: Flattened data structures are often more compatible with various data analysis and machine learning algorithms. Many algorithms require a tabular or flat structure, and by flattening the data, it becomes readily usable for such algorithms.
Simplified debugging and troubleshooting: Working with flattened data structures can make debugging and troubleshooting easier. It reduces complexity and enables more straightforward identification and resolution of issues related to data quality, consistency, or specific data points.
Improved data sharing: Flattening nested structures can facilitate data sharing with others. It simplifies the process of exchanging data, collaborating on projects, and sharing insights with stakeholders who may not be familiar with complex nested representations.
Easier data export and integration with external systems: Flattened data structures are often more compatible and easier to export or integrate with external systems, such as databases, data warehouses, or reporting tools. It simplifies the process of data migration and integration.

By flattening nested structures, data becomes more accessible, easier to process, and compatible with a wider range of applications and tools. It promotes simplicity, efficiency, and effective data utilization across various domains and use cases.

Overall, flattening nested data structures can make working with complex data sets more manageable and efficient.

By the way, did you know why a data scientist become a pancake chef?

Because they loved flattening data almost as much as they loved flattening pancakes! 🥞😄

References

PEP 448 -- Additional Unpacking Generalizations.
Python Docs: Unpacking Argument Lists.
PySpark functions used:
- select - Selects a subset of columns from a DataFrame.
- map_keys - Extracts the keys from a map column.
- alias - Renames a column.
- select - Selects a subset of columns from a DataFrame.
- map_keys - Extracts the keys from a map column.
- alias - Renames a column.

Should I use Mongo DB API or Cosmos SQL API in Azure Cosmos DB?

Ivan G — Thu, 30 Mar 2023 15:26:46 +0000

It's a question I asked myself for a long time, so I've decided to answer it for myself and for someone else.

What is Cosmos DB?

Cosmos SQL is a data model within Cosmos DB that supports SQL-like queries against JSON data. It provides a way to store and query structured data in a distributed environment, and it can be a good choice for certain types of applications.

Features

Multi-Model Database. Cosmos DB supports multiple data models such as document, key-value, graph, and column-family, allowing developers to use the most appropriate data model for their specific use case.
Globally Distributed. Cosmos DB provides global distribution of data across multiple regions and zones, with automatic replication and failover capabilities. This allows applications to provide low latency and high availability for users around the world.
Automatic Scaling. Cosmos DB provides automatic scaling of throughput and storage, based on the needs of the application. This means that as the application grows, Cosmos DB can automatically scale to handle the increased workload.
Consistency Levels. Cosmos DB provides a range of consistency levels to choose from, allowing developers to balance between data consistency and availability.
Multi-API Support. Cosmos DB supports multiple APIs such as SQL, MongoDB, Cassandra, Azure Tables, and Gremlin, making it easy to use with existing applications and tools.
Security. Cosmos DB provides advanced security features such as encryption at rest and in transit, role-based access control, and virtual network integration.
Analytics and Visualization. Cosmos DB provides built-in analytics and visualization capabilities with Azure Synapse Link, allowing developers to perform real-time analytics on their data.
Developer Productivity. Cosmos DB provides a range of tools and SDKs to make it easy for developers to get started with the service, including Azure Portal, Azure CLI, Azure PowerShell, and various programming language SDKs.

Overall, Cosmos DB provides a comprehensive set of features for building modern, scalable, and highly available applications in the cloud.

Can I Mix and Match APIs?

Yes and No.

Yes

You can use both SQL and MongoDB APIs in Azure Cosmos DB at the same time. Azure Cosmos DB is designed to be a multi-model database service that supports multiple APIs, including SQL, MongoDB, Cassandra, Azure Table, and Gremlin. This means that you can store data using different data models, and access that data using the appropriate API for your application.

For example, you could store structured data in a SQL API container, and unstructured data in a MongoDB API container within the same Cosmos DB account. This provides flexibility in how you structure and access your data, while still taking advantage of the benefits of a globally distributed, multi-model database service.

No

You cannot use different APIs in the same container in Azure Cosmos DB. Each container in Cosmos DB is associated with a specific API and data model, and data stored in a container must conform to that data model.

For example, if you create a container using the MongoDB API, you can only store data in that container using the MongoDB data model and API. Similarly, if you create a container using the SQL API, you can only store data in that container using the SQL data model and API.

Summary

The short answer is no. See expanded answer above. An instance of data is tightly bound to API, and has to be set at the creation time. You cannot change API level later.

When you create a container in Cosmos DB, you choose a specific API and data model that is associated with that container. The data stored in that container must conform to the data model and API that was chosen.

If you need to use a different API for your data, you will need to create a new container with the appropriate API and data model, and then migrate your data to the new container. Depending on the amount of data and the complexity of your application, this migration process can be a complex task.

Which one is Better?

There are some scenarios where using MongoDB in Cosmos DB might be a better choice. One reason is that MongoDB has a more flexible data model than Cosmos SQL. MongoDB supports nested documents and arrays, which can be more difficult to model in Cosmos SQL. Additionally, MongoDB has a richer set of query operators and aggregation functions, which can make it easier to work with complex data.

Another reason to use MongoDB in Cosmos DB is if you're already using MongoDB in your application and want to take advantage of Cosmos DB's global distribution, low latency, and automatic scaling features. By using the MongoDB API in Cosmos DB, you can continue to use the same MongoDB drivers and tools that you're already familiar with, while also gaining the benefits of Cosmos DB's global distribution and scalability.

Ultimately, the choice between using Cosmos SQL or MongoDB in Cosmos DB depends on your specific application requirements and data model. You should evaluate both options to determine which one is the best fit for your needs.

In Layman's Terms, Which One Should I Go For?

Short Answer

MongoDB.

Long Answer

It depends :) See feature comparison table below.

	SQL API	MongoDB API
Data Model	Tabular, column-family	Document-based
Query Language	SQL-like	Document-based
Indexing	Supports indexing on specific columns	Supports indexing on fields within documents
Transactions	Supports multi-document transactions	Supports single-document transactions
Consistency	Provides strong consistency by default	Provides eventual consistency by default
Scalability	Highly scalable, distributed database	Highly scalable, distributed database
Programming Model	Similar to traditional relational databases	Optimized for working with JSON documents

Tools

Cosmos SQL

Azure Portal. The Azure Portal provides a web-based interface for managing and querying data in Cosmos DB. However, it honestly feels clunky and is only suitable for very simple ad-hoc queries.
Cosmos DB SQL Studio. Very simple unofficial extension for VS Code.
Cosmos DB Explorer. A promising Windows Desktop application, however I found it crashing often and in general really unstable. And it's Windows-only.

As a developer who works with Cosmos DB, I have faced many challenges and frustrations with the lack of proper tools for this database service. It's worth noting I have tried really hard to find a good tool for Cosmos DB. There are none, even official ones. And it's been a while since it exists.

These limitations are not insurmountable, but they require extra work and code to overcome. And this is where a good tool for Cosmos DB would come in handy.

Unfortunately, such a tool does not exist yet. The official tools provided by Microsoft are either outdated, incomplete, or buggy. For example, the Azure portal only supports the SQL API and has a limited UI for browsing and querying data. The Azure Data Studio extension only supports the MongoDB API and has a poor performance and user experience. The Azure Cosmos DB Emulator only works on Windows and does not support all the features of the service.

Therefore, I think there is a huge gap in the market for a good tool for Cosmos DB. A tool that would make the life of developers easier and more productive. A tool that would leverage the full potential of Cosmos DB and help to create better applications. A tool that would be updated regularly and supported by a dedicated team.

If you are a developer who works with Cosmos DB, or if you are interested in creating such a tool, please let me know. I would love to hear your thoughts and feedback on this topic.

Mongo

There are hundreds of developer tools available for working with MongoDB, depending on your preferences and development environment. Here are some popular ones:

MongoDB Compass. A graphical user interface for MongoDB. https://www.mongodb.com/products/compass
Robo 3T. A lightweight, open-source MongoDB GUI client. https://robomongo.org/
Studio 3T. A comprehensive MongoDB GUI client with advanced features such as SQL support, aggregation pipeline builders, and query autocompletion. https://studio3t.com/
NoSQLBooster for MongoDB. A smart GUI client for MongoDB with features such as IntelliShell, visual query builder, and schema analyzer. https://nosqlbooster.com/
Aqua Data Studio: A multi-platform database IDE that supports MongoDB, with features such as visual analytics, data import/export, and SQL debugging. From the maker of Cosmos DB but with no Cosmos SQL support :) https://www.aquadatastudio.com/
Navicat for MongoDB. A GUI tool for managing MongoDB databases with features such as data modeling, data synchronization, and query building. https://www.navicat.com/en/products/navicat-for-mongodb
Datagrip. A multi-platform database IDE that supports MongoDB, with features such as code completion, code analysis, and schema editor. https://www.jetbrains.com/datagrip/

These are just a few examples of the many developer tools available for working with MongoDB.

Pricing

It's difficult to compare the average prices of Cosmos DB with SQL API and Mongo API directly because the pricing model for Cosmos DB is based on several factors including the choice of API, data storage, throughput, and regions. However, here are some factors to consider:

Data storage: Both SQL API and Mongo API use the same pricing model for data storage in Cosmos DB, which is based on the amount of data stored per month. The price per GB of data stored is the same regardless of which API is used.
Request units: Request units (RUs) are a measure of the throughput capacity of a Cosmos DB database. Both SQL API and Mongo API use the same pricing model for RUs. The cost of RUs varies depending on the performance level chosen, which is based on the number of RUs per second required for your application.
Regions: The cost of using Cosmos DB also depends on the number of regions selected for your deployment. Each region has its own pricing, so the more regions you select, the higher the cost.

In general, the cost of using SQL API and Mongo API in Cosmos DB will depend on the specific requirements of your application and the amount of data storage and throughput required. It's important to carefully consider your application's needs and compare pricing options before choosing an API.

Verdict - mostly equal.

Is It Faster to Enumerate an Array With "foreach" or "for" in C#?

Ivan G — Thu, 22 Dec 2022 10:10:47 +0000

Here's an interesting question - having an array of integers, is it faster to enumerate it, to perform a simple operation on an element, with for or foreach?

To answer this, first of all I've created a benchmark that covers 3 use cases:

Use for loop and get an element by it's index.
Use foreach on the array.
Use foreach on the array, but before enumerating case the array to IEnumerable forcibly. I don't know why I had a hunch that might behave slightly different, but I did it anyway.

Here is the testing code:

#LINQPad optimize+

void Main()
{
    Util.AutoScrollResults = true;
    BenchmarkRunner.Run<Enumeration>();
}

[ShortRunJob]
[MinColumn, MaxColumn, MeanColumn, MedianColumn]
[MemoryDiagnoser]
[MarkdownExporter]
public class Enumeration
{
    int[] _source;

    [Params(10, 100, 1000, 10000, 100000)]
    public int Length;


    [GlobalSetup]
    public void Setup()
    {
        _source = Enumerable.Range(0, Length).ToArray();
    }


    [Benchmark]
    public void EnumerateAsArray()
    {
        int count = 0;
        for(int i = 0; i < Length; i++) {
            int element = _source[i];
            count += 1;
        }
    }

    [Benchmark]
    public void EnumerateWithForeach()
    {
        int count = 0;
        foreach(int i in _source) {
            count += 1;
        }
    }

    [Benchmark]
    public void EnumerateWithForeachAfterCasting() {
        int count = 0;
        foreach (int i in (IEnumerable)_source) {
            count += 1;
        }
    }
}

And the result:

Method	Length	Mean	Error	StdDev	Min	Max	Median	Gen0	Allocated
EnumerateAsArray	10	3.473 ns	1.4879 ns	0.0816 ns	3.385 ns	3.546 ns	3.490 ns	-	-
EnumerateWithForeach	10	3.081 ns	0.6979 ns	0.0383 ns	3.057 ns	3.125 ns	3.062 ns	-	-
EnumerateWithForeachAfterCasting	10	360.151 ns	112.8064 ns	6.1833 ns	353.018 ns	363.983 ns	363.452 ns	0.0648	272 B
EnumerateAsArray	100	49.681 ns	10.2353 ns	0.5610 ns	49.237 ns	50.311 ns	49.495 ns	-	-
EnumerateWithForeach	100	42.489 ns	15.0294 ns	0.8238 ns	41.728 ns	43.364 ns	42.375 ns	-	-
EnumerateWithForeachAfterCasting	100	3,365.735 ns	1,017.7344 ns	55.7855 ns	3,330.609 ns	3,430.059 ns	3,336.535 ns	0.5798	2432 B
EnumerateAsArray	1000	404.754 ns	101.6160 ns	5.5699 ns	399.244 ns	410.382 ns	404.638 ns	-	-
EnumerateWithForeach	1000	366.522 ns	44.1604 ns	2.4206 ns	364.396 ns	369.157 ns	366.015 ns	-	-
EnumerateWithForeachAfterCasting	1000	34,436.210 ns	16,553.4298 ns	907.3493 ns	33,513.147 ns	35,326.984 ns	34,468.500 ns	5.7373	24032 B
EnumerateAsArray	10000	4,011.964 ns	1,402.6901 ns	76.8862 ns	3,924.527 ns	4,069.007 ns	4,042.358 ns	-	-
EnumerateWithForeach	10000	3,644.260 ns	350.8573 ns	19.2317 ns	3,632.706 ns	3,666.461 ns	3,633.613 ns	-	-
EnumerateWithForeachAfterCasting	10000	340,725.798 ns	170,058.5539 ns	9,321.4832 ns	330,179.199 ns	347,861.084 ns	344,137.109 ns	57.1289	240033 B
EnumerateAsArray	100000	40,460.423 ns	9,494.4089 ns	520.4206 ns	40,090.979 ns	41,055.597 ns	40,234.692 ns	-	-
EnumerateWithForeach	100000	36,107.100 ns	6,022.3198 ns	330.1037 ns	35,901.172 ns	36,487.848 ns	35,932.281 ns	-	-
EnumerateWithForeachAfterCasting	100000	3,892,458.333 ns	4,019,109.7373 ns	220,300.9666 ns	3,652,532.422 ns	4,085,627.734 ns	3,939,214.844 ns	570.3125	2400038 B

The findings are impressive - enumerating an array with for or foreach is not much different, but surprisingly foreach is slightly faster (around 12%). Whereas when casting to IEnumerable and then performing iteration is way way slower than anything else! In fact, it's about 107 times slower than normal array iteration!

To understand what's happening, I'm going to disassembly the generated code.

EnumerateAsArray

This is IL representation:

IL_0000  ldc.i4.0   
IL_0001  stloc.0     // count 
IL_0002  ldc.i4.0   
IL_0003  stloc.1     // i 
IL_0004  br.s  IL_0017 
IL_0006  ldarg.0   
IL_0007  ldfld  Enumeration._source 
IL_000C  ldloc.1     // i 
IL_000D  ldelem.i4   
IL_000E  pop   
IL_000F  ldloc.0     // count 
IL_0010  ldc.i4.1   
IL_0011  add   
IL_0012  stloc.0     // count 
IL_0013  ldloc.1     // i 
IL_0014  ldc.i4.1   
IL_0015  add   
IL_0016  stloc.1     // i 
IL_0017  ldloc.1     // i 
IL_0018  ldarg.0   
IL_0019  ldfld  Enumeration.Length 
IL_001E  blt.s  IL_0006 
IL_0020  ret

As you can see, this is what's happening:

Create two variables - count and i.
Keep comparing array length to i (IL_0017). When i is less than array length, jump to IL_0006.
Load array element.
Increment the count.
Increment i.

Simple enough.

EnumerateWithForeach

IL source:

IL_0000  ldc.i4.0   
IL_0001  stloc.0     // count 
IL_0002  ldarg.0   
IL_0003  ldfld  Enumeration._source 
IL_0008  stloc.1   
IL_0009  ldc.i4.0   
IL_000A  stloc.2   
IL_000B  br.s  IL_0019 
IL_000D  ldloc.1   
IL_000E  ldloc.2   
IL_000F  ldelem.i4   
IL_0010  pop   
IL_0011  ldloc.0     // count 
IL_0012  ldc.i4.1   
IL_0013  add   
IL_0014  stloc.0     // count 
IL_0015  ldloc.2   
IL_0016  ldc.i4.1   
IL_0017  add   
IL_0018  stloc.2   
IL_0019  ldloc.2   
IL_001A  ldloc.1   
IL_001B  ldlen   
IL_001C  conv.i4   
IL_001D  blt.s  IL_000D 
IL_001F  ret

It's not much different to before, but .NET runtime makes slight optimisation, because it knows you are going to need to value and not need the index (foreach always retrieves the value). So that's good!

EnumerateWithForeachAfterCasting

IL source:

IL_0000  ldc.i4.0   
IL_0001  stloc.0     // count 
IL_0002  ldarg.0   
IL_0003  ldfld  Enumeration._source 
IL_0008  callvirt  IEnumerable.GetEnumerator () 
IL_000D  stloc.1   
IL_000E  br.s  IL_0020 
IL_0010  ldloc.1   
IL_0011  callvirt  IEnumerator.get_Current () 
IL_0016  unbox.any  Int32 
IL_001B  pop   
IL_001C  ldloc.0     // count 
IL_001D  ldc.i4.1   
IL_001E  add   
IL_001F  stloc.0     // count 
IL_0020  ldloc.1   
IL_0021  callvirt  IEnumerator.MoveNext () 
IL_0026  brtrue.s  IL_0010 
IL_0028  leave.s  IL_003B 
IL_002A  ldloc.1   
IL_002B  isinst  IDisposable 
IL_0030  stloc.2   
IL_0031  ldloc.2   
IL_0032  brfalse.s  IL_003A 
IL_0034  ldloc.2   
IL_0035  callvirt  IDisposable.Dispose () 
IL_003A  endfinally   
IL_003B  ret

Wow! You don't need to be an IL expect to see this will be slow, because it's full of callvirt calls. Calling virtual methods is expensive! This time you are forcing C# to use IEnumerable pattern, and that's exactly what it does. To represent this code in C# 1, it translates to the following:

int count = 0;
IEnumerator enumerator = ((IEnumerable)_source).GetEnumerator ();
try
{
    while (enumerator.MoveNext ())
    {
        int num = (int)enumerator.Current;
        count++;
    }
}
finally
{
    IDisposable disposable = enumerator as IDisposable;
    if (disposable != null)
    {
        disposable.Dispose ();
    }
}

So a lot of method calls and unboxing, all are extremely expensive.

Summary

If you can, prefer foreach to for loops, they will be slightly faster.
Never cast arrays to IEnumerable and then enumerate, as this expands into a lot of virtual calls using IEnumerator pattern.

Install .NET 6 runtime on clean Alpine Linux

Ivan G — Fri, 23 Sep 2022 11:45:18 +0000

These are step-by-step instructions as they seems to be scattered all over the internet. I will start with a completely clean, stock Alpine distro.

Install

Dependencies

Alpine does not have .NET 6 installers or sudo, so you have to install dependencies as admin first:

isb:/home/alg# apk add bash icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
(1/13) Installing ncurses-terminfo-base (6.3_p20211120-r1)
(2/13) Installing ncurses-libs (6.3_p20211120-r1)
(3/13) Installing readline (8.1.1-r0)
(4/13) Installing bash (5.1.16-r0)
Executing bash-5.1.16-r0.post-install
(5/13) Installing libgcc (10.3.1_git20211027-r0)
(6/13) Installing libstdc++ (10.3.1_git20211027-r0)
(7/13) Installing icu-libs (69.1-r1)
(8/13) Installing krb5-conf (1.0-r2)
(9/13) Installing libcom_err (1.46.4-r0)
(10/13) Installing keyutils-libs (1.6.3-r0)
(11/13) Installing libverto (0.3.2-r0)
(12/13) Installing krb5-libs (1.19.3-r0)
(13/13) Installing libintl (0.21-r0)
Executing busybox-1.34.1-r3.trigger
OK: 49 MiB in 35 packages

.NET 6

.NET 6 itself does not need admin privileges to install. First, download install script:

isb:~$ wget https://dot.net/v1/dotnet-install.sh
Connecting to dot.net (20.84.181.62:443)
Connecting to dotnet.microsoft.com (13.107.246.64:443)
saving to 'dotnet-install.sh'
dotnet-install.sh    100% |************************************************************************| 57823  0:00:00 ETA
'dotnet-install.sh' saved

give it exec permission

isb:~$ chmod +x dotnet-install.sh

and install the latest runtime:

isb:~$ ./dotnet-install.sh -c Current --runtime aspnetcore
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.

dotnet-install: wget extra options are unavailable for this environment
dotnet-install: Attempting to download using aka.ms link https://dotnetcli.azureedge.net/dotnet/aspnetcore/Runtime/6.0.9/aspnetcore-runtime-6.0.9-linux-musl-x64.tar.gz
dotnet-install: wget extra options are unavailable for this environment
Connecting to dotnetcli.azureedge.net (152.199.19.161:443)
saving to '/tmp/dotnet.XXXKEaFfa'
dotnet.XXXKEaFfa     100% |************************************************************************| 37.5M  0:00:00 ETA
'/tmp/dotnet.XXXKEaFfa' saved
dotnet-install: Extracting zip from https://dotnetcli.azureedge.net/dotnet/aspnetcore/Runtime/6.0.9/aspnetcore-runtime-6.0.9-linux-musl-x64.tar.gz
dotnet-install: Installed version is 6.0.9
dotnet-install: Adding to current process PATH: `/home/alg/.dotnet`. Note: This change will be visible only when sourcing script.
dotnet-install: Note that the script does not resolve dependencies during installation.
dotnet-install: To check the list of dependencies, go to https://docs.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
dotnet-install: Installation finished successfully.

At this point .net is fully installed.

Add to path

Alpine is using ash shell, so you need to create .profile in your home folder and add the following:

export DOTNET_ROOT=$(pwd)/.dotnet
export PATH=$PATH:$DOTNET_ROOT

Now reload the shell and you're done.

Short way

Run as su

su
apk add bash icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib
exit

Run as user

wget https://dot.net/v1/dotnet-install.sh
chmod +x dotnet-install.sh
./dotnet-install.sh -c Current --runtime aspnetcore
vi .profile
# paste 2 lines:
# export DOTNET_ROOT=$(pwd)/.dotnet
# export PATH=$PATH:$DOTNET_ROOT
source .profile

Stress Testing with cURL

Ivan G — Thu, 15 Sep 2022 12:06:11 +0000

There are plenty of tools for stress testing, read RapidAPI, paw, SoapUI, Postman, rest-assured, JMeter and so on! I'm sure they are amazing, however that's all big and heavy, slow, sometimes paid tools!

If all you need is just simple (or not, depending on what you want) stress testing, cURL is your hidden gem! cURL works pretty much everywhere, and even Windows has curl installed out of the box!

To remind, to issue a simple cURL request, just issue it like curl http://localhost/path1.

A bit more

What if you need to test path1, path2 and path3? Easy:

curl http://localhost/path{1,2,3}

that will issue:

http://localhost/path1
http://localhost/path2
http://localhost/path3

that can be also replaced with curl http://localhost/path[1-3] to the same effect, i.e. you can use ranges!

Ranges can be mixed i.e.

curl http://localhost/path{1,2,3}/subpath[1-3]

will issue

http://localhost/path1/subpath1
http://localhost/path1/subpath2
http://localhost/path1/subpath3
http://localhost/path2/subpath1
http://localhost/path2/subpath2
http://localhost/path2/subpath3
http://localhost/path3/subpath1
http://localhost/path3/subpath2
http://localhost/path3/subpath3

you can also use letters i.e. http://localhost/path-[a-z] will issue queries from http://localhost/path-a to http://localhost/path-z (26 requests total).

That is already cool!

All the requests are executed sequentially so if that's ok with you (and most of the times it is) you should be fine.

Serious offense

Ok, what about a parallel load? xargs will come to rescue! Specifically the multi-threaded version. As you should know, xargs (stands for extended arguments) allows to fork execution from an input to an execution per line. Combined with seq you can start an offensive 😜seq can generate input you want (numbers only) i.e. seq 1 5 will produce 5 lines:

then piping this into xargs allows 5 executions i.e. seq 1 5 | xargs -I{} curl http://localhost/path{} will invoke sequentially:

http://localhost/path1
http://localhost/path2
http://localhost/path3
http://localhost/path4
http://localhost/path5

but passing -P will allow you to paralellise requests! So this is how you issue 500 requests with 5 always going in parallel:

seq 1 500 | xargs -P5 -I{} curl http://localhost/path{}

If you need to absolutely exhaust the target, pass a higher number into -P, or to max it out pass 0 as parallelism. From official man pages:

If max-procs is 0, xargs will run as many processes as possible at a time.

You do lose some ability of more powerful cURL ranges, especially with characters, as seq only produces numbers. However even this can be fixed.

seq supports -f switch allowing you to pass a custom format string, however that's not very useful because you are already formatting with xargs (-I{})

You can do even better - generate a lot of word permutations! Take crunch for example (may need to install it i.e. apt install crunch) and generate words 5 character length:

crunch 5 5 -o 5words.txt

The file is quite large (67mb) and takes a few seconds to generate, hence we save it locally. Then take this "dictionary" as an input of a stress test:

cat 5words.txt | xargs -n1 -P5 -I{} curl http://localhost/words/{}

That will issue 11881376 (11 million!) requests in parallel! More than enough if you ask me.

So is there a need for a dedicated stress tool? I don't think so.

PowerShell Friends

1..10 | % -Parallel { curl http://localhost/path$_ }

1..10 is a "range operator", equivalent to seq 1 10 (at least something is shorter in PS ;)

% is an alias for Foreach-Object which can run in parallel mode optionally.

-Parallel flag is only available in PowerShell 7 and higher. One can use -ThrottleLimit parameter to further limit number of parallel "threads", otherwise it's unlimited (equivalent to xargs' -P)

Ranges in PowerShell can also do letters, but it's not obvious - [char]'a'..[char]'z' will produce

a
b
c
d
e
f
g
h
i
j
k
l
m
n
o
p
q
r
s
t
u
v
w
x
y
z

You can do a reverse range by doing 5..1 which produces

Scala Mind Map

Ivan G — Thu, 14 Jul 2022 11:16:00 +0000

This post has no words but mostly a mind map I've created over years when learning Scala. It's serves as a quick reminder for majority of Scala features. You can zoom it indefinitely thanks for it being exported in SVG.

P.S. Watch this page for further updates.

Face Detection with OpenCV (for complete Dummies)

Ivan G — Wed, 23 Mar 2022 15:59:50 +0000

This is a very dumb and straightforward tutorial that is meant to be a complete newbie intro into face detection. I'll be using OpenCV for this, and python for simplicity, however if you want to scale you might want to rewrite this in C++. Python is great for prototyping so I'll stick with this.

Install Dependencies

You can grab opencv-python from pypi - this is an unoptimised version that works on CPU - more than enough for prototyping.

Write Code

First, import OpenCV and load the image you want to work with, optionally resizing it. You need to grayscale the image before applying further classification.

import cv2 as cv

img1 = cv.imread("band-small-faces.jpg")

# resize if needed
# img2 = cv.resize(img1, None, fx=0.3, fy=0.3, interpolation=cv.INTER_CUBIC)
img2 = img1  # not resizing in this case
img3 = cv.cvtColor(img2, cv.COLOR_BGR2GRAY)

This is the image provided as input:

Then, you need something called a Haar classifier that does the magic:

haar_cascade_face = cv.CascadeClassifier("./data/haarcascades/haarcascade_frontalface_default.xml")

As an input it needs a model to work with. I'm passing a model that detects frontal faces. I've attached available models for you to download (apparently you need to create data/haarcascases folder where your python script is).

If you get an error similar to

cv2.error: OpenCV(4.5.5) ...opencv-python\opencv-python\opencv\modules\objdetect\src\cascadedetect.cpp:1689: error: (-215:Assertion failed) !empty() in function 'cv::CascadeClassifier::detectMultiScale'

it usually means that path to the xml is invalid (thanks for useful messages!).

And to display model results:

for (x, y, w, h) in faces_rects:
    cv.rectangle(img2, (x, y), (x + w, y + h), (0, 255, 0), 3)

cv.imshow("Display", img2)
k = cv.waitKey(0)

As you can see, most of the faces were detected! Last person's leg looks like a face (you need to play with model parameters to rule this out). Last face wasn't detected because it's not frontal! Maybe use another model if you need to detect more.

Same way you can detect other things with OpenCV, just supply another pre-trained model. For instance, face expressions, eyes, eyeglasses, bodies, plate numbers etc.

Recruiters! We Were Always Remote!

Ivan G — Thu, 17 Mar 2022 11:42:38 +0000

As COVID-19 is ending/easing, recruitment companies are trying to lower salaries / day rates by justifying the position is now "mostly remote" or "remote". This is bullshit.

We were always working remote.

Taking away most of the commute does not make it remote.

Pre-covid we used to waste a big chunk of our day but going to the office, getting tired before work starts, and getting into uncomfortable workplace.

We put noise-cancelling headphones on to block the office environment.

We spent majority of meetings online on Teams, Slack or whatever the corporate messaging platform is.

We work during commute whenever possible in terrible conditions - uncomfortable busy trains, buses etc. most of the time not being able to. And if we can't, we would use our phones to reply to emails and instant messaging apps to help with customer requests. How the hell is this not remote?

And then coming home late mostly to get to bed and wake up to commute again.

So what's the cost saving for us, developers, by working "remote"?

We benefit from it mostly by saving on mental health, and I can't think of more to be honest. There are side benefits as well for spending more time with families (whoever has one) and self-care. But who are really benefitting from this is companies we work for - now they need to pay zero costs for our gym memberships and other benefits we take to stay sane.

What are the other benefits companies have?

Developers stay focused and productive, as they are free to set up their own workplace in a way they want.

If you as a company used to pay for commute, now this is gone too.

Most of the time developers do work longer hours than usual, because they can. This crowd is not going to work to clock hours, most of us are working the job because we*genuinely like it*. It's usual to see people working much longer hours now, and not being tired.

Then, office space is expensive, free coffee is expensive, and all the small things combined like office supplies, power, hardware, furniture, you name is adding up to quite a bit. And you need to hire people and pay them salaries to run the office. Extra people are very expensive.

But you also get extra productivity i.e. free developer hours which are really really expensive.

And I've just started this morning rant and given more time can find many more reasons.

Therefore think again, do you still want to offer me lower salary? You better do the job yourself. Fully remote.

We need to rethink this as a community, and raise prices for our services as companies are now getting more in return than usual.

C++: lvalue/rvalue for Complete Dummies

Ivan G — Mon, 06 Dec 2021 22:28:54 +0000

I find the concepts of lvalue and rvalue probably the most hard to understand in C++, especially after having a break from the language even for a few months. So this is an attempt to keep my memory fresh whenever I need to come back to it. This topic is also super essential when trying to understand move semantics.

Essentials

lvalue

There are plenty of resources, such as value categories on cppreference but they are lengthy to read and long to understand. In general, lvalue is:

Is usually on the left hand of an expression, and that's where the name comes from - "left-value".
Something that points to a specific memory location. Whether it's heap or stack, and it's addressable.
Variables are lvalues, and usually variables appear on the left of an expression. For example in an expression int x = 1; x is lvalue. x is also pointing to a memory location where value 1 is.
- Another example is int* y = &x. In this case y is lvalue as well. x is also lvalue, but &x is not!
Early definitions of lvalue meant "values that are suitable fr left-hand-side or assignment" but that has changed in later versions of the language. For instance const int a = 1; declares lvalue a but obviously it cannot be assigned to, so definition had to be adjusted. Later you'll see it will cause other confusions!

Some people say "lvalue" comes from "locator value" i.e. an object that occupies some identifiable location in memory (i.e. has an address).

rvalue

rvalue is something that doesn't point anywhere. The name comes from "right-value" because usually it appears on the right side of an expression.
It is generally short-lived. Sometimes referred to also as "disposable objects", no one needs to care about them.
rvalue is like a "thing" which is contained in lvalue.
Sometimes rvalue is defined by exclusion rule - everything that is not lvalue is rvalue.
lvalue can always be implicitly converted to lvalue but never the other way around.
rvalue can be moved around cheaply

Coming back to express int x = 1;:

x is lvalue (as we know it). It's long-lived and not short-lived, and it points to a memory location where 1 is. The value of x is 1.
1 is rvalue, it doesn't point anywhere, and it's contained within lvalue x.

Here is a silly code that doesn't compile:

int x;
1 = x; // error: expression must be a modifyable lvalue

which starts making a bit more sense - compiler tells us that 1 is not a "modifyable lvalue" - yes, it's "rvalue". In C++, the left operand of an assignment must be an "lvalue". And now I understand what that means.

A definition like "a + operator takes two rvalues and returns an rvalue" should also start making sense.

"A useful heuristic to determine whether an expression is an lvalue is to ask if you can take its address. If you can, it typically is. If you can't, it's usually an rvalue."

Effective Modern C++

References

Are references lvalues or rvalues? General rule is:

lvalue references can only be bound to lvalues but not rvalues

assumes that all references are lvalues.

In general, there are three kinds of references (they are all called collectively just references regardless of subtype):

lvalue references - objects that we want to change.
const references - objects we do not want to change (const references).
rvalue references - objects we do not want to preserve after we have used them, like temporary objects. This kind of reference is the least obvious to grasp from just reading the title.

The first two are called lvalue references and the last one is rvalue references.

As I said, lvalue references are really obvious and everyone has used them - X& means reference to X. It's like a pointer that cannot be screwed up and no need to use a special dereferencing syntax. Not much to add.

One odd thing is taking address of a reference:

int i = 1;
int& ii = i;    // reference to i
int* ip = &i;   // pointer to i
int* iip = &ii; // pointer to i, equivent to previous line

Basically we cannot take an address of a reference, and by attempting to do so results in taking an address of an object the reference is pointing to.

Another weird thing about references here. To initialise a reference to type T (T&) we need an lvalue of type T, but to initialise a const T& there is no need for lvalue, or even type T! For const references the following process takes place:

Implicit type conversion to T if necessary.
Resulting value is placed in a temporary variable of type T.
Temporary variable is used as a value for an initialiser.

To demonstrate:

int& i = 1;        // does not work, lvalue required
const int& i = 1;  // absolutely fine
const int& i {1};  // same as line above, OK, but syntax preferred in modern C++

Now it's the time for a more interesting use case - rvalue references. Starting to guess what it means and run through definition above - rvalue usually means temporary, expression, right side etc. Rvalue references are designed to refer to a temporary object that user can and most probably will modify and that object will never be used again.

A classic example of rvalue reference is a function return value where value returned is function's local variable which will never be used again after returning as a function result. It's completely opposite to lvalue reference:

rvalue reference can bind to rvalue, but never to lvalue

Rvalue reference is using && (double ampersand) syntax, some examples:

string get_some_string();
string ls {"Temporary"};

string&& s = get_some_string();  // fine, binds rvalue (function local variable) to rvalue reference
string&& s { ls };              // fails - trying to bind lvalue (ls) to rvalue reference
string&& s { "Temporary" };      // fails - trying to bind temporary to rvalue reference

You could also thing of rvalue references as destructive read - reference that is read from is dead. This is great for optimisations that would otherwise require a copy constructor.

Going Deeper

Newest versions of C++ are becoming much more advanced, and therefore matters are more complicated. Generally you won't need to know more than lvalue/rvalue, but if you want to go deeper here you are.

So, there are two properties that matter for an object when it comes to addressing, copying, and moving:

Has Identity (I). The program has the name of, pointer to, or reference to the object so that it is possible to determine if two objects are the same, whether the value of the object has changed, etc.
Moveable (M). The object may be moved from (i.e., we are allowed to move its value to another location and leave the object in a valid but unspecified state, rather than copying).

Cool thing is, three out of four of the combinations of these properties are needed to precisely describe the C++ language rules! Fourth combination - without identity and no ability to move - is useless. Now we can put it in a nice diagram:

So, a classical lvalue is something that has an identity and cannot be moved and classical rvalue is anything that we allowed to move from. Others are advanced edge cases:

prvalue is a pure rvalue.
grvalue is generalised rvalue.
xvalue is extraordinary or expert value - it's quite imaginative and rare.

Usually std::move(x) is an xvalue, like in the following example:

void do_something(vector<string>& v1)
{
    vector<string>& v2 = std::move(v1);
}

It both has an identity as we can refer to it as v1 and we allowed it to be moved (std::move). It's still really unclear in my opinion, real headcracker I might investigate later.

But below statement is very important and very true:

For practical programming, thinking in terms of rvalue and lvalue is usually sufficient. Note that every expression is either an lvalue or an rvalue, but not both.

The C++ Programming Language

Reference Geek-Out

If you take a reference to a reference to a type, do you get a reference to that type or a reference to a reference to a type? And what kind of reference, lvalue or rvalue? And what about a reference to a reference to a reference to a type?

using rr_i = int&&;     // rvalue reference
using lr_i = int&;      // lvalue reference

using rr_rr_i = rr_i&&;  // int&&&& is an int&&
using lr_rr_i = rr_i&;   // int&&& is an int&
using rr_lr_i = lr_i&&;  // int&&& is an int&
using lr_lr_i = lr_i&;   // int&& is an int&

Meaning the rule is simple - lvalue always wins!. This is also known as reference collapse.

What is int*&? It's a reference to a pointer.

References

C++ 17: IF and SWITCH With Initialisation

Ivan G — Thu, 02 Dec 2021 16:35:02 +0000

In C++ 17, if statement can have an initialisation clause i.e. instead of

if(condition) {}

this:

if(init; condition) {}

Why? I could just write

init;
if(condition) {}

and it's almost the same. Initialisation clause is valid until the ned of the if part. To demonstrate:

if (int i = 1; i == 0)
{
    // OK
    cout << "if 1" << endl;
}
else
{
    // OK
    cout << "else:" << i << endl;
}

// left the scope, so the following line won't compile:
cout << i << endl;

This also means RAII applies to the initialisation phase i.e. if init instantiates a class on stack, a destructor will be called.

This is useful in many scenarios, for instance guard initialisation:

if (std::lock_guard<std::mutex> g{mtx}; !coll.empty()) {
    // safely use the collection
}

You can achieve the same in pre C++17 codebase with the following code, which is equivalent to above:

{
    std::lock_guard<std::mutex> g{mtx};

    if (!coll.empty()) {
        // safely use the collection
    }
}

it's just syntaxically longer.

Important note - the object in initialisation clause must have a name, and if it doesn't, it's automatically created and destroyed before the first statement inside if executes. Basically, it's destroyed as soon as possible, so no resources are wasted. It would make sense, as you are not referencing it.

Same goes for switch statement, absolutely no differences from if:

switch(init; condition) {}

C++ Structured Binding - From Zero to Hero

Ivan G — Wed, 01 Dec 2021 14:00:37 +0000

Structured binding allows to initialise multiple entities by members of another object, for instance:

struct stb_node
{
    int index = 0;
    string value;
};

stb_node node1{1, "First"};
cout << node1.index << ", " << node1.value << endl;

auto [u, v] = node1;
cout << u << ", " << v << endl;

Produces output:

1, First
1, First

When you break in a debugger, it shows that there are two local variables int u and string v created. u and v are called structural bindings. The purpose of structural bindings is to make code more readable by binding the value directly to names.

This becomes more apparent when working with maps:

map<int, string> m = ...;

// pre - C++17 way
for (const auto& el : m)
{
    cout << el.first << ", " << el.second << endl;
}

// C++17 way
for (const auto& [index, value] : m)
{
    cout << index << ", " << value << endl;
}

In the second syntax option, the code is carrying the context that it's working with index and value, unlike el.first and el.second which doesn't make sense. Of course you can create references to key and value i.e. auto& index = el.first (and that's what I often do to improve readability) but that makes code more verbose.

Note that decltype(index) is int and decltype(value) is string - there is no special magic types involved, these are just other names for struct members.

Also, as with normal assignment, auto [u, v] = node1 creates a copy of struct members, with all the consequences i.e. modifying u modifies a copy of u.

Qualifiers

To avoid copying and enforce constant expressions, you can use qualifiers, for instance:

const auto [u1, v1] = node1;
u1 = 1;

fails on line 2 when trying to modify a copy of node1.index.

const auto& [u2, v2] = node1;
u2 = 2;

fails on line 2 when trying to modify a reference of node1.index.

More Usages

In general, structured bindings can be used with:

Structs with public data members as above.
Raw C-style arrays.
"Tuple-like" objects.

For structs or classes, the decomposed members must be members of the same class definition. Inheritance is not supported. So this won't work:

struct B {
    int a = 1;
    int b = 2;
};

struct C : B {
    int c = 3;
}

auto [i, j, k] = C{};   // won't work, gives an error "type "C" has no components to bind to"

For arrays, binding only works for arrays of known size:

int arr[] = { 1, 2 };
auto [e1, e2] = arr;    // OK
auto [e1] = arr;        // won't compile, gives an error "there are more elements than there are binding names"

std::tuple decomposes nicely i.e.

std::tuple<char,float,std::string> do_something();
// ...
auto [a,b,c] = do_something();

std::pair is just like before, and is logically a subset of tuple.

Redefining Values

Looking at the example before, how do you call do_someting again and assign to a, b, c? You can't. However, if the types match, use std::tie:

std::tuple<char, float, std::string> do_something();
// ...
auto [a, b, c] = do_something();
// redefine:
std::tie(a, b, c) = do_something();

Of course you could just introduce new set of variables, however you wouldn't be able to do that in a loop, like:

std::boyer_moore_searcher bm{sub.begin(), sub.end()};
for (auto [beg, end] = bm(text.begin(), text.end()); beg != text.end(); std::tie(beg,end) = bm(end, text.end())) {
    // ...
}

Thoughts on Rust at First, but then Ramblings on Top Programming Languages

Ivan G — Wed, 29 Sep 2021 15:03:39 +0000

Below post has started from me thinking about Rust on a weekend, and then expanded a bit. I was merely trying to understand whether C/C++ will die eventually and be replaced with something like Rust.

First of all, Rust is great, good performance, interesting language, more accessible than C.

However, thinking more about it - forces developer to start from clean slate.

There is no interop with C++. There is with C, meaning that you can wrap C++ in C calls, make a dynamic library and only then call it from Rust. Wrapping is not always a good idea, as C is unsafe and C++ is safe, plus you will have to translate some rich C++ data structures into C, meaning performance overhead. Also this prevents overall program compiler optimisation, meaning again performance loss. Also you will have to wrap C interface from Rust, again another performance loss.

System programming close to metal and OS is primarily done in C, so Rust is a very weird citizen there. Like Linux and Windows is all C, and according to various sources there is no vision to use Rust.

The promised safety is really arguable. Usually you will hear arguments that Rust is safer than C++. But that’s just not true for Modern C++. It could be true when comparing Rust to the state of C++ maybe 20 years ago.

New successful languages like TypeScript, Scala, Kotlin all have one thing is n common - they are built on heritage and are completely comfortable in their area. I mean for example TypeScript is actually JavaScript on steroids. You can take js file, change extension, and that will be a valid TypeScript program. It’s not ideal, but it will work. Apparently you can also use any JavaScript library from TypeScript and vice versa. You can mix and match in one program. This is big.

Scala is another example, as it builds on JVM, can use any native Java library, and can be compiled to use from classic Java as well! Same with Kotlin. And the wast amount of platforms JVM runs on.

Python is… well, still good. Most of Python libraries are actually written in C++ with python wrapping on top. Python is old and well developed. It even runs on JVM.

Golang is just golang, it does not claim to be a replacement for anything, bit fits well into the area of writing small simple utilities for command line where others would be too heavy. Some do write more complicated code, for instance Docker is done in Go, however it is not common. You would not write anything performance critical in Go either, mostly due to unpredictable garbage collection issues. And it’s not powerful enough to scale to really complex code. Remember that Docker is essentially an orchestrator gluing together different parts of OS to make it more accessible to mortals. It does not actually run when container runs.

C# is a weird beast. It builds on top of Java principles, but doesn’t run on JVM. It’s really way ahead of Java in terms of features and productivity, but used to only compile for Windows. Today cross platform support is great, but it may be too late for the party. It may end up like Windows Phone, it’s hard to tell. I hope not. Btw Windows Phone was way more superior than iOS and Android, but that didn’t matter at the end. It was just really late to the party.

Where does Rust leave us? I’m really confused, but it seems nowhere. Yes ecosystem is growing and there is just a lot of good stuff. But always almost enough, not completely enough. Sure if you are building software that is a utility or web server it’s ok to use, but not something really deep, no way to reuse rock solid code that worked for decades already. You’ll have to create some weird workarounds or write stuff that existed for decades from scratch. Scratch that.

Rust is not supported on as many platforms as C/C++ is. Meaning embedded and IoT is not fully accessible.

Tooling is just not there. There’s no cool IDEs like Visual Studio (which btw works perfect for Linux programming) or CLion or Eclipse. We have some half baked support in IDEA, Vs code and some others. But no really cool tooling C++ has grown to.

Do I hate Rust? Oh no, not at all. I think it’s a beautiful language. Very clear, with no legacy like C++.

Is it superior to C++? No, I don’t think so. Everything Rust does C++ does, if you use more or less recent C++ version. Even 10 year old C++ will do. But, the problem with C++ is novice programmers - the language is just so rich, it’s much easier to escape out of the sage box and do something stupid. But that’s just language evolution, eventually Rust will be there too as history shows.

Would I choose Rust for my next project? It depends. If I need to interface with OS or use third party C/C++ libraries than no. I would probably use it for command line utils or self sufficient cross platform network utilities. But then why won’t I just stick with C++ if I already use it? So far only as a learning exercise.

Well, I think I’ll stick with C++, but I’m open for criticism.

P.S. Originally published on my blog.