DEV Community: Mark Nefedov

Debugging Containers Without Shell Access: Quick Tips

Mark Nefedov — Sat, 25 Jan 2025 16:31:35 +0000

1. Run Commands Inside the Container’s Namespace

Every container runs in isolated namespaces. Use nsenter to "enter" these namespaces from the host:

# Get the container’s PID (Docker example)
PID=$(docker inspect -f '{{.State.Pid}}' your_container)  

# Inspect processes inside the container’s PID namespace
sudo nsenter -p -t $PID ps aux  

# Check network sockets in the container’s network namespace
sudo nsenter -n -t $PID ss -tnlp

Replace ps or ss with any host-installed tool (e.g., tcpdump, strace).

2. Access Container Files via `/proc`

Container filesystems are mounted under /proc/$PID/root:

# View container files
ls /proc/$PID/root/etc/nginx/  

# Edit configs directly from the host
vim /proc/$PID/root/app/config.yaml

Why This Works

https://www.man7.org/linux/man-pages/man7/namespaces.7.html

Understanding {:via, Registry, {...}} vs Registry.register/3 in Elixir

Mark Nefedov — Sun, 03 Nov 2024 17:55:33 +0000

Understanding `{:via, Registry, {...}}` vs `Registry.register/3` in Elixir

If you've worked with processes and registries in Elixir, you've likely encountered both the {:via, Registry, {...}} syntax and the Registry.register/3 function. At first glance, these two may appear to do the same thing — allowing you to register processes with a registry. However, they have different purposes and use cases. In this post, we'll explore what each of these does, how they differ, and when to use each.

Background: Why Use a Registry?

In Elixir, registries offer a way to name processes and access them by a unique identifier rather than just a process ID (PID). Instead of passing around PIDs, which can be tricky to manage and track, you can register a process under a name in a registry. This allows you to reference it more easily and enables features like process discovery and message passing.

Elixir’s built-in Registry module provides functions for managing named registries, and with it, you can register processes and retrieve them by a unique identifier (like a room name, session ID, etc.). Now, let’s look at the two methods.

Method 1: `{:via, Registry, {...}}` Tuple

The {:via, Registry, {...}} tuple is a special syntax used for starting or referencing a process with a registry. This tuple signals to Elixir that you want to look up or register a process via a custom registry and associate it with a key.

The {:via, Registry, {...}} pattern is typically used as the name argument when starting a process with GenServer.start_link/3 or when sending a message to a process registered with a registry.

Here's an example:

GenServer.start_link(MyModule, [], name: {:via, Registry, {ServerAlpha.RoomRegistry, room_id}})

In this example, we’re instructing the GenServer to:

Register its PID with the registry ServerAlpha.RoomRegistry.
Associate the process with the key room_id.

Now, any process can find or send messages to this process by referencing it through {:via, Registry, {ServerAlpha.RoomRegistry, room_id}}, rather than needing the process’s PID directly.

Why Use It?

Using {:via, Registry, {...}} with GenServer.start_link/3 makes the registration part of the startup process, eliminating the need for a separate registration call. It’s a convenient and consistent way to automatically register a process in a registry upon its creation.

Method 2: `Registry.register/3`

The Registry.register/3 function, on the other hand, is a direct function call used to manually register the current process with a registry. You call this function within the process you want to register, typically right after it starts.

Here’s how it might look:

{:ok, _} = Registry.register(ServerAlpha.RoomRegistry, room_id, somedata)

This line registers the current process (from where this function is called) with ServerAlpha.RoomRegistry under the key room_id. Typically, the somedata could be any term, such as room metadata.

This approach requires the process to register itself after it has been started, making it more suitable for cases where the registration process involves additional logic or metadata.

Why Use It?

With Registry.register/3, you can choose when and where to register the process, allowing more flexibility. This can be especially useful if you need to set custom metadata for the registered process or if the registration conditionally depends on certain initialization steps.

Key Differences: `{:via, Registry, {...}}` vs `Registry.register/3`

Now that we understand each method, let’s summarize their differences:

Feature	`{:via, Registry, {...}}`	`Registry.register/3`
Purpose	Start or reference a process with a registry.	Manually register the current process.
Automatic vs. Manual Registration	Automatic registration upon process start.	Manual call to register after process start.
Usage Context	Use with `GenServer.start_link/3` or similar.	Call within process for conditional registration.
Metadata Support	Not intended for custom metadata.	Allows additional metadata upon registration.

When to Use Each

Use {:via, Registry, {...}} when you want to make the registry registration part of the process's startup flow. This method is concise and reliable for consistently registering a process upon its creation, making it a great choice for processes with predictable, uniform registration needs.
Use Registry.register/3 when you need more control over the registration. For example, if you need to perform some additional setup before registering or want to include custom metadata with the registration, Registry.register/3 offers that flexibility.

Practical Example: A Room-Based Server

Let’s imagine a chat server where each room has its own GenServer process. You want to register each room process in a registry so that clients can look up rooms by room IDs.

Using {:via, Registry, {ServerAlpha.RoomRegistry, room_id}}, you could start a GenServer for a room and automatically register it under its room ID:

def start_room(room_id) do
  GenServer.start_link(RoomServer, room_id, name: {:via, Registry, {ServerAlpha.RoomRegistry, room_id}})
end

Clients can then look up or interact with room servers directly by referencing {:via, Registry, {ServerAlpha.RoomRegistry, room_id}}.

Alternatively, if your GenServers require some initialization before they’re registered, you might start each GenServer without registering it, and then call Registry.register/3 once it’s fully set up:

def init(room_id) do
  # Perform setup
  Registry.register(ServerAlpha.RoomRegistry, room_id, :undefined)
  {:ok, %{}}
end

This way, registration only happens after the necessary setup steps are complete.

In Summary

While {:via, Registry, {...}} and Registry.register/3 may look similar at first, they have distinct roles in process management:

Use {:via, Registry, {...}} for automatic, inline registration when starting a process.
Use Registry.register/3 for more control, allowing manual registration and customization.

Go 1.20 memory arena

Mark Nefedov — Sat, 22 Jul 2023 12:05:22 +0000

Enhancing Golang Memory Management with Memory Arenas

One of the groundbreaking features of Go, compared to other compiled languages, is its automatic memory management of unused objects through garbage collection. However, this can lead to performance degradation by handing over control to the memory management process. Until Go 1.20, no alternative mechanism was provided. With the introduction of Go 1.20, an experimental memory management solution called Memory Arena has been added. This article explores the key aspects of using Memory Arenas in Go 1.20 to combine safe dynamic memory allocation and reduce the impact of integrated memory management on application performance.

Enabling Memory Arena Support

To enable support for the new memory management mechanism, add the following environment variable:

export GOEXPERIMENT=arenas

Now, to allocate memory, use the new arena module:

package main

import "arena"

type Person struct{
  Lastname string
  Firstname string
}

func main() {
  mem := arena.NewArena()
  defer mem.Free()

  for i:=0; i<10; i++ {
    obj := arena.New[Person](mem)
    print(obj)
  }
}

As demonstrated, object addresses are allocated sequentially from a single memory area, and the entire allocated arena is freed after calling Free(). With proper usage, this improves code performance since garbage collection will not be called for the arena.

Copying Data and Creating Slices

If you need to copy data to a regular heap, use the Clone method, which creates a copy of the structure from the arena to regular dynamic memory (for example, when you need to return the processing result to the main application). You can also create slices in the arena by specifying the initial size and potential capacity using arena.MakeSlice(mem, initial, capacity).

Allocating Memory with Reflect

To allocate memory based on a type from reflect, use the new method reflect.ArenaNew(mem, typ) which returns a pointer to an object of the given type allocated in the arena stored in mem.

Error Detection and Address Sanitizers

Detect errors when using the arena (for example, reading or writing a value to a structure after freeing the arena) using mechanisms like go run -asan (Address Sanitizer) or go run -msan (Memory Sanitizer). For example:

package main

import "arena"

type Data struct {
  value int32
}

func main() {
  mem := arena.NewArena()
  v := arena.New[Data](mem)
  mem.Free()
  v.value = 1
}

When running with asan/msan, it will show an error of incorrect pointer usage after freeing the arena.

Storing Strings in the Arena

To store strings in the arena, create a memory region from a byte sequence and copy the string contents into it, like this:

src := "original"

mem := arena.NewArena()
defer mem.Free()

bs := arena.MakeSlice[byte](mem, len(src), len(src))
copy(bs, src)
str := unsafe.String(&bs[0], len(bs))

Using the Arena for Primitive Data Types

The arena can also be used for storing structures, primitive data types, or their sequences. In this case, the interaction is no different from working with a pointer to a variable:

package main

import "arena"

func main() {
  mem := arena.NewArena()
  defer mem.Free()
  v := arena.New[int32](mem)
  *v = 10
  println(*v)
}

Similarly, the behavior of slices in the arena is no different from regular slices in Go:

package main

import "arena"

func main() {
  mem := arena.NewArena()
  defer mem.Free()
  v := arena.MakeSlice[int32](mem,50,100)
  v[49] = 10;
  v = append(v, 20)
  println(v[49])        //10
  println(v[50])        //20
  println(len(v))       //51
  println(cap(v))       //100
}

Detecting Memory Leaks

To detect memory leaks when using arenas, use the usual profiling mechanisms in Go (go tool pprof for visualizing memory allocation sampling, which can be saved via functions in the runtime/pprof module). From the point of view of memory allocation, working with an arena is similar to allocating a single memory block (which can grow in size). When the arena is freed, all objects allocated in it become inaccessible.

Performance improvements can be expected in cases where the application intensively allocates memory (for example, when storing binary trees or other related data structures). However, it is assumed that the allocated data structures are long-lived and exist until the arena is freed entirely (the garbage collector is not applied to the arena, and allocated objects in the contiguous memory area are not cleaned up).

Conclusion

In summary, the introduction of Memory Arenas in Go 1.20 provides an experimental memory management solution that allows for safe dynamic memory allocation while reducing the impact of integrated memory management on application performance. Key aspects of using Memory Arenas include enabling support through an environment variable, allocating memory using the new arena module, copying data and creating slices, allocating memory with reflect, error detection using Address and Memory Sanitizers, storing strings and primitive data types in the arena, and detecting memory leaks using profiling mechanisms in Go.

Memory Arenas offer potential performance improvements in cases where applications intensively allocate memory, such as storing binary trees or other related data structures. However, it is important to note that the allocated data structures are expected to be long-lived and exist until the arena is freed entirely, as the garbage collector does not apply to the arena and allocated objects in the contiguous memory area are not cleaned up.

Setup screen session for interviewee monitoring

Mark Nefedov — Fri, 14 Jul 2023 12:47:31 +0000

This tutorial provides a step-by-step guide on setting up a shared screen session for a Linux interview. By following these steps, you can give the interviewee access to a Linux box while monitoring their inputs. Although the steps are mostly similar for all Linux distributions, minor adjustments may be required for different shells.

Step 1: Create a User for the Interviewee

First, create a new user account for the interviewee using the useradd command. The -m flag creates a home directory for the new user.

useradd -m interviewee

Step 2: Install the `screen` Utility

The screen utility allows you to create and manage multiple terminal sessions. If it's not already installed on your system, use the appropriate package manager command for your Linux distribution:

For SUSE:

zypper in screen

For Debian/Ubuntu:

apt install screen

For RHEL/Fedora:

dnf install screen

Step 3: Modify the `.profile` File to Create a `screen` Session at User Login

Next, update the .profile file in the interviewee's home directory to automatically create a screen session when they log in. Use the echo command to append the necessary line to the file:

echo "screen -S interviewee-session" >> /home/interviewee/.profile

Make sure to replace /home/interviewee with the actual path to the interviewee's home directory if it's different on your system.

Step 4: Access the Shared Screen Session

Now you can give the interviewee their credentials, and they can log in to the system. To monitor their actions, switch to the user account you created and access the shared screen session using the following command:

screen -x interviewee-session

That's it! You've successfully set up a shared screen session for a Linux interview.

Dotnet authentication clock skew.

Mark Nefedov — Tue, 20 Jun 2023 06:39:12 +0000

If you use dotnet JWT authentication check your TokenValidationParameters, and make sure you have set ClockSkew to reasonable values. By default, it is set to 300 seconds (5 minutes). That make your JWT tokens valid for extra 5 minutes.

/// <summary>
        /// Gets or sets the clock skew to apply when validating a time.
        /// </summary>
        /// <exception cref="ArgumentOutOfRangeException">If 'value' is less than 0.</exception>
        /// The default is <c>300</c> seconds (5 minutes).
        [DefaultValue(300)]
        public TimeSpan ClockSkew
        {
            get
            {
                return _clockSkew;
            }

            set
            {
                if (value < TimeSpan.Zero)
                    throw LogHelper.LogExceptionMessage(new ArgumentOutOfRangeException(nameof(value), LogHelper.FormatInvariant(LogMessages.IDX10100, LogHelper.MarkAsNonPII(value))));

                _clockSkew = value;
            }
        }

Nginx TCP transparent proxy

Mark Nefedov — Mon, 19 Jun 2023 08:07:57 +0000

So we have a setup like this:

Set gateways to local and external networks:



network:
  ethernets:
    ens3:
      addresses:
        - 192.168.229.10/24
      gateway4: 192.168.229.15 # NGINX transparent proxy
      nameservers:
        addresses:
        - 192.168.222.8
        - 192.168.222.2
      routes:
        - to: 192.168.222.0/24
          via: 192.168.229.254 # Known gateway to internal network
  version: 2

Create a policy where all packets marked with 1 will be routed to loopback interface for nginx processing.

ip rule add fwmark 1 lookup 100

Firewall marks are a feature of the Linux kernel's networking stack that allow you to "mark" packets with a certain value, which can then be used for making routing decisions. This can be useful for complex network setups where you need to route traffic differently based on criteria other than the destination IP.

In this case, the rule is saying that any packets with a firewall mark of 1 (which would have been set by a previous step, perhaps an iptables rule) should use routing table 100.

ip route add local 0.0.0.0/0 dev lo table 100

This command adds a route to table 100 that sends all traffic (0.0.0.0/0 represents all IP addresses) to the loopback interface (lo).

The local keyword is used to indicate that these packets are to be treated as local and not to be forwarded. This is typically used when the system itself will be processing the packets, rather than forwarding them on to another network.

So this rule is saying that any traffic which gets directed to table 100 (such as by the previous ip rule command) should be sent to the local loopback interface.

firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -p tcp -s 192.168.229.10/31 --sport 3232 -j MARK --set-xmark 0x1/0xffffffff

This command adds a permanent direct rule to the firewalld configuration, which marks TCP packets with a source IP address in the range 192.168.229.10/31 and a source port of 3232 in the PREROUTING chain of the mangle table. The mark value is set to 0x1 with a mask of 0xffffffff.

Add directives nginx configuration:



server {
    listen 3232;
    proxy_bind $remote_addr transparent;
    proxy_pass sftp1;
}

Enable NGINX transparent proxy handling.

TLDR

Change gateway address to NGINX to send all responses to spoofed packets back to NGINX.
Configure iptables/nftables to mark incoming traffic from upstream.
Create a rule and a routing table to send marked traffic local processing for nginx to pick it up.

Shard Key Best Practices: Ensuring Balanced and Efficient Data Distribution

Mark Nefedov — Mon, 12 Jun 2023 11:58:16 +0000

Alright folks, we're going to dive into the nitty-gritty world of distributed databases today. You know, those complex systems that split data across multiple nodes or shards (fancy name for pieces) to ensure things run smoothly and efficiently. One of the big players in this game is the shard key. It's a specific field or fields that decides how data gets divvied up across the nodes.

The Importance of Shard Keys in Distributed Databases

A shard key is a specific field or set of fields used to determine how data is partitioned across the nodes in a distributed database system. The choice of a shard key is central to ensuring a balanced distribution of data and optimizing the performance of read and write operations. Therefore, a carefully chosen shard key can lead to better utilization of resources, improved query performance, and increased scalability.

High Cardinality: The Key to Even Distribution

Cardinality is a term that originates from mathematics, specifically set theory, where it's used to count the number of elements in a set. In the context of databases, however, cardinality takes on a slightly different meaning. It's still about counting, but here it refers to the number of unique values in a column or a set of columns.

For example, consider a database that stores information about people. If there's a column for "gender" and you only record "male", "female", and "prefer not to say", the cardinality of the "gender" column is 3 because there are 3 unique values. But if there's a column for "social security number", the cardinality would be very high (assuming you have many records), because each person has a unique social security number.

High cardinality refers to columns with values that are very uncommon or unique. A column with high cardinality would have values that are typically distinct or unusual. This could be an email address or a transaction id, where each record has a unique value.

On the other hand, low cardinality refers to columns with values that are very common or repeating. For instance, a "yes/no" column has a cardinality of 2.

In the context of shard keys in distributed databases, a high cardinality is preferred because it provides a greater number of potential partitions for the data, allowing for a more even distribution across the nodes in the database cluster. If you had a low cardinality shard key, you'd end up with less potential partitions, and that could lead to an uneven distribution of data, which could impact the performance of your database.

In a nutshell, cardinality is all about the uniqueness of data in a column, and when choosing a shard key, going for high cardinality can help you ensure a more balanced and efficient distribution of data in a distributed database system.

Low Frequency: Avoiding Shard Key Hotspotting

Frequency, in the simplest terms, is how often a particular value occurs in a dataset. So, when we talk about low frequency in relation to databases, we're referring to values in a column that don't show up very often.

Imagine you have a database table that logs the activities of users on a website. If there's a column called "activity_type" that records whether a user "logged in", "logged out", "posted a comment", "liked a post", etc., the frequency of each activity is how often it appears in that column. If "logged in" and "logged out" are very common activities, they have high frequency. On the other hand, if "deleted account" is a rare activity, it has low frequency.

Now, let's bring shard keys into the picture. When selecting a shard key for a distributed database, choosing one that has low frequency values is typically beneficial. Here's why: if a shard key value appears very frequently, it means many records will have the same shard key value and they will all be directed to the same shard or node. This can create an imbalance in the data distribution, where one shard is handling a lot of data (and therefore work), while others have less data and might be under-utilized.

This could also lead to "hotspotting", where one node becomes a hotspot due to a high volume of read/write requests. Hotspots can seriously degrade the performance of your database. So, selecting a shard key with low frequency values can help avoid this situation by ensuring a more even distribution of data across the shards.

So, to sum it up, when we say "low frequency" in the context of shard keys, we're talking about choosing a field that has a lot of unique or less commonly occurring values, which helps to balance the load across the nodes in a distributed database system and prevent any one node from being overwhelmed.

Use Case Considerations

While high cardinality and low frequency are general principles, the ideal shard key will also depend on the specific use case, including the nature of the data, the expected query patterns, and the read/write workload. For example, if the database mainly supports read operations and the data is not updated frequently, a different sharding strategy might be more effective. Often a composite shard key (made up of multiple fields) might be used to achieve better distribution and performance.

Here are a couple of examples:

User ID in a User Profile Table: Suppose you have a table storing user profiles, where each row represents a unique user. In this case, using 'user_id' as the partition key would be an ideal choice. Each 'user_id' is unique to an individual user (high cardinality), and since each user_id will only show up once in the user profile table, it has a low frequency. This ensures a balanced distribution of data across the Cassandra nodes.
Transaction ID in an E-commerce Transactions Table: In an e-commerce application, you could have a table logging every transaction. Each transaction would have a unique 'transaction_id', which would be a perfect candidate for the partition key. Similar to the previous example, this provides high cardinality (each transaction is unique) and low frequency (each 'transaction_id' shows up only once).

For instance, if you frequently retrieve data based on 'user_id' and 'transaction_date', you might choose a composite partition key consisting of these two fields. This would still provide high cardinality (assuming you have many users and transactions spread across different dates), and each combination of 'user_id' and 'transaction_date' would have a low frequency, helping to evenly distribute data across your nodes.

Switch statements are a lie.

Mark Nefedov — Mon, 05 Jun 2023 09:25:52 +0000

Switch Statements Are a Lie

For those who first encounter the switch statement in their early programming journey, it might appear as a magic construct that significantly simplifies logic by allowing a multi-way branch. However, as with many things in the programming world, what you see is not always what you get. While switch statements in high-level languages like C or C++ have their syntax and semantics, the story behind the scenes, in the realm of the compiler, is far more complex. Let's unravel this mystery.

The Illusion of Jump Tables

You might have been taught or intuitively believed that the compiler translates a switch statement into a jump table in assembly, which should allow for constant-time selection of code blocks regardless of the number of cases. Indeed, in a perfect world where all case values are densely packed, starting from zero or some manageable offset, this would be a reasonable and efficient translation. Unfortunately, the real-world situations are often less ideal, and a jump table is not always the go-to solution.

The Factors Preventing Jump Table Generation

So, what factors can derail the creation of a jump table for a switch statement in the compiler's realm? Here are some major considerations:

Density of case values: If the case values are scattered over a wide range, the jump table, which usually covers all values from the lowest to the highest, could become overwhelmingly large and impractical. Consider a switch statement with two cases: 1 and 1,000,000. The jump table for this would theoretically require a million entries!
Size of the range of case values: Even if the case values are densely packed, a large range of values could still make a jump table infeasible due to memory limitations.
Number of cases: With only a few cases, it might be more efficient to generate a series of if-else statements (also known as a binary search tree), as the overhead of creating, initializing, and using a jump table might outweigh its benefits.
Compiler's optimization level: Depending on the optimization settings, the compiler may favor execution speed, code size, or a balance between them. Higher optimization levels are more likely to generate a jump table, prioritizing speed, while lower levels may prefer a series of comparisons to conserve memory.
Order of case labels: Some compilers might rely on case labels being in ascending order to generate a jump table.
Compiler's internal heuristics: Finally, compilers use complex internal heuristics that take into account all the above factors and possibly many others. These heuristics are part of the compiler's design and tend to evolve as compiler technology progresses.

Deconstructing the Lie

So, what does a compiler do when a jump table is off the table? Often, it resorts to a series of comparisons and jumps - essentially an if-else-if chain or a binary search tree, depending on the specifics. This transformation might surprise programmers who believed in the illusion of the "switch-as-a-jump-table."

Let's consider a C++ switch statement. When translated into assembly, you would see a series of cmp (compare) and je (jump if equal) instructions corresponding to each case. If the case matches the input value, the program jumps to the associated code block. Otherwise, it continues to the next comparison. If none of the cases match, the program reaches the equivalent of the default clause. The actual instruction set and structure might vary based on the architecture and the compiler, but the principle remains the same.

Let's illustrate this with a tangible example. Consider the following C++ code snippet: https://godbolt.org/z/qhWrd3dP3

#include <iostream>
#include <ctime>
#include <cstdlib>

int main() {
    srand(time(0));
    int value = rand() % 101;

    switch(value / 10) {
        case 0: std::cout << "Value is 0\n"; break;
        case 1: std::cout << "Value is 10\n"; break;
        case 2: std::cout << "Value is 20\n"; break;
        case 3: std::cout << "Value is 30\n"; break;
        case 4: std::cout << "Value is 40\n"; break;
        case 5: std::cout << "Value is 50\n"; break;
        case 6: std::cout << "Value is 60\n"; break;
        case 7: std::cout << "Value is 70\n"; break;
        case 8: std::cout << "Value is 80\n"; break;
        case 9: std::cout << "Value is 90\n"; break;
        case 10: std::cout << "Value is 100\n"; break;
        default: std::cout << "Value is out of range\n";
    }

    return 0;
}

        cmp     edx, 5
        je      .L7
        cmp     eax, 50
        jle     .L24
        cmp     edx, 8
        je      .L16
        cmp     eax, 80
        jg      .L17
        cmp     edx, 6
        je      .L18
        cmp     edx, 7
        jne     .L13

This code generates a random number between 0 and 100 and uses a switch statement to select the appropriate output message. One might expect the compiler to translate this into a jump table. However, examining the resulting assembly code reveals that the compiler actually transforms the switch into a series of comparisons and conditional jumps.

In the assembly output, the cmp and je instructions correspond to each case. If the value matches, the program jumps to the corresponding code block. If none of the case values match, the code reaches the "default" block, which outputs the "Value is out of range" message.

This serves as a potent illustration that what we imagine as a straightforward jump table can, in fact, be something quite different under the hood.

Embracing the Truth

Given this understanding, does it mean switch statements are useless? Absolutely not! Regardless of how they're implemented under the hood, switch statements can still greatly enhance the readability and maintainability of your code by providing a clear, compact way of handling multiple conditions based on a single value. However, it's important for programmers to understand the truth behind switch statements and consider performance implications in critical code paths. Understanding your compiler's behavior can help you write more efficient and effective code.

So, the next time you write a switch statement, remember it's not always what it seems. You're not directly commanding the hardware, but giving suggestions to a complex compiler system that's doing its best to translate your high-level intentions into efficient low-level operations. The art of programming lies not only in mastering the language syntax but also in understanding the underlying processes that bring your code to life.

Interview tips I wish I've known.

Mark Nefedov — Sun, 04 Jun 2023 13:58:29 +0000

It is okay to be nervous.

Many people struggle with expressing their feelings. One helpful tip is to begin your conversation with something like this: "I'm very interested in working for this company. However, I am a bit nervous, so I hope you understand". This lets the interviewer know you may need a bit more time to focus, and it also earns you extra points for communication skills.

Explain how you think.

I struggle with thinking quickly on the spot. If you pose an algorithm question and put me on a whiteboard, I won't be able to give you an answer right away. At work, I'm much more used to spending an entire day researching different solutions before I even come up with something. You can say: " I will try my best to answer all your questions quickly, but when I actually work, I would like to take my time thinking and looking at the problem from many different angles." A lot of people can relate to this, and they probably will give you much more time and space to think clearly. You will also get bonus points in many jobs like engineering, having patience and being methodical is a preferred quality.

When encountering a difficult question, acknowledge it.

Instead of panicking, losing focus, and worrying about making a poor impression, simply say, "This seems like a challenging question to me". This shows your interviewer that you understand the complexity of the question. Many people fail to communicate even this much. Even if you don't have the right solution right away, at least it shows that you are able to see the bigger picture. You can propose to simplify question at first and solve a bit more basic example so you can get a better understanding of the problem, and from there work towards final solution. This will give you points for critical thinking and being able to break down complex problems into a smaller chunks.

Admit when you don't know an answer.

If you don't know an answer, admit it, and try to explain a potential solution in general terms. For example, if you're asked about Spring Boot authentication, but you've only worked with ASP.NET, you could suggest a solution based on your existing knowledge: "If Spring authentication is similar to .NET's, I think the answer could be something like this…"

Ask probing questions.

In a working environment you will often first get a task that won't have all the details necessary for you to go ahead and come up with the solution, so you will be required to ask your co-workers and your superiors further questions about what we are trying to achieve with this particular task and in what way do we approach it by asking your interviewer similar questions. Let your interviewer know you can have a well thought out discussion. You will be surprised how much useful information you’ll find out from your interviewers. Keep in mind that a lot of questions are intentionally given vague, and that the expectation is for you to try and clarify them before you try to answer.

Talk through your thinking.

Ensure that the interviewer has as much information as possible about how you're approaching a problem. Most of the time, even if you don't have the final solution, interviewers will be satisfied with your overall response. If you couple this with the previous advice about asking clarifying questions, you'll leave a positive impression. At the end of the day, the interviewers are trying to answer one simple question: "Would I want to work with this person?" Regardless of the problem you're working on during the interview, strive to answer that question first.

How to find all structs that Derive known macro in Rust

Mark Nefedov — Wed, 24 May 2023 18:44:26 +0000

I've recently embarked on an endeavor to add migration generation to my favorite Rust ORM Welds.

Rust's powerful macro system allows us to extend the language in ways that can greatly enhance productivity and maintainability. In this article, I'll focus on a specific use case: finding all the structs in a codebase that derive a particular macro, in this case the WeldsModel macro.

The solution provided in the question uses the syn crate, a parsing library for Rust syntax, and the walkdir crate, which provides an iterator over the files in a directory and its subdirectories. Here, we'll walk through this solution and explain how it works.

The Main Function

First, let's look at the main function:

fn main() {
    for entry in walkdir::WalkDir::new("welds/examples")
        .into_iter()
        .filter_map(Result::ok)
    {
        let path = entry.path();
        if path.is_file() && path.extension().unwrap() == "rs" {
            let mut file = File::open(path).expect("Unable to open file");
            let mut contents = String::new();
            file.read_to_string(&mut contents)
                .expect("Unable to read file");
            let syntax_tree = syn::parse_file(&contents).unwrap();
            let weldsmodel_structs = find_weldsmodel_structs(syntax_tree);
            for weldsmodel_struct in weldsmodel_structs {
                println!("Found struct: {}", weldsmodel_struct.ident);
            }
        }
    }
}

The main function here uses the WalkDir iterator from the walkdir crate to iterate over all the files in the welds/examples directory and its subdirectories. It filters out directories and non-Rust files (path.extension().unwrap() == "rs"), then reads the contents of each Rust file into a String.

The syn::parse_file function is then used to parse this string into a syntax tree, which represents the structure of the Rust code in the file. This syntax tree is passed to the find_weldsmodel_structs function, which returns a Vec<ItemStruct> containing all structs in the file that derive the WeldsModel macro. ItemStruct is a syn data model that describes struct in a Rust AST, they will be used later in the project as a data source to build schema from.

Finally, the main function prints out the name of each struct found.

The find_weldsmodel_structs Function

Next, let's look at the find_weldsmodel_structs function:

fn find_weldsmodel_structs(syntax_tree: syn::File) -> Vec<ItemStruct> {
    let mut weldsmodel_structs = Vec::new();
    for item in syntax_tree.items {
        if let syn::Item::Struct(item_struct) = item {
            if has_weldsmodel_derive(&item_struct) {
                weldsmodel_structs.push(item_struct);
            }
        }
    }
    weldsmodel_structs
}

This function iterates over all items in the syntax tree, checks if each item is a struct, and if so, checks if the struct derives the WeldsModel macro using the has_weldsmodel_derive function. If both conditions are met, the struct is added to the weldsmodel_structs vector.

The has_weldsmodel_derive Function

Finally, the has_weldsmodel_derive function:

fn has_weldsmodel_derive(item_struct: &ItemStruct) -> bool {
    let attrs = &item_struct.attrs;
    for attr in attrs {
        let meta = &attr.meta;
        if meta.path().is_ident("derive") {
            let nested = attr
                .parse_args_with(Punctuated::<Meta, Token![,]>::parse_terminated)
                .unwrap();
            for nested_meta in nested {
                if nested_meta.path().is_ident("WeldsModel") {
                    return true;
                }
            }
        }
    }
    false
}

This function checks if a given struct item has the WeldsModel derive attribute. It does this by looking at each attribute of the struct, and if the attribute's path is "derive", it parses the arguments of the attribute with parse_args_with. This gives a Punctuated list of Meta items, each representing an argument of the derive attribute. If any of these arguments' path is "WeldsModel", the function returns true, otherwise false.

TLDR

That's the overall solution: use WalkDir to iterate over all Rust files in a directory, use syn::parse_file to parse the contents of each file into a syntax tree, and then check each ATS node attributes and meta to find structures you need.

Integrate Elxir Phoenix client with JS app.

Mark Nefedov — Tue, 16 May 2023 16:54:23 +0000

If you have an existing JavaScript or TypeScript application, and you’d like to add a Phoenix JavaScript client to it, you can do so by following these steps:

Install the Phoenix JavaScript client by running the following command in your React project’s directory:

npm install --save phoenix

In your React application, you’ll need to import the Socket and LongPoll modules from the phoenix package, as well as the phoenix package itself. For example:

import {Socket} from "phoenix"

Create a new instance of the Socket class, passing in the URL of your Phoenix server’s endpoint as the first argument.

let socket = new Socket("ws://localhost:4000/socket");

Connect to the socket by calling the connect method on the socket object. For example:

socket.connect();

Once the socket is connected, you can join channels and bind to events to receive messages from the server. For example, you can join a channel named “room:lobby” and listen for the “new_msg” event like this:

let channel = socket.channel("room:lobby", {});
channel.join()
  .receive("ok", resp => { console.log("Joined successfully", resp) })
  .receive("error", resp => { console.log("Unable to join", resp) });

channel.on("new_msg", payload => {
  console.log(payload);
});

To send a message back to the server over this channel, you can use push method:

channel.push("new_msg", {body: "This is a test message"});

With these steps, you should now be able to interact with your Phoenix server using the JavaScript client in your React application.

Please keep in mind that you might have to modify the server side to handle new_msg event, depend on your server implementation.

Typescript

You can install the @types/phoenix package, which provides the TypeScript declarations for the Phoenix client. You can do this by running the following command in your project’s directory:

npm install --save-dev @types/phoenix

This package provides TypeScript declarations for the Phoenix JavaScript client, so that you can take advantage of TypeScript’s type checking and editor support when working with the Phoenix client in your application.

You could also use some pre-built package like phoenix-ts that is written in Typescript, but this is not an official library of phoenix.

It’s also worth noting that these packages may be updated less frequently than the official phoenix package, so you may want to keep an eye on that and consider updating if necessary.

Using Vulkan without linking to it. (volk alternative for vulkan.hpp)

Mark Nefedov — Tue, 16 May 2023 16:53:30 +0000

Instead of using the full fledged Vulkan SDK, we only can use the Vulkan-headers package and load actual Vulkan implementation from the driver.

To do this, we need to enable vulkan.hpp dynamic dispatcher by default by setting VULKAN_HPP_DISPATCH_LOADER_DYNAMIC to 1, adding VULKAN_HPP_DEFAULT_DISPATCH_LOADER_DYNAMIC_STORAGE to the source file and updating VULKAN_HPP_DEFAULT_DISPATCHER as we load in more Vulkan features.

#define VULKAN_HPP_DISPATCH_LOADER_DYNAMIC 1
#include "vulkan/vulkan.hpp"

// Vulkan defined macro storage for dispatch loader
VULKAN_HPP_DEFAULT_DISPATCH_LOADER_DYNAMIC_STORAGE

int main()
{
    vk::DynamicLoader dl;
    PFN_vkGetInstanceProcAddr vkGetInstanceProcAddr = dl.getProcAddress<PFN_vkGetInstanceProcAddr>("vkGetInstanceProcAddr");
    VULKAN_HPP_DEFAULT_DISPATCHER.init(vkGetInstanceProcAddr);
    vk::Instance instance = vk::createInstance({}, nullptr);
    // initialize function pointers for instance
    VULKAN_HPP_DEFAULT_DISPATCHER.init(instance);
    // create a dispatcher, based on additional vkDevice/vkGetDeviceProcAddr
    std::vector<vk::PhysicalDevice> physicalDevices = instance.enumeratePhysicalDevices();  
    vk::Device device = physicalDevices[0].createDevice({}, nullptr);
    // function pointer specialization for device
    VULKAN_HPP_DEFAULT_DISPATCHER.init(device);
}

DEV Community: Mark Nefedov

Debugging Containers Without Shell Access: Quick Tips

1. Run Commands Inside the Container’s Namespace

2. Access Container Files via /proc

Why This Works

Understanding {:via, Registry, {...}} vs Registry.register/3 in Elixir

Understanding {:via, Registry, {...}} vs Registry.register/3 in Elixir

Background: Why Use a Registry?

Method 1: {:via, Registry, {...}} Tuple

Why Use It?

Method 2: Registry.register/3

Why Use It?

Key Differences: {:via, Registry, {...}} vs Registry.register/3

When to Use Each

Practical Example: A Room-Based Server

In Summary

Go 1.20 memory arena

Enhancing Golang Memory Management with Memory Arenas

Enabling Memory Arena Support

Copying Data and Creating Slices

Allocating Memory with Reflect

Error Detection and Address Sanitizers

Storing Strings in the Arena

Using the Arena for Primitive Data Types

Detecting Memory Leaks

Conclusion

Setup screen session for interviewee monitoring

Step 1: Create a User for the Interviewee

Step 2: Install the screen Utility

Step 3: Modify the .profile File to Create a screen Session at User Login

Step 4: Access the Shared Screen Session

Dotnet authentication clock skew.

Nginx TCP transparent proxy

Set gateways to local and external networks:

ip rule add fwmark 1 lookup 100

ip route add local 0.0.0.0/0 dev lo table 100

firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -p tcp -s 192.168.229.10/31 --sport 3232 -j MARK --set-xmark 0x1/0xffffffff

Add directives nginx configuration:

TLDR

Shard Key Best Practices: Ensuring Balanced and Efficient Data Distribution

The Importance of Shard Keys in Distributed Databases

High Cardinality: The Key to Even Distribution

Low Frequency: Avoiding Shard Key Hotspotting

Use Case Considerations

Switch statements are a lie.

Switch Statements Are a Lie

The Illusion of Jump Tables

The Factors Preventing Jump Table Generation

Deconstructing the Lie

Embracing the Truth

Interview tips I wish I've known.

It is okay to be nervous.

Explain how you think.

When encountering a difficult question, acknowledge it.

Admit when you don't know an answer.

Ask probing questions.

Talk through your thinking.

How to find all structs that Derive known macro in Rust

The Main Function

The find_weldsmodel_structs Function

The has_weldsmodel_derive Function

TLDR

Integrate Elxir Phoenix client with JS app.

Typescript

Using Vulkan without linking to it. (volk alternative for vulkan.hpp)

2. Access Container Files via `/proc`

Understanding `{:via, Registry, {...}}` vs `Registry.register/3` in Elixir

Method 1: `{:via, Registry, {...}}` Tuple

Method 2: `Registry.register/3`

Key Differences: `{:via, Registry, {...}}` vs `Registry.register/3`

Step 2: Install the `screen` Utility

Step 3: Modify the `.profile` File to Create a `screen` Session at User Login