DEV Community: Dima Danilov

The switch from Vim

Dima Danilov — Sat, 23 May 2020 21:00:00 +0000

For the past five years, my go-to text editors have been Vim and gVim/MacVim. Currently, I work on macOS with Ubuntu, as a target OS, running on the local VMware Fusion virtual machine.
Personally, I prefer to use the macOS GUI over Ubuntu and do not enjoy constantly switching between windows,
so I used to connect to the machine with ssh -XY user@server and run gVim there. Since macOS supports X Window System, I was able to open the gVim window in macOS as a “native” application.

However, when I had to edit a file locally on macOS, naturally I used MacVim.

Over time, it became increasingly inconvenient because the behavior and appearance of gVim and MacVim had minor differences.
Additionally, the X Window System simply does not fit the macOS ecosystem well.

Visual Studio Code

I started to search for efficient alternatives and almost ended up switching to Visual Studio Code.
Don’t get me wrong - VS Code is an awesome text editor with features that a Vim user can only dream of, but I got used to my Vim shortcuts and plugins. I know there is Vim support in VS Code, but it is not the same.

Neovim

Eventually, I discovered Neovim.

Neovim is a refactor, and sometimes redactor, in the tradition of Vim (which itself derives from Stevie). It is not a rewrite but a continuation and extension of Vim.

Nvim always includes ALL features, in contrast to Vim (which ships with
various combinations of 100+ optional features). Think of it as a leaner
version of Vim's "HUGE" build. This reduces the surface area for bugs and
removes a common source of confusion and friction for users.

Once Neovim is installed, it behaves the same way as Vim and supports all of its plugins.
Basically, it is a drop-in replacement of Vim.

Neovim also supports remote plugins that communicate via msgpack-rpc.
The RPC messages can be sent through various channels, such as Unix socket, TCP socket, or stdin/stdout.

Neovim GUI clients are implemented as remote plugins and most of them communicate through stdin/stdout.
Each GUI client launches nvim process and sends it commands in msgpack format via stdin, and nvim
replies back via stdout with information on how to redraw the screen.

Neovim + Neovim-Qt

Neovim-Qt is a compact Neovim GUI client written in C++ with Qt5.
If I need to edit a file locally, I open Neovim-Qt, which works as described above.

Additionally, Neovim-Qt is able to connect to a Neovim instance, operating in server mode.
In order to edit files on the virtual machine, I launch a nvim process within the virtual environment, enabling listening on the given IP/port:

$ssh user@server
$nvim --listen : \
      --headless #headless means "don't start a user interface"

and then start Neovim-Qt with the following parameters on macOS:

$nvim-qt --server <ip>:<port>

Running all of these commands manually every time I need to open a text editor is quite tedious.
Therefore, I decided to automate the process with a simple script that provides the alias rgvim, where r stands for “remote”.

Finally, I can use a single GUI for both operating systems!

Screenshots:

Local macOS environment:

Connected to a remote instance:

std::shared_ptr is an anti-pattern

Dima Danilov — Sat, 11 Jan 2020 22:00:00 +0000

A bit of history

The first time a shared pointer was introduced in boost library in the year 1999. Even before boost had release numbers! Back then the only alternative standard C++ could provide was std::auto_ptr. It was so bad that it was rarely used, got deprecated in C++ 11 and was eventually removed in C++ 17. So the only way to have a nice smart pointer in C++ 98/03 was to use boost::shared_ptr. In C++ 11 boost::shared_ptr finally made it to the standard library.

A few words about how `shared_ptr` works

If the reader knows how std::shared_ptr works they may skip this section. From a very simplified point of view, a shared pointer has two pointers: one to an object in the heap that it owns and another to a reference counter of shared instances.

Every time a shared pointer’s copy-constructor is called the counter is incremented
Every time a shared pointer’s assignment operator is called the counter of the right-hand pointer is incremented and of the left-hand is decremented
Every time a shared pointer’s destructor is called the counter is decremented
If the counter equals zero the object is deleted

The ideal usage

With the shared pointer a programmer may create a variable and pass it wherever they want without caring about memory deallocation because at some point it happens automatically. It is awesome, isn’t it? Let us see!

Incrementation of the reference counter and multi-threading

In order to satisfy thread-safety requirements, the reference counter is usually implemented as atomic. So every time a
shared pointer is passed by value its atomic counter is incremented and decremented. Obviously, an incrementation of the atomic counter is relatively
expensive.

This issue may be addressed by passing shared pointers by const reference. Thus, you do not increment/decrement the counter and save CPU cycles. Let’s see the difference in performance by running the following code:

#include <memory>

using namespace std::chrono;
using shared_ptr_t=std::shared_ptr<int>;

void shared_ptr_receiver_by_value(shared_ptr_t ptr) {
    (void)*ptr;
}

void shared_ptr_receiver_by_ref(const shared_ptr_t& ptr) {
    (void)*ptr;
}

void test_copy_by_value(uint64_t n) {
    auto ptr = std::make_shared<int>(100);
    for(uint64_t i = 0u; i < n; ++i) {
        shared_ptr_receiver_by_value(ptr);
    }
}

void test_copy_by_ref(uint64_t n) {
    auto ptr = std::make_shared<int>(100);
    for(uint64_t i = 0u; i < n; ++i) {
        shared_ptr_receiver_by_ref(ptr);
    }
}

int main(int argc, char *argv[]) {
    uint64_t n = (argc == 3 ) ? std::stoull(argv[2]) : 100;
    auto t1 = high_resolution_clock::now();
    if(atoi(argv[1]) == 1) {
        test_copy_by_value(n);
    } else {
        test_copy_by_ref(n);
    }
    auto t2 = high_resolution_clock::now();
    auto time_span = duration_cast<duration<int64_t, std::micro>>(t2 - t1);
    std::cout << "It took me " << time_span.count() << " microseconds.\n";
    return 0;
}

And the output is:

$ ./cpu_atomic_copy.bin 1 999999
It took me 3616 microseconds.
$ ./cpu_atomic_copy.bin 2 999999
It took me 2 microseconds.

The difference is breathtaking!!!
But wait, if I do not update the reference counter my shared pointer does not track the reference count! The shared pointer’s counter can be changed outside of the current function scope only if there is a reference to the pointer in another thread. Thus, the only reason to pass a shared pointer by value is when passing it to another thread. So when the first thread stops and all its objects are destructed or the thread just
stops using the pointer then the shared pointer’s atomic counter is decremented. But the stored object is not deleted
and the other thread(s) may continue using it. However, we usually join threads in the end. So that we know when they
stop and do not need any of the resources shared with them. It means that you probably need to pass a shared pointer to
a thread only if you run a detached thread. Or the thread that passes shared_ptr does not use it anymore. Though, in
this case, you might prefer using std::unique_ptr.

Shared pointer initialization

This is the initialization of std::shared_ptr I usually see:

// let us assume that new int(42) does not throw
auto ptr = std::shared_ptr<int>(new int(42));

What happens in the code?

Memory in the heap is allocated for the integer 42 and its pointer is stored in the shared pointer’s stored pointer
Memory in the heap is allocated for the reference counter and its pointer is stored in the second inner pointer of the shared pointer

Why is it bad?

You make two memory allocations in the heap for one stored object
The shared pointer’s data is located in two different parts of the heap. Which potentially may lead to a higher cache miss rate. However, I haven’t succeeded to prove it yet

How to fix it?
std::make_shared comes to help:

auto ptr = std::make_shared<int>(42);

It looks almost the same but this code makes only one allocation of a contiguous piece of memory used for storing both the stored object and the reference counter. The picture below shows the difference in the memory layout of shared pointers created in the two ways.

Let us check the memory allocation with the following simple code:

#include <iostream>
#include <memory>
#include <vector>


void test_shared_ptr(size_t n) {
    std::cout << __FUNCTION__ << "\n";
    std::vector<std::shared_ptr<size_t>> v;
    v.reserve(n);
    for(size_t i = 0u; i < n; ++i) {
        v.push_back(std::shared_ptr<size_t>(new size_t(i)));
    }
}

void test_make_shared(size_t n) {
    std::cout << __FUNCTION__ << "\n";
    std::vector<std::shared_ptr<size_t>> v;
    v.reserve(n);
    for(size_t i = 0u; i < n; ++i) {
        v.push_back(std::make_shared<size_t>(i));
    }
}

int main(int argc, char *argv[]) {
    size_t n = (argc == 3 ) ? atoi(argv[2]) : 100;
    if(atoi(argv[1]) == 1) {
        test_shared_ptr(n);
    } else {
        test_make_shared(n);
    }
    return 0;
}

Let’s check it with valgrind:

$valgrind --tool=memcheck ./memory_allocation.bin 1 100000
==3005== Memcheck, a memory error detector
==3005== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3005== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==3005== Command: ./memory_allocation.bin 1 100000
==3005==
test_shared_ptr
==3005==
==3005== HEAP SUMMARY:
==3005==     in use at exit: 0 bytes in 0 blocks
==3005==   total heap usage: 200,003 allocs, 200,003 frees, 4,873,728 bytes allocated
==3005==
==3005== All heap blocks were freed -- no leaks are possible
==3005==
==3005== For counts of detected and suppressed errors, rerun with: -v
==3005== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)


$valgrind --tool=memcheck ./memory_allocation.bin 2 100000
==3010== Memcheck, a memory error detector
==3010== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3010== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==3010== Command: ./memory_allocation.bin 2 100000
==3010==
test_make_shared
==3010==
==3010== HEAP SUMMARY:
==3010==     in use at exit: 0 bytes in 0 blocks
==3010==   total heap usage: 100,003 allocs, 100,003 frees, 4,073,728 bytes allocated
==3010==
==3010== All heap blocks were freed -- no leaks are possible
==3010==
==3010== For counts of detected and suppressed errors, rerun with: -v
==3010== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

As you may see there are 200,003 allocs while using constructor versus 100,003 allocs with std::make_shared.
More info about std::make_shared

Reference cycles

Consider the next piece of code:

#include <iostream>
#include <memory>


struct B;

struct A {
    ~A() { std::cout << "dtor ~A\n"; }
    std::shared_ptr<B> b;
};

struct B {
    ~B() { std::cout << "dtor ~B\n"; }
    std::shared_ptr<A> a;
};

void test() {
    auto ptrA = std::make_shared<A>();
    auto ptrB = std::make_shared<B>();
    ptrA->b = ptrB;
    ptrB->a = ptrA;
}
int main() {
    test();
    return 0;
}

Output:

$ ./reference_cycle.bin
$

Its output is empty which means the destructors are never called due to a reference cycle: ptrA points to ptrB and vice versa. Of course, the code is naive and a programmer may easily find and break the cycle. But if you have multiple shared objects passing from one class to another you can get into this situation. However, the problem of reference cycles is not new and in order to address it the C++ standard library provides std::weak_ptr.

struct B_;

struct A_ {
    ~A_() { std::cout << "dtor ~A_\n"; }
    std::shared_ptr<B_> b;
};

struct B_ {
    ~B_() { std::cout << "dtor ~B_\n"; }
    // one of the pointers becomes weak
    std::weak_ptr<A_> a;
};

void test_() {
    auto ptrA = std::make_shared<A_>();
    std::cout << "A address: " << ptrA.get() << std::endl;
    auto ptrB = std::make_shared<B_>();
    ptrA->b = ptrB;
    ptrB->a = ptrA;
    std::cout << "Number of references to A: "
              << ptrB->a.use_count()
              << std::endl
              << "A address: "
              << ptrB->a.lock().get()
              << std::endl;
}

int main() {
    test_();
    return 0;
}

The output shows that all the destructors are called now:

$ ./reference_cycle.bin
A address: 0x7f83564017d8
Number of references to A: 1
A address: 0x7f83564017d8
dtor ~A_
dtor ~B_

More info about std::weak_ptr

`std::make_shared` together with `std::weak_ptr`

So now we know that std::make_shared saves memory allocation and std::weak_ptr may prevent reference cycles. But what happens when we use them together?

struct A_ {
    // Let us add char buffer str to class A_
    char str[256];
    A_() {
      strcpy(str, "AAAAAAAAA");
    }
    ~A_() { std::cout << "dtor ~A_\n"; }
    std::shared_ptr<B_> b;
};

struct B_ {
    ~B_() { std::cout << "dtor ~B_\n"; }
    std::weak_ptr<A_> a;
};

std::tuple<std::weak_ptr<A_>,A_*>
create_make_shared_and_return_weak_ptr() {
    auto ptrA = std::make_shared<A_>();
    std::cout << "A_ address: " << ptrA.get() << std::endl
              << "A_::str = " << ptrA->str
              << std::endl;
    auto ptrB = std::make_shared<B_>();
    ptrA->b = ptrB;
    ptrB->a = ptrA;
    std::cout << "Number of references to A_: " << ptrB->a.use_count()
        << std::endl;
    return {ptrB->a, ptrB->a.lock().get()};
}

void test_make_shared_with_weak() {
    auto[weak, raw_ptr] = create_make_shared_and_return_weak_ptr();
    std::cout << "Returned from create_make_shared_and_return_weak_ptr"
              << std::endl;
    auto strong = weak.lock();
    std::cout << "Number of references to A_: "
              << strong.use_count()
              << std::endl;
    std::cout << "Stored address: " << strong.get() << std::endl;
    std::cout << "Value of A_::str stored by original address of A_: "
              << raw_ptr
              << " is: " << raw_ptr->str << std::endl;
}

int main() {
    test_make_shared_with_weak();
    return 0;
}

In the code above the function create_make_shared_and_return_weak_ptr creates two shared pointers and then returns a copy of the weak pointer ptrB->a together with the raw pointer to the same object(it does not affect the results but we need it further). What do you expect to happen after the flow returns from create_shared_and_return_weak_ptr? We know that there were no reference cycles in the function and after exiting there are no instances of shared pointers. I’d expect the stored objects to be destructed and the memory released. But let’s see what happens:

$ ./reference_cycle.bin
A_ address: 0x7fd37ac01918
A_::str = AAAAAAAAA
Number of references to A_: 1
dtor ~A_
dtor ~B_
Returned from create_make_shared_and_return_weak_ptr
Number of references to A_: 0
Stored address: 0x0
Value of A_::str stored by original address of A_: 0x7fd37ac01918 is: AAAAAAAAA

As you see the destructors are called and the weak pointer received from function create_make_shared_and_return_weak_ptr holds a null pointer which is expected. Nevertheless, the value stored in the variable A_::str is still available by its original address(0x7fc71ec01878). How come???

Here is the explanation from cppreference.com:

If any std::weak_ptr references the control block created by std::make_shared after the lifetime of all shared owners
ended, the memory occupied by T persists until all weak owners get destroyed as well, which may be undesirable if
sizeof(T) is large.

Basically, it means that the destructor ~A_() is called explicitly but delete for the stored pointer is not!
The reason is that a shared pointer created with std::make_shared stores both the stored object and control block in a contiguous piece of memory. As a result, the two can be deleted only together. But if there is any weak pointer the control block cannot be deleted otherwise the weak pointer would not have information about the reference count. So the solution is to destruct the stored object but not to delete the whole memory block until there are weak pointers.

Thus, C++ provides two very nice features but using them together may lead to inefficient code. However, if you use std::weak_ptr it might be a good idea to reconsider the design of your application.

Design problems

So, for now, we know that shared pointer is relatively expensive to copy by value, requires specific instantiation to save memory allocations, and in the case of incorrect usage may lead to cycle references. And we already know most of the problems can be addressed by using the right language constructions.

Despite this, there is another problem - design. In general, C++ is a language that expects a programmer to have full control of used resources and objects’ lifecycles. Shared pointers make the application’s memory model more complex and
couplings between its parts are hard to track. Thus, the whole application becomes more bug-prone.

Conclusion

After reviewing all the cases above I came to the conclusion:

Try to follow the single ownership principle
Prefer to use objects with automatic storage duration
If you need a pointer, try using unique_ptr
If you have to use shared_ptr make sure you don’t overuse it and keep in mind it’s features

Credits

Thanks to Sergey Pastukhov and Orian Zinger for help.

How to enable in-band FEC for Opus codec

Dima Danilov — Mon, 02 Dec 2019 22:00:00 +0000

As a reader may know UDP network protocol does not support either message retransmission or acknowledge. It means that
all these problems are supposed to be handled at the application level. In the audio/video transmission domain the problem
is usually solved with forward error correction technique.
The idea is to add some redundant data to a message or messages so that if one of the messages is lost, a receiver could
replicate it from a combination of other messages and the redundant data. This redundant data is called FEC.

A developer is supposed to decide how much FEC to add to the messages at the encoder side and how to decode
it at the decoder side. It sounds like a piece of work!
So Opus codec developers decided to make our life easier and added in-band FEC support to the codec.

In order to make the OPUS encoder to add FEC a user has to set the following configuration:

Packet time(ptime) has to be not less than 10ms otherwise OPUS works in the CELT mode and not in SILK.
Don't use very high bitrates. For example, I use 24 kbps.
In order to use FEC, your bitrate should be higher. For example, if the sample rate is 8kHz then the bitrate has to be 12 kbps or 24 kbps. The encoder needs a higher bitrate to have room for LBRR packets containing FEC.
FEC must be enabled via OPUS_SET_INBAND_FEC.
The encoder must be told to expect packet loss via OPUS_SET_PACKET_LOSS_PERC.

Here is an example of configuring the OPUS encoder.

opus_encoder_ctl(encoder, OPUS_SET_INBAND_FEC(TRUE));
opus_encoder_ctl(encoder,OPUS_SET_PACKET_LOSS_PERC(opus_packet_loss));

I configure opus_packet_loss via config file but it may be configured in realtime.
For example, you can update OPUS according to RTCP
statistics.

Now, after the encoder is ready, let's configure the decoder.
First of all, you need to know if a packet is lost.
The most natural way to know it is to check the sequence number of RTP packets.
When you know that the previous packet is lost you are supposed to decode the current packet twice:
first time with FEC turned on to reproduce the lost packet, and then with FEC turned off to decode the current packet.

/* Decode the lost packet */
opus_decoder_ctl(decoder, OPUS_GET_LAST_PACKET_DURATION(frame_size));
opus_decode(    decoder,
        buffer, /* buffer to decode */,
        length, /* number of bytes in buffer */
        sampv,  /* output buffer */
        frame_size,
        1);   /* in-band FEC is turned on */
play_buffer(buffer);
/*Decode the current packet*/
opus_decode(    decoder,
        buffer, /* buffer to decode */,
        length, /* number of bytes in buffer */
        sampv,  /* output buffer */
        frame_size,
        0);   /* in-band FEC is turned off */
play_buffer(buffer);

Summary

We learned how to enable in-band FEC for Opus codec but the last thing to talk about is the pros and cons of using it!

Pros:

No need to develop an in-house mechanism of adding FEC to your packets.
Most of the third parties like SIP servers(Asterisk) or WebRTC support Opus in-band FEC out of the box.
FEC may increase the audio quality even when there is a packet loss.

Cons:

Tricky to configure.
There is no obvious way to see if traffic actually carries in-band FEC except generating packet loss and listening to an audio sample. I used a sine wave generated by: ffmpeg -f lavfi -i "sine=frequency=1000:sample_rate=8000:duration=5" output.wav.
Since Opus packet contains information only about the prior packet in-band FEC can replicate only a single packet loss. The problem is that usually packets are lost in a burst.
Enabling FEC increases bitrate and bandwidth.

Using strace for debugging, part 2

Dima Danilov — Sun, 18 Aug 2019 21:00:00 +0000

This is the second post about debugging with strace.

In our solution, we have a process that runs another process.
The parent communicates with the child process via TCP socket.
In order to know whether the child is ready for communication the parent reads
its stdout and looks for the readiness indication string, e.g. "I'm ready".
Everything worked fine but suddenly my colleague and I started observing a strange behavior:
the child process was stuck after a while.

So I ran strace -p 1666, where 1666 is the child' id.
strace output:

$sudo strace -p 25485
strace: Process 25485 attached
write(1, "Some important information...."..., 101

The child process was trying to write something to
stdout(1 is the file descriptor of stdout) but was unable to do it!
I tried to reproduce the issue by running the process manually.
And to our surprise, the issue did not reproduce! So I started looking at the way our application was starting the process:

cmd = ['./child.py']
handle = subprocess.Popen( cmd, stdout=subprocess.PIPE,
                           stderr = subprocess.STDOUT,
                           close_fds=True, shell=True)
wait_for_child_readiness(handle)
#do_some_real_work()
handle.wait()

It looked quite normal: Start a process and then forward stdout to stderr
and pipe both to the parent process. And then the parent process reads the output:

def wait_for_child_readiness(proc_handle):
        last_stdout = ""
        while "CHILD IS READY" not in last_stdout :
                last_stdout = proc_handle.stdout.read(512)
                sys.stdout.write(last_stdout)
                sys.stdout.flush()
        sys.stdout.write("\nParent detected child's readiness!\n")

Then we read the output until we find the indication string.
But wait! The child process continues writing to stdout/stderr and the pipe continues sending
the output to the parent process. But the parent process does not read it.
And then pipe overflow happens and the child process cannot write anything to stdout!

The solution was to continue reading the output of the child process.
Hence once you start reading child process output don't stop doing that!;)

P.S. The way to know the pipe size on Linux is to check:

$cat /proc/sys/fs/pipe-max-size
1048576

The default value for this file is 1048576 (1 MiB)

http://man7.org/linux/man-pages/man7/pipe.7.html

Using strace for debugging

Dima Danilov — Mon, 03 Jun 2019 21:00:00 +0000

Debugging is an important part of software engineering and every
developer has a few debugging tools in his toolbelt.
Usually, I use logs and a debugger and that's enough for my everyday work.
But lately, I had to work with an open source SIP client called baresip.
Baresip is a very stable open-source SIP client with an excellent design supporting plugins.

After a while, I discovered that sometimes baresip was unable to start playing
audio for outgoing calls because of system error "Device or resource is busy".
I noticed it did not happen when the client was answering an incoming call.
My first impression was that baresip tried to open audio for playing incoming audio twice.
I tried to find this place in the code but since I was not familiar with the code I did not
manage to do that. And then I thought of strace.

strace is a userspace tool that prints all system calls of a given process.
You may start a process inside strace or you can attach it to a running process.
I ran baresip with strace: strace -o strace.log baresip -e 7100, where:

-o filename Write the trace output to the file filename rather than to stderr.
strace.log is the output log file
baresip is the process under investigation
-e is the command line argument saying to baresip to dial the number 7100.

And then I looked for "Device or resource is busy" in the log file.

And as we can see it happens in line 1308. By the way, vim has a builtin syntax highlight for strace logs.

So I looked for open("/dev/snd/pcmC0D0p", O_RDWR|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC).

As you see the open returns file descriptor "14". So let's look whether baresip closes the file descriptor:

Next open call:

And the corresponding close call in line 1484:

And if you still follow me here is the next open call:

So as you may see the next open function call happens in line 1308 while close is called in line 1484.
Meaning baresip opens the audio device before closing it.
Looks like we found out the problematic function calls!

As far as I understand baresip opens the audio device to play
incoming audio but why does it open it before? So let's take a look at what happens around the second open call. In line 1120 we see that baresip opens and then reads ringback.wav file which is
a ringback tone, the audible ringing that is heard by the calling party after dialing:

And after that, it obviously opens the audio device in order to play the file.
And while the audio device is playing the ringtone another thread receives a response
from the callee and opens the same audio device in order to play the incoming audio.
And then the problem happens.

This is how with very limited knowledge about the application I managed to find out the root cause.
I didn't fix the issue because it required significant changes in baresip' design but I
found a quick workaround by disabling the ringback in a config file.

Python garbage collector and the other magic

Dima Danilov — Wed, 08 Aug 2018 21:00:00 +0000

Once upon a time, in the company where I work there was a bug in production that caused to the Too many files open error.

I was able to locate the class responsible for the issue, it was a class working with os.pipefile descriptors, let’s call it OsPipeHolder. You may find the simplified version of the code below:

#Listing #1

#!/usr/bin/python
import os


class OsPipeHolder(object):
    def __init__( self ):
        read, write = os.pipe()
        self._read = os.fdopen( read, "r" )
        self._write = os.fdopen( write, "w" )
        self.isClosed = self.is_closed

    def is_closed(self):
        return self._read and self._write

    def close(self):
        self._read.close()
        self._write.close()
        self._read = None
        self._write = None

    def __del__(self):
        print "You've deleted me!!!"
        if not self.is_closed():
            self.close()


if __name__ == "__main__":
    pipe = OsPipeHolder()
    del pipe

The problem was that the application made a retry in the event of
failure and created a new instance of OsPipeHolder class.

As you may see from the code above there is __del__ method that should be called by the garbage collector.

But… for no reason, it is never called and the file descriptors are left open:

$ ./OsPipeHolder.py
$

As the reader may know, Python’s garbage collector destroys objects not referenced from the stack(1 or fewer references).

Despite we create only one instance of the object in the line #30 and do not copy it elsewhere, I propose to verify the number of references with sys.getrefcount:

#Listing #2

import os
import sys

class OsPipeHolder(object):
...

if __name__ == "__main__":
    pipe = OsPipeHolder()
    #we need -1 since passing an object to getrefcount
    #creates an additional reference
    print "Refcount:", (sys.getrefcount(pipe) -1)               
    del pipe

Run results #2:

$ ./OsPipeHolder_refcount.py
Refcount: 2

As you see, every time we create an instance of OsPipeHolder Python creates two references!
So, maybe there is an internal reference inside the object itself.

In order to check it I have decided to print the information on all the attributes of OsPipeHolder:

#Listing #3

#!/usr/bin/python
import os
import sys

class OsPipeHolder(object):
...

if __name__ == "__main__":
    pipe = OsPipeHolder()
    #we need -1 since passing an object to getrefcount
    #creates an additional reference
    print "Refcount:", (sys.getrefcount(pipe) -1)          
    for i, attribute in enumerate(dir(pipe)):
            msg = "%d. Attribute name: %s\tinfo: %s" % (i, attribute, (getattr(pipe, attribute)))
            print msg

    del(pipe)

Run results #3:

$ ./OsPipeHolder.py
Refcount: 2
0. Attribute name: __class__ info: 
1. Attribute name: __del__ info: >
...
21. Attribute name: close info: >
22. Attribute name: isClosed    info: >
23. Attribute name: is_close    info: >

At first glance, everything looks good, but in the line #22 we see that isClosed is a reference to the method is_close.
This is the inner reference cycle we were looking for!

Let’s comment it out:

#Listing #4

#!/usr/bin/python
import os
import sys

class OsPipeHolder(object):
    def __init__( self ):
        read, write = os.pipe()
        self._read = os.fdopen( read, "r" )
        self._write = os.fdopen( write, "w" )
        #self.isClosed = self.is_closed
...

Run results #4:

$ ./OsPipeHolder.py
Refcount: 1
You've deleted me!!!

Yey, finally our method __del__ was called!

But we still have two issues:

We cannot remove isClosed since it was added for backward compatibility purposes
Python is supposed to handle reference cycles easily!

Let’s start with the first.

I found the solution in Python sources.
In order to create an alias to a function, you simply declare isClosed as a "class-level attribute"(line #16).

#Listing #5

#!/usr/bin/python
import os
import sys

class OsPipeHolder(object):
    def __init__( self ):
        read, write = os.pipe()
        self._read = os.fdopen(read, "r")
        self._write = os.fdopen(write, "w")

    def is_closed(self):
        return self._read and self._write

    isClosed = is_closed

    def close(self):
        self._read.close()
        self._write.close()
        self._read = None
        self._write = None

    def __del__(self):
        print "You've deleted me!!!"
        if not self.is_closed():
            self.close()

if __name__ == "__main__":
    pipe = OsPipeHolder()
    print "Refcount:", (sys.getrefcount(pipe) -1)
    del(pipe)

Run results #5:

$ ./OsPipeHolder.py
Refcount: 1
You've deleted me!!!
$ ./OsPipeHolder.py
Refcount: 1
You've deleted me!!!

As you see, we have only one reference to the instance and the garbage collector calls our __del__ method!
But why isn’t it called in the original code?
Let's take a look at the memory layout of our object:

As you see we have one reference(Ref #1) to the object from the stack and another inner reference (Ref #2).
After the Ref #1 is deleted and we have no other references from the stack the garbage collector is supposed to call the __del__ method despite the inner reference.
Why does it not happen?

Python documentation is your best friend and has an answer for everything!

A list of objects which the collector found to be unreachable but could not be freed (uncollectable objects).
By default, this list contains only objects with __del__() methods.
1 Objects that have __del__() methods and are part of a reference cycle
cause the entire reference cycle to be uncollectable,
including objects not necessarily in the cycle but reachable only from it.

Python docs: https://docs.python.org/2/library/gc.html#gc.garbage

As you see, the __del__ method itself was the root cause!

Summary

Don’t create aliases with self since it leads to a redundant reference(only alias = method_name and not self.alias = self.method_name)
Remember the Zen of Python saying: “Explicit is better than implicit.” (call close method explicitly or use the with statement)
Python is not C++ so do not implement RAII there
Think in the Pythonic way and read the docs!:)

P.S. Originally I posted it at Medium

DEV Community: Dima Danilov

The switch from Vim

Visual Studio Code

Neovim

Neovim + Neovim-Qt

Screenshots:

std::shared_ptr is an anti-pattern

A bit of history

A few words about how shared_ptr works

The ideal usage

Incrementation of the reference counter and multi-threading

Shared pointer initialization

Reference cycles

std::make_shared together with std::weak_ptr

Design problems

Conclusion

Credits

How to enable in-band FEC for Opus codec

Summary

Pros:

Cons:

Using strace for debugging, part 2

Using strace for debugging

Python garbage collector and the other magic

Summary

A few words about how `shared_ptr` works

`std::make_shared` together with `std::weak_ptr`