DEV Community: Kevin Albertson

Test Driven Learning

Kevin Albertson — Tue, 14 Mar 2023 00:58:37 +0000

Idea: learn a new code base by writing tests. Read all of the public API. Come up with questions and try to answer them by writing tests.

I tried this idea with the libbson. I help maintain libbson. I was curious to see what (if anything) I would learn by surveying and testing the public API.

This is a summary of the results.

Discover API

I learned about the existence of the following:

BCON_EXTRACT to extract values from a bson_t.
bson_unichar_t and related API to represent a Unicode codepoint in a uint32_t.
bson_writer_t to write a sequence of BSON documents to a buffer.

Find Surprising Behavior

bson_copy_to (<stack allocated>, <heap allocated>) results in a leak. For example:

static void test_bson_copy_to(void) {
    bson_t *dst = bson_new(); // dst is heap allocated.
    bson_t src = BSON_INITIALIZER; // src is stack allocated.
    BCON_APPEND(&src, "a", BCON_INT32(1));
    bson_copy_to(&src, dst); // struct for `bson_t` is leaked!
    ASSERT_BSON_EQUAL(&src, dst);
    bson_destroy(&src);
    bson_destroy(dst);
}

bson_validate double validates UTF-8 values. bson_iter_visit_all validates UTF-8, as does the visitor functions.

Found that BSON_CHECK_VERSION is incorrectly documented. "is greater than" should be "is greater than or equal to". Fixed with (https://github.com/mongodb/mongo-c-driver/pull/1219).

Answer Open Questions

As I read through API, I wrote down many open questions and answered them with tests:

Q: Does bson_append_array reject documents with non integer string keys?
- A: No. But bson_append_array warns if the first key is not "0".
Q: Does bson_append_utf8 error if given invalid UTF-8? A: No.
Q: If the default bson_context_t is used, is a child process likely to produce the same bson_oid_t?
- A: No. The default bson_context_t disables the PID cache.
Q: Does the max len option of bson_json_opts_new produce invalid UTF-8 characters if on multi-byte character boundary? A: Yes.
Q: Does bson_concat support self-concatenation? A: Yes
Q: Can bson_iter_find_descendant and bson_iter_recurse use the same iterator for input and output? A: Yes
Q: Will parsing an integer value in JSON promote to int64 if it does not fit in int32? A: Yes.
Q: How does bson_string_t grow allocations? A: By power of 2.

Learn more C

Learned about static keyword applied to array sizes: https://hamberg.no/erlend/posts/2013-02-18-static-array-indices.html

Learned that aligned_alloc requires alignment to be a power of 2 >= sizeof(void*), and memory requested be a multiple of alignment.

Conclusion

I think this was a valuable time investment.

For further reading, I recommend David Golden's A better way to learn a new codebase. That article inspired this idea.

"Specify /EHsc" warning

Kevin Albertson — Mon, 20 Feb 2023 01:01:45 +0000

Using cmake to build C++ with custom CMAKE_CXX_FLAGS on MSVC may result in a warning to Specify /EHsc.

Here is an example:

# CMakeLists.txt

cmake_minimum_required (VERSION 3.23)
project(ehsc)
add_executable(ehsc ehsc.cpp)

// ehsc.cpp
#include <iostream>

class NeedsDestruction {
public:
    NeedsDestruction () {
        std::cout << "Constructor called" << std::endl;
    }

    ~NeedsDestruction () {
        std::cout << "Destructor called" << std::endl;
    }
};

void throws () {
    NeedsDestruction nd;
    throw std::exception("exception");
}

int main () {
    try {
        throws ();
    } catch (std::exception e) {
        std::cout << "caught exception" << std::endl;
    }
}

Configure with a custom CMAKE_CXX_FLAGS:

cmake -S. -Bcmake-build -DCMAKE_CXX_FLAGS="/Wall"
cmake --build cmake-build --target ehsc

Building results in many warnings suggesting to Specify /EHsc:

warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc

Why does specifying CMAKE_CXX_FLAGS result in this warning? Because cmake specifies default flags for MSVC.
Setting CMAKE_CXX_FLAGS appears to have the side-effect of overwriting the default MSVC flags set by cmake.
For example, if CMAKE_CXX_FLAGS is not set, this is the output of printing CMAKE_CXX_FLAGS with cmake 3.23.1 on my Windows machine:

CMAKE_CXX_FLAGS=/DWIN32 /D_WINDOWS /W3 /GR /EHsc

Excluding /EHsc from the build flags appears particularly problematic. /EHsc is the documented default for the exception handling model:

Use an /EH compiler option to specify the exception handling model to use in a C++ project. Standard C++ exception handling (/EHsc) is the default in new C++ projects in Visual Studio.

Without /EHsc, the destructor in the example is not called when the exception is thrown. Executing does not print the expected Destructor called:

throws ... begin
Constructor called
caught exception

How I work

Kevin Albertson — Sun, 12 Feb 2023 16:47:12 +0000

Here is a sample timeline of my typical day:

Respond to Urgent Requests

I start the day by responding to urgent requests. I check slack and e-mail and only respond if the request is urgent or fast to respond to. The less I can divide my attention the more effective I can be at completing the Daily Plan.

Make a Daily Plan

The Daily plan is three items. The first item is the highest priority.

Here is a sample Daily Plan written in the margin of a work notebook:

Plan achievable tasks. It is better to err towards small achievable tasks, than have big tasks that go unfinished. The more you finish the Daily Plan, the more you will trust yourself to stick to your Daily Plan.

For big tasks, I set daily tasks to spend time. E.g. if X is a task that will take several days, my daily task will be "Spend 30 minutes on X" rather than "Do X".

Estimate to complete the Daily Plan with half of the time available in the day. E.g. if you are in meetings for 2 hours, you have 6 hours of working time. Estimate to spend 3 hours on the Daily Plan. This padding accounts for unexpected interruptions. If the Daily Plan is finished early, work on other backlog items.

Review Input Streams

At the end of the day, I review a checklist of input streams. Streams are reviewed in order of urgency:

Slack. Respond to all messages.
Github. Review quick low complexity code reviews.
Calendar. Accept/decline/reschedule all invites for the next day.
E-mail. Respond/archive/delete/label all e-mails. Non-urgent e-mails requiring a response are often labeled with "Later".
TODO list.
Jira.

After reviewing the checklist, I have the state in my head of all pending items. Make the Daily Plan for the next day.

Here is a sample checklist to end the day, with the next day's Daily Plan:

How I take notes

Kevin Albertson — Thu, 02 Feb 2023 01:46:00 +0000

Keep notes close to code

I keep notes in the same directory as code. This makes it easy to reach. Here is a sample for notes taken when working on Wireshark:

The newest notes are at the top of the file. This results in less scrolling to see recent notes or add new notes.

Each logical section is separated with the line containing --.

I use a global .gitignore_global to ignore this file from git. The file is named KEVINALBS-README.md.

Keep TODO lists and Open Questions on paper

I find hand writing TODO lists helpful. It permits checking off a TODO list item.

I write Open Questions on paper in this format:

The "A:" with empty space means it is an open question. I check off the "Q:" when the question is answered. I think I am more likely to remember Open Questions when I hand write them. That cues me to look for answers if I see them. Hand writing is a larger barrier than typing. I am more likely to make a shorter and clearer question when hand writing.

Draft external communication

I draft descriptions for Pull Requests in my text editor. This permits writing them as I work on code. Example:

I draft descriptions for Jira tickets in my text editor. Jira does not have a "save draft" feature. And often I may not have all information, or time / energy to write a full ticket in one sitting.

Time box work

Kevin Albertson — Sun, 29 Jan 2023 20:04:17 +0000

I work as a technical lead. I started working remotely in 2020. My job permits flexible working hours. In the start of 2020, I worked for periods of varying hours interspersed with long breaks. I never felt like I was "done" with work. This schedule stressed me and made me less productive. This post describes habits that have helped me define work-life boundaries and best utilize my time working.

Work fairly

Consider compensation. I am salaried and expected to work 40 hours a week. I uphold my agreement and work for 40 hours a week. During those 40 hours, I work to the best of my ability. If I need to work less, I request PTO. I may work more than 40 hours on exception if there is a need. Working more than 40 hours devalues my time.

Working 8 hours a day gives a clear "done" criteria. Once I have worked 8 hours, I sign off. I have upheld my side of the agreement with work.

One way to develop this habit is by tracking time. Another way to develop this habit is working a regular schedule.

Work a regular schedule

I set a schedule of working 8:00am-4:30pm. Other employees can rely on me being online and responsive during my working hours. I set the working hours in Google Calendar. If meetings are scheduled outside of my working hours, I ask to reschedule.

I take breaks to refresh. Between tasks, I get up and walk around for ~5 minutes. I take a 30 minute lunch. Lunch is scheduled on my calendar. I do not check slack or e-mail during lunch.

I do not use working more than 40 hours as an excuse for working less than 40 hours the next week. I continue to work my regular schedule.

One counterargument to consider: I have varying energy during the day. I have the most energy in the morning, and less in the afternoon. I may be more productive by taking a long nap in the afternoon, and working later in the evening. But also consider: why use all of the highest energy on work? If I only work when I have energy, I will have no energy left for my personal life. I prefer working for a continuous 8 hour period, signing off, then using my remaining energy in my personal life.

Time box tasks

Time box tasks to clarify "done" criteria for tasks.

Once the requirements are met, there are always optional improvements to make. I do the best work I can in the time allotted. Testing and self-reviewing code are requirements. Skipping tests may result in bugs that cost more time later. Skipping self-review may result in more time spent for my peers reviewing.

For example: I am writing a test to compare byte output of a function.

ASSERT(got.len == expected.len && 0 == memcmp(got.data, expected.data, expected.len));

That line took ~5 seconds to write. It meets the minimum requirements. If the assertion fails, it prints this message:

FAIL:/home/kevin/code/test_c_driver/test-timebox.c:14  main()
  Condition 'got.len == expected.len && 0 == memcmp(got.data, expected.data, expected.len)' failed.

The message may be more helpful if the data is also printed on failure. I add that:

// bytes_to_hex returns string for the hex representation of `in`.
// The returned string must be freed with `free`.
static char *bytes_to_hex(const uint8_t *in, size_t len) {
    char *out = malloc(len * 2 + 1);
    const char *hex_table = "01234567890ABCDEF";
    char *start = out;
    for (size_t i = 0; i < len; i++) {
        uint8_t v = in[i];
        *out++ = hex_table[v / 16];
        *out++ = hex_table[v % 16];
    }
    *out = '\0';
    return start;
}

The assertion becomes:

ASSERTF(got.len == expected.len && 0 == memcmp(got.data, expected.data, expected.len),
        "got     : %s\n"
        "expected: %s",
        bytes_to_hex(got.data, got.len), bytes_to_hex(expected.data, expected.len));

That took ~2 minutes. If the assertion fails, it prints this message:

FAIL:/home/kevin/code/test_c_driver/test-timebox.c:33  main()
  Condition 'got.len == expected.len && 0 == memcmp(got.data, expected.data, expected.len)' failed.
got     : 626E6E626172
expected: 666E6E626172

That looks better. After refactoring other ASSERTF statements, this becomes verbose. I write a macro to reduce the repetition:

#define ASSERT_MEMEQUAL(got, expected)                                                             \
    ASSERTF(got.len == expected.len && 0 == memcmp(got.data, expected.data, expected.len),         \
            "got     : %s\n"                                                                       \
            "expected: %s",                                                                        \
            bytes_to_hex(got.data, got.len), bytes_to_hex(expected.data, expected.len));

And the assert becomes:

ASSERT_MEMEQUAL(got, expected);

That took another ~30 seconds.

The whole task took ~2 minutes 35 seconds. The extra effort added value. Good test assertion messages can save time diagnosing test failures later.

This could further be improved. Tests with large data print large horizontal strings on failure. bytes_to_hex could add newlines. Assert failures can print a side-by-side diff. I can audit other tests that compare bytes and use the new ASSERT_MEMEQUAL. How far I go with improvements is informed by how much time I want to spend on this task.

Handling notifications

Turn off notifications off-hours. Only check notifications off-hours if you are prepared to act on them.
Example: I kick off a CI test run just before signing off. That night I feel anxious about the results and want to check. I want the tests to pass. But I ask myself, "If the tests failed, do I have energy to investigate?" The answer is "No". Checking the test results will not change my behavior in that moment. Seeing failing tests will stress me. So I do not check the results.

Turn on notifications on-hours. For someone in on-call, on-hours may be 24 hours. Be responsive on-hours. Respond to direct messages with a specific ETA "I will investigate this afternoon" rather than a vague "I will investigate later". If you do not respond, the sender can check in at that time. If it is urgent, the sender can request you prioritize it sooner.

Order of init and del

Kevin Albertson — Fri, 25 Nov 2022 14:02:34 +0000

The order of calls to __del__ surprised me. In Python:

class A:
    def __init__ (self):
        print ("A.__init__")
    def __del__ (self):
        print("A.__del__")

class B:
    def __init__ (self):
        print ("B.__init__")
    def __del__ (self):
        print ("B.__del__")

def fn():
    a = A()
    b = B()

fn ()
"""
Produces this output:
A.__init__
B.__init__
A.__del__
B.__del__
"""

I expected __del__ to be called in reverse order (ABBA). The reverse order matches C++ behavior.

An alternative to guarantee order is __enter__ and __exit__ and use of the with block:

class A:
    def __enter__ (self):
        print ("A.__enter__")
    def __exit__ (self, exc_type, exc_val, exc_tb):
        print("A.__exit__")

class B:
    def __enter__ (self):
        print ("B.__enter__")
    def __exit__ (self, exc_type, exc_val, exc_tb):
        print ("B.__exit__")

def fn():
    with A() as a, B() as b:
        pass

fn ()

"""
Produces this output:
A.__enter__
B.__enter__
B.__exit__
A.__exit__
"""

The standard guarantees the order:

With more than one item, the context managers are processed as if multiple with statements were nested:
with A() as a, B() as b:
    SUITE
is semantically equivalent to:
with A() as a:
    with B() as b:
        SUITE

Things learned from K&R C

Kevin Albertson — Wed, 16 Nov 2022 13:12:31 +0000

Pointers

Pointer arithmetic on void * is not allowed since the size of pointed-to data is unknown.

"dangling pointer" == points to a deleted object
"wild pointer" == ~~points to uninitialized~~ uninitialized pointer

Converting from an array to a pointer loses size information

char str[256];
printf ("%zu\n", sizeof str); // 256
char *pstr = str;
printf  ("%zu\n", sizeof pstr); // 4 or 8 (size of pointer).

Memory management

"static" == allocated at compile time
"automatic" == allocated on the stack
"dynamic" == allocated on the heap

Types of behavior

"implementation-defined" == implementation must document. E.g. propagation of sign bit on >> operation.
"undefined behavior" == no requirements. E.g. signed overflow.
"unspecified behavior" == two or more possible behaviors E.g. order of argument evaluation.
"locale-specific behavior" == depends on local conventions. E.g. whether tolower returns true for characters other than 26 lowercase Latin letters.

The number of bits in a byte is implementation defined.

An assignment is an expression evaluating to the left-hand side. E.g. (c = getchar()) evaluates to c.

extern can be used inside a function definition. E.g.

int x;
void foo() {
    extern int x;
}

External and static variables are initialized to 0.

String literals cannot be modified. E.g.

char *s = "abc";
s[0] = `d`; // not allowed

C89 int division with negative numbers is implementation defined. 7 / -2 can be -3 or -4 (round up or down). C99 defines as -3 (toss the decimal after division).

If fn is called but not declared, the compiler assumes it is defined as int fn().

Arithmetic and comparison of pointers that do not point to the same array is undefined behavior (exception is one past the last element in the array).

You may omit the first dimension of a multi-dimensional array in parameters. E.g. void f(int arr[][13])

argv[argc] is NULL.

sizeof does not require parenthesis for objects. sizeof object vs. sizeof (type name).

C supports bit fields. The order of bits is implementation defined. E.g.

struct {
    unsigned int is_keyword : 1;
    unsigned int is_extern : 1;
    unsigned int is_static : 1;
}

printf format is %[-][<width>][.][<precision>][h|l][<conversion>]. <width> or <precision> may be *.

FILE* is OS independent. File descriptors are Unix specific.

Bit shift wraps values

Kevin Albertson — Fri, 19 Aug 2022 00:57:35 +0000

#include <inttypes.h>
#include <stdint.h>
#include <stdio.h>

int main () {
    for (uint32_t shift = 0; shift < 35; shift++) {
        uint32_t u32 = UINT32_MAX >> shift;
        printf ("1 >> %02" PRIu32 " = %" PRIu32 "\n", shift, u32);
    }
}

This example may be run here.

I expected that UINT32_MAX >> 32 to result in 0. But the output disagrees:

1 >> 00 = 4294967295
1 >> 01 = 2147483647
1 >> 02 = 1073741823
1 >> 03 = 536870911
1 >> 04 = 268435455
1 >> 05 = 134217727
1 >> 06 = 67108863
1 >> 07 = 33554431
1 >> 08 = 16777215
1 >> 09 = 8388607
1 >> 10 = 4194303
1 >> 11 = 2097151
1 >> 12 = 1048575
1 >> 13 = 524287
1 >> 14 = 262143
1 >> 15 = 131071
1 >> 16 = 65535
1 >> 17 = 32767
1 >> 18 = 16383
1 >> 19 = 8191
1 >> 20 = 4095
1 >> 21 = 2047
1 >> 22 = 1023
1 >> 23 = 511
1 >> 24 = 255
1 >> 25 = 127
1 >> 26 = 63
1 >> 27 = 31
1 >> 28 = 15
1 >> 29 = 7
1 >> 30 = 3
1 >> 31 = 1
1 >> 32 = 4294967295
1 >> 33 = 2147483647
1 >> 34 = 1073741823

1 >> 32 is undefined behavior by the C standard 6.5.7 of left shift:

If the value of the right operand is negative or is greater than or equal to the width of the promoted left operand, the behavior is undefined.

TIL