DEV Community: aodinokov

C Reflection Magic: Simple Logging with A Wrapper for Printing Arbitrary Functions Arguments and Results

aodinokov — Thu, 06 Jun 2024 13:51:04 +0000

This article is a research report which covers some potential implementation aspects of writing a helper wrapper which will automatically log arguments and results of the arbitrary C function. This is one of the examples why reflection may be useful even in C. The implementation is based on the Metac project. The introduction of it was given in this article. The research has some good results, but it still in progress. The comments on how it could be done in a better way are appreciated.

Logging is one of the important ways of debugging. Making proper logging is a key to understanding what potentially went wrong without using a debugger. But it’s annoying to print out all the arguments of each function and its result. C reflection with Metac could potentially have an ability to do this, because debugging information provided by DWARF has all the data about the type of each argument. Check it out. Here is the testing application:

#include <stdio.h>
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>

#include "metac/reflect.h"

int test_function1_with_args(int a, short b) {
    return a + b + 6;
}
METAC_GSYM_LINK(test_function1_with_args);

int main() {
    printf("fn returned: %i\n", test_function1_with_args(1, 2));

    return 0;
}

We want to make some kind of wrapper to print arguments of test_function1_with_args. Metac will generate its reflection info since METAC_GSYM_LINK(test_function1_with_args); is in the code. For simplicity int and short argument types are selected. The first idea how we could create a wrapper is - create a macro:

void print_args(metac_entry_t *p_entry, ...) {
// use va_args and debug information about types to print value of each argument
}

#define METAC_WRAP_FN(_fn_, _args_...) ({ \
        print_args(METAC_GSYM_LINK_ENTRY(_fn_), _args_); \
        _fn_(_args_); \
    })

int main() {
    // use wrapper instead of printf("fn returned: %i\n", test_function1_with_args(1, 2));
    printf("fn returned: %i\n",
        METAC_WRAP_FN(test_function1_with_args, 1, 2));

    return 0;
}

This wrapper so far handles only arguments, but it’s ok for the first step. Lets try to implement print_args. Here is the first naive attempt:

void print_args(metac_entry_t *p_entry, ...) {
    if (p_entry == NULL || metac_entry_has_paremeter(p_entry) == 0) {
        return;
    }

    va_list args;
    va_start(args, p_entry);

    printf("%s(", metac_entry_name(p_entry));

    // output each argument
    for (int i = 0; i < metac_entry_paremeter_count(p_entry); ++i) {
        if (i > 0) {
            printf(", ");
        }

        // get i-th arg
        metac_entry_t * p_param_entry = metac_entry_by_paremeter_id(p_entry, i);
        if (metac_entry_is_parameter(p_param_entry) == 0) {
            // something is wrong
            break;
        }
        // if it’s … argument just print … - there is no way so far to handle that
        if (metac_entry_is_unspecified_parameter(p_param_entry) != 0) {
            // we don't support printing va_args... there is no generic way
            printf("...");
            break;
        }

        // get arg name and info about arg type
        metac_name_t param_name = metac_entry_name(p_param_entry);
        metac_entry_t * p_param_type_entry = metac_entry_parameter_entry(p_param_entry);
        if (param_name == NULL || param_name == NULL) {
            // something is wrong
            break;
        }

        // lets handle only base_types for now
        if (metac_entry_is_base_type(p_param_type_entry) != 0) {
            // take what type of base type it is. It can be char, unsigned char.. etc
            metac_name_t param_base_type_name = metac_entry_base_type_name(p_param_type_entry);

// if _type_ is matching with param_base_type_name, get data using va_arg and print it.
#define _base_type_arg_(_type_, _pseudoname_) \
    do { \
        if (strcmp(param_base_type_name, #_pseudoname_) == 0) { \
            _type_ val = va_arg(args, _type_); \
            metac_value_t * p_val = metac_new_value(p_param_type_entry, &val); \
            if (p_val == NULL) { \
                break; \
            } \
            char * s = metac_value_string(p_val); \
            if (s == NULL) { \
                metac_value_delete(p_val); \
                break; \
            } \
            printf("%s: %s", param_name, s); \
            free(s); \
            metac_value_delete(p_val); \
        } \
    } while(0)
    // handle all known base types
    _base_type_arg_(char, char);
    _base_type_arg_(unsigned char, unsigned char);
    _base_type_arg_(short, short int);
    _base_type_arg_(unsigned short, unsigned short int);
    _base_type_arg_(int, int);
    _base_type_arg_(unsigned int, unsigned int);
    _base_type_arg_(long, long int);
    _base_type_arg_(unsigned long, unsigned long int);
    _base_type_arg_(long long, long long int);
    _base_type_arg_(unsigned long long, unsigned long long int);
    _base_type_arg_(bool, _Bool);
    _base_type_arg_(float, float);
    _base_type_arg_(double, double);
    _base_type_arg_(long double, long double);
    _base_type_arg_(float complex, complex);
    _base_type_arg_(double complex, complex);
    _base_type_arg_(long double complex, complex);
#undef _base_type_arg_
        }
    }
    printf(")\n");
    va_end(args);
    return;
}

If we run it we will see:

% ./c_print_args
test_function1_with_args(a: 1, b: 2)
fn returned: 9

It works! But it handles only base types. And we want it to be universal.

The main challenge here is with this line:

 _type_ val = va_arg(args, _type_);

C's va_arg macro requires the type of the argument to be known at compile time. However, reflection information only provides type names at runtime. Can we trick it? va_arg is a macros which covers a builtin function. The second parameter is a type (very non-typical thing). But why does this thing at all needs the type? The answer is - to understand the size and to be able to take it from the stack. We need to cover all possible sizes and to get a pointer to the next argument. On Metac side we know the size of argument - we can use this snippet to get it:

        metac_size_t param_byte_sz = 0;
        if (metac_entry_byte_size(p_param_type_entry, &param_byte_sz) != 0) {
            // something is wrong
            break;
        }

As a next idea let's make the macro which will cover 1 size and make sure that we handle it properly:

        char buf[32];
        int handled = 0;
#define _handle_sz_(_sz_) \
        do { \
            if (param_byte_sz == _sz_) { \
                char *x = va_arg(args, char[_sz_]); \
                memcpy(buf, x, _sz_); \
                handled = 1; \
            } \
        } while(0)
        _handle_sz_(1);
        _handle_sz_(2);
        _handle_sz_(3);
        _handle_sz_(4);
// and so on ...
        _handle_sz_(32);
#undef _handle_sz_

With this approach we covered different sizes from 1 to 32. We could generate a code and cover arguments sized till any arbitrary number, but in most cases people use pointers rather than passing arrays/structures directly. For the sake of our example we’ll keep 32.
Lets refactor our function to make it more reusable split it into 2 vprint_args and print_args similarly to ‘vprtintf’ and printf:

void vprint_args(metac_tag_map_t * p_tag_map, metac_entry_t *p_entry, va_list args) {
    if (p_entry == NULL || metac_entry_has_paremeter(p_entry) == 0) {
        return;
    }

    printf("%s(", metac_entry_name(p_entry));

    for (int i = 0; i < metac_entry_paremeter_count(p_entry); ++i) {
        if (i > 0) {
            printf(", ");
        }

        metac_entry_t * p_param_entry = metac_entry_by_paremeter_id(p_entry, i);
        if (metac_entry_is_parameter(p_param_entry) == 0) {
            // something is wrong
            break;
        }
        if (metac_entry_is_unspecified_parameter(p_param_entry) != 0) {
            // we don't support printing va_args... there is no generic way
            printf("...");
            break;
        }

        metac_name_t param_name = metac_entry_name(p_param_entry);
        metac_entry_t * p_param_type_entry = metac_entry_parameter_entry(p_param_entry);
        if (param_name == NULL || p_param_type_entry == NULL) {
            // something is wrong
            break;
        }

        metac_size_t param_byte_sz = 0;
        if (metac_entry_byte_size(p_param_type_entry, &param_byte_sz) != 0) {
            // something is wrong
            break;
        }

        char buf[32];
        int handled = 0;
#define _handle_sz_(_sz_) \
        do { \
            if (param_byte_sz == _sz_) { \
                char *x = va_arg(args, char[_sz_]); \
                memcpy(buf, x, _sz_); \
                handled = 1; \
            } \
        } while(0)
        _handle_sz_(1);
        _handle_sz_(2);
//...
        _handle_sz_(32);
#undef _handle_sz_

        if (handled == 0) {
            break;
        }

        metac_value_t * p_val = metac_new_value(p_param_type_entry, &buf);
        if (p_val == NULL) {
            break;
        }
        char * v = metac_value_string_ex(p_val, METAC_WMODE_deep, p_tag_map);
        if (v == NULL) {
            metac_value_delete(p_val);
            break;
        }
        char * arg_decl = metac_entry_cdecl(p_param_type_entry);
        if (arg_decl == NULL) {
            free(v);
            metac_value_delete(p_val);
            break;
        }

        printf(arg_decl, param_name);
        printf(" = %s", v);

        free(arg_decl);
        free(v);
        metac_value_delete(p_val);

    }
    printf(")");
}

void print_args(metac_tag_map_t * p_tag_map, metac_entry_t *p_entry, ...) {
    va_list args;
    va_start(args, p_entry);
    vprint_args(p_tag_map, p_entry, args);
    va_end(args);
    return;
}

The reader may notice that we added p_tag_map as the first argument. This is for the further research - it's not used in this article.

Lets now try to create a part which handles the result. Unfortunately typeof isn’t supported till C23 (gcc extension as an option, but it won't work with clang) and we have a dilemma - do we want to keep our METAC_WRAP_FN notation as is, or it’s ok to pass it one more argument - type of the function result to be used as a buffer. Probably we could use libffi to handle this in a universal way - Metac knows the type, but it’s not clear how to put the returned data into the buffer of the proper size. For simplicity let’s change our macro:

#define METAC_WRAP_FN_RES(_type_, _fn_, _args_...) ({ \
        printf("calling "); \
        print_args(NULL, METAC_GSYM_LINK_ENTRY(_fn_), _args_); \
        printf("\n"); \
        WITH_METAC_DECLLOC(loc, _type_ res = _fn_(_args_)); \
        print_args_and_res(NULL, METAC_GSYM_LINK_ENTRY(_fn_), METAC_VALUE_FROM_DECLLOC(loc, res), _args_); \
        res; \
    })

Now we’re passing _type_ as a first argument to store the result. If we pass incorrect type or arguments - the compiler will complain about this _type_ res = _fn_(_args_). This is good.
Printing out the result is a trivial task, we already did that in the first article. Let’s also update our test functions to accept some different types of parameters.
Here is the final example code.

If we run it we’ll get with the comments:

% ./c_print_args

# show args of base type arg function
calling test_function1_with_args(int a = 10, short int b = 22)
fn returned: 38

# show args if the first arg is a pointer
calling test_function2_with_args(int * a = (int []){689,}, short int b = 22)
fn returned: 1710

# using METAC_WRAP_FN_RES which will print the result. using pointer to list
calling test_function3_with_args(list_t * p_list = (list_t []){{.x = 42.420000, .p_next = (struct list_s []){{.x = 45.400000, .p_next = NULL,},},},})
fn returned: 87.820000

# another example of METAC_WRAP_FN_RES with int * as a first arg
calling test_function2_with_args(int * a = (int []){689,}, short int b = 22)
test_function2_with_args(int * a = (int []){689,}, short int b = 22) returned 1710

# the log where 1 func with wrapper calls another func with wrapper
calling test_function4_with_args(list_t * p_list = (list_t []){{.x = 42.420000, .p_next = (struct list_s []){{.x = 45.400000, .p_next = NULL,},},},})
calling test_function3_with_args(list_t * p_list = (list_t []){{.x = 42.420000, .p_next = (struct list_s []){{.x = 45.400000, .p_next = NULL,},},},})
test_function3_with_args(list_t * p_list = (list_t []){{.x = 42.420000, .p_next = (struct list_s []){{.x = 45.400000, .p_next = NULL,},},},}) returned 87.820000
test_function4_with_args(list_t * p_list = (list_t []){{.x = 42.420000, .p_next = (struct list_s []){{.x = 45.400000, .p_next = NULL,},},},}) returned -912.180000

It’s seen that Metac prints for us the deep representation of the arguments as well as results. In general it works, though there are some flaws like a need to handle each size of argument separately.

Here are some additional limitations:

clang doesn't expose debug information about external functions like printf. That means - our wrapper won't work with that as-is. We may need to introduce some additional tricks.
functions with unspecified arguments ... won't show such arguments. there is no generic way, but potentially we may want to give a way to provide a callback to extract information for such cases.
there is no (yet?) support for the cases of linked arguments, e.g. when we pass pointer and length as 2 separate but logically connected arguments .

If you have any suggestion on how it could be more generic - please comment. Thanks for reading!

C Reflection: When the Good Old DWARF Makes Your Elves Face Their Unconscious Truth

aodinokov — Thu, 30 May 2024 18:29:12 +0000

The article is dedicated to the ability of major compilers like gcc or clang to be a source for reflection information for C applications, which makes possible C reflection implementation like Metac. This works for elf, macho and pe formats on the corresponding platforms Linux, macOS and Windows.

Traditionally, C hasn't embraced reflection capabilities like some other programming languages. This is because C prioritizes efficiency and control. However, the lack of reflection doesn't necessarily mean a lack of introspection capabilities. Debuggers, for instance, rely heavily on debug information embedded within executable files. This information goes even beyond what reflection needs, encompassing details like types defined in the code, line numbers, source code references, symbol information and even location of variables and function arguments on the stack. One of the most common debug information formats is called DWARF and it’s a native for ELF format way to expose all needed for debuggers data. Even better, this format also works for Mach-O (MacOs executable format) and PE (Windows executable format).

This begs the question: can't applications utilize this data directly to gain self-awareness? Here's why it's not as straightforward as it seems:

DWARF has too much information - reflection requires just a subset.
Shipping executables with full debug information can be undesirable due to increased size and potential security concerns.
Utilities like strip can remove debug information, rendering the application unusable.
It may be worth separating reflection information which may be only for part of the application from debug information.

This implies a need for a specialized tool that can efficiently read, filter, and convert DWARF data (or its equivalent in other formats) into a format usable by the application. Enter Metac.
Metac leverages this existing DWARF (or equivalent) information within the executable format to provide a targeted form of reflection for C code. This allows applications to access a relevant subset of the data, promoting introspection without the drawbacks of full debug information. Metac bridges the gap and allows C programs to query that data at runtime, enabling them to extract information about their own types, variables, and functions. This newfound self-awareness empowers C programs for more efficient debugging, dynamic behavior, and potential future functionalities.

While DWARF data is available for object files on other platforms, macOS presents a slight hurdle. On macOS, DWARF information is typically not generated by default and requires the dsymutil tool to create it explicitly for linked executable binary ONLY.

To ensure consistent behavior across platforms, Metac takes a two-step approach that will work on macOS and compatible with other platforms:

Build with DWARF Generation: The application is first built with special flags (-g3 -D_METAC_OFF_) that enable DWARF generation but disable Metac functionalities during this stage.
Extract and Integrate DWARF Data: After the initial build, the DWARF information is extracted from the executable. Then, an additional C file containing reflection information is generated based on this data. Finally, the application is rebuilt with this additional file to include the necessary reflection capabilities and with Metac functionalities enabled.

This multi-step process might seem complex, but it's automated within a Makefile, simplifying the workflow for developers. It’s important to remember that Metac is getting DWARF from the complete built and linked application even though this process can be changed for other platforms.

Let's delve into a practical example. Imagine a C program that manages a complex data structure, like a linked list. Traditionally, debugging any issues within the structure requires manual code inspection. However, with Metac, the program can introspect its own linked list, examining elements like pointers and values. This allows for targeted debugging and manipulation of the list at runtime.

Here's a simplified code example demonstrating how Metac could be used to examine a variable of type struct test:

// main.c
#include <stdio.h>  // printf
#include <stdlib.h> // free
#include <math.h>   // M_PI, M_E
#include "metac/reflect.h"

struct test {
    int y;
    char c;
    double pi;
    double e;
    short _uninitialized_field;
};

int main(){
    // we need to use this construction to wrap variable declaration
    // to get its type information
    WITH_METAC_DECLLOC(decl_location,
        struct test t = {
            .y = -10,
            .c = 'a',
            .pi = M_PI,
            .e = M_E,
        };
    )
    metac_value_t *p_val = METAC_VALUE_FROM_DECLLOC(decl_location, t);

    char * s;
    s = metac_entry_cdecl(metac_value_entry(p_val));
    // next will output "struct test t = "
    printf("%s = ", s);
    free(s);

    s = metac_value_string(p_val);
    // next will output "{.y = -10, .c = 'a', .pi = 3.141593, .e = 2.718282, ._uninitialized_field = 0,};\n"
    printf("%s;\n", s);
    free(s);

    metac_value_delete(p_val);

    return 0;
}

Explanation:

We include the metac/reflect.h header for using Metac functions.
We define a structure called test with various member variables.
In main, we create a variable t of type struct test and initialize its members. Note: the construction WITH_METAC_DECLLOC just makes sure that the arbitrary C-code from the second argument is located on the same line with the declaration location variable decl_location.
We use METAC_VALUE_FROM_DECLLOC(decl_location, t) to get a metac_value_t representing the value of t.
metac_entry_cdecl(metac_value_entry(p_val)) retrieves a C-style string representing the declaration of t (e.g., struct test t).
metac_value_string(p_val) retrieves a string representing the actual value of t with its member values (e.g., {y=-10, c='a', pi=3.141593, e=2.718282, _uninitialized_field=0}).
We free the allocated memory for both strings using free.
Finally, we call metac_value_delete(p_val) to clean up resources used by Metac.

This example demonstrates how Metac can be used to:

Extract type information about a variable at runtime.
Retrieve the actual value of the variable and its members.

This is just a basic example, but it showcases the power of Metac for C code introspection.

In order to build that it will be necessary to have Metac on the host where the build process is happening.

Here is KBUILD-like Makefile to build the example:

ifeq ($(M),)
METAC_ROOT=../..

all: test target

target:
    $(MAKE) -C $(METAC_ROOT) M=$(PWD) target

clean:
    $(MAKE) -C $(METAC_ROOT) M=$(PWD) clean

test:
    $(MAKE) -C $(METAC_ROOT) M=$(PWD) test

.PHONY: all clean test
endif

rules+= \
    target \
    _meta_c_app \
    c_app.reflect.c \
    c_app \

LDFLAGS-c_app=-Lsrc -lmetac
LDFLAGS-_meta_c_app=-Lsrc -lmetac

in_c_app+=main.o

TPL-_meta_c_app:=bin_target
IN-_meta_c_app=$(in_c_app:.o=.meta.o)
POST-_meta_c_app=$(METAC_POST_META)

TPL-c_app:=bin_target
IN-c_app=$(in_c_app) c_app.reflect.o
DEPS-c_app=src/libmetac.a

TPL-c_app.reflect.c:=metac_target
METACFLAGS-c_app.reflect.c+=run metac-reflect-gen $(METAC_OVERRIDE_IN_TYPE)
IN-c_app.reflect.c=_meta_c_app

TPL-target:=phony_target
IN-target:=c_app

Explanation:

The part from ifeq to endif works exactly like in KBUILD. It’s possible to run make all METAC_ROOT=<path to the Metac root> in order to build this example.
The rest of the file is used to define rules which are going to be generated to build the example using multi-step process described in the beginning. The variable rules list those:

Rule target:

TPL-target:=phony_target
IN-target:=c_app

which is generated as .PHONY and that requires the final target c_app to be built using the corresponding rules.

Rule c_app:

TPL-c_app:=bin_target
IN-c_app=$(in_c_app) c_app.reflect.o
DEPS-c_app=src/libmetac.a

is a rule to build executable binary out of main.o and c_app.reflect.o and libmetac.a. Make knows how to build main.o automatically from main.c. Where do we get c_app.reflect.o from? From c_app.reflect.c:

Rule c_app.reflect.c:

TPL-c_app.reflect.c:=metac_target
METACFLAGS-c_app.reflect.c+=run metac-reflect-gen $(METAC_OVERRIDE_IN_TYPE)
IN-c_app.reflect.c=_meta_c_app

is a rule which employs a metac tool with arguments run metac-reflect-gen $(METAC_OVERRIDE_IN_TYPE). Input file is _meta_c_app. This combination will instruct metac to read DWARF data from _meta_c_app. Parameter METAC_OVERRIDE_IN_TYPE is used to specify if metac must expect elf, macho or pe as input. metac-reflect-gen is a go-template module name which generates c_app.reflect.c.

Rule _meta_c_app:

TPL-_meta_c_app:=bin_target
IN-_meta_c_app=$(in_c_app:.o=.meta.o)
POST-_meta_c_app=$(METAC_POST_META)

Similar to c_app, but it uses main.meta.o as source. The only difference between main.meta.o and main.o is that the first was built with flags -g3 -D_METAC_OFF_.

If we run make on macOS we should see:

% make
/Library/Developer/CommandLineTools/usr/bin/make -C ../.. M=/Users/user/Workspace/metac/examples/c_app_simplest test
/Library/Developer/CommandLineTools/usr/bin/make -C ../.. M=/Users/user/Workspace/metac/examples/c_app_simplest target

cc -I./include -c -MMD -MF /Users/user/Workspace/metac/examples/c_app_simplest/main.d -MP -MT '/Users/user/Workspace/metac/examples/c_app_simplest/main.o /Users/user/Workspace/metac/examples/c_app_simplest/main.d' -o /Users/user/Workspace/metac/examples/c_app_simplest/main.o /Users/user/Workspace/metac/examples/c_app_simplest/main.c

cc -I./include -g3  -D_METAC_OFF_ -c -MMD -MF /Users/user/Workspace/metac/examples/c_app_simplest/main.meta.d -MP -MT '/Users/user/Workspace/metac/examples/c_app_simplest/main.meta.o /Users/user/Workspace/metac/examples/c_app_simplest/main.meta.d' -o /Users/user/Workspace/metac/examples/c_app_simplest/main.meta.o /Users/user/Workspace/metac/examples/c_app_simplest/main.c

cc /Users/user/Workspace/metac/examples/c_app_simplest/main.meta.o -Lsrc -lmetac -o /Users/user/Workspace/metac/examples/c_app_simplest/_meta_c_app

(which dsymutil) && dsymutil /Users/user/Workspace/metac/examples/c_app_simplest/_meta_c_app || echo "Couldn't find dsymutil"
/usr/bin/dsymutil

./metac run metac-reflect-gen -s 'path_type: "macho"' -s 'path: "/Users/user/Workspace/metac/examples/c_app_simplest/_meta_c_app"' > /Users/user/Workspace/metac/examples/c_app_simplest/c_app.reflect.c

cc -I./include -c -MMD -MF /Users/user/Workspace/metac/examples/c_app_simplest/c_app.reflect.d -MP -MT '/Users/user/Workspace/metac/examples/c_app_simplest/c_app.reflect.o /Users/user/Workspace/metac/examples/c_app_simplest/c_app.reflect.d' -o /Users/user/Workspace/metac/examples/c_app_simplest/c_app.reflect.o /Users/user/Workspace/metac/examples/c_app_simplest/c_app.reflect.c

cc /Users/user/Workspace/metac/examples/c_app_simplest/main.o /Users/user/Workspace/metac/examples/c_app_simplest/c_app.reflect.o -Lsrc -lmetac -o /Users/user/Workspace/metac/examples/c_app_simplest/c_app

Now if we run the application we’ll see:

% ./c_app
struct test t = {.y = -10, .c = 'a', .pi = 3.141593, .e = 2.718282, ._uninitialized_field = 0,};

The example can be found here.
More information on how to use metac can be found here

Conclusion:
Metac isn't just a tool; it's a path to self-improvement for your C code. With DWARF's insights and metac's interpretation, your programs can shed light on their "unconscious" behaviors and unlock their full potential.