C Testing and Analysis

I'm a solid believer in having appropriate tests for large code repos. They're a pain to make, but as long as the code you're testing doesn't change frequently, automated tests will pay for themselves in spades. Additionally, code analysis tools are best to get set up on a new project as opposed to trying to add them to an existing project, so we'll also explore a few analysis options on this page.

Testing Frameworks

I already did a little digging into test frameworks, and pretty much landed on Unity and FFF, but it's worth noting that many others exist depending on your needs. My goals were to find frameworks that were as simple as possible while still providing a great deal of power and flexibility. It's worth noting that Unity is mainly intended for embedded software, so this could very easily be a mistake on my part, but I'd rather start with something too simple and need to reinvent a wheel than something too complex and get bogged down in the learning curve.

Unity

Unity is just 3 files: unity.c, unity.h, unity_internals.h. To define a test file, you'd create a .c file with a main() function that looks close to the following:

int main(void) {
    // 1) Start our test run
    UNITY_BEGIN();

    // 2) Run each of our test functions
    RUN_TEST(test_DoesTheThing);
    RUN_TEST(test_DoesTheOtherThing);
    RUN_TEST(test_DoesTheLastThing);

    // 3) Complete our test run
    return UNITY_END();
}

All of your test functions have a series of assertions they can call, which includes:

• TEST_ASSERT_TRUE(condition)
• TEST_ASSERT_FALSE(condition)
• TEST_ASSERT_EQUAL_*(expected, actual) -- replace the * with common data types, including INT, INT8, UINT, ...
• TEST_ASSERT_NULL(actual)
• TEST_ASSERT_GREATER_THAN(threshold, actual) (and *_LESS_THAN)
• ...there's quite a lot more, including some for arrays, bits, floats, strings, etc. Refer to their Github README.md for a full list.

Here's an example implementation of the test_DoesTheThing() test function which we're supposedly running above in our main function:

void test_DoesTheThing(void) {
    TEST_ASSERT_EQUAL_INT(3, 1 + 2);
}

Including Unity in our build

As mentioned earlier, Unity is shipped as 3 files unity.c, unity.h, and unity_internals.h. To compile and run our tests, we need to make sure these files are part of our build command. Since I'm using Meson for Breezy, the build configuration will look a little something like this (assuming the Unity files are inside the ./unity directory relative to our Meson build file):

# Compile 'unity/unity.c' as a static library
unity_lib = static_library('unity',
  'unity/unity.c',
  include_directories : include_directories('unity'))

# Declare our unity library as a dependency so other parts of our build can pull it in.
unity_dep = declare_dependency(
  link_with           : unity_lib,
  include_directories : include_directories('unity'))

# List our unity dependency as a dependency to our test executable.
exe = executable('test_utils', 'test_utils.c', dependencies: [ unity_dep ])
test('Test Utilities', test_utils_exe)

FFF

FFF, the "Fake Function Framework", or as I like to call it, "Ffffffff", is just a header file: fff.h. It's used to mock functions in our tests, tracking things like call counts, arguments, and return values. All of it's magic is done using a couple macros:

• FAKE_VOID_FUNC(name, ...arg_types) - used to mock a function that returns void.
• FAKE_VALUE_FUNC(return_type, name, ...arg_types) - used to mock a function that returns a value.

To mock a function, you just pass your function name (and arguments) to one of those macros, and then you have access to a variable named <function-name>_fake. This *_fake variable has the following properties on it:

• call_count
• arg_history_len
• arg_histories_dropped
• return_val (only when using FAKE_VALUE_FUNC)
• custom_fake

Let's do the learn-by-example approach for these. Let's say you have these functions defined:

// You're looking to test this
function void my_func() {
    int x = get_int();
    int result = add(x, 2);
    echo(result);
}

// You're looking to mock these, which my_func() calls into:
function void echo(int message) { ... }
function int get_int(void) { ... }
function int add(int a, int b) { ... }

Your test file might look something like this:

// Import / initialize FFF
#include "fff/fff.h"
DEFINE_FFF_GLOBALS;

// Mock our three functions
FAKE_VOID_FUNC(echo, int);            // Creates "echo_fake"
FAKE_VALUE_FUNC(int, get_int);        // Creates "get_int_fake"
FAKE_VALUE_FUNC(int, add, int, int);  // Creates "add_fake"

// Run our test
void test_my_func() {
    // When "get_int()" is called, it should return "3".
    get_int_fake.return_val = 3;

    // Call the function we're testing
    my_func();

    // Ensure all three functions were called exactly once
    TEST_ASSERT_EQUAL_INT(1, get_int_fake.call_count);
    TEST_ASSERT_EQUAL_INT(1, add_fake.call_count);
    TEST_ASSERT_EQUAL_INT(1, echo_fake.call_count);

    // The "add" function received the arguments "3" and "2"
    TEST_ASSERT_EQUAL_INT(3, add_fake.arg0_val);
    TEST_ASSERT_EQUAL_INT(2, add_fake.arg1_val);

    // Reset our fakes
    RESET_FAKE(echo);
    RESET_FAKE(get_int);
    RESET_FAKE(add);
    // ...and also reset some internal FFF structures
    FFF_RESET_HISTORY();
}

You might've noticed I used two new macros in that example: RESET_FAKE(...) and FFF_RESET_HISTORY(). They should be pretty self-explanatory, but just note that they exist and you should use them.

Build Updates

To integrate FFF into our build, we'd update our Meson build file with the following:

# Declare our FFF header file as a dependency so other parts of our build can pull it in.
fff_dep = declare_dependency(include_directories : include_directories('fff'))

# List our FFF dependency as a dependency to our test executable.
exe = executable('test_utils', 'test_utils.c', dependencies: [ fff_dep ])
test('Test Utilities', test_utils_exe)

Code Analysis

There are quite a few code analysis tools that are ripe for the picking. Adding all of them to the same project would be a terrible idea as they all have a lot of overlap with each other. The best approach is to select a few that cover a good variety of situations, and just roll with those.

Me being a total noob to the many tools available, I took some guidance from my average friend, Claude. After some back-and-forths and experimentation, I ended up going with the following:

• Static Analysis - analyzes source code at compile time.
• Various compiler flags (there's a lot...)
• Clang-tidy, which was already built into CLion.
• Dynamic Analysis - analyzes a program during runtime. (...or in my case, when executing test suites.)
• Address Sanitizer (ASan)
• Undefined Behavior Sanitizer (UBSan)

These tools will hopefully catch all of my dumb mistakes before I spend 8 hours crying over a segfault, but if not, then I might look into adding more to this list or replacing these in the future.

I'll also be keeping Valgrind handy as a manual tool for troubleshooting issues when needed. It's not compatible to run alongside ASan (which I found out the hard way), which is why I chose not to run it as part of the test suite. I could've created a separate Meson build directory that runs Valgrind instead of ASan (and I actually did before deleting it), but I decided the benefits likely wouldn't be worth the overhead or maintenance.