Building C Code

I remember back in my college days trying to learn C, and not really understanding the build process. Sure, I had a command I could run that would create an executable file, but I could never keep straight the different stages of the build, where libraries were loaded from, nor really understood the point of higher-level build tools. Now that I'm (marginally) more mature and working on a project that will need something more complex than a basic gcc command, I wrote this up to help out.

Main Stages

Building a C program has roughly four stages:

1. Preprocessing - Resolves all directives, which are the lines beginning with #. It's still C code after this step.
2. Compilation - Converts the preprocessed source code into assembly code. This is completed by the C compiler (cc1).
3. Assembly - Converts the assembly code into binary machine code, resulting in object files (.o). This step is done by the assembler (as).
4. Linking - Combines all the object files, including those from needed libraries, into a single executable. This is done by the linker (ld).

You execute these four stages by running something that builds C code (like gcc or clang). You're able to do all the steps in one shot, creating a my_program executable from ./main.c with the following command:

gcc -o ./my_program ./main.c

So if we can just hammer out all four stages in one command, why bother separating them? The simple answer is that you might not always want to do them all at once. For example, if you have a large project, it's common to create object files from each of your source files, and then link them all together as the final step:

gcc -c ./file1.c  # Creates ./file1.o
gcc -c ./file2.c  # Creates ./file2.o

gcc -o ./my_program ./file1.o ./file2.o

The main reason for splitting up the build is efficiency. As your C projects grow in size, there's no reason to rebuild the entire project just because you changed one file. You should only need to rebuild that one file, and anything that depends on it. By splitting the build up, you're able to accomplish that. (This approach is usually managed automatically by a build tool, which we'll get into further down.)

Static vs Dynamic Libraries

If every program we wrote needed to do everything from scratch, development would slow down enough to make a snail 🐌 look like a racecar 🏎️. Just about any project of reasonable size depends on other libraries to provide some prewritten code. Libraries come in two flavors: static and dynamic.

• Static Libraries are packaged into your program during the linking step of your build. They're .a files, which are just archives of .o files that were bundled with ar.
• Dynamic / Shared Libraries are not fully packaged into your program. The linker makes note that your program requires the specific library, and then at runtime, the dynamic linker (ld.so) finds and loads the dynamic libraries into memory that you need. Dynamic libraries are .so ("shared object") files.

Static and dynamic libraries have opposite tradeoffs. The upside to using static libraries is that your program already contains everything that it needs to run, but comes with the drawback that the common library code gets duplicated by every program that needs it. Using dynamic libraries keeps your program as light as possible, but requires the target machine to have the necessary libraries (and versions) preinstalled.

You can add libraries to your gcc build with the -l flag. Ex: gcc -o ./main ./main.c -lexample. When attaching a library (ex: -lexample), your system is searched for both dynamic (libexample.so) and static (libexample.a) versions of that library, and whichever is found is used. If both are found, then by default it'll use the dynamic libraries, but it's possible to force static libraries by specifying some build flags:

• -static to make all libraries static
• -Wl,-Bstatic -lexample to make a single library static
• -Wl,-Bdynamic -lexample to make a single library dynamic

Locating Libraries

When looking for libraries at build time, ld checks the following places (in order):

1. Directories passed as -L flags on the command line. Ex: gcc main.c -L/opt/example -lexample
2. The LIBRARY_PATH environment variable, which is different from the LD_LIBRARY_PATH runtime variable explained below.
3. Any paths that are built-in to ld, such as /usr/lib, /usr/local/lib, and /lib. You can inspect these with:

$ ld --verbose | grep SEARCH_DIR

When looking for dynamic libraries at runtime, ld.so checks the following places (in order):

1. RPATH - A field that's embedded in the binary. Deprecated in favor of RUNPATH.
2. The LD_LIBRARY_PATH environment variable - useful for development and testing, but that's typically it.
3. RUNPATH - A field that's embedded in the binary. Replaces RPATH and carries a lower priority.
4. /etc/ld.so.cache - This is an index of libraries on your system, generated by running sudo ldconfig, which looks for *.so files inside the directories specified by /etc/ld.so.conf or /etc/ld.so.conf.d/*.conf.
5. Default system paths, like /lib, /usr/lib, and their 64-bit variants.

If your build is failing due to a missing library, there are a few tools you can use to try and locate it yourself; always check for slight variations on the name, as sometimes their names aren't what you'd expect. Also, make sure you also have the *-dev packages installed from your package manager.

# Use pkg-config when possible for development, as it knows about library names and include paths
pkg-config --list-all | grep -i "search-term"
pkg-config --libs --cflags libraryname

# Searches for dynamic libraries cached by /etc/ld.so.cache
ldconfig -p | grep -i "search-term"

# Search for libraries installed but not registered with ldconfig
sudo find /usr -name "*.so*" 2>/dev/null | grep -i "search-term"
sudo find / -name "*search-term*.so*" 2>/dev/null

To inspect which shared libraries are tied to an existing executable, you can use ldd:

$ ldd ./my_program

Build Tools

As projects become larger and split across multiple files, manually managing and running gcc commands can get unwieldy. A few different build tools exist that are designed to make this easier.

Makefile

This is an old one which new projects typically don't go for. It still works, but is typically slower than the alternatives and can be more difficult to maintain as the project grows and changes over time.

It works by defining one or more targets, a list of files each target depends on, and one or more commands that should be run to execute each target (oftentimes gcc commands). When you run make, any targets that have had their dependencies modified will have their command(s) rerun.

CMake

CMake is a bit higher-level. It generates build files which then get executed to build the actual project. The build files it generates are typically Makefiles or Ninja files (which we'll talk more about in a sec). I've never used CMake, so I won't be going into more details here, but it's a popular build tool that exists if you want to learn more about it on your own.

Meson

Meson is my favorite based on my (very limited) experiences. It's similar-ish to CMake in that it takes high-level configurations and turns them into Ninja build files. Its claims to fame are about speed and readability, which is exactly what Breezy strives for. Additionally, I really like that it doesn't pollute the source directories with interim build files (like .o files). All built/generated files are contained within a build directory of your choosing.

To use Meson, you first create a build file (meson.build), then use that to create a build directory (meson setup ./builddir). The build directory that Meson creates contains the Ninja build files that are run to build your executable. The last step, which is the only step that needs to be done going forward, is to build your code by running meson compile from inside your build directory. This executes the Ninja build files.

Ninja build files can be thought of like Makefiles. They list out the build commands (ex: gcc), the targets, etc, but they're much more efficient at creating a parallelized dependency graph and executing the commands. Ninja has no concept of C or its compilers - it just runs whatever commands Meson put in its build files.

Defining your Meson build

A very basic build file should have a project definition that specifies the source language, along with instructions for creating the end artifact (likely an executable):

project('breezy-os', 'c')

executable('breezy', './main.c')

For multiple source files, you can declare them in a list, and include that list in your executable command:

project('breezy-os', 'c')

src_files = [ './main.c', './utils.c' ]

executable('breezy', sources: src_files)

To add dependencies to our build, it follows a similar pattern, but calls into the dependency(...) function which looks up the library with pkg-config:

project('breezy-os', 'c')

src_files = [ './main.c', './utils.c' ]
deps = [ dependency('libdrm'), dependency('gbm') ]

executable('breezy', sources: src_files, dependencies: deps)

I'll cover some testing frameworks on a different notes page, but you can configure Meson to run your test suite by compiling your test file as an executable, telling Meson to run it with test(), and then actually triggering that piece of the build using the command meson test:

test_utils_exe = executable('test_utils', 'test_utils.c')
test('Test Utilities', test_utils_exe)

$ meson compile
$ meson test

To make this a bit easier to manage as you create more and more test files, you can make use of Meson's foreach loop:

foreach t : ['string_utils', 'math_utils']
  exe = executable('test_' + t, 'test_' + t + '.c')
  test(t, exe)
endforeach

As projects grow in size, you'll want to split them into pieces to make them more maintainable. The way to do this with Meson is to create subdirectories, each of which has its own meson.build file. Note that the child build file should not define its own project(), and all variables are shared between the build files. The subdirectory's build file will define a dependency (using declare_dependency()) that other build files can then make use of:

util_src_files = [ './math_utils.c', './list_utils.c' ]
utils_lib = static_library('utils', sources: util_src_files)
utils_dep = declare_dependency(link_with: utils_lib)

Then in the parent's build file, you'd execute the child directory's build with subdir(), and list its dependency by name in your main executable:

subdir('utils')

executable('breezy', sources: src_files, dependencies: [utils_dep])

Last, it's generally a good practice to put all of your header files into one namespaced directory, and then define one global_inc variable in your top-level meson.build file which can be used everywhere throughout your build. In other words, all of Breezy's custom header files will be inside ./include/breezy/*.h, and I'll define my "global includes" variable like this:

global_inc = include_directories('include')

And then use it from all of my meson.build files like so:

# Parent build creating an executable
executable('breezy', sources: src_files, include_directories: global_inc)

# Child module when declaring itself as a dependency
utils_dep = declare_dependency(link_with: utils_lib, include_directories: global_inc)

And my C files will import these headers using:

#include "breezy/my_utils.h"