Linux Rendering

When I first started this project, I had no idea there were so many technologies and pieces involved with rendering even a simple rectangle on the screen. I'll do my best to break down what each piece does down below, and some info for using it. Keep in mind this is only a very high-level overview for each, as they could all use their own entire website explaining their details. I might make deeper dives into these technologies if I have a need to dig more into them, but for now, this'll do.

UI Library

An application is written using a UI library such as GTK or Qt. If you're familiar with the linux ecosystem, you might've heard of these before; someone would use one of these libraries to create their ✨ dream app ✨. These libraries know how to speak the protocol of the compositor (or windowing system), which is what draws your application windows in the correct location on the screen. Cool, let's go a layer deeper.

Compositor

The compositor does as the name implies - it "composits" multiple images onto the screen, where each image would be a user application. A windowing system manages all the windows you have open, determines where to draw them, and how to send keyboard/mouse inputs to them. ...wait, those sound like they both do similar things... And to an extent, they do. There are two common protocols / pieces of software that perform this role on linux: X11 and Wayland.

• X11 is the older framework, and is comprised of many moving pieces. It separates out these responsibilities to different components, hence the need to differentiate between a "compositor" and "windowing system".
• The newer protocol, Wayland, combines responsibilities into one piece of software that does everything from window rendering to input handling. This one piece of software is generally called a "compositor".

So in the context of X11, "compositor" refers specifically to rendering, but in the context of Wayland, it refers to rendering and input. Since this project is creating a Wayland project, we'll be using the term "compositor" to refer to everything related to managing and displaying applications.

Graphics Library

Wayland is just a protocol though. It's a set of messages that both the compositor and application libraries (like GTK) agree on a meaning for. The compositor's code needs to actually "do the things", and for that, it needs help. That's right, more libraries! The next layer down is a graphics library, and the two most common ones for linux are OpenGL and Vulkan.

What I gathered from some quick reading (and zero experience) is that OpenGL is an older library, but much simpler to learn. Vulkan is a newer one, more complex, but has the capacity to be optimized more than OpenGL.

It's also worth noting, these are technically graphics API protocols, not implementations of them. To actually call the functions and have them do something, you'd need to have a library imported which implements one of these protocols. Mesa is by far the most popular implementation for linux, and you'll see their name come up very frequently.

The graphics library's functions are all related to converting "concepts" into an image buffer. What do I mean by that? Well, you can ask it to render a rectangle with a given size, position, and color onto a buffer that represents a bunch of pixels. The concept is "the rectangle, it's position, etc". Given that concept, the graphics library knows which pixels to make which color within the buffer. It's obviously more than just "drawing rectangles", but that's the idea - converting concepts into colorful pixels.

It's worth noting that we'd also create some shaders to help guide OpenGL into how to render our beautiful artwork. These shaders are mini-programs written using a special shader language (like OpenGL Shading Language / OpenGL SL), but that's all I'm going to talk about them here.

Linux Kernel / DRM

We still have quite a gap to cover in the area between the graphics library and the image shown on your GPU. We were talking about how OpenGL colors the pixel data stored in a buffer, but where does that buffer come from? Surely it must be stored somewhere in memory, just like any other variable. It's technically possible to allocate this in you RAM memory just like you normally would any other variable (ex: with malloc), but that's going to be slowwwwwww.

Luckily for us, the GPU has a bunch of VRAM (video memory) that we can use. We can't use a simple malloc to allocate it though.

The linux kernel generally locks down application access to hardware, and that includes the GPU. Could you imagine if the random person who made that fun game you just downloaded could hijack your display output, and replace all the current contents with whatever it wanted? That'd be pure chaos! The linux kernel has a subsystem specifically dedicated to graphics that allows for some direct access from user applications, but also limits other access. The subsystem is called "DRM", or Direct Rendering Manager. It has two pieces that we care about: KMS (Kernel Mode Setting) and GEM (Graphics Execution Manager).

• KMS covers your display / scanout; it protects the pathway between your GPU and monitor. Only one application is allowed to take ownership of that pathway, preventing anything else from accessing it until the main application relinquishes its grasp. That "main application" would be our Compositor.
• GEM is responsible for managing the memory available on your GPU. All applications are allowed to request memory. The GEM is what allocates it, ensures nobody else can access it, and ultimately frees it when the application is done with it.

GBM

It's a bit of a pain to communicate with these two DRM components. You'd normally need to submit a bunch of generic ioctl commands with specific codes in order to request things, but luckily there's an easier way. There's a library called libdrm which is a thin wrapper around these ioctl commands. Even that library can be tricky to use though, so there's yet another API that sits on top called GBM (Generic Buffer Management). This is a platform-agnositic set of APIs that allow you to allocate GPU memory with ease. (well, with "easier".) Remember our good friends at "Mesa" who publish the OpenGL library? Well, they create this one too. When you install these Mesa libraries, they're generally all tightly coupled for whatever platform / GPU you're targeting so that our application code can be generic instead of hardware-specific, hence the name: Generic Buffer Management.

Summary

...Ok, this has been a lot. Surely that must be all there is to it, right? ...right? Yeah, ok, it's finally enough to paint the full picture. To recap:

1. An application is written using a UI library such as GTK.
2. That UI library speaks the Wayland protocol with the Wayland compositor to negotiate how to render the application and receive inputs.
3. The Wayland compositor communicates with OpenGL to draw all the applications into a big pixel buffer, which is reserved on the GPU's VRAM by calling into GBM (Generic Buffer Management).
4. The GBM library talks to the GEM component of the DRM linux subsystem in order to allocate and manage that GPU memory.
5. The Wayland compositor takes ownership of the display output by going through the KMS component of the DRM linux subsystem.
6. The Wayland compositor uses that ownership to render the giant buffer of pixels onto your screen, resulting in potentially magical things.

Bonus Soup 🥣

So now that we have all that laid out and with a perfectly clear understanding, let me add two more acronyms to this soup.

• DRI (Direct Rendering Infrastructure) isn't another library thankfully, but is a figurative box that contains the graphics library (ex: OpenGL), libdrm, and the DRM subsystem itself. The only reason I'm mentioning it is because when setting up all of these things to be used, we find our graphics card inside the /dev/dri/ directory, aka: / "devices" / "direct rendering infrastructure".
• EGL (Embedded-system Graphics Library) unfortunately is another library, and one that we use. I'm just going to describe it as "glue code" here, as it sits between a few of the components we've discussed (OpenGL, GBM, libdrm), and helps them communicate better by managing certain things. So while it is important, it doesn't really contribute to understanding the overall flow of data, which is why I left it out earlier.

Fin

The Linux rendering stack is far from simple, so if you're still feeling a bit fuzzy on it, that's to be expected. The second devlog on this project will cover setting up most of this stack, so that might help a smidge. The below links might also be good to refer to as needed.