How do system programmers think?

Since the underlying system is, practically, not reliable.

Whilst there are unreliable systems, it doesn't sound like you are interacting with that much unreliability. What you are interacting with is complexity. The job of a programmer is to manage complexity.

So, formally speaking, I must make a judgement on an infinite language of execution traces. This is again an impossible task.

You will have to categorize the infinite language of execution traces into a finite set of categories. Whilst there are infinitely many ways of doing this categorization, you will have to pick one that fits your requirements.

Enumerating all the relevant conditions and precisely understanding what they mean is an impossible task.

It's not impossible. It may be lengthy, but for every system I have interacted with there is a finite amount of documentation.

If what you observe isn't covered by documentation, record what you did to make what you observed. That is a start to making your own documentation.

In almost all projects, you don't need to do this all up front. Unless your project is safety critical, you can try something, see if it meets you requirements, and tweak that.

What else can I do? [to avoid runtime errors] How do people who write major system programs think about this kind of problems?

Unit tests.

Conceptually the specify example you give could theoretically be amenable to static checks on your code, ie a better compiler.

The other common way that you already address, is to wrap the problematic code ensuring that the exception is handled, avoided or otherwise would generate a compile time error. ie perhaps with a builder class.

C is a fairly low level language so its lacking in some of the compile time abstractions that haskel or an OOP language has.

But in general, You can just add a unit/integration test to check that no error is thrown at runtime for given sets of parameters.

You can limit the range of these parameters by wrapping the direct function calls in units which restrict the ways in which they can be used and only calling and testing those units rather than the underlying functions.

For instance with your example: (excuse my pseudo code)

get_layer_surface takes the problematic parameter surface object<wl_surface> which must be uninitialized. But we could wrap the call in a function which instead takes a new class: wl_surface_uninitialised written by ourselves.

This class would have its own methods to attach/commit a buffer which would return a wl_surface_initialised object.

wrap all this up in some immutability, add unit tests to make sure it works and you now have the compile time checks you were looking for.

So first of all, I don't think this question has much to do with hardware. Vulkan or Wayland are not different from any other C lib, in the sense: from the caller's perspective you just call functions and you are expected to follow rules. It looks to me like this is more about differences between langauges like Haskell and C.

The C, C++ and to some extend Rust (as in: unsafe Rust) are very different from languages like Haskell, Java, C# (except for unsafe C#) or Python. The big difference is that these low level languages have a built-in concept of undefined behaviour. The latter languages do not. Every line of code written in Haskell, Java, C# or Python has a well defined behaviour. This is an extremely useful property, which C does not have.

C does not protect you from doing stupid things. For example you are allowed to go outside of the array bounds. The reason for this is performance: the compiler will always assume that you (as a programmer) behave as expected, so that it can properly optimize the code. Without this assumption every time you access an element of an array by index, the compiler would have to generate code that verifies that the index is inside bounds. But that requires additional book keeping and takes non-zero time. And indeed, higher level languages actually do that. And that is one of the reasons they are slower than C, C++ or Rust.

But this has a nasty side effect: the compiler and runtime are less helpful. Code crashing is actually a good situation. You at least know you have a bug. But going outside array does not always mean crash unfortunately. Sometimes the code looks like its doing exactly what expected. Until it runs on production. And that is very bad.

This means that C, C++ and Rust demand a lot more from the software engineer. They demand lots of discipline. And sure, tests help. But they won't catch all undefined behaviours. Sure, analyzers like valgrind help. But they won't catch all undefined behaviours. In general, the "does my code contain undefined behavior" problem is undecidable (for example because of pointer arithmetic). Even though there are tools that catch most common cases.

What else can I do? How do people who write major system programs think about this kind of problems?

They have lots of years of experience, and they try to avoid common pitfalls. They write tests wherever it is possible. And they use tools, e.g. static and dynamic analyzers. That's all.

But don't be misguided: even hardcore coders make mistakes all the time. In 2025, the Linux kernel recorded around ~5000 security-specific bugs. Half of them due to concurrency issues.

You can also try to isolate problematic pieces of code, and actually call them from a higher level language through ffi. For example Python does this all the time with all the AI and numerical libs, which are actually implemented in C.

Rust was created to address some of those issues, while still being at at low enough level. If you can use Rust instead of C then I strongly encourage you to do that. Both Vulkan and Wayland have Rust bindings. With Rust you can create safe abstractions around unsafe code. In a way that is impossible with C. For example Rust references are always valid in safe Rust. Rust will also prevent you from shooting yourself in many many situations (e.g. the borrow checker). Not all though.

The main point stands though: this is simply hard.

synchronous correctness If the program crashes, I can argue that it is not my fault.

You absolutely should never assume that. That's delusional.

diachronous correctness If there is a fault in my program, I can decisively fix it while introducing, on average, less than 1 new fault.

Well, that would be ideal and tests help here a lot. But such thing can never be guaranteed.

TLDR: How to manage complexity? With intuition, incorrect abstractions and grit.

I find appeal to language or technology naive.

A real problem here is the task. When dealing with external systems, communication protocol has to fit both systems and is always low level. There are plenty of serialization, message passing and other IPC tools, but those never address the problem of state space and there is always a need to bridge the abstraction gap between low-level protocol and high-level ideas programmer would like to work with.

State is only one subset of the communication protocol parameter space. Sure, adding a single bit of state doubles API surface, but so does a boolean parameter.

You have never observed similar problems in Haskell, because the average task in Haskell is plain and stupid in comparison. Compare web-page rendering, where there is one input and one output with handling an IO interrupt that could be triggered by any of the devices in the system and has to produce drastically different results depending on the situation, and most actions result in crash and you will see my point. Alternatively imagine Linux kernel state as an object in an IO monad - that also illustrates the difference nicely.

Do not get me wrong - web devs do deal with complexity, sure. But it comes from different problem space of business logic and usability, not from protocol complexity or state management.

And here comes the annoying truth - software engineers are hired to manage complexity and the normal tools and recipes apply to manage your particular example. Usually the main tool is abstraction layering.

• External system has state? Wrap it in an idempotent layer at cost of minor performance hit.
• Too many parameters? Hide them behind a simplified layer, that has none at cost of reduced flexibility.
• Too many preconditions? Introduce validation layer, take performance hit and reduce usability.

The list goes on and is ridiculously well-trodden.

With each layer of abstraction, you get less flexible, but more robust system. There is a great risk to select a wrong abstraction for a particular job and that's why we are being paid to understand the problem space in enough detail and to redo the systems over when we inevitably choose wrong.

There are plenty of books and opinions on selection of abstractions and layering, but unfortunately, they are either too vague or too specialized to be useful for a particular task. We read them to train our intuition and broaden horizons.

My main point here is - system engineers deals with complexity just like everybody else, they just have a different problem space.

I can represent state either explicitly, with another variable next to it, implicitly by the sequence of operations, or, if I'm feeling adventurous, implicitly by some other state that I already know will be updated at a similar time.

So, I could write the setup for the surface object either as a sequence

1. send commit without buffer attached
2. wait for reply
3. acknowledge reply
4. configure buffer according to the demands of the compositor

at the end of the sequence, I have a working connection to the compositor, and because nothing can interrupt that sequence, I do not have to care about the intermediate state.

I could also build an event-driven program, where the reply is handled in the generic event handler, and any drawing functions check whether the buffer is already allocated -- if it isn't, then I don't even know what the output viewport size is going to be, so any attempt at drawing would be premature.

I can combine these approaches and set up the rendering pipeline including the final viewport size as part of the sequence, and drawing functions check for the presence of a working pipeline; the presence of the buffer then becomes an implementation detail that lets me abstract the drawing away from the presentation.

In an event-driven program I can also express the state of the rendering pipeline by proxy, for example by only subscribing to events that cause me to begin rendering when it is safe to do so.

That has some potential for bugs if the proxy is not consistent with the state of the system (e.g. I have an event handler registered that tries to draw, but I don't have a surface), but a lot of people prefer this implicit style over silently absorbing inconsistent states, at least for debug builds, because it makes errors more visible.

A practical approach to this problem is to adopt a different paradigm for expressing your problem.

You are not dealing with clean mathematical abstractions. You are operating very finite machinery: it happens to be electronic, rather than mechanical, and it operates very fast, but it still has a bunch of moving parts that can fail.

An effective way to write code for this is to think of your task as a search for errors, which happens to accomplish work as a side-effect.

This is very different from Haskell, where a complete lack of side effects is, I understand, a major point of the language. This may have something to do with Haskell, and other functional languages, being quite rare in any kind of programming for commercial purposes.

So, tldr, real-world programming is hard? (I'm the upvoter btw, not the downvoter.)

The reality of computer programming is not to produce something that works correctly for the life of the universe, but to assist human activity for the time being and to "save labour", and do so with enough reliability to be useful.

Haskell programs may be guaranteed to run in their own terms, but the hardware those programs drive is not guaranteed to work correctly, and the program itself is not guaranteed to be correct (in the sense of doing what it should, or striking a reasonable balance in the circumstances between competing and irreconcilable goals).

You've clearly acquired involvement in programming via academia (hence the use of Haskell), where there has been decades of lack of common sense about what real-world programming is about.

You might therefore have found that you are talented in ways that won't make you a good industrial programmer, or you might be perfectly capable of being a good industrial programmer but are about to find that an academic course of study (and association with academics) has conveyed you very little relevant experience and competency and has in fact twisted your concepts and expectations, and you are now about to start again.

There is no simple way of boiling down a professional area of practice, but amongst various things that professionals do do, is managing the complexity of solutions so as not to exhaust the available budget of programmer labour available (i.e. your labour), and controlling projects which (when kept within a budget) will work too unreliably to be useful (computers can be unreliable either because they produce wrong results too often or with too great consequences, or because they stop processing and require intervention and alteration too frequently or urgently).

Doing these tasks skilfully is what the job consists of.

There's obviously lots more that can be said, but obviously one of the main mistakes you're making is assuming your programs should be truly perfect and that programming computers will be easy, and worrying that this is not the case. Experienced computer programmers don't think like that.

EDIT:

On the points added to the question, the problem is that a "crash" is only conventional in definition. You can avoid a "crash" simply by removing or swallowing all the exceptions from your program.

Proving that a program "terminates" is rarely useful. Some programs are supposed to run indefinitely. Others must not merely terminate before the death of the Sun, but within an acceptable budget of time.

As for reducing bugs by decomposition, the problem is that you can solve every problem with another independent component, except the problem of too many independent components.

And the vast majority of useful programs have to be globally stateful, respond to external events, and evolve their states in ways that are not fully determined at the outset of operation.

What you're mentioning are the common canards peddled by academics about functional languages like "global state", "side effect" data flows, and so on, and experienced practioners know (to be blunt) it's all a load of nonsense!

Professional practice is about working competently with what they dub "global state" and "side effects", including managing the complexity of these and moderating their usage to what is reasonably essential (rather than eliminating them on the theory they are inessential).