How to develop a Blender export Add-on

Okapi Renderer is a toy renderer I developed. It is closed-source, but there is also a stripped-down Open-Source version of it.

It can render simple scenes using ambient occlusion or native path tracing, with support for diffuse and mirror materials. The feature set is currently very minimal. Nevertheless, to make Okapi a bit more useful, I implemented a Blender to Okapi export add-on.

I want to summarize my learnings in this post series. Since Okapi is only a toy renderer, the insights of this article can also be used to implement similar exporters (e.g. for your own renderer). I also identified a few similar projects:

• GitHub - stig-atle/io_scene_pbrt: Exporter for blender that exports the scene into pbrt's ascii file format.
• GitHub - NicNel/bpbrt4: pbrt-v4 render engine/exporter for Blender
• nori/ext/plugin at master · wjakob/nori
• GitHub - mitsuba-renderer/mitsuba-blender: Mitsuba integration add-on for Blender
• GitHub - joeyskeys/btop: btop is a blender addon for PBRT aiming for better user experience
• GitHub - giuliojiang/pbrt-v3-blender-exporter: A Blender exporter for PBRTv3 on Linux

I used Blender version 4.4.3 for this. The Blender Python API seems to be rather stable across different versions of Blender. Nevertheless, code that worked in Blender 2.x does not necessarily work in Blender 4.x.

First struggle: Getting started with Blender Python API

1. Start Blender
2. Go to the “Scripting” tab
3. “Text” -> “New”
4. Paste print("Hello World!")
5. Run the script via “Run script”

You will see no output if you run the above Hello World script. To see the output, "Hello World!" you need to start Blender via a terminal. All Python print statements are then redirected to your terminal.

You can change the above program to

import sys
print("Blender's Python version:", sys.version)

to see the current Python version used by Blender.

Test-driven development with Blender Python API

Assume you have a file named test.py with the following content:

"""
    SPDX-FileCopyrightText: Copyright 2025 Julian Amann <dev@vertexwahn.de>
    SPDX-License-Identifier: Apache-2.0
"""

import sys
import unittest


class LookAtMatrix(unittest.TestCase):
    def test_look_at(self):
        self.assertTrue(True)


if __name__ == "__main__":
    print("Blender's Python version:", sys.version)

    sys.argv = [__file__] + (
        sys.argv[sys.argv.index("--") + 1 :] if "--" in sys.argv else []
    )
    unittest.main()

You can run blender --background --python test.py -- --verbose to run the contained test. On my system, I get the output:

Blender 4.4.3 (hash 802179c51ccc built 2025–04–29 15:12:13)
Blender's Python version: 3.11.11 (main, Feb 6 2025, 17:26:54) [GCC 11.2.1 20220127 (Red Hat 11.2.1–9)]
test_look_at (__main__.LookAtMatrix.test_look_at) … ok
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Ran 1 test in 0.000s
OK

To be continued

See you in the next part (which is WIP).

Rendering (186.101, 2021S) Computer Graphics at TU Wien

Rendering Lecture 00 — Introduction

Color Theory

BRDFs

Normal maps

Render Systems

pbrt-v4 code walkthrough

Mitsuba 3

Conferences

AWE XR

Motivation

Digital Lego City: The Alley | Episode 1

Ep.1: The pioneers of computer graphics 1960–1970

Ep.2: The pioneers of computer graphics — 1980s

• Missing IDE integration — e.g. Visual Studio (“If it does not work in my IDE/OS/Environment I will not use it”)
• Missing support for C++20 modules
• Missing out-of-the-box support/easy integration for needed libraries (e.g. OpenCV)
• rules_cc has some quirks
• Fear that Google stops to support Bazel in the future
• Time-consuming to learn a new build system/Learning curve/Invest in new technology — most people do not have fun caring about a build system and just want to code C++
• Java Runtime is not supported on the platform you want to develop
• Bazel is not written in C++/Rust

I tried to sort the list from the top blocker down to the most unimportant one. I have no statistics on this. It is just my gut feeling that I developed after talking with a handful of C++ developers and Bazel users. Maybe other important things are completely missing here on the list and I am pretty sure that some people would prioritize the list differently.

Missing out-of-the-box support/easy integration for needed libraries

Let’s talk about the “Missing out-of-the-box support/easy integration for needed libraries” blocker. At the time of this writing, the Bazel Central Registry contains at least 21 C++ libraries. I think there is a kind of pattern when a new C++ package manager is born. First, there are a lot of compression libraries, such as zlib, libdeflate, xz, etc. since many other things depend on compression such as file storage or file transmission. There is also a network libraries wave. After that, a wave of image libraries is introduced. libpng, libgif, libtiff, libwebp, etc. you name it. The reason here is that images are important in many applications. Maybe one day we see also an AI wave. Nevertheless, the bazelization of libraries and adding them to the Bazel Central Registry seems to be a manual process that needs manual effort.

To avoid this we have also e rules_foreign_cc that support CMake, configure_make, make, Meson, and ninja (at the time of writing this). I do not use those rules, since my gut feeling is that this is only a temporary solution. I have no experience of how well rules_foreign_ccwork in a real setting where you want to support Windows, Linux, and macOS builds at the same time. I would assume that these rules cannot be better than the pure usage of the underlying build tools itself. I go always for an 100% pure Bazel approach not depending on an additional build system when it comes to C++ libraries.

It would be nice If we could find an automated way to take over the build from other 3rd party library managers such as Conan 2. Maybe a strace could be used to track down what is built and linked under the hood and from this BUILD files can be automatically derived.

Missing IDE integration

If you work on Windows with Visual Studio and do not want to leave this environment, then there is currently no good integration. I left Visual Studio and Windows and went over to CLion and Ubuntu. CLion comes with a great Bazel integration. I only visit Windows to build my project using Bazel to ship Windows binaries.

I wrote a very short article about how you can use Visual Studio to build your VS projects. It is doable, but there is no out-of-the-box solution that works without any friction:

Using Visual Studio 2022 to build Bazel projects

CLion has the advantage that I do not need an advanced AddIn for better syntax highlighting and refactoring. I tried CLion also in the past on Windows, but the CLion Bazel support on Windows was not good there. On macOS I made also good experiences with CLion + Bazel.

Besides this there are cool Bazel integrations/helpers to get a comfortable Bazel experience in some environments:

GitHub - MobileNativeFoundation/rules_xcodeproj: Bazel rules for generating Xcode projects.

Missing support for C++20 modules

There is currently (at the time of writing this) no C++20 module integration. Nevertheless, it seems that is something that is currently worked on and I assume this will land in Bazel release at some time in the future.

There are also some rules available that give a module support right now:

GitHub - eomii/rules_ll: An Upstream Clang/LLVM-based toolchain for contemporary C++ and heterogeneous programming

Conclusion

There are some blockers — but I think they are not show-stoppers.

Feel free to leave a comment here or to contact me to let me know your blockers.

In this post, I will describe one easy way not to say hacky way to use Visual Studio 2022 with Bazel.

First create basic C++ example project. Create a folder hello_world. Add a.bazelversion file to this folder with the content 8.0.0 . Add amain.cpp file with the following content:

#include <iostream>

int main() {
    std::cout << "Hello World!" << std::endl;
    return 0;
}

Add an empty MODULE.bazel file. Add a BUILD.bazel file with the following content:

cc_binary(
    name = "hello_world",
    srcs = ["main.cpp"],
)

The expected file structure should look like this:

.
├── .bazelversion
├── BUILD.bazel
├── MODULE.bazel
└── main.cpp

Start Visual Studio 2022 and select “Create a new project”:

Select “Makefile Project”:

Set “Project name” and “Solution name” to HelloWorld, set “Location” to the Hello World example, enable the checkmark “Place solution and project in the same directory”, and hit the “Create” button:

Set “Build command line” to bazel run //:hello_world. Set “Clean command line” to bazel clean --expunge. Set “Output” to hello_world.exe. Hit the “Next” button. Overtake the same setting for Release. Click the “Finish” button:

This process has to be repeated for every build/test target you need.

A working example that you can download and test can be found here.

In the past there was Lavender a open source tool to automate this process. I did work well for me in the past, but unfortunately, it is unmaintained currently. Nevertheless, I think this is one approach for a lightweight integration of Bazel into Visual Studio.

UPDATE: Since CLion is now free for non-commercial use and CLion has made progress in Windows and Bazel support, it is also a valid option to use CLion on Windows. Under the hood, still the VS2022 compiler is used to compile your code, and you will get the VS2022 error and warning messages, but you get on top of that really good Bazel support, e.g. debugging works out of the box:

The first video shows tile-based rendering:

As you can see from the video different tiles of the final image are rendered in parallel. In contrast, there is progressive rendering where the whole image is rendered apparently simultaneously:

Here is some pseudocode that shows how tile-based rendering can be implemented:

tbb::parallel_for(0, tg.tile_count(), [&](int index) {
    FilmTileDescription ftd = tg.tile_description(index);

    FilmTile film_tile{ftd.offset, ftd.size, channel_count, filter.get(), film->tile_bounds()}; // where to get the correct channel count?

    auto sampler(scene->sampler()->clone());
    int spp = sampler->sample_count();
    auto tile_size = ftd.size;

    for(int y = 0; y < tile_size.y(); ++y) {
        for(int x = 0; x < tile_size.x(); ++x) {
            for(int sample_index = 0; sample_index < spp; ++sample_index) {
                Point2f sample_position = Point2f(ftd.offset.x() + x, ftd.offset.y() + y) + sampler->next_2d();
                Ray3f ray = sensor->generate_ray(sample_position);
                auto color = integrator->trace(scene, sampler.get(), ray, 0);
                film_tile.add_sample(sample_position, color.data());
            }
        }
    }

    mutex.lock();
    film->add_tile(film_tile);
    mutex.unlock();
});

Every tile is visited in a parallel for loop. In this example, oneTBB is utilized to achieve this. For every tile, every pixel of a tile is visited and sampled multiple times according to the value of maximum samples per pixel (spp). Once the whole tile is rendered it is added to the final image (film).

In contrast, to change this to progressive rendering, the above code only has to be modified minimally. The sample per pixel loop will become the outermost loop and tiles get added to the “big picture” once a specific tile has been rendered with one sample per pixel. Instead of visiting every tile only once we visit every tile for every sample:

for(int sample_index = 0; sample_index < spp; ++sample_index) {
    LOG_INFO("Progressive rendering: {}/{} SPP", sample_index, spp);


    tbb::parallel_for(0, tg.tile_count(), [&](int index) {
        FilmTileDescription ftd = tg.tile_description(index);

        FilmTile film_tile{ftd.offset, ftd.size, channel_count, filter.get(), film->tile_bounds()}; // where to get the correct channel count?

        auto sampler(scene->sampler()->clone());
        int spp = sampler->sample_count();
        auto tile_size = ftd.size;

        for(int y = 0; y < tile_size.y(); ++y) {
            for(int x = 0; x < tile_size.x(); ++x) {          
                Point2f sample_position = Point2f(ftd.offset.x() + x, ftd.offset.y() + y) + sampler->next_2d();
                Ray3f ray = sensor->generate_ray(sample_position);
                auto color = integrator->trace(scene, sampler.get(), ray, 0);
                film_tile.add_sample(sample_position, color.data());
            }
        }

        mutex.lock();
        film->add_tile(film_tile);
        mutex.unlock();
    });
}

The progressive and the tile-based variants render individual tiles. The difference is when a tile gets added to the final image. For the progressive variant, this is done after every pixel has been sampled one time and for the tile-based approach, it is done when all pixels have been sampled the predefined maximum number of samples per pixel.

The reason why the progressive variant is still tile-based is that the above approach assumes that samples are splatted across different pixels using some filter kernel. This leads to the problem that our tiles have also an overlap near the tile edges with other tiles and need to be accumulated in a proper and synchronized way.

Tile to final image synchronization

To get rid of the tile to final image synchronization we could think about a pattern or order in which parallel execution can be performed without having to do synchronization. Let us consider a 4x4 grid:

+---+---+---+---+
| A | B | C | D |
+---+---+---+---+
| E | F | G | H |
+---+---+---+---+
| I | J | K | L |
+---+---+---+---+
| M | N | O | P |
+---+---+---+---+

Assuming that the border that affects other pixels from a tile does not influence more than one neighborhood tile, it is clear that we could easily render tile A and tile P in parallel without caring about synchronization issues. Also, the sets A, M, D, and P should work in parallel without synchronization. If we assume that the border of influence of a tile is not wider than one-half of a tile we could render A, C, I, K in parallel. We could probably come up with a formula like:

// Assumes integer division
int next_look_free_tile_x = (index*2) % tiles_per_width;

Once A, C, I, K are rendered we can offset the whole pattern by one tile in the x-direction, then in y-direction, and then in x- and y-direction. Assuming that our final image can be split into many tiles we can find enough work for all parallel processing units.

Prerequisites

I assume you have basic knowledge about the following things:

• Boost C++ Libraries
• Bazel
• Bazel registries
• Bazel Central Registry (BCR)

Motivation

Easy use of Boost C++ libraries using Bazel.

Design Decisions

Division into Bazel modules

Instead of offering one “big” Boost dependency to be able to use Boost C++ libraries (e.g. bazel_dep(name = "boost", version = "1.87.0") , it was decided to create a Bazel module for each individual Boost library, e.g. bazel_dep(name = "boost.array", version = "1.87.0") , bazel_dep(name = "boost.assert", version = "1.87.0") , bazel_dep(name = "bind", version = "1.87.0") , etc. Nevertheless, it is still intended to have bazel_dep(name = "boost", version = "1.87.0") which will refer to all individual modules. The advantage of this setting is that if you only depend on a single Boost library you do not need to fetch everything of Boost (in the case of Boost 1.88.0 more than 150 MB). Moreover, Boost itself also manages each single library in an own GitHub repo, e.g. https://github.com/boostorg/pfr. This paves the way to get native Bazel support into Boost in the future.

How to avoid mixing different boost versions?

To avoid that different boost versions are mixed, e.g. boost.algorithm@1.87.0 is using boost.array@1.86 a module named boost.pin_versionwas introduced. boost.pin_version@1.88.0.bcr.1 looks like this:

# boost.pin_version is not a real Boost module.
# Its whole purpose is to ensure that Boost modules of one specific Boost version get not mixed with another one (e.g. 1.88.0 with 1.83.0)

module(
    name = "boost.pin_version",
    version = "1.88.0.bcr.1",
    bazel_compatibility = [">=7.6.0"],
    compatibility_level = 108800,  # Can remain constant as nodeps prevent version skew
)

bazel_dep(name = "boost.accumulators", version = "1.88.0.bcr.1", repo_name = None)

# List of Boost modules is based on https://pdimov.github.io/boostdep-report/boost-1.88.0/module-overview.html.
# Dependency reports for other versions can be found at https://pdimov.github.io/boostdep-report/.
[bazel_dep(name = boost_module, version = "1.88.0.bcr.1", repo_name = None) for boost_module in [
    "boost.accumulators",
    "boost.algorithm",
    "boost.align",
    "boost.any",
    "boost.array",
    "boost.asio",
    [[ ... all other Boost libraries of this specifc version..  ]]
    "boost.wave",
    "boost.winapi",
    "boost.xpressive",
    "boost.yap",
]]

boost.pin_version@1.88.0.bcr.1 points to all other Boost Modules that can be combined with each other. It is assumed that a version 1.88.0.bcr.1 of boost.pin_version only references to Boost libraries of version 1.88.0.bcr.1 . The good thing about boost.pin_version is that it can refer to modules that do not currently exist. We follow here strictly the policy that boost.pin_version refers only to Boost libraries of the same version as boost.pin_version , i.e. boost.pin_version@1.88.0.bcr.1 only refers to boost.accumulator@1.88.0.bcr.1 , boost.algorithm@1.88.0.bcr.1 etc. boost.pin_version@1.88.0.bcr.1 can not refer to a Boost library whose version is different from 1.88.0.bcr.1 . At the same time, every Boost library module references the boost.pin_versionas a dependency. This means there is a mutual dependency between pin_version, and its dependencies. That has the benefit, that version get not mixed

OLD:

Naming conventions for branches and pull requests

Use as a naming convention for your branches and pull requests the pattern library@1.2.3 e.g. boost.algorithm@1.83.0 . If you want to fix an issue in an already published module, use .bcr.1 , .bcr.2 , etc. as a postfix to the version number, e.g. boost.algorithm@1.83.0.bcr.1 .

Comments on pull request

There is a CI check in the BCR that makes sure that no unstable URLs are used to reference to the sourcecode of a module. Unfortunately, Boost libraries published on GitHub use currently unstable URLs. To avoid ignore the warning from the BCR CI add to your GitHub pull request the comment @bazel-io skip_check unstable_url .

MODULE.bazel: Use the right compatibility level

The compatiblity_level should correspond to the Boost version. For example, Bazel modules for Boost 1.83.0 libraries should have the compatibility level 108300.

module(
    name = "boost.algorithm",
    version = "1.83.0",
    bazel_compatibility = [">=7.2.1"],
    compatibility_level = 108300,
)

presubmit.yml: Test on many platforms

Test on at least these platforms:

  platform:
    - debian10
    - debian11
    - macos
    - macos_arm64
    - ubuntu2004
    - ubuntu2204
    - ubuntu2404
    - windows

One difference between the various existing image formats is their support for color depth. This can be either designed for low dynamic range (LDR) or high dynamic range (HDR).

PNG (Portable Network Graphics), for example, is an LDR image format. 8 bits, i.e. 256 different color values, can be saved per color channel. PNGs have a color channel for red, green, blue, and optionally an alpha channel. This gives a total of 3 * 8 bits = 24 bits or 4 * 8 bits = 32 bits per color value. PNGs also offer support to store grayscale images. If you want to find out more about the feature set of PNG file format consider libpng which offers a reference implementation of the PNG format.

In contrast, there are also HDR formats. EXR is a representative of such a format. It can use up to 32 bits per color channel. This means that for an image with a red, green, blue, and alpha channel, 4 * 32 bits = 128 bits are available for each color value.

HDR formats allow a much greater contrast range (difference between brightest and darkest colors) than LDR formats. This makes them a prominent image format when it comes to storage of the output result of the rendering process.

PFM

One of the simplest HDR formats is the Portable Float Map (PFM). The file header is written as a plain text file, i.e. you can view and read it with a simple text editor. To indicate that the file is a PFM file the first line of the file starts with PF or Pf. PF indicates that the image has three color channels (red, green, blue), whereas Pf indicates that the image is monochrome, i.e. it only has a single color channel. Then a Unix-style carriage return follows (hex code 0x0a). The next line defines the width and height of the image followed again by a Unix-style carriage return. The third line defines the byte order. The byte order can be either little-endian (which is indicated by -1.0) or big-endian (which is indicated by 1.0). Again there is a Unix-style carriage return. After this, a series of 4-byte IEEE 754 single floating point values follows. The values are sorted from left to right and from bottom to top to

Here is an example header:

PF
200 200
-1.0

You can easily extract from a PFM file the first three lines using the head command:

head -3 test.pfm

You can also have a look at the hex representation of a PF file via:

head -n 3 /home/vertexwahn/Desktop/test.pfm | hd

Here is an example output of the command:

00000000  50 46 0a 32 30 30 20 32  30 30 0a 2d 31 2e 30 0a  |PF.200 200.-1.0.|
00000010

From the hex dump, you can see the Unix-style carriage return values (0x0a)

Example

Here is an example C++ program that writes a PFM image with the size of 200×200 pixels.

#include <iostream>
#include <fstream>
#include <memory>

namespace shrew {
    struct Color3f {
        explicit Color3f() : values{0.f, 0.f, 0.f} {}
        explicit Color3f(float red, float green, float blue) :
            values{red, green, blue} {}
        float values[3];
    };

    class Image3f {
    public:
        Image3f(int width, int height) : width_{width}, height_{height},
                                         data_{new Color3f[width*height]} {}
        [[nodiscard]] int width() const { return width_; }
        [[nodiscard]] int height() const { return height_; }
        void set_pixel(int x, int y, const Color3f& color) {
            data_[x+y*width_] = color;
        }
        [[nodiscard]] Color3f get_pixel(int x, int y) const {
            return data_[x+y*width_];
        }
        [[nodiscard]] const Color3f* data() const { return data_.get(); }
        [[nodiscard]] size_t byte_size() const {
            return width_ * height_ * sizeof(float) * 3;
        }
    private:
        int width_, height_; // size dimensions of the image
        std::unique_ptr<Color3f[]> data_; // pixel data is sorted left to right, top to bottom
    };

    Image3f flip_horizontally(const Image3f& image) {
        Image3f flipped{image.width(), image.height()};
        for (int y = 0; y < image.height(); ++y) {
            for (int x = 0; x < image.width(); ++x) {
                auto color = image.get_pixel(x,image.height()-y-1);
                flipped.set_pixel(x, y, color);
            }
        }
        return flipped;
    }
    
    void store_pfm(const Image3f& image, std::string_view filename) {
        std::ofstream file(filename.data(), std::ios::binary);
        file << "PF" << "\n" << image.width() << " " << image.height() << "\n"
             << "-1.0" << "\n";

        Image3f flipped = flip_horizontally(image);

        file.write(reinterpret_cast<const char*>(flipped.data()),
                   static_cast<std::streamsize>(flipped.byte_size()));
    }
}

using namespace shrew;

int main() {
    Image3f image{200, 200};
    for(int y = 0; y < image.height(); ++y) {
        for(int x = 0; x < image.width(); ++x) {
            image.set_pixel(x, y, Color3f{1.f, 1.f, 0.f});
        }
    }

    for(int y = 0; y < image.height()/2; ++y) {
        for(int x = 0; x < image.width()/2; ++x) {
            image.set_pixel(x, y, Color3f{1.f, 0.f, 0.f});
        }
    }

    for(int y = 0; y < image.height()/2; ++y) {
        for(int x = image.width()/2; x < image.width(); ++x) {
            image.set_pixel(x, y, Color3f{0.f, 1.f, 0.f});
        }
    }

    store_pfm(image, "test.pfm");
}

Note that in the above example Image3f stores images from left to right and bottom to top. This means that the top right pixel is addressed by image.get_pixel(0,0) and the pixel at the right bottom via image.get_pixel(image.width()-1,image.height()-1). PFM images are stored differently. They have a bottom-to-top order. Therefore, the image is horizontally flipped in the store_pfm method.

The output of the generated PFM image looks like this:

Note that tev is used here as an image viewer.

A C++ function to store PFM float data that is independent of the Image3f and Color3f classes could also look like this:

void store_pfm(int width, int height, const float* data, 
               std::string_view filename) {
    std::ofstream file(filename.data(), std::ios::binary);
    file << "PF" << "\n" << width << " " << height << "\n" << "-1.0" << "\n";
    std::streamsize byte_size = width * height * sizeof(float) * 3;
    file.write(reinterpret_cast<const char*>(data),
               static_cast<std::streamsize>(byte_size));
}

Even if C++ is not your favorite programming language it should not be hard to port this code to a different language. Also, ChatGPT should be able to support you in the endeavor.

The above implementation has some pitfalls. For instance, there are no out-of-bound checks. What happens if a user of the Image3f class wants to access a pixel with a negative pixel position? The implementation still needs some tweaks but the main goal was here to give you an understanding of how a basic HDR file format works. Another thing that I assumed is that the program is always running on a little-endian machine. If you run the program on a machine with Big Endian layout you will be in trouble.

EXR

EXR is a more advanced file format to store HDR images. One library that implements reading and writing this format is OpenEXR.

OpenEXR usage example

Here is an example of how OpenEXR can be used to store an .exr file:

#include "OpenEXR/ImfChannelList.h"
#include "OpenEXR/ImfOutputFile.h"
#include "OpenEXR/ImfRgbaFile.h"
#include "OpenEXR/ImfStringAttribute.h"

#include <memory>
#include <string_view>

using namespace Imf;
using namespace Imath;

namespace shrew {
    struct Color3f {
        explicit Color3f() : values{0.f, 0.f, 0.f} {}
        explicit Color3f(float red, float green, float blue) :
            values{red, green, blue} {}
        float values[3];
    };

    class Image3f {
    public:
        Image3f(int width, int height) : width_{width}, height_{height},
                                         data_{new Color3f[width*height]} {}
        [[nodiscard]] int width() const { return width_; }
        [[nodiscard]] int height() const { return height_; }
        void set_pixel(int x, int y, const Color3f& color) {
            data_[x+y*width_] = color;
        }
        [[nodiscard]] Color3f get_pixel(int x, int y) const {
            return data_[x+y*width_];
        }
        [[nodiscard]] Color3f* data() { return data_.get(); }
        [[nodiscard]] const Color3f* data() const { return data_.get(); }
        [[nodiscard]] size_t byte_size() const {
            return width_ * height_ * sizeof(float) * 3;
        }
    private:
        int width_, height_; // size dimensions of the image
        std::unique_ptr<Color3f[]> data_; // pixel data is sorted left to right, top to bottom
    };

    void store_exr(Image3f &image, std::string_view filename) {
        Header header(image.width(), image.height());
        header.insert("comments", Imf::StringAttribute("Generated by my awesome App"));

        ChannelList &channels = header.channels();
        channels.insert("R", Imf::Channel(Imf::FLOAT));
        channels.insert("G", Imf::Channel(Imf::FLOAT));
        channels.insert("B", Imf::Channel(Imf::FLOAT));

        FrameBuffer frame_buffer;
        size_t comp_stride = sizeof(float);
        size_t pixel_stride = 3 * comp_stride;
        size_t row_stride = pixel_stride * image.width();

        char *data = reinterpret_cast<char *>(image.data());
        frame_buffer.insert("R", Imf::Slice(Imf::FLOAT, data, pixel_stride, row_stride));
        data += comp_stride;
        frame_buffer.insert("G", Imf::Slice(Imf::FLOAT, data, pixel_stride, row_stride));
        data += comp_stride;
        frame_buffer.insert("B", Imf::Slice(Imf::FLOAT, data, pixel_stride, row_stride));

        OutputFile file(filename.data(), header);
        file.setFrameBuffer(frame_buffer);
        file.writePixels(image.height());
    }
}

using namespace shrew;

int main() {
    Image3f image{200, 200};
    
    for(int y = 0; y < image.height(); ++y) {
        for(int x = 0; x < image.width(); ++x) {
            image.set_pixel(x, y, Color3f{1.f, 1.f, 0.f});
        }
    }

    for(int y = 0; y < image.height()/2; ++y) {
        for(int x = 0; x < image.width()/2; ++x) {
            image.set_pixel(x, y, Color3f{1.f, 0.f, 0.f});
        }
    }

    for(int y = 0; y < image.height()/2; ++y) {
        for(int x = image.width()/2; x < image.width(); ++x) {
            image.set_pixel(x, y, Color3f{0.f, 1.f, 0.f});
        }
    }

    store_exr(image, "test.exr");
}

To be able to compile this program you need the OpenEXR library and link it to your application. The Bazel build system can fetch OpenEXR, build it and link it to your application. Assuming that the above source code is stored in a file named main.cpp you can build and run this program via Bazel by creating the following files:

mkdir openexr_example
cd openexr_example
echo '7.2.0' > .bazelversion
echo 'build --enable_platform_specific_config
build:macos --cxxopt=-std=c++2b
build:linux --cxxopt=-std=c++20
build:windows --cxxopt=/std:c++20
' > .bazelrc
echo 'cc_binary(
    name = "Demo",
    srcs = ["main.cpp"],
    deps = ["@openexr//:OpenEXR"],
)' > BUILD.bazel
echo 'bazel_dep(name = "openexr", version = "3.2.4")' > MODULE.bazel

The above bash script creates the following files:

.bazelversion:

7.2.0

.bazelrc:

build --enable_platform_specific_config
build:macos --cxxopt=-std=c++2b
build:linux --cxxopt=-std=c++20
build:windows --cxxopt=/std:c++20

BUILD.bazel:

cc_binary(
    name = "Demo",
    srcs = ["main.cpp"],
    deps = ["@openexr//:OpenEXR"],
)

MODULE.bazel:

bazel_dep(
    name = "openexr", 
    version = "3.2.4"
)

Now you can run the application via:

bazel run //:demo

Modern low dynamic range formats

A prominent option for low dynamic range (LDR) images is the PNG format. There are many different options for storing LDR images. One of the newer options is WebP. WebP provides similar to PNG a lossless image compression. Here is an example of a PNG image:

For the above example image the WebP variant is about 0.5 MB smaller:

Format | Size
PNG | 1.7 MB
WebP | 1.2 MB

If you serve a lot of images via a web service per day and have to pay for network traffic and CPU usage image compression can get very important cost-wise. When it comes to compression often there is a trade of between compression speed, decompression speed, and file size.

Here is an example of how image data can be stored as a WebP file using the webp library:

bool store_webp(const char *filename, const Image4b &image) {
    int stride = image.width() * static_cast<int>(sizeof(Color4b));
    uint8_t* out = nullptr;
    auto encoded_size = WebPEncodeLosslessRGBA(image.data(), image.width(), image.height(), stride, &out);
    FILE* file = fopen(filename, "wb");
    fwrite(out, 1, encoded_size, file);
    fclose(file);

    return true;
}

The full source code for this can be found here. A good library that supports many image formats is OpenImageIO.

Spectral images

Spectral render engines produce spectral images. Instead of storing this data in an RGB image format one approach could be to store this data in spectral image data format. The paper An OpenEXR Layout for Spectral Images gives an overview of the current state of affairs. A spectral viewer can be found here.

To filter or not to filter? That is the question.

Usually during rendering some filtering process takes place to avoid effects such as aliasing. The problem with the procedure is that after the rendering result is stored as an image you can not change the filtering process. For instance, if you used as a reconstruction filter a tent filter with a radius of 2 pixels and now want to change to 3 pixels you have to rerender the whole scene. The filtering takes only a small fraction of the whole rendering time. If we had stored all samples with the corresponding values we could maybe reuse them when we just change the reconstruction filter. I was always wondering if it makes sense to store individual samples in an image format. Of course, this can get expensive if we render for instance with 8192 samples per pixel. That would mean an image that is filtered 1 MB in total size will blow up to 8 GB. Maybe if the rendering time is really big (> 12h) and we spend a lot time tweaking reconstruction filtering such a format would make sense.

References

• PFM Portable FloatMap Image Format
• Unofficial PBM format for HDR images, PFM (Portable Float Map)

Terraform manages its state in a terraform.tfstate file. This works well as long as only a single developer works on a single machine. If you want to execute Terraform from a different machine Terraform is not aware of which resources are already in place in which state they are, since the other machine does not have the state file. One option would be to check in the terraform.tfstate file to a version control system such as Git. But this can lead to other problems. Two different users might concurrently try to modify the state. Since there is no locking mechanism, this might lead to problems. Besides this, there are also other issues such as that the terraform.tfstate file can leak passwords. Instead of storing the Terraform state on your local machine or a Git repository another option is to store it on AWS.

Migrate from local state to AWS-managed state

Local management of state

Assuming our setup manages a DynamoDB table on AWS. Our setup could look like this:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.7.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

resource "aws_dynamodb_table" "favorite_songs_dynamodb_table" {
  name           = "FavoriteSongs"
  read_capacity  = 10
  write_capacity = 10
  hash_key       = "Title"

  attribute {
    name = "Title"
    type = "S"
  }

  attribute {
    name = "Artist"
    type = "S"
  }

  attribute {
    name = "PlayTimeInSeconds"
    type = "N"
  }

  global_secondary_index {
    name               = "ArtistPlayTimeInSecondsIndex"
    hash_key           = "Artist"
    range_key          = "PlayTimeInSeconds"
    write_capacity     = 10
    read_capacity      = 10
    projection_type    = "INCLUDE"
    non_key_attributes = ["Title"]
  }
}

Please note the choice of the hash_key and global_secondary_index here is completely random and comes due to my lack of deeper knowledge of DynamoDB. Consider it as some kind of “dummy” for a resource that we want to manage. If you are interested in more details in DynamoDB consider the official documentation. Now let's focus on Terraform state management again.

Let's create a local state. Execute:

terraform init
terraform plan -out=plan

There should be now a terraform.tfstate file. By executingterraform apply "plan" you should find a new table named FavoriteSongs under tables in the DynamoDB section in the AWS Management Console.

Creating an S3 Bucket for storing Terraform state

For storing the Terraform state we will use a S3 Bucket. This can be created by:

resource "aws_s3_bucket" "terraform_state" {
  bucket = "terraform-favorite-songs-state"

  lifecycle {
    prevent_destroy = true # prevent deleting this S3 bucket by accident via terraform delete
  }
}

Please note by setting the prevent_destroy attribute we prevent that the S3 bucket can be deleted without manual intervention.

Furthermore, we enable versioning for the S3 bucket:

resource "aws_s3_bucket_versioning" "versioning" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

And encryption:

resource "aws_s3_bucket_server_side_encryption_configuration" "encryption" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

To prevent users from modifying the Terraform state in parallel we need support for locking. We will use a DynamoDB table for this:

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform_favorite_songs_terraform_state_locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

As a last step, we need to tell Terraform to use the S3 bucket for the storage of the Terraform state file.

terraform {
  backend "s3" {
    bucket         = "terraform-favorite-songs-state"
    key            = "global/s3/terraform.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "terraform_favorite_songs_terraform_state_locks"
    encrypt        = true
  }
}

When now doing a terraform apply we get a now a notification that we have first to do a terraform init . The reason for this is that we are going to move now our local state to an AWS-managed state. After terraform init a terraform apply can be performed. One more problem that can occur now is that Terraform complains now about the missing S3 bucket or DynamoDB referenced in the S3 backend state. To circumvent this you first have to comment out the backend section, then run terraform again, and then comment it in again.

After this the terraform.tfstate file on the local disk should be empty and you should find a Terraform state file in the AWS S3 bucket.

Making it more complicated

In the case, we have two AWS accounts — one for testing and another one for production and try to apply the same Terraform configuration to both accounts we will run into some trouble. The reason for this is the S3 storage. The name of the S3 storage needs to be unique — you cannot use the same S3 Bucket name in two different accounts.

Therefore we introduce a file name testing.tfbackend . The file contains all variables specific to the backend configuration variables that are specific for the test environment and related:

bucket         = "terraform-favorite-songs-state"
key            = "global/s3/terraform.tfstate"
region         = "eu-west-1"
dynamodb_table = "terraform_favorite_songs_terraform_state_locks"

We change the backend block this way:

terraform {
  backend "s3" {
    encrypt        = true
  }
}

Now we run terraform init -backend-config=testing.tfbackend . Via the -----backend-config we can provide different attribute values using a file. This way we can have one file for our test and another one for your production environment to avoid name duplication of S3 buckets.

In my personal blog which is based on Hugo I did some minor modifications to support TeX. This enables me to easily embed math formulas in my blog posts. Here is an example:

The syntax for embedding such formulas looks like this:

The ${\chi}^2$ test can be used to determine if an observed sample distribution matches an expected sample distribution. 

For instance, given a sample generator that should generate uniform distributed 2D samples ($s_n$) within the domain $[0,1) \times x[0,1)$ the ${\chi}^2$ test can be used to check if the generated samples are really uniform distributed.
As a null hypothesis, we can formulate that the sample generator generates uniform distributed samples.

<div>$$ H_o: \forall s_n: p(s_n) = \text{const}
$$</div>

The sample generator can be tested by computing the difference between the expected sample frequency ($s_e$) and the observed sample frequency ($s_o$) according to (assuming $s_e \ge 5$ ):

In my blog, MathJax is used to render TeX formulas. It would be very nice to have support for TeX in Medium. To integrate MathJax in a personal web page it takes less than 3 lines of code. When searching for solutions on how to integrate math equations in Medium you find many not satisfying — only have baked solutions. One approach is to convert your equation into a raster image and embed it as an image on Medium. This way you lose the ability to do quick changes/iterations and have to repeat this cycle once you find an error in your equation. Hopefully, you have the original equation stored as a backup — otherwise, you have to retype it. Besides this, you need another tool to type it. I assume that storing a TeX equation takes less memory and traffic than transmitting images of equations. Furthermore, it would attract more people to Medium. What do you fear Medium people? Are you afraid of mathematical formulas? Better user experience, less traffic, and less data to store. Do it!

Upmath shows a very nice way how to integrate TeX support in Markdown. Meanwhile, also GitHub supports a similar style to add TeX formulas to Markdown files. DEV uses KaTeX.

Embeds are not a good solution for mathematical formulas

Embeds are Medium’s way of embedding content from other parties. There are +300 providers.

Using embeds

For instance, there is a provider for Twitter that allows you to embed Tweets in your Medium post, such as this one:

Vertexwahn on Twitter: "FlatlandRT is a 2D ray tracer visualization tool. Just released version 1.1.0: https://t.co/9T4xH6lTDD pic.twitter.com/2TsRk4XX7Z / Twitter"

FlatlandRT is a 2D ray tracer visualization tool. Just released version 1.1.0: https://t.co/9T4xH6lTDD pic.twitter.com/2TsRk4XX7Z

All you have to do is to copy the tweet URL, paste it to your Medium editor, and then it gets automatically converted into an embedded tweet. On the list of supported providers, you can also find https://texblocks.com/. Which gives you TeX support for medium. Unfortunately, it does not work for complex formulas, e.g.

becomes:

Besides the TeX rendering is not working there is also some annoying empty space around the formula (or you do not see anything at all in some browsers).

Also, other people struggled with this issue:

Using LaTeX on Medium

Add support for tables

There are no tables in Medium. WTF? Again, you have to use workarounds:

• 5 Tips for Embedding Tables in Your Medium Posts
• 3 Tips to Sharing Beautiful Tables on Medium Post

The most promising one seems to be this one:

Show table on Medium

The same arguments as for “Mathematical Formulas” can be used here. Embedding tables via a syntax, such as

| Month    | Savings  |
| -------- | -------- |
| January  | $2250    |
| February | $180     |
| March    | $6420    |

or providing a table editor, would lead to faster iterations in writing (since no external hacks or tools for tables have to be used). It could save memory and traffic since some users will simply screenshot a table and embedded as an image.

Tear down the paywall

Assume you have on average 100 readers per month for your written articles. Furthermore, let us assume that about 1/10 of those readers have a Medium membership. Putting your article behind the paywall would mean that you will immediately lose 90% of your readers. Unfortunately, the only way to earn money with your own writings on Medium seems to be putting them behind the paywall. Why is it not possible to earn money with an article that is not behind the paywall? It is clear that you can not get paid for 100 readers, but at least for the 10% that have a paid subscription. Let the content creator decide how content should be shared with the community. The current model leads to the effect that 90% of the readers get frustrated when facing the paywall. It is likely that in the future non-members will simply ignore “Medium” search results, and go for other blog services. If a non-member accesses Medium very frequently maybe also advertisements make sense to compensate for the fall of the paywall. Maybe a model such as non-members get always advertisements, logged-in non-paing members have 3 free advertisements, etc. would make sense here. A non-paying site visitor is better than no visitors at all.

Free membership for writers with many views/reads

Writers who have more than a certain number of views/reads on their articles should be offered a free membership.

Last words

If I need to describe the design of Medium: It's black and white with a focus on text and simplicity. Adding the previously described features may change the DNA of Medium. Users would get a more complicated user interface for editing stories, look & feel would change since you could play around with tables, formulas, etc. Maybe this is the chance for another platform that is more focused on technical writing. There are several competitors, such as DEV.

For Hugo, there is a Medium Theme. Maybe I will come up with my own solution in the future. Hugo also supports diagrams. Unfortunately, this takes additional effort on my side, and will lose this way the great Medium editor and read statistics. Please let me know what you think about my proposed additions and if you have found any alternatives.

On StackOverflow, I stumbled over the question “Is putting all project dependencies inside project’s repository good practice?”.

Actually, I have one public Git repository named FlatlandRT that tries to follow this idea. The file structure of this repository looks like this (generated via tree -L 2):

.
├── azure-pipelines.yml
├── devertexwahn
│   ├── ci
│   ├── core
│   ├── coverage.sh
│   ├── flatland
│   ├── imaging
│   ├── math
│   ├── okapi
│   └── WORKSPACE.bazel
├── docs
│   └── images
├── LICENSE
├── README.md
└── third_party
    ├── abseil-cpp
    ├── bazel-skylib
    ├── bazel-toolchain-0.8
    ├── Catch2
    ├── eigen-3.4.0
    ├── fmt
    ├── gflags
    ├── glog
    ├── googletest
    ├── hypothesis
    ├── Imath-3.1.7
    ├── libjpeg-turbo-2.1.4
    ├── libpng-1.6.39
    ├── nasm-2.14.02
    ├── openexr
    ├── pcg-cpp
    ├── pugixml-1.13
    ├── rules_boost
    ├── rules_pkg-0.9.0
    ├── software-bill-of-materials.md
    ├── xtensor
    ├── xtl
    └── zlib-1.2.13

The idea of the folder third_party is that it contains all external dependencies. Every third-party dependency (e.g. library, tool, etc.) should be added in source code form to the third_party folder.

There are some good reasons for this:

• Legal issues: Most open-source projects require you to reproduce the original license and copyright notes. By simply copying everything you downloaded and distributing this again you are most likely already fulfilling all legal constraints for the redistribution of an external dependency in source code form. Another scenario: It can happen that for some legal reasons some license notes have to be checked for certain files — this is without the original source not possible. It can also happen that a third-party dependency changes its license model — in this case, it is good to have a backup/prove that the source was distributed under a certain license in the past.
• Reproducibility: It can happen that an external dependency is deleted and can not be restored anymore. Especially when using smaller Open Source Projects that only have a single maintainer.
• Offline build: Allows a build of the software without an internet connection. If you have all dependencies in your repo you do not need to redistribute them in other ways. E.g. fetching them with a packages manager, downloading them, etc.
• Easy modification: Allows easy modification of the source code. If the code of all third-party libs is in third_party folder and all third-party libs are built from source it is very easy to do quick changes and fixes and test them with your codebase. Refactorings are not blocked by a complicated release and versioning process.

Managing third-party libraries

Naming conventions

I came up with some naming conventions for folder names in the third_party folder:

If a specific release version of a library is used this version becomes part of the directory name. For instance, for zlib version 1.2.13 the directory name becomes zlib-1.2.13. This also allows to have different release versions of a library at the same time in the third-party folder.

In the case a specific commit hash was used (e.g. from a Git repository) the corresponding commit hash is added as a suffix to the folder name. For instance, when using the commit4c8fe9fabbd149dff42f854c07fabbe286f93a8 of the repository rules_license the corresponding folder name becomes rules_license-84c8fe9fabbd149dff42f854c07fabbe286f93a8. This way it becomes clear how the repository was materialized.

Unfortunately, adding the commit as a suffix does not work pretty well with Git and GitHub. For instance when there is an update of GoogleTest all files that have not changed are shown as moved. That pollutes the diff view on Git and GitHub and makes it difficult to find the “real” changes. Therefore, there is a third convention. Use only the library name, e.g. googletest and track the current commit hash in the file software-bill-of-materials.md. This file tracks for every folder the corresponding commit hash:

Update of libraries

I wrote a shell script that auto-updates my libraries:

#!/usr/bin/env bash

#
#   SPDX-FileCopyrightText: 2022-2023 Julian Amann <dev@vertexwahn.de>
#   SPDX-License-Identifier: Apache-2.0
#

set -euxo pipefail

git_repos_with_bom_update=(
    "https://github.com/abseil/abseil-cpp"
    "https://github.com/AcademySoftwareFoundation/openexr"
    "https://github.com/bazelbuild/bazel-skylib"
    "https://github.com/bazelbuild/bazelisk"
    "https://github.com/catchorg/Catch2"
    "https://github.com/gflags/gflags"
    "https://github.com/google/benchmark"
    "https://github.com/google/glog"
    "https://github.com/google/googletest"
    "https://github.com/imneme/pcg-cpp"
    "https://github.com/jbeder/yaml-cpp"
    "https://github.com/martis42/depend_on_what_you_use"
    "https://github.com/nelhage/rules_boost"
    "https://github.com/nlohmann/json"
    "https://github.com/oneapi-src/oneTBB"
    "https://github.com/pytorch/cpuinfo"
    #"https://github.com/grailbio/bazel-toolchain" # Issues on macOS
)

for repo in "${git_repos_with_bom_update[@]}"
do
    bazel run //modernizer:update_git_archive_v2 -- "${THIRD_PARTY_DIR}" "${repo}"
done

The script that fetches a library and updates the commit hash looks like this:

#!/usr/bin/env bash

#
#   SPDX-FileCopyrightText: 2022-2023 Julian Amann <dev@vertexwahn.de>
#   SPDX-License-Identifier: Apache-2.0
#

set -euxo pipefail

# Provide third_party folder and GitHub repo url as command line arguments
if [ "$#" -ne 2 ]; then
    echo "Wrong number of parameters detected"
    echo "Usage: $0 <third_party_dir> <git_hub_url>"
    exit 1
fi

# First command line argument
third_party_dir=$1 # First command line argument e.g. ~/dev/Piper/third_party
git_hub_url=$2 # E.g. "https://github.com/imneme/pcg-cpp"

# Get github repo name from git_hub_url
git_hub_repo_name=$(echo "$git_hub_url" | sed -e 's/.*\///g')

# Create a temporary directory
tmpdir=$(mktemp -d)

# Get rid of temporary files when script exits
trap "rm -rf $tmpdir" EXIT

# Clone repo to tmp folder
cd "$tmpdir"
git clone "$git_hub_url"

# Determine hash
cd "$git_hub_repo_name"
commit_hash=$(git rev-parse HEAD)

# Determine hash without cloning
#git ls-remote "$git_hub_url" | \
#   grep refs/heads/master | cut -f 1

# Delete last version of the third party dependency
cd "$third_party_dir"
old_dir=$(ls | grep "$git_hub_repo_name")
old_hash=$(ls | grep "$git_hub_repo_name" | sed 's/$git_hub_repo_name-//')
echo "Old hash is" "$old_hash"
rm -rf "$old_dir"

# Download an extract latest third party dependency to third_party folder
cd "$tmpdir"
curl -L "$git_hub_url"'/archive/'$commit_hash'.zip' --output '$git_hub_repo_name-'$commit_hash'.zip'
sha256=$(sha256sum '$git_hub_repo_name-'$commit_hash'.zip' | cut -d ' ' -f 1)
echo "$sha256"
unzip -o '$git_hub_repo_name-'$commit_hash'.zip' -d "$third_party_dir"

# Rename folder to exclude hash
mv "$third_party_dir"/"$git_hub_repo_name"-"$commit_hash" "$third_party_dir"/"$git_hub_repo_name"

# Change commit hash
cd "$third_party_dir" || exit -1
grep -n "$git_hub_repo_name" "software-bill-of-materials.md"

lineNum="$(grep -n "$git_hub_repo_name" software-bill-of-materials.md | head -n 1 | cut -d: -f1)"
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
    sed -i $lineNum's/'$git_hub_repo_name'.*/'$git_hub_repo_name'-'$commit_hash'/' software-bill-of-materials.md
else
    sed -i '' $lineNum's/'$git_hub_repo_name'.*/'$git_hub_repo_name'-'$commit_hash'/' software-bill-of-materials.md
fi

There is also a nightly CI pipeline that runs this script:

The GitHub workflow looks something like this:

#
#   SPDX-FileCopyrightText: Copyright 2022-2023 Julian Amann <dev@vertexwahn.de>
#   SPDX-License-Identifier: Apache-2.0
#

name: Update third-party dependencies
on:
  schedule:
    - cron: "0 0 * * *"
  workflow_dispatch:
  #push: {}
    
jobs:
  build:
    runs-on: ubuntu-22.04

    steps:
      - name: Show OS version
        run: |
          lsb_release -a
          
      - name: Show Bazel version
        run: |
          bazelisk version

      - name: Show GCC version
        run: |
          gcc --version

     
      - uses: actions/checkout@v3

             
      - name: Run third-party dependencies update script
        run: |
          ./update_third_party_dependencies.sh "/home/runner/work/Piper/Piper"

      - name: Add Linux specific Bazel ignore file
        run: |
          cd devertexwahn
          cp ".bazelignore.linux" ".bazelignore"

     
      - name: Build and test using GCC11 on Ubuntu 22.04
        run: |
          cd devertexwahn
          bazelisk build --config=gcc11 --config=buildbuddy_remote_cache -- //...
          bazelisk test --config=gcc11 --config=buildbuddy_remote_cache -- //...
          bazelisk build --config=gcc11 --compilation_mode=dbg --config=buildbuddy_remote_cache -- //...
          bazelisk test --config=gcc11 --compilation_mode=dbg --config=buildbuddy_remote_cache -- //...
          bazelisk build --config=gcc11 --compilation_mode=opt --config=buildbuddy_remote_cache -- //...
          bazelisk test --config=gcc11 --compilation_mode=opt --config=buildbuddy_remote_cache -- //...

      - name: Build and test using Clang14 on Ubuntu 22.04
        run: |
          cd devertexwahn
          # Compile using Clang14 (fastbuild, debug and optimized)
          bazelisk build --config=clang14  --config=buildbuddy_remote_cache -- //...
          bazelisk build --config=clang14  --config=buildbuddy_remote_cache --compilation_mode=dbg -- //... 
          bazelisk test --config=clang14  --config=buildbuddy_remote_cache --compilation_mode=dbg -- //...
          bazelisk build --config=clang14  --config=buildbuddy_remote_cache --compilation_mode=opt -- //...
          bazelisk test --config=clang14  --config=buildbuddy_remote_cache --compilation_mode=opt -- //...
        
      - name: Remove Linux specific Bazel ignore file
        run: |
          cd devertexwahn
          rm ".bazelignore"

      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Auto update of dependencies

There are also other tools such as Copybara that could be used to move the changes from other repositories including the whole change history into my FlatlandRT mono repository. I decided against having all changes including all commit messages from all external libraries since this would pollute too much my usage of Git and GitHub. In detail: Most commits would be from changes in external libraries and not my project. To be clear here if I would have the proper tooling to handle this better I would like to have every single commit. The practical and manageable solution for me at this point (given the current tools Git and GitHub) is to have so-to-say auto-update snapshots every evening that summarize all changes.

Handling of large artifacts

In some cases copying a third-party dependency is not practical. For instance, when considering Qt or Boost. Boost`s source code is compressed over 100 MB already. Since GitHub is used to store the source code, it is a bit unhandy to add more than 100 MB to the repository only for a single dependency. For Qt, the situation is even worse. If I could I would also add Boost and Qt to the third-party folder, but size limits from Git and GitHub prevent me from doing this.

In the end, handling large artifacts is a limitation of Git and GitHub. I really see here the chance for a new version control system and hosting platform to solve this issue that enables this use case. A similar problem arises when you have big data assets.

“Plain” Git is not a good choice when a repository contains many big data assets (such as textures, 3D models, virtual environments, trained neural nets, audio files, etc.) for different reasons, such as:

• Fetch/clone speed goes down
• Users can probably not checkout the repo because of limited disk space (e.g. when repo is > 1 TB)

Nevertheless, it is desirable to be able to manage data and source code together in one repository.

There is Git LFS, but it has some drawbacks, e.g.:

• referenced files can get deleted by accident since they are not part of the “real” repo
• misc git LFS issues, when switching between branches
• Users sometimes forget to also check in Git LFS files
• No good migration path if reference files move later to another location (old commits will then point to invalid file locations)

Recently, Git scalar was added to Git. Maybe this is a first approach to address some problems of this.

Anyway, I was also thinking about a light-weight solution to this problem, such as adding a simple script that downloads the “big” stuff, e.g.:

echo "Fetch boost..."
curl "https://boostorg.jfrog.io/artifactory/main/release/1.82.0/source/boost_1_82_0.tar.gz" --output boost_1_80_0.tar.gz
tar -xf boost_1_82_0.tar.gz

But this again brings problems such as that files outside of the repo can get changed by accident and then do not fit anymore to the corresponding commit hash. In the end, atomic commits would be desirable to pin everything together. In my current setting Boost is fetched from the internet as a tar.gz file and hash-checked via Bazel.

Summary

I really like to have a mono repository with all external dependencies included. Since there are some tooling limitations and I do not have time to come up with my own version control system one has to live with some compromises here. Nevertheless, I think a version control system that gives atomic commits and can easily handle an insane amount of data would be beneficial in many areas and for many projects (e.g. machine learning). Maybe the solution would be something like GitHub, but with the difference that there is only ONE BIG mono repo for all users with ONE CI for everyone.