Motivation

Finding a good balance between code and data in Rendering. What is the necessary code that should be written ? Why ?

In rendering many areas can be described in a fast and robust way using data. A pipeline (in D3D12/Vulkan lingo) for example is a collection of different states: depth stencil, alpha blend, rasterizer, shaders, etc. All those state can be hard-coded or defined in data. Moving them to data can help with the visibility of them, that instead of being buried somewhere into the code can be retrieved before even running the application.

As a bigger-scope example, a frame-graph can be implicitly defined inside the code, if different areas, or in data. Recent posts about it started raising attention to the problem, especially after the introduction of lower-level APIs like D3D12 and Vulkan and their resource barriers. I’ve personally used something like json (xml back in the day) since 2009, after asking myself the very silly question:

what is the biggest dependency in rendering?Render Targets!

Since then I saw only in the Codemasters postprocess system (since Dirt 2) a similar approach, and have never being able to advocate towards it. The only full use case I have is my personal indie game (a full deferred rendering pipeline with many different rendering needs) all defined in a json file (render_pipeline.json). Anyway, a couple of examples of this data-driven mentality can be found here:

http://bitsquid.blogspot.com/2017/03/stingray-renderer-walkthrough-7-data.html

I chose to see what is a good way of describing low-level rendering resources, the bricks towards data-driven rendering. I’ve already tried defining them in a json file, but wanted something more direct — something I can copy easily with minimal parsing.

I found 4 possible approaches:

1. Custom data language
2. Already existing data language
3. Json (already used)
4. Hard-coding everything

In this experiment I’ve chosen Flatbuffers for the easy of use, the good performances and the feature set that seems complete. As an exercise, I wanted to create some UI based on the data coming from Flatbuffers without having to write too much code.

Flatbuffers

Flatbuffers is a serialization library developer by Google used by many companies.

https://google.github.io/flatbuffers/

Compared to Protocol Buffers (still developed by Google) it tries to go towards a very simple parsing/unpacking (actually ABSENT in Flatbuffers, so much faster to read/write) and serialization speed.

Flatbuffers is mainly a compiler that accepts .fbs (FlatBuffers Schema) files and can generate code for serialization purposes.

The advantage is that it automatically generates the parsing files in the language you prefer (C++, Java, C#, Go, C, Lua, Javascript, Rust) without you needing to write the always tedious serialize/deserialize methods.

It is largely based on either simple c-structs or tables with offsets for more complex object.

The objective here will be to create a schema file, define a couple of resources (like textures) and use those to automatically generate UI. I will be using the SDL + ImGUI sample from the amazing ImGUI as a base.

The flow will be the following:

1. Write schema files
2. Generate reflection informations
3. Parse schemas
4. Generate UI

Schema Files

Let’s write our first schema file. A bigger version (that I am using for my low-level renderer) is included in the github repository.

namespace rendering;

enum TextureFormat : ushort { UNKNOWN, R32G32B32A32_TYPELESS, R32G32B32A32_FLOAT, R32G32B32A32_UINT, R32G32B32A32_SINT, R32G32B32_TYPELESS, R32G32B32_FLOAT, R32G32B32_UINT, R32G32B32_SINT, R16G16B16A16_TYPELESS, R16G16B16A16_FLOAT, R16G16B16A16_UNORM, R16G16B16A16_UINT, R16G16B16A16_SNORM, R16G16B16A16_SINT, R32G32_TYPELESS, R32G32_FLOAT, R32G32_UINT, R32G32_SINT, R10G10B10A2_TYPELESS, R10G10B10A2_UNORM, R10G10B10A2_UINT, R11G11B10_FLOAT, R8G8B8A8_TYPELESS, R8G8B8A8_UNORM, R8G8B8A8_UNORM_SRGB, R8G8B8A8_UINT, R8G8B8A8_SNORM, R8G8B8A8_SINT, R16G16_TYPELESS, R16G16_FLOAT, R16G16_UNORM, R16G16_UINT, R16G16_SNORM, R16G16_SINT, R32_TYPELESS, R32_FLOAT, R32_UINT, R32_SINT, R8G8_TYPELESS, R8G8_UNORM, R8G8_UINT, R8G8_SNORM, R8G8_SINT, R16_TYPELESS, R16_FLOAT, R16_UNORM, R16_UINT, R16_SNORM, R16_SINT, R8_TYPELESS, R8_UNORM, R8_UINT, R8_SNORM, R8_SINT, R9G9B9E5_SHAREDEXP, D32_FLOAT_S8X24_UINT, D32_FLOAT, D24_UNORM_S8_UINT, D24_UNORM_X8_UINT, D16_UNORM, S8_UINT, BC1_TYPELESS, BC1_UNORM, BC1_UNORM_SRGB, BC2_TYPELESS, BC2_UNORM, BC2_UNORM_SRGB, BC3_TYPELESS, BC3_UNORM, BC3_UNORM_SRGB, BC4_TYPELESS, BC4_UNORM, BC4_SNORM, BC5_TYPELESS, BC5_UNORM, BC5_SNORM, B5G6R5_UNORM, B5G5R5A1_UNORM, B8G8R8A8_UNORM, B8G8R8X8_UNORM, R10G10B10_XR_BIAS_A2_UNORM, B8G8R8A8_TYPELESS, B8G8R8A8_UNORM_SRGB, B8G8R8X8_TYPELESS, B8G8R8X8_UNORM_SRGB, BC6H_TYPELESS, BC6H_UF16, BC6H_SF16, BC7_TYPELESS, BC7_UNORM, BC7_UNORM_SRGB, FORCE_UINT }

attribute "ui";

struct RenderTarget {
    width                   : ushort (ui: "min:1, max:16384");
    height                  : ushort;
    scale_x                 : float;
    scale_y                 : float;
    format                  : TextureFormat;
}

There are few things here to discuss.

1. Enums. Flatbuffers can generate enums with string version of each values and conversions between enum and string.
2. Struct. It is exactly like C/C++: a simple struct that can be memcopied. Different than a Table (that can point to other structs and Tables).
3. Attributes. This can be used to define custom parsable attributes linked to a member of a struct/table. They can be used, for example, to drive the UI generation.

Generating Reflection Informations

After we generated the schema file, we can serialize it and load/save it from disk. But we need reflection data to be able to automatically generate the UI we need! There are two main reflection mechanisms in Flatbuffers: mini-reflection and full-reflection. We will use both to generate a UI using ImGUI and see the differences.

Mini-Reflection

This is the simplest of the two and works by generating an additional header file for each .fbs file we use. The command line is the following:

flatc --cpp RenderDefinitions.fbs --reflect-names

This will generate the RenderDefinitions_Generated.h file that must be included in your application and has the downside of needing you to recompile every time you change the data.

Also, and this is the biggest downside, I could not find any way to parse custom per-member attributes.

I hope I am wrong, but could not find any documentation on the topic: everything seems to point towards the full reflection mechanism.

So why bothering with the mini-reflection ?

Mini-reflection generates code, and this became useful for one of the most tedious C/C++ code to write: enums!

I can’t count how many times I wrote an enum, I wanted the string with the same value for it (for example to read from a json file and get the proper enum value) and every time an enum is changed is painful.

So a lesson from the mini-reflection is to have a code-generator for enums for C/C++, and I will show an example soon in another article.

Back to the enums, Flatbuffers generates:

1. Enum
2. Name array
3. Value array
4. Enum to name method

A nice property of the generated code for the enum is that it is easy to copy-paste in any c++ file — no Flatbuffers involved!

This is my first choice now when I want to write an enum in any c++ application.

Full-reflection

This is the most used (or at least documented) form of reflection in Flatbuffers.

It use a very elegant solution, totally data-driven: it reads a reflection schema file that can parse…ANY other schema!

This very Inception-esque mechanism gives the full access to all the types, including Attributes.

By executing this command:

flatc.exe -b --schema reflection.fbs RenderDefinitions.fbs

the RenderDefinitions.bfbs (binary fbs) file is generated.

This is the file that needs to be read to fully reflect the types inside the .fbs file. The order of operations is the following:

1. Generate a binary fbs with flatc (with the command line shown)
2. Load the bfbs file generated
3. Load the schema from the bfbs
4. Reflect

The fbfs file contains all the informations from the schema: types, enums, attributes.

Parsing schemas and Generating UI

For both reflection mechanisms the objective is the same: given a type (RenderTarget) generate an editor that can edit properties and potentially load/save them.

Mini-Reflection

The UI generation is pretty straightforward with mini-reflection.

Each type defined in the .fbs file contains a type_name-TypeTable() method that gives accent to a TypeTable.

This contains a list of per-member type, name and default values.

What is really missing here is the attributes, that could be used to generate custom UI in a more specific way (eg. adding a min/max/step to a slider).

The code doing this is in the github sample.

There are few interesting points here.

ImGui usability

In order to use ImGui to modify a struct, I had to create the class FlatBuffersReflectionTable to instantiate a struct with a similar layout than the Flatbuffers struct.

This is annoying but I could not find a way around different than this.

With this in-place, a ImGUI slider can point to a memory area that can be used to save/load the data. Let’s begin by retrieving the TypeTable:

const TypeTable* rt_table = rendering::RenderTargetTypeTable();

The TypeTable is what is included in the generated header and contains the reflection informations. Listing the members and their type is pretty straight-forward:

for ( uint32_t i = 0; i < type_table.num_elems; ++i ) {
    const flatbuffers::TypeCode& type_code = type_table.type_codes[i];
    ImGui::Text( "%s: %s", type_table.names[i], flatbuffers::ElementaryTypeNames()[type_code.base_type] );
    sprintf_s( s_string_buffer, 128, "%s", type_table.names[i] );
    
    if ( type_code.sequence_ref == 0 ) {
        if ( type_table.type_refs[type_code.sequence_ref] ) {
            const flatbuffers::TypeTable* enum_type = type_table.type_refs[type_code.sequence_ref]();
             ImGui::Combo( s_string_buffer, (int32_t*)reflection_table.GetData( i ), enum_type->names, enum_type->num_elems );
        }
    }
    else {
        switch ( type_code.base_type ) {
             case flatbuffers::ET_BOOL:
            {
                ImGui::Checkbox( s_string_buffer, (bool*)reflection_table.GetData( i ) );
                break;
            }
         }
    }
}

The interesting parts:

flatbuffers::TypeCode* contains the reflection information for a type.

Given a type_code, sequence_ref can be used to check if it is an enum, pointer, or primitive type. In this case is used for enum, showing a combo with all the selectable values.

Base_type contains instead the primitive type. In this example a bool can be mapped to a checkbox. This uses the custom reflection_table class to have a memory area for ImGUI.

For mini-reflection this is basically it.

Full-reflection

Code here is longer but it follows the 4 steps highlighted before.

All the code is inside the ReflectUIFull method.

Here the binary fbs file and its corresponding schema are loaded.

// 1. Obtain the schema from the binary fbs generated
std::string bfbsfile;    
flatbuffers::LoadFile("..\\data\\RenderDefinitions.bfbs", true, &bfbsfile );     
const reflection::Schema& schema = *reflection::GetSchema( bfbsfile.c_str() );

The schema can be used to list the types:

// 2. List all the types present in the fbs.    
auto types = schema.objects();    
for ( size_t i = 0; i < types->Length(); i++ ) {        
   const reflection::Object* type = types->Get( i );
   ImGui::Text( "    %s", type->name()->c_str() );    
}

(Using the auto here because I am lazy. The type is some multiple templates of offsets…) We can also list all the enums:

auto enums = schema.enums();    
for ( size_t i = 0; i < enums->Length(); i++ ) {        
    const reflection::Enum* enum_ = enums->Get( i );
    ImGui::Text( "    %s", enum_->name()->c_str() );    
}

A problem I found (with a workaround in the code) is that enums do not have an easily to access array of string values.

So I generated one for the sake of example, but I am far from happy with the solution!

Going forward, we can get the type we want to reflect (notice the full namespace.type):

auto render_target_type = types->LookupByKey( "rendering.RenderTarget" );
and begin the work on each field:
auto fields = render_target_type->fields();    
if ( fields ) {
    // 5.1. List all the fields        
    for ( size_t i = 0; i < fields->Length(); i++ ) {
            auto field = fields->Get( i );
            ...

and the UI can be generated.

For each field, the primitive type can be accessed with the following:

reflection::BaseType field_base_type = field->type()->base_type();

and again, I found a workaround to know if a type is primitive or an enum.

Last piece of the puzzle: attributes!

auto field_attributes = field->attributes();
if ( field_attributes ) {
    auto ui = field_attributes->LookupByKey( "ui" );
    if ( ui ) {
      ImGui::Text("UI attribute: %s", ui->value()->c_str());
    }
}

These can be parsed as strings and can be used to drive UI code (like a slider with min, max and steps).

Conclusions

In the end, I’ve managed to generate UI based on a type without too much code.

There was some reverse-engineering to do because I could not find proper documentation (I possibly miss some links to a in-depth example of reflection!) but nothing major.

The full source code:

(https://github.com/JorenJoestar/FlatbuffersReflection)