Skip to content

Multiview

This example uses the multiview extension to draw the same texture multiple times simultaneously. This is useful in VR applications where it is necessary to render the same scene twice from different cameras. When a multiview render pass happens, the different renderings will be sent to a 3D texture. In the shaders, you can differentiate between which pass you are rendering with gl_ViewIndex. For example, the following code is what makes the triangles turn in opposite directions:

1
float rotationSign = gl_ViewIndex == 0 ? -1.0 : 1.0;

Filename: multiview/doc/shadersnippet.vert

In a 3D VR game or application, this could be a ternary/conditional which translates and slightly rotates the view matrix if it's on the second layer.

Once the 3D texture has been rendered to and contains the two views on each layer, we need to render each layer to the screen, one view on the left half and one view on the right. This process does not involve multiview, just sampling from each layer of the 3D texture and re-rendering them in a squashed half-screen views. In order to achieve this, we do two passes with the same pipeline, updating a push constant which says which layer to sample from.

Here is the shader code which receives the push constant:

1
2
3
4
5
6
7
8
9
layout(push_constant) uniform PushConstants {
    int arrayLayer;
} pushConstants;

void main()
{
    vec3 color = texture(colorTexture, vec3(texCoord, pushConstants.arrayLayer)).rgb;
    fragColor = vec4(color, 1.0);
}

Filename: multiview/doc/shadersnippet.vert

Initialization

In order to set up splitscreen multiview rendering, we must do the following:

  1. Create two separate pipelines:
    • One with the multiview shader.
    • One with the shader that samples from the multiview-ed 3D texture.
  2. Configure each pipeline to have the correct push constant and bind group layouts.
  3. Bind the texture from the first pipeline to the second pipeline for sampling.

Creating the multiview pipeline is almost identical to previous examples of creating pipelines in KDGpu, with the new addition of adding the push constant range to the bind group:

1
2
3
4
    // Create a pipeline layout (array of bind group layouts)
    m_mvPipelineLayout = m_device.createPipelineLayout(PipelineLayoutOptions{
            .pushConstantRanges = { m_mvPushConstantRange },
    });

Filename: multiview/multiview.cpp

And setting up the multiview pass options so that multiview is enabled:

1
2
3
4
5
6
7
8
9
    m_mvPassOptions = {
        .colorAttachments = {
                { .view = m_multiViewColorOutputView,
                  .clearValue = { 0.3f, 0.3f, 0.3f, 1.0f },
                  .finalLayout = TextureLayout::ShaderReadOnlyOptimal },
        },
        .depthStencilAttachment = { .view = m_multiViewDepthView },
        .viewCount = 2, // Enables multiview rendering
    };

Filename: multiview/multiview.cpp

m_multiViewColorOutputView and m_multiViewDepthView are initialized when the scene is initialized and when the window is resized.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
void MultiView::createMultiViewOffscreenTextures()
{
    m_multiViewColorOutput = m_device.createTexture(TextureOptions{
            .type = TextureType::TextureType2D,
            .format = m_mvColorFormat,
            .extent = { m_window->width(), m_window->height(), 1 },
            .mipLevels = 1,
            .arrayLayers = 2,
            .samples = SampleCountFlagBits::Samples1Bit,
            .usage = TextureUsageFlagBits::ColorAttachmentBit | TextureUsageFlagBits::SampledBit,
            .memoryUsage = MemoryUsage::GpuOnly,
    });
    m_multiViewDepth = m_device.createTexture(TextureOptions{
            .type = TextureType::TextureType2D,
            .format = m_mvDepthFormat,
            .extent = { m_window->width(), m_window->height(), 1 },
            .mipLevels = 1,
            .arrayLayers = 2,
            .samples = SampleCountFlagBits::Samples1Bit,
            .usage = TextureUsageFlagBits::DepthStencilAttachmentBit,
            .memoryUsage = MemoryUsage::GpuOnly,
    });

    m_multiViewColorOutputView = m_multiViewColorOutput.createView(TextureViewOptions{
            .viewType = ViewType::ViewType2DArray,
    });
    m_multiViewDepthView = m_multiViewDepth.createView(TextureViewOptions{
            .viewType = ViewType::ViewType2DArray,
    });
}

Filename: multiview/multiview.cpp

On resize/initialize we must also re-create the bindgroup that is correctly bound to the new textures we created in the previous step.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
void MultiView::updateFinalPassBindGroup()
{
    // Create a bindGroup to hold the Offscreen Color Texture
    m_fsqTextureBindGroup = m_device.createBindGroup(BindGroupOptions{
            .layout = m_fsqTextureBindGroupLayout,
            .resources = {
                    {
                            .binding = 0,
                            .resource = TextureViewSamplerBinding{ .textureView = m_multiViewColorOutputView,
                                                                   .sampler = m_multiViewColorOutputSampler },
                    },
            },
    });
}

Filename: multiview/multiview.cpp

Per-Frame Rendering Logic

Rendering to the off-screen multiview textures is the first thing we do on render, and very similar to rendering in previous examples. The only addition is the need to update the push constant.

1
2
3
4
5
6
7
    // MultiView Pass
    auto mvPass = commandRecorder.beginRenderPass(m_mvPassOptions);
    mvPass.setPipeline(m_mvPipeline);
    mvPass.setVertexBuffer(0, m_vertexBuffer);
    mvPass.pushConstant(m_mvPushConstantRange, &rotationAngleRad);
    mvPass.draw(DrawCommand{ .vertexCount = 3 });
    mvPass.end();

Filename: multiview/multiview.cpp

The majority of the new logic occurs during the full screen pass, where we render each layer of off-screen texture in two separate passes. We use KDGpu::RenderPassCommandRecorder::setViewport to squash the different views onto either half of the screen.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
    // FullScreen Pass
    m_fsqPassOptions.colorAttachments[0].view = m_swapchainViews.at(m_currentSwapchainImageIndex);
    auto fsqPass = commandRecorder.beginRenderPass(m_fsqPassOptions);
    fsqPass.setPipeline(m_fsqPipeline);
    fsqPass.setBindGroup(0, m_fsqTextureBindGroup);

    // Left Eye
    fsqPass.setViewport(Viewport{
            .x = 0,
            .y = 0,
            .width = halfWidth,
            .height = float(m_window->height()),
    });
    const int leftEyeLayer = 0;
    fsqPass.pushConstant(m_fsqLayerIdxPushConstantRange, &leftEyeLayer);
    fsqPass.draw(DrawCommand{ .vertexCount = 6 });

    // Right Eye
    fsqPass.setViewport(Viewport{
            .x = halfWidth,
            .y = 0,
            .width = halfWidth,
            .height = float(m_window->height()),
    });
    const int rightEyeLayer = 1;
    fsqPass.pushConstant(m_fsqLayerIdxPushConstantRange, &rightEyeLayer);
    fsqPass.draw(DrawCommand{ .vertexCount = 6 });

    // Call helper to record the ImGui overlay commands
    renderImGuiOverlay(&fsqPass);

    fsqPass.end();

Filename: multiview/multiview.cpp

Note that it is necessary to re-set the color attachment's view before beginning the second pass, and that we send a different value to the push constant for each pass (that's the index of the layer that the shader should sample from).

Updated on 2023-12-22 at 00:05:36 +0000