Extremely slow performance when processing virtual terminal sequences

[cmuratori]

Windows Terminal version (or Windows build number)

1.8.1521.0

Other Software

No response

Steps to reproduce

Using any command line utility that produces virtual terminal sequences for setting the colors of individual characters, the performance of the terminal drops by a factor of around 40.

To measure this effect precisely, you can use the F2 key in termbench and observe the performance difference between color-per-character output and single-color output:

https://github.com/cmuratori/termbench/releases/tag/V1

Expected Behavior

Despite the increased parsing load, modern CPUs should not have a problem parsing per-character color escape codes quickly. I would expect the performance of the terminal to be able to sustain roughly the same frame rate with per-character color codes as without, and if there was a performance drop, I wouldn't expect it to be anything close to 40x.

Actual Behavior

The speed of per-character color output is 40x slower than the speed of single-color output.

[skyline75489]

Thanks for the amazing benchmark tool! I'm sure @miniksa will be interested in trying it out.

Yeah the current performance of colored output is not as fast as non-colored ones. However, the bottleneck is not in parsing but rather rendering, in my opinion. Well, to be specific, when it comes to terminal, there's a lot of things that may hurt the performance, for example ConPTY, DxRenderer, memory allocation, etc. Anyway this is a good tool to measure the performance of both conhost & terminal.

[cmuratori]

Can you explain what you think is the slow part of rendering characters in multiple colors (for those of us unfamiliar with how the renderer of your console works)?

[skyline75489]

If it means anything, here's a sample WPR trace running your benchmark tool:

As you can see, roughly 70% of CPU time is consumed by RenderThread, which in this case (Windows Terminal) relys DxRenderer for the actual rendering job. The OutputThread (where the VT parsing & related work reside) only takes 10% ~ 20% of the CPU time. Note that this pattern of CPU usage can be seen among almost every output-heavy-with-color programs (cmatrix, for example).

Regarding colored vs non-colored text, my initial observation is that drawing colored text frame takes longer time than non-colored ones, which leaves less CPU for the OutputThread and causes the FPF to drop.

[skyline75489]

Deep down in the RenderThread, we use DirectX to do the actual drawing. In a perfect world, we'd like to see RenderThread takes most of the CPU time, which means we are not wasting too much CPU on VT processing. But even if we managed to make RenderThread consume 90% of the CPU, this still won't gives you a near close rendering performance for colored text comparing non-colored text.

Another example that might helps: if we cat a very long file, which gives us non-colored output, the WRP trace usually indicates about 60%~70% of CPU is consumed by OutputThread and only 30% of CPU consumed by RenderThread. See how this is going here? Rendering non-colored text is simply a much more cheaper operation for the renderer, which helps the FPS a lot.

In conclusion, I know there's a lot of space for performance tuning in the Windows Terminal, but I honestly can't expect the performance of colored text to be close comparing to non-colored text. Even with a hardware-accelerated solution like DirectX, we still faces performance bottleneck from the rendering stack & GPUs.

[cmuratori]

Two things:

1. Can you expand that entire Render::PaintFrame trace? I assume it has more detailed attribution of time there?

2. I'm not sure I understand what you're suggesting. Are you actually saying that you think a GPU will slow down substantially if it has to use a different color for each character?

[mmozeiko]

I cannot profile Terminal .exe as there are no symbols for it available on Microsoft pdb servers.
But I wrote dumb vt terminal using CreatePseudoConsole that does not render anything - all it does it sits in a loop doing ReadFile from application output. Basically for (;;) { ReadFile(pipe, ...); } and completely ignores output. So there is no VTE parsing, no rendering, no formatting. Nothing. Only thing that runs is conhost.exe.

In task manager I see that there is ~4MB/s traffic to my "terminal" from Casey's termbench.exe. And I get around 250-300ms per frame which seems pretty slow. Task manager shows that conhost.exe is bottleneck. It is using 100% of one core (3.12% on 32x core Ryzen):

When I run ETW on conhost.exe for my "terminal" I see following things:

1. A lot of time is spent in std::vector resizing:

2. A lot of time is spent in std::stringstream:

3. A lot of time is spent in RenderThread:

The confusing part to me in this RenderThread call stack are all those string formatting functions like _SetGraphicsRenditionRGBColor. Why is conhost.exe formatting VT sequences like this? Shouldn't my "terminal" receive direct bytes in pipe from what termbench application is sending?

Again - there is zero rendering happening in my terminal. There are no DirectX calls at all. Only conhost.exe is bottleneck here.

[skyline75489]

@cmuratori To answer you question:

1. The deepest call that consumes the CPU is

   terminal/src/renderer/dx/CustomTextRenderer.cpp

   Line 798 in fb597ed

   d2dContext->DrawGlyphRun(baselineOrigin, glyphRun, glyphRunDescription, brush, measuringMode);

   
   This is how Windows Terminal essentially uses to draw text.
2. I'm seeing the fact that drawing colored-text consumes more CPU time in the renderer. I can't say it's because of DirectX or GPU, or both. I'm no DX expert so I can only guess both.

[skyline75489]

@mmozeiko To helps you understand how things work under the hood, the RenderThread you see is actually used by conhost.exe to "render" VT sequences for the Windows Terminal. That's why you see all the text-related things. This is part of the ConPTY mechanism. A more detailed introduction can be found here.

Shouldn't my "terminal" receive direct bytes in pipe from what termbench application is sending?

Unfortunely no. There's currently no way to bypass the ConPTY layer. This does hurt the performance but it's necessary at the moment for the terminal to work properly. There's discussion about ConPTY "passthough" in #1173.

[cmuratori]

1. The deepest call that consumes the CPU is https://github.com/microsoft/terminal/blob/fb597ed304ec6eef245405c9652e9b8a029b821f/src/renderer/dx/CustomTextRenderer.cpp#L798

Is it not possible to post the entire trace?

[skyline75489]

@cmuratori it's possible. Just me being lazy about it because I've seen too many of those traces.

The WPR traces actually varies, depending on the content being drawn, the font and probably also the GPU performance. Check out the screenshot here #6206 (comment) if you're interested. This PR helps the performance with Cacafire, but for cmatrix (or more practially, vim) it does not mean too much.

[mmozeiko]

@skyline75489
I cannot test this, because I have no idea how to rebuild conhost.exe, but from my benchmark it is visible that all this terminal stuff can be sped up a lot by improving string processing on ConPTY layer. Nothing needs to be changed in DirectX rendering. Just need to avoid expensive string allocations and operations. This will help all terminal application - not only Windows Terminal. Currently if somebody wants to implement more efficient rendering they really cannot, because they will be bottlenecked by these issues in ConPTY layer.

For example, _SetGraphicsRenditionRGBColor should be changed to something like this (probably can do something even better, but I wrote this in github comment):

HRESULT VtEngine::_SetGraphicsRenditionRGBColor(const COLORREF color, const bool fIsForeground) noexcept
{
    DWORD const r = GetRValue(color);
    DWORD const g = GetGValue(color);
    DWORD const b = GetBValue(color);

#define FMT_BYTE(x)                           \
    if (x >= 100) *ptr++ = (x/100) + '0';     \
    if (x >= 10)  *ptr++ = ((x/10)%10) + '0'; \
    *ptr++ = (r%10) + '0';

    char buffer[10+3+3+3];

    char* ptr = buffer;
    *ptr++ = '\x1b';
    *ptr++ = '[';
    *ptr++ = fIsForeground ? '3' : '4';
    *ptr++ = '8';
    *ptr++ = ';';
    *ptr++ = '2';
    *ptr++ = ';';
    FMT_BYTE(r);
    *ptr++ = ';'
    FMT_BYTE(g);
    *ptr++ = ';'
    FMT_BYTE(b);
    *ptr++ = 'm';
    
#undef FMT_BYTE

    return _Write({ buffer, ptr - buffer });
}

No std::string and no vsnprintf functions. Rest of file uses too much of std::string just for trivial constant string literals used in formatter string.

[skyline75489]

@mmozeiko Yeah you have a good point here actually. What I want to say is that there's a lot things under the hood than just "processing terminal sequences". The performance of ConPTY layer is also very important for the console subsystem, as you mentioned it helps all terminal applications.

Do understand that some of the tuning tricks are not used for both readability and maintainability of the project. That being said, if we found something worth investigating, I think we'd all be happy to squeeze as much CPU as we can to improve the performance.