How to improve latency?

[alatteri]

Hello,

I've been using UG for many years, and thank you so much for creating, maintaining and improving it.
I've recently seen an integration that someone is doing using UG and Comprimato J2K, and their latency is around 7-8 frames. I know they are using BMD SDI out, but unsure of what their ingest card is, but likely BMD too.

We are using 12bit RGB 444 SDI -> BMD in (4K Extreme or 8K cards) / out (UltraStudio Mini 4K) , and while we have not had the ability to test with Comprimato, no matter what codec I try (x264/x265/Cineform) I can't get below around 14 frames latency.
We do need to have 10/12 bit 444, RGB preferred, but YUV acceptable. I've tried even setting gop=1 to in theory remove temporal latency. I've not yet tested NVENC in a long time. The issue is our receivers are Intel NUCs (no discrete GPU), the last time I tried, they could not properly decode NVenc stream in software.

Any tips and how I can improve this?

[MartinPulec]

Hi Alan,

the 14 frames latency is in my opinion very high. I am trying to understand, where it can come from - could you provide more details (video resolution, command used)?

gop=1 might not be needed, UG already disables B-frames, P-frames should not introduce additional latency (are predicted from previous frame(s))

I've done quick test with current code - using 2160p24 with 10-bit YCbCr (I don't have too much equipment that would support 444 RGB in UHD), Something like uv -t decklink -c lavc:enc=libx265 and uv -d gl on same computer - see pictures libx265 and libx264, the previewing Video Assist 4K is displaying the same (split) input that the BMD Decklink is capturing. If you look closer, the notebook didn't manage to compress H.265 at full rate but still the latency is 5 frames. Of course, this is artificial content and the compression characteristics will be very different with real image.

But the point of the previous paragraph is - can you ensure that the delay originates really from the compression/decompression? 444 RGB at higher bit-depth and the content certainly makes a difference so the latency may increase. But 14 frames look rather that the frames are somewhere buffered or so.

I've not yet tested NVENC in a long time. The issue is our receivers are Intel NUCs (no discrete GPU), the last time I tried, they could not properly decode NVenc stream in software.

Just FYI, there are some updates in UG, I've made somehow intra-refresh working (properly) with h264_nvenc (hevc partially). If the problem with decode was that performance one (decoded by single CPU core) I am not sure whether there is some update (likely on the decoding side; from a quick test it doesn't seem to be so)

[alatteri]

Hi Martin...

My encoder/decoder are headless Ubuntu Server install and separate machines, so I can't test in your scenario. Below are the command line syntaxes used for each. That might provide some understanding. I wonder if specifically declaring R12L is causing a delay via a slow color conversion? Some use cases we do not declare a channel-map.

uv -m 1316 -t decklink:codec=R12L -c libavcodec:encoder=libx265:crf=22\:vbv-bufsize=60000\:vbv-maxrate=30000 -f V:rs:200:240 -s embedded --audio-capture-format channels=8 --audio-codec=OPUS:bitrate=192K -f A:mult:3 -P 40000

uv -d decklink sync -r embedded --audio-channel-map 6:0,7:1 --audio-scale none -P 50000 --param use-hw-accel,resampler=soxr,decoder-use-codec=R12L

[MartinPulec]

Well, in my opinion the sync parameter is adding the delay. You can try to decrease min/max scheduled frames but I think that there is a possibility to decrease the latency at most by like 2 frames. I am not sure if it can be improved significantly in the "synchronized" mode. If I remember correctly, reducing the number of scheduled frames may lead to instability.

[alatteri]

I'll try to run a latency test today without sync.

[MartinPulec]

[out-of-topic remark] when trying various codecs it may be worth trying also FFV1 (via libavcodec) for 12-bit RGB. But this is really OT, I don't suppose that it may help here.

[alatteri]

As I understand FFV1, it is lossless, and very little ability to control bitrate, so the data load is massive and not really useful reasonable internet streaming. When I was testing Cineform recently, bitrates for a 2K SDI 12bit signal were in the 150mb/s range, which is barely practical for our use case.

[MartinPulec]

Yep, you are right - I thought FFmpeg honored my 10M bitrate setting when I was testing. But it was not true, it just coincidentally generated the stream that was approx. 10 (actually 12) Mbps by chance.

[alatteri]

I ran fresh tests, bypassing hd-rum reflector, and with minimal command line parameters, and I can get ~9 frames latency @24fps. I'm going to add on my full parameters and the affect.

encoder:
uv -t decklink -c libavcodec:encoder=libx265 -s embedded --audio-capture-format channels=8 --audio-codec=OPUS:bitrate=192K 10.55.118.51

decoder:
uv -d decklink --param use-hw-accel

[MartinPulec]

Okay, I've tried multiple things but unfortunately I was not able to get a clean path in which I was able to pass 2160p24 12-bit RGB with sufficient performance. Due to insufficient performance, I've also got 6-10 frame latency even without the compression. Anyways, this may be the point that the latency is given by the performance. If you are comparing to results with cmpto_j2k, it is can be also important that the implementation is altogether on the GPU.

[alatteri]

Hi Martin,

2K DCI is our majority use case, so testing even HD latencies is valid, and much less compute intensive than 4K.

With 2K and a fast Ryzen cpu, I’m unable to get below 9 frames glass-to-glass on local LAN.
Adding decklink sync brings that to 12 frames.

I’ve only tried R12L 444 as that is our desired colorspace, but I am open to full range R10K.

thank you again for everything you do for this project. It helps many people in my profession.

[MartinPulec]

I did 2 more attempts with 2K DCI if acceptable with these results:

1. R10k without compression - 3 frames
2. R10k with libx265 (default opts, Intel i7-11800) - 5 frames
3. R12L without the compression - 3 frames

For cases 1,2 decklink 4K was used, case 3 8K Pro. Input was -t testcard2:noise:m=2d24. If you look at the pictures below, the left Smartscope display is presenting the source; its loop-through SDI is passed to capture Decklink, sent over loopback interface (with or without compression) and displayed back - wired to the right Smartscope output).

Details

12-bit has slightly shifted colors - I believe that may not support 12-bit colors.

[alatteri]

Hi Martin... thank you for running the tests. Would you please provide me the full command syntax used so I can replicate?

[MartinPulec]

No problem, my setup was quite easy. For the cases 1 and 2 above, I've used couple of DeckLink 4K Extreme devices, one output used just for signal generator:
uv -t testcard2:m=2d24:noise:c=R10k -d decklink:d=a
(explicit ports not needed - if both vcap/disp and implicit loopback is used, random, non-default ones are selected)

and then:

# capture
uv -t decklink:d=a -m 9000 localhost             # case 1 - Jumbo frames needed because I didn't get full FPS over lo; OR:
uv -t decklink:d=a -c lavc:enc=libx265 localhost # case 2
# display
uv -d decklink:d=b

Physically the SDI outA of decklink:d=a was connected to Smartview Duo, the loop-through connected to inA of the same decklink. The outA of decklink:d=b was connected the right display of SmartView.

For the case 3 (above) I've used a different machine with 8K Pro (because 4K Extreme seem to support R12L just on output). Device indices was then different (with half-duplex, I've used connectors a-c of the single card).