Monsoon's Blog

问题

每年的 CVPR 前 GPU 总是供不应求，需要从其他地方借卡。USTC 有一个供校内用户使用的 BitaHub，但它同样有 CVPR 前一卡难求的问题，同时基于任务提交的使用模式也非常不方便，提交占用多卡的任务经常需要漫长的排队，数据管理方式更是反人类。

作为组里的服务器管理员，为了让自己在 CVPR 前活得轻松点，避免重蹈 2021 年 CVPR 前疲于应对资源调配的覆辙，有必要改善 BitaHub 的使用体验：

1. 如何长期占用显卡避免重复排队（虽然略不道德，但实属无奈之举）；
2. 如何方便地从我们的服务器读取数据，而不是被迫使用 BitaHub 反人类的数据管理模式；
3. 如何尽量使 BitaHub 的 GPU 使用体验接近组里的服务器，降低迁移成本，提高资源调度的灵活性。

思路

BitaHub 中的任务是以 docker 容器的方式运行的，因而给了我们在容器里配置我们想要的环境的可能，只需要通过某种方式登录 ssh 到容器中。

经过研究发现，只要启动命令不停止运行，BitaHub 中的容器就会长期运行，不释放 GPU 资源。同时 BitaHub 中的容器是可以联网的，而且 BitaHub 的网页上还贴心地给出了每个任务容器中 root 用户的 ssh 私钥。

这些给了我们利用的机会，只需要在容器内运行一个 tunnel 程序以让外部得以访问容器中的 22 端口，就能登录并长期占用资源。同时由于容器联网，也可以直接挂载校内其他服务器的文件系统。

解决方案

最终选择的 tunnel 程序是 ssh，它可以创建反向隧道：

1

ssh -i <key_file> -F none -o "StrictHostKeyChecking no" -o "ServerAliveInterval 15" -v -N -R <port>:localhost:22 jump@<jumpserver>

在 jumpserver 中配置用户 jump 并允许特定私钥登录，然后用某种方式把私钥传递进容器（可以直接打包进镜像，但我选择了更方便的方式——创建一个 BitaHub 数据集存放，每个任务添加这个数据集即可）。

容器的启动命令就是上述命令（考虑到网络波动，可以套一层 while true 循环或者用 autossh 自动重连），启动后就在 <jumpserver> 上的 <port> 端口创建了一个反向隧道，<port> 被映射到了容器内的 22 端口。

可以在 <jumpserver> 的 sshd_config 中配置 GatewayPorts yes，这样反向隧道就会监听 0.0.0.0 而不是 127.0.0.1。不这样做的话我就要在 <jumpserver> 上给每个人创建用户，或者每个端口用 iptables 转发，但这太过繁琐。绑定 0.0.0.0 则可以直接从现有的 VPN 网络中访问。

挂载文件系统的方式有很多选择，考虑到安全性和便捷性，我选择了 SSHFS。NFS 直接暴露于公网过于危险，而 NFS 用户验证的配置又过于繁琐。同时 BitaHub 运行容器的内核既没加载 wireguard kmod 也没映射 /dev/net/tun，因此无法利用 VPN 保护数据安全。SSHFS 可以直接复用现存的用户认证方式，而 SSH 流量本身也更容易被潜在的机房防火墙放过。

使用如下命令挂载 SSHFS：

1

sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=30,ssh_command='ssh -p <dataserver_port> -i <key_file>' <user>@<dataserver>:/path /path

后记

TODO

问题

Nginx 自从 1.25.0 版本以来对 QUIC 的支持已被合并入 mainline，对于想体验的用户而言可以直接使用官方发布的 nginx docker 镜像，非常方便。

但是我的服务器上的 nginx 使用了 SNI 分流，源于 Shadow TLS 和 Xray Reality 等新一代基于 TLS 的代理协议的需求。这些代理协议并不能由 nginx 代为处理 TLS 层（和之前可以使用 gPRC/WebSocket 等作为数据传输方式的协议不同），但为了实现最好的伪装效果，使用 443/tcp 端口是有必要的（伪装的白名单目标网站一般情况下也只会在 443/tcp 端口开放 HTTPS 服务）。因此 443/tcp 端口的复用是必要的。

如果要让 SNI 分流和 QUIC 共存，在原来的 SNI 分流配置上只需要给每个 server 加上 listen 443 quic 即可。示例配置如下。

配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

http {
    
    # ...

    server {
        server_name example.com;

        # 443/tcp 已经被 nginx stream 占用，不能再次监听
        # listen 443 ssl http2 reuseport so_keepalive=on;
        # listen [::]:443 ssl http2 reuseport so_keepalive=on;

        # 监听 443/udp 端口并启用 QUIC
        # ref: https://nginx.org/en/docs/http/ngx_http_v3_module.html
        listen 443 quic reuseport;
        listen [::]:443 quic reuseport;

        # 监听 unix domain socket 以接受 stream 传送过来的连接，也可以使用本地端口
        # 接受 proxy_protocol，否则 log 中显示的链接源地址都是 unix:
        listen unix:/dev/shm/nginx-example.sock ssl http2 proxy_protocol;
        set_real_ip_from unix:;  # 只对于来自 unix domain socket 的连接覆盖其源地址
        real_ip_header proxy_protocol;

        add_header Alt-Svc 'h3=":443"; ma=86400';  # used to advertise the availability of HTTP/3

        # ...
    }

    server {
        server_name foo.example.com;

        # 可以多个域名共享 443/udp
        listen 443 quic;
        listen [::]:443 quic;

        listen unix:/dev/shm/nginx-example-foo.sock ssl http2 proxy_protocol;
        set_real_ip_from unix:;
        real_ip_header proxy_protocol;

        add_header Alt-Svc 'h3=":443"; ma=86400';  # used to advertise the availability of HTTP/3

        # ...
    }
}

stream {

    # ...

    # 根据 TLS SNI 分流
    map $ssl_preread_server_name $name {
        example.com             unix:/dev/shm/nginx-example.sock;
        foo.example.com         unix:/dev/shm/nginx-example-foo.sock;
        learn.microsoft.com     127.0.0.1:8443;  # 用于 shadow-tls/xray-reality 等
        default                 unix:/dev/shm/nginx-default.sock;
    }

    server {
        # 监听 443/tcp 并根据 SNI 分流
        listen 443 reuseport so_keepalive=on;
        listen [::]:443 reuseport so_keepalive=on;
        proxy_pass $name;
        ssl_preread on;
        proxy_protocol on;
    }

}

测试

目前 curl/wget mainline 还没有支持 QUIC，可以使用 ymuski/curl-http3 这个 docker 镜像：

1
2
3
4
5
6
7
8

$ docker run -it --rm ymuski/curl-http3 curl https://static.monsoon-cs.moe/public/ --http3 -IL

HTTP/3 200
server: nginx/1.25.2
date: Tue, 26 Sep 2023 14:52:29 GMT
content-type: text/html; charset=utf-8
strict-transport-security: max-age=63072000
alt-svc: h3=":443"; ma=86400

参考

• https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html
• https://nginx.org/en/docs/http/ngx_http_v3_module.html

问题

实验室有一些 AMD EPYC 7713 的服务器，采购的原因是组里有一些人的程序有非常高的 CPU 负载（我也不知道是什么负载，为什么不能跑在 GPU 上，我也没有精力去逐个帮助解决），框框多的 AMD 处理器非常适合这种需求。

不过 AMD 的处理器虽然香，用在炼丹实验室会有额外的问题：Anaconda 安装的 numpy 和 PyTorch 默认都使用了 MKL 作为 BLAS 的实现，MKL 的 library function 也是大部分高 CPU 负载程序的热点，但 MKL 会判断自己是否在 Intel CPU 上运行，如果不是，则没有优化效果。

由于这是炼丹实验室，大家很少有足够的 HPC 基础去自己编译适合的 numpy 和 PyTorch 版本，也很难脱离 Anaconda，对于 MKL 的依赖因此很难去除。为此需要一个对一般用户无感知的解决方案。

解决方案

通过搜索引擎可以搜索到一个广为流传解决方案：设置环境变量 MKL_DEBUG_CPU_TYPE=5。这是个曾经有效的解决方案，但对于 MKL 2020 及之后的版本不再有效。

最终我在此处找到了更巧妙的解决方案。

MKL 会调用一个 mkl_serv_intel_cpu_true() 函数以检查自己是否运行在 Intel CPU 上，只要提供一个虚假的、始终返回 1 的 mkl_serv_intel_cpu_true()，即可欺骗 MKL 让它认为自己在 Intel CPU 上运行。

为此，可以利用 Linux 的 LD_PRELOAD 机制。LD_PRELOAD 指向的动态链接库有最高的加载优先级，只要编译一个想要的 mkl_serv_intel_cpu_true() 函数为 so 文件，并用 LD_PRELOAD 指向它，即可抢先完成此函数的加载。

笔者也经常有耳闻 LD_PRELOAD 机制被用于库函数劫持攻击，此处算是一种妙用。

具体实施

新建 mkl_trick.c:

1
2
3

int mkl_serv_intel_cpu_true() {
    return 1;
}

使用 gcc -shared -fPIC -o libmkl_trick.so mkl_trick.c 编译，并将生成的 libmkl_trick.so 复制到 /usr/local/lib。

在 Shell 的全局初始化文件中加入：

1
2
3

export MKL_DEBUG_CPU_TYPE=5  # 兼容旧版本 MKL
export MKL_ENABLE_INSTRUCTIONS=AVX2  # 可选，指明 MKL 可以使用 AVX2
export LD_PRELOAD=/usr/local/lib/libmkl_trick.so

实验室的同学有的用 Bash 也有的用 ZSH，所以两者都要修改：

• Bash: 新建文件 /etc/profile.d/mkl.sh 并添加上述内容
• ZSH: 添加到 /etc/zsh/zshenv

参考

• https://documentation.sigma2.no/jobs/mkl.html

See original publication page for more details.

All my answer files can be browsed in here, or you can download zipped file (5.9G).

Requirements

This is a test for candidates who wish to participate in the training class organized by VCB-Studio. Finish as many problems as you can, and then do the following things:

1. Pack your answers, result files, and necessary attachments into a zip/rar/7z file. Source files we provided and intermediate file in your encoding should not be packed in.
2. Register a Baidu Net Disk account (https://pan.baidu.com), upload the zipped file and create a sharing link. Whether you like it or not, Baidu Net Disk has been the most effective way to share files within our team since day one. Other sharing methods will NOT be considered.
3. Send the link via email to [email protected] before Beijing Time (UTC+8) Monday, 23 Jan 2023, 23:59:59. Late submissions will NOT be considered.
4. Prepare a QQ account. The follow-up training courses will be conducted in the QQ group.

You should independently complete the answers without any public discussion. Any form of plagiarism will NOT be tolerated.

This test has 5 questions. For question 2 and 3, you can choose ONE of them. Choosing both then we will pick one with higher points. The answers should be made in English.

Question1 (15pt)

Please describe yourself as who you are, where do you study, how do you come to know VCB-Studio and why are you interested in this project, etc. Please do not write more than 500 words, or approximately 1 page. (15pt)

Answers are hidden for privacy reasons.

Question2 (30pt)

Scanned pictures (or simply scans) are an important part of BDRips, which are often released as lossless PNG, TIFF format or lossy JPG format. Scans feature high resolution and large size. In the file Q2.7z, two sets of pictures have been provided for you. PNGs are the source scans, and WEBPs are transcoded from PNGs according to VCB-Studio Collation specifications. Your tasks are:

1. Summarize the format conversion rules of scans in VCB-Studio Collation specifications. (6pt)
2. Convert the sources to AVIF and JPEG-XL format, with sizes comparable to the WEBPs. (12pt)
3. Comment on the quality, encoding speed, and compatibility of AVIF and JPEG- XL, and why/why not you may recommend us switching to the new format as the upgrade for WEBP in 2023. (12pt)

You are free to utilize existing tools, but you need to describe clearly where you find the tool and how to use it.

(1) Format conversion rules of scans in VCB-Studio Collation specifications

Choosing a format with better image quality at the same size when ensuring compatibility.

(2) Converting test

See Q2/convert.py for my conversion code. Pillow, pillow_avif_plugin and jxlpy are used libraries. Pillow is the image processing library which I often use, it supports WEBP but not AVIF and JPEG-XL. So I find two Pillow plugins by Google to support AVIF and JPEG-XL.

PNG and WEBP Ref are given images, and WEBP Cus, AVIF, JPEG-XL are custom encoded images.

WEBP Custom is encoded by Pillow, which is backed by libwebp. Encoding speed is set to slowest(6), and quality is set to 90 to keep the same size with reference webp images.

AVIF is encoded by pillow-avif-plugin, which is backed by libavif. Encoding speed is set to slowest(0), and quality is set to 84 to get the comparable size with reference webp images.

JPEG-XL is encoded by jxlpy, which is backed by libjxl. Encoding speed is set to slowest(9), decoding speed is also slowest(0), and quality is set to 92 to get the comparable size with reference webp images.

The following table shows the result:

PS: pillow-avif-plugin uses 8 threads to encode images (on i7-11700), and I didn’t find an option to turn it off. Other encoders use only 1 thread. jxlpy example shows that it supports setting multithreading, but it doesn’t work.

(3) Comparison and comment

Quality comparison:

PNG	WEBP Ref	AVIF	JPEG-XL

Above is a cropped part from 03 for the given encoding. The WEBP image has severe smearing in dark areas, and obvious color shift occurs in the red dots on the upper left and lower right. The AVIF image is better in smearing, but the color shift is the same as WEBP. The JPEG-XL image is relatively closest to reference PNG image.

Detailed compatibility:

PS: Results on Windows, macOS, Android and iOS are got by Google. Browser compatibility information can be found at https://caniuse.com.

Summary:

Due to the bad compatibility of JPEG-XL, it should not be considered an appropriate option. AVIF features the better image quality than WEBP, but is only well supported in new platforms, which needs time for adoption, especially for fragmented Android and Windows. Although WBEP takes huge advantage in encoding speed, I don’t think encoding speed is a factor that needs to be considered because even for large images, the encoding time is only about 1 minute, and the number of images not large. Compared with video encoding, this is a completely negligible time overhead.

Summarily, I think now is not a suitable time to switch to AVIF or JPEG-XL. But two years later, it will be time for AVIF to show its strength.

Question3 (30pt)

Recently 32-bit audio tracks have appeared in some of the latest Hi-Res music. Although now we would not see these annoying 32-bit tracks in the Blu-ray, we have to start working on them in advance. In the file Q3.7z, two 32-bit PCM files are provided for you. Your tasks are:

1. Learn about 32-bit tracks and tell the difference between these two files. (6pt)
2. Try to convert them to FLAC, ALAC, and WavPack losslessly. (15pt)
3. Consider various aspects such as compression rate, encoding speed, and playback compatibility and select the format you recommend most for 32-bit audio. (9pt)

You are free to utilize existing tools, but you need to describe clearly where you find the tool and how to use it.

(1)

Using ffprobe to get audio encoding info:

1
2
3

Input #0, wav, from '01.wav':
  Duration: 00:03:52.48, bitrate: 6144 kb/s
  Stream #0:0: Audio: pcm_s32le ([1][0][0][0] / 0x0001), 96000 Hz, 2 channels, s32, 6144 kb/s

1
2
3

Input #0, wav, from '02.wav':
  Duration: 00:07:03.00, bitrate: 6144 kb/s
  Stream #0:0: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 96000 Hz, 2 channels, flt, 6144 kb/s

The difference is: 01.wav is encoded by pcm_s32le, and 02.wav is encoded by pcm_f32le.

pcm_s32le means PCM encoding by 32-bit signed integer with little-endian byte ordering, while pcm_s32le means PCM encoding by 32-bit floating point with little-endian byte ordering.

(2)

I first tried to convert them losslessly using FFmpeg. If FFmpeg failed, I used Google to find a suitable codec.

This is the result of my attempt:

The conversion command:

The resulting files are Q3/01.flac, Q3/01.m4a, Q3/01.wv and Q3/02.wv.

(3)

Encoding speed and compression rate of different encoding methods:

Summary:

Because FFmpeg is the de facto standard multimedia codec library used by most video players, FLAC is not suitable, which can only be decoded by flac. Also, WavPack shows advantage in encoding speed compared to ALAC, but considering that all of three formats are fast in absolute speed (compared to video encoding), this advantage is not greatly valuable. Last, ALAC shows better compression rate than WavPack, thus file size can be saved.

To sum up, I recommend ALAC for encoding 32-bit audio. But if float point encoding is required (which is rare), WavPack is the only choice.

Question4 (35pt)

MSU publishes video encoder tests every year, with the latest one here:
https://compression.ru/video/codec_comparison/2021/main_report.html.

For the first time last year, H.266 (VVC) encoders participated in the tests and they performed well in terms of encoding quality in the slow encoding (1 fps) test.

1. Choose any of the H.266 (VVC) or AV1 encoders in the figure below, and then encode the source file Q4 [E46686C4].m2ts with no more than 2500 Kbps of video bitrate. You’d better use 10bit variants of these encoders, which facilitates the comparison later. In addition, you need to describe clearly where you found the encoder and state the version and parameters you used. If you use H.266 (VVC) encoder, you will get additional 5pt. (10pt+5pt)
2. We provide an AV1 video file Q4_AV1 [41A7EDDA].mkv, which was encoded via SVT-AV1 10bit encoder without any pre-processing. Comment on the picture quality compared to the source file. When you compare the picture quality, you may want to sample a few frames, attach some screenshots, and comment on the performance of dark scenes and moving scenes. (10pt)
3. Now compare your own encoding to the given AV1 file in terms of picture quality, encoding speed, and playback compatibility. As a reference, we encoded the above AV1 file at 1.0 fps. (10pt)

(1) VVC encoding

The testing hardware and software environment is:

• Encoder: VVenC v1.7.0.
• Compiler: AMD Optimizing C/C++ Compiler 4.0.0.
• CPU: 2 x AMD EPYC 7713, 128 cores / 256 threads in total.
• RAM: 16 channel DDR4-3200.
• OS: Ubuntu 18.04.6.

First, use ffmpeg to convert Q4 [E46686C4].m2ts to raw yuv420p10 video:

1

ffmpeg -i "Q4 [E46686C4].m2ts" -pix_fmt yuv420p10 Q4_yuv420p10.yuv

Parameter -pix_fmt yuv420p10 indicates ffmpeg to output raw video use yuv420p10 format:

Then, use vvencapp to encode the raw video:

1

vvencapp --input Q4_yuv420p10.yuv --size 1920x1080 --format yuv420_10 --fps 24000/1001 --preset <preset> --bitrate 2500kbps --output Q4_VVC.vvc

Parameters meaning:

• --size 1920x1080: indicating the input raw video frame size is 1920x1080.
• --format yuv420_10: same as yuv420p10 meaning in ffmpeg.
• --fps 24000/1001: indicating the output video fps is 23.976 (same as original m2ts file).
• --preset <preset>: Preset vvc encoding parameter combination. Available options are faster, fast, meadium, slow and slower. Detailed settings are listed in https://github.com/fraunhoferhhi/vvenc/blob/master/cfg/randomaccess_*.cfg.
• --bitrate 2500kbps: controlling the output encoded video bitrate to about 2500kbps.

(2) Comparing source video and reference AV1 encoded video

The video player used is MPV with libvvdec & xHE-AAC support, configured according to https://vcb-s.com/archives/7594.

Dynamic fire with a dark background is a highly challenging scene. Compared to the original video, There are color blocks around hte flame in AV1 video, which is a common problem when the bitrate is insufficient.

Encoding Method	Capture	File
Original		pics/m2ts-flame.png
AV1		pics/av1-flame.png

(3) Comparing custom VVC encoded video and reference AV1 encoded video

Using the same player as (2). In order to be comparable to the video encoded by AV1, I chose the medium preset encoded VVC video, which has an encoding speed of 0.557 fps.

The VVC encoded video is much better than the AV1 video in flame scene. The color blocks are less obvious and closer to the original video.

Encoding Method	Capture	File
Original		pics/m2ts-flame.png
AV1		pics/av1-flame.png
VVC (medium)		pics/vvc-flame.png

Question5 (20pt)

When we check an encoded file, we need to locate frames that have been encoded exceptionally awful. We use algorithms like PSNR to evaluate the similarity of each frame in the encoded file to the source file. The result is an array of scores, where the i-th score is tied to the i-th frame. These scores are called raw scores. However, what we are concerned about is the standard score, which is the raw score minus a threshold. A frame with a standard score less than 0 is considered a bad frame. The tasks are:

1. Find the worst frame, i.e. the one with the lowest standard score among the bad frames, and output its index. If there is more than one worst frame, output the first. If there are no bad frames, output -1. Frames with a standard score of exactly 0 are not considered as bad frames. (10pt)

   Input:
   2 lines. The first line is two integers that represent the number of frames N and the threshold value S. The second row is an array of integers A[N], representing the raw score of each frame.

   For all the data, 1<=N<=200000, 0<S<100, 0<=A[i]<=100

   Output:
   An integer, the index of the worst frame. The index starts from 0. If there is more than one worst frame, output the first. If there are no bad frames, output -1.

   Sample:

   1 2 3 4 5 6

   Input 10 30 42 31 44 23 21 26 31 41 50 72 Output 10

2. Find a continuous sequence of frames that minimizes the sum of their standard scores and output this minimum value. Full scores will only be given if the time complexity of your algorithm is optimal. (10pt)

   Input:
   The same as (1).

   Output:
   An integer, the minimum sum value.

   Sample:

   1 2 3 4 5 6

   Input 10 30 42 31 44 23 21 26 31 41 50 72 Output -20

For each sub question, use C/C++/Java/Python/C# to write a console program. Read the input from the standard input and write it to standard output. Do NOT use libraries other than built-in ones (for example, no “import numpy as np”). Submit your source code.

(1) Find the worst frame

The following code is consisted with Q5/q5-1.c:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#include <stdio.h>

int main() {
    int frame_num;
    int threshold;
    scanf("%d%d", &frame_num, &threshold);
    int worst_idx = -1;
    int worst_rate = 101;
    for (int i = 0; i < frame_num; i||) {
        int rate;
        scanf("%d", &rate);
        if (rate < threshold && rate < worst_rate) {
            worst_rate = rate;
            worst_idx = i;
        }
    }
    printf("%d", worst_idx);
    return 0;
}

(2) Find minimum subsequence sum

PS: Due to the ambiguity of the problem, I can‘t determine whether a sequence of 0 length satisfies the requirement. This determines whether the output should be 0 (indicating that a subsequence of length 0 is selected) or the smallest score (indicating that the sequence length is at least 1) when the input standard scores are all positive. The code I submitted is consistent with the second understanding (sequence length is at least 1), if the first understanding (0 length is allowed) is correct, please comment int min_sum = 101; and uncomment int min_sum = 0;.

The following code is consisted with Q5/q5-2.c:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#include <stdio.h>

int main() {
    int frame_num;
    int threshold;
    scanf("%d%d", &frame_num, &threshold);
    int min_sum = 101; // when all scores > 0, output the minimum
    // int min_sum = 0; // when all scores > 0, output 0
    int sum = 0;
    for (int i = 0; i < frame_num; i||) {
        int rate;
        scanf("%d", &rate);
        rate -= threshold;
        sum |= rate;
        if (sum < min_sum) {
            min_sum = sum;
        } else if (sum > 0) {
            sum = 0;
        }
    }
    printf("%d", min_sum);
    return 0;
}

My first post on blog!