Hi)

May be it's just bit-rate setting issue. Can you attach sample 11MB file?
I will check the compression ratio of ffmpeg result and the original.

Here's a link to the file:
https://drive.google.com/file/d/14LsSdtVxrxzvc09oNo74OcZutMCcJI7D/view?usp=sharing

I think you are right, I'm just testing a few more options (changing qp_min and qp_max, and desired_frame_bytes, vbv_size_bytes) and I'm able to get it down to ~2 MB with your libraries without a huge quality loss. I'm not entirely sure how to go about setting these flags though, do you have any suggestions that might be sorta similar to "fast" / "slow" profiles?

Yep, looks like it.

ffmpeg -i minih264-output.mp4 minih264-ffmpeg.mp4
gives 770kb on my side or 6849 kbps.

With h264enc_x64 -kbps6849 minih264_1920x1080.yuv out.264 I'm able to produce comparable size.
But quality is much worse. Issue here is vbv buffer size, rate control gives much less bits for intra frame than ffmpeg.

So I adjust vbv buffer size to 2 seconds based on kbps:

    create_param.vbv_size_bytes = cmdline->kbps*1000/8*2; // 2 seconds vbv buffer for quality, so rate control can allocate more bits for intra frame

And now quality is comparable.

Other things to adjust for performance/quality is run_param.encode_speed and create_param.temporal_denoise_flag.
qp_min-qp_max range 10-51 is max supported range, so rate control can set any possible quantizer (lower is better).
You can also adjust this range to change minimum and maximum quality, but then not all values available to rate control. And vbv overflow or underflow can happen in more scenarios.