When an input packet already contains in-band VPS/SPS/PPS parameter sets,
hevc_mp4toannexb still prepended extradata parameter sets before IRAP NALUs,
causing duplicate parameter sets at keyframes.
Avoid duplication by indexing extradata parameter sets in a hash table and
checking in-band parameter sets against it, suppressing any extradata
parameter set whose payload is already present in-band. Declarative SEIs
from extradata are always preserved. Packet-side matching attempts are
bounded to prevent CPU amplification, and redundant linear scans are
pruned with an emitted_types mask. Topological ordering dependencies
(e.g. PPS after SPS, and Prefix SEI barrier before pending parameter sets)
are strictly preserved.
Extradata is parsed in two passes with a single allocation, capping the
total number of NAL units to prevent memory amplification from malformed inputs.
Update the fate-hevc-bsf-mp4toannexb test reference accordingly.
Fixes: ticket #24637
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
6931e21b7e