parse_dat(), parse_cube(), and parse_cinespace() multiply an untrusted LUT size
before allocate_3dlut() validates it, which can overflow int.
Use the validated lutsize2 computed by allocate_3dlut() instead.
Fixes: signed integer overflow
(cherry picked from commit 92cd5c9781)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array access
Fixes: 9aj_hqdn3d_dynamic_res.mjpg / 9aj_generate_hqdn3d_dynamic_res_mjpg.py
Fixes: wWDsy2oDvMuR
Found-by: Adrian Junge (vurlo) <adjun37@gmail.com>
(cherry picked from commit f0f634b658)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array access
Fixes: 7aj_swaprect_odd17_nv12.nut / 7aj_generate_swaprect_odd17_nv12.py
Fixes: VRAXYvKtmKa8
Found-by: Adrian Junge (vurlo) <adjun37@gmail.com>
(cherry picked from commit a7e38b617b)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array access
Fixes: 8aj_floodfill_dynamic_size.pgm / 8aj_generate_floodfill_dynamic_size_pgm.py
Fixes: 3MleMXjGZvu3
Found-by: Adrian Junge (vurlo) <adjun37@gmail.com>
(cherry picked from commit 24c322fdb2)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Two runtime paths could compute out-of-range source coordinates for
degenerate projection geometry, causing heap-buffer-overflow reads
Fixes: out of array read
Fixes: assertion failure
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a73d648f6e)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array read
Fixes: assertion failure
Fixes: mQzloVqnivHQ
Found-by: Anthony Hurtado
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b3712addc9)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array access
Fixes: JbvzNObhorBp
Fixes: 030e140145 (lavfi: add quirc filter)
Found-by: Adrian Junge (vurlo)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 4da9812e25)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The hand written boundary mirroring reflected an out of range index only
once, which is insufficient when the image dimension is smaller than the
filter half width (filt_w/2 == 8). A 1x1 input made the index reach 8
and -7, reading out of the src[]/temp[] arrays. Use avpriv_mirror(),
which mirrors repeatedly and stays in range for any dimension.
Fixes: out of array access
Fixes: repro.sh
Fixes: HuQn51lLiVJX
Fixes: 38aea9b041 (avfilter: add vif filter)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 56309e476a)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
A format-only conversion leaves the auto use_filters value unresolved.
Regression since 5d0748243f.
Fix issue #23737
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
(cherry picked from commit c57660fb18)
Assisted-by: Fairy
Fixes: error: no type named 'system_error' in namespace 'std'
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
(cherry picked from commit c6498178bb)
Since the input and output format can differ (e.g. 444 -> 420), we need to
reference the correct subsampling for the partially applied filter.
Keep track of this in the CUDATex itself.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit 01972b4f85)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
Instead of going via an AVFrame at all. This will allow us to fix the
intermediate chroma plane size for split downscaling.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit 420a9e90b8)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
Instead of re-creating this object every frame.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit e79e9f06ba)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
At this point, s->hwctx and CudaFunctions * are available.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit 4289a29bb0)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
I want to disentangle the internal logic from AVFrame, because some
intermediate states (e.g. for partially subsampled chroma with simultaneous
scaling) may not directly map to a valid AVPixelFormat.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit fef976b197)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
Make the next commit a bit easier to review.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit 61750318db)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
This is already done by cudascale_filter_frame().
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit 0c3f04a97c)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
NVDEC and CUVID now output AV_PIX_FMT_P012 (12-bit 4:2:0), AV_PIX_FMT_P212
(12-bit 4:2:2) and AV_PIX_FMT_YUV444P10MSB / AV_PIX_FMT_YUV444P12MSB
(10/12-bit 4:4:4) for high-bit-depth content, but these CUDA filters
rejected the formats in their supported-format lists, breaking pipelines
such as "-hwaccel cuda ... -vf scale_cuda" on 12-bit input.
These formats use 16-bit sample storage, and the filters select their CUDA
kernel by byte-storage size and plane layout, not by the number of valid
bits, so they can reuse the existing 16-bit kernels:
- scale_cuda: P012/P212 -> "semiplanar16", YUV444P10MSB/YUV444P12MSB ->
"planar16".
- transpose_cuda: the ushort/ushort2 kernels are chosen from the pixel
descriptor (byte size + channel count); just allow the new formats.
- thumbnail_cuda: P012 reuses the P010/P016 path and the MSB 4:4:4 formats
reuse the YUV444P16 path; P012 is added to the 4:2:0 chroma-histogram
scaling as well. (thumbnail has no 4:2:2 path, so P212 is not added.)
The CUDA deinterlacers (bwdif_cuda, yadif_cuda) already accept any format
with <= 2 bytes per sample and <= 2 channels, so they need no change. The
8-bit-only filters (overlay_cuda, pad_cuda, bilateral_cuda, chromakey_cuda,
colorspace_cuda) do not support high bit depths and are left untouched.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
(cherry-picked from commit df94900c98)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
This may be faster or slower than the existing specialized kernels,
so I opted not to prefer it by default. I also deliberately didn't expose
additional filter function capabilites yet.
The main motivating reason here is to get correct anti-aliasing behavior
when downscaling, which is currently completely broken.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit 5d0748243f)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
This can be useful for any sort of separable filtering with arbitrary
weights.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit 469281fa50)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
Ignored for now by the existing fixed function kernels.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit d98cc462f6)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
Useful for GPU-based filters, which may also need to compute filter weights.
Since we cannot cross-link to internal functions, we need to recompile this
helper inside libavfilter.c.
Signed-off-by: Niklas Haas <git@haasn.dev>
(cherry-picked from commit b400791442)
Signed-off-by: Marvin Scholz <epirat07@gmail.com>
If s->stop is set, the return value would be overwritten
before being checked. This bug was introduced in the switch
to AV_TX in 014ace8f98.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
(cherry picked from commit 0992c19c30)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Reproduced with a small output (e.g. size=2x2) under ASan.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit d133b4a231)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This is a behavior preserving change for all non-overflowing cases.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit f7368f97b9)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Slice based filter workers compute their per-thread row/sample/channel
boundaries as total * jobnr / nb_jobs. The total * jobnr product is
evaluated in int and overflows signed int for large dimensions and many
slice threads, before the division by nb_jobs brings it back in range.
(cherry picked from commit 218b4771a3)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
deinterlace_slice() computed per-thread row boundaries with int
multiplication height * (jobnr + 1). With a tall frame and many filter
threads the product overflows signed int before the division by nb_jobs.
Use int64_t for the intermediate product before converting back to int
row indices.
Found-by: Kery (Qi Kery <qikeyu2001@outlook.com>)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 200e0cba67)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: integer overflow
Fixes: out of array access
Found-by: Claude (Anthropic), reported by Omkhar Arasaratnam <omkhar@linkedin.com>.
(cherry picked from commit 537e87e3c2)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
For glyphs whose source is already in bitmap form (color emoji fonts such
as NotoColorEmoji.ttf), FT_Glyph_To_Bitmap(..., destroy=0) returns the
input pointer unchanged. The result is that glyph->bglyph[idx] aliases
glyph->glyph (and analogously border_bglyph[t] may alias border_glyph).
glyph_enu_free then called FT_Done_Glyph on both, double-freeing the
underlying object.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 9efca1d946)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
