This filter works by running the SAD function on the first plane, which
works for any (native endian) format that starts with an unshifted luma
plane. This notably includes formats like nv12, yuva420p, and grayscale
formats, as well as luma-alpha formats. The last addition is debatable,
but the precedent here is the existing support for RGBA, which also
dilutes scene scores by including the alpha component in the SAD
calculation). There were also some notable missing packed RGBA formats,
such as ARGB, as well as 16-bit variants.
This is an exhaustive listing as of the current set of formats.
Tested by looping over `format=<pixfmt>,scdet` and verifying that the
scene SAD score doesn't (meaningfully) change, except for expected
deviations (e.g. larger scores for full range vs limited range, and
lowered scores for packed alpha formats).
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
Over HTTP, seeking to a trailing chunk can make avio_size() stale, so the PCM
duration guard fails and duration is N/A. Cache size before seeking.
Fixes#24122
Formats with the same dimensions may expose different frame rates. The
selection loop previously updated selected_format for every size match but
kept selected_range from an earlier format. This could apply a range to a
format that does not support it and make device configuration fail.
Only update selected_format together with its matching range. Track a
size-only match separately so the muxed-device fallback remains unchanged.
Reported-by: MW-de <mw-de@noreply.code.ffmpeg.org>
Suggested-by: MW-de <mw-de@noreply.code.ffmpeg.org>
Fixes: https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/22694
Signed-off-by: iSold Leo <s@qwqlog.com>
The warp calculation is currently done for every plane, but doesn't depend on
the plane contents, just the dimensions; so this is wasted work when plane
sizes don't vary (eg: RGB, 4:4:4 YUV).
This will not have a visible impact on performance at realistic interpolation
rates as the optical flow generation is the limiting factor until you are
doing something silly like 30->900fps.
But wasted work is wasted work, so best to stop doing it.
We have previously only supported some packed RGB and planar YUV, so the logic
for extracting the luma was not flexible enough to handle other scenarios.
To generalise this for planar RGB, we need to bind every colour plane to the
grayscale shader and derive luma from it.
Additionally, we add support packed 4:4:4 YUV formats.
In textureless regions the optical flow engine has no data to track and
its regularizer invents a smooth, often large, and internally
self-consistent flow field. Backward warping then traces a background
pixel along that invented flow onto a nearby moving object and pulls
object pixels into what should be empty background, leaving a ghost
outline at a distance from the object where nothing is actually moving.
Forward/backward flow consistency does not catch this because the
invented flow round-trips cleanly.
I have no idea if this is a common limitation in optical flow implementations
but it's definitely happening with the nvidia hardware, so this change
attempts to introduce a heuristic to work around it.
When the luma at a given location is sufficiently identical in both frames,
and the optical flow magnitude is sufficiently large, we will now assume that
the location is static and the flow is bogus. Genuine motion will usually lead
to luma differences, and low magnitude flow will be respected either way.
Again, this is all highly empirical, and the constants are justified purely by
by fiddling around with specific samples.
The grayscale inputs, flow images and optical flow session were a single shared
instance, so each source pair's optical flow execution had to wait for the
previous pair's interpolations to finish reading the flow images before it could
overwrite them, serialising the optical flow engine against the compute queue.
Double-buffer the per-pair resources across two slots (the optical flow session
bakes in its image bindings, so each slot owns its own session, grayscale and
flow images), indexed by the pair generation. One pair's optical flow execution
can then run while the previous slot's flow images are still being sampled by
the earlier pair's interpolations. The write-after-read fence on slot reuse is
tracked per slot (interp_done) instead of against the single shared instance.
As the optical flow engine is a single serial unit, more slots can't increase
throughput.
The grayscale, optical flow and interpolation stages are currently serialised
with host fence waits (ff_vk_exec_wait) after each of the the grayscale and
optical flow submissions.
We can remove these waits and manage everything with timeline semaphores. In
particular, we must correctly handle the case where more than one frame is
being interpolated per optical flow pair. In these situations, we have multiple
interpolations waiting on the optical flow generation, and then we have the
next optical flow generation waiting on all the previous interpolations.
This requires the ff_vk_exec_add_dep_signal_sem() we added in the previous
commit.
We also introduce a second context to the optical flow exec pool so that the
next command buffer can be recorded without a host-wait for the previous one
to retire - but of course, optical flow is serial so the queued one won't
actually execute until the previous one completes.
Add a helper to register a timeline-semaphore signal with an explicit value on
an execution context, mirroring ff_vk_exec_add_dep_wait_sem() on the wait side.
The fruc_vulkan filter will use this to chain timeline semaphores across its
pipelined cross-queue submissions, where an interpolation must signal a
specific (generation-keyed) value rather than the implicit binary signal.
Replace the placeholder temporal blend in the interpolation shader with a
motion-compensated algorithm. This is finally the point where we are actually
using the optical flow information.
For each output pixel the source position on the f0 and f1 grids is recovered
by fixed-point Picard iteration before the samples are blended. The number of
iterations and the relaxation factor are established through empirical testing
on a couple of ad-hoc samples, representing highly uniform and non-uniform
motion respectively.
The approach here is hardly profound, and not meant to be. There are for sure
many more advanced algorithms for doing motion interpolation from optical flow,
and we can happily replace it with, or add, something more advanced once all
the machinery is in place.
Add the two options that tune the NV optical flow session:
* perf: performance level (slow/medium/fast), previously hard-coded to slow.
Trades motion-estimation quality for speed.
* grid: output grid size in pixels (auto/1/2/4/8), previously always the finest
grid the device supported. A coarser grid computes fewer flow vectors, so it
is faster and uses less memory at the cost of detail.
These settings are important as you can't handle high resolution input in
realtime at the highest quality settings. For example, 2160p24 movie content
needs reduced settings (eg: pref=medium,grid=2) for the filter to run
comfortably.
Add the Vulkan optical-flow machinery on top of the filter skeleton:
* a grayscale-extraction compute shader
* the VK_NV_optical_flow session and its persistent input/flow images
* the cross-queue (compute -> optical flow -> compute) submission with
binary-semaphore ordering
* an interpolation compute shader dispatched per output frame which blends each
new frame instead of repeating the nearest source frame
Notable characteristics:
* At this stage we are doing a simple proportional blend that doesn't actually
look at the optical flow information. That will happen in the next commit
* The optical flow is stored in a weird nvidia specific fixed point image
format. Yay?
* Optical flow is calculated based purely on Luma. For YUV format, this is
trivially obtained, but for RGB input, we should put some effort into getting
accurate Luma. But we only need relative Luma differences, so we don't have
to use the exactly correct colourspace conversion; I've just hard-coded
BT.709
* We're doing host-synchronisation for initial simplicity, but this will get
replaced in subsequent changes
* The accepted format set is deliberately narrow to begin with: planar and
semi-planar YUV, gray, and packed RGB whose sampled channels are already
in component order. Subsequent changes will expand the support set.
* We rely on VK_KHR_maintenance9 to enable implicit ownership transfers for
images between the compute and optical flow queue families.
ff_vk_init_sampler() always used VK_SAMPLER_MIPMAP_MODE_LINEAR for samplers with
normalized coordinates. But it is possible to have an image view that uses
normalized coordinates while not supporting linear filtering (eg: R16G16_SINT).
The simple solution is to always use nearest mipmap filtering if the image
filter is nearest.
This change introduces the basic Vulkan filter machinery, while preserving the
original vf_framerate timing logic. At this stage, the filter simply picks the
temporally closest input frame for the output.
Note that while input frames are not modified in anyway, they must be
deep-copied for output. This ensures that they match the output context, which
may differ from the input context in important way, and avoids extending the
lifetime of hardware decoded frames that are part of the decoder's DPB.
This requires exposing vkCmdCopyImage in the Vulkan function loader, and a
compute queue family and execution pool to record the copy on.
The filter depends on vulkan_1_4 for its Vulkan-Headers requirement; while this
is not required by any of the code in this commit, the final optical-flow
pipeline will need VK_KHR_maintenance9, which first appeared with Vulkan 1.4.
To avoid confusion, the requirement is asserted from the beginning.
The output timebase is derived, as in vf_fps / vf_framerate, by reducing the
source timebase against the requested frame rate. That reduction can collapse to
a zero time base - av_reduce() bounds its result, so the numerator can underflow
to zero, leaving 0/1 - when the source timebase shares no useful factors with
the requested rate, leaving an unusable output timebase that breaks the
downstream PTS rescaling.
Detect the degenerate result and fall back to the plain 1/fps timebase derived
directly from the requested rate, so config_output() always yields a valid
timebase.
This change enhances the `fps` parameter to support the `source_fps`
expressions from `vf_fps`, which are highly useful if you want to request a
multiple of the base frame rate.
The NTSC/PAL constants from `vf_fps` are not brought over. These are very
unlikely to be useful targets for a filter that is increasing the framerate of
a video.
This is the minimal level of renames required to get the new filter to function
as a clone of vf_framerate. The context type is inlined from the header, but
there are no actual functional changes.
The fruc_vulkan filter will re-use the frame timing logic from vf_framerate -
it works so we don't need to do something different. To maximise reviewability,
this commit simply copies the file into place, so that all the changes in the
subsequent commits can be more easily understood.
Enable VK_NV_optical_flow if it is present, as an optional extension. The
extension was added in 1.3.230 which is older than the minimum required
header release, so we don't need #ifdef guards.
AV_PIX_FMT_GBRAPF16 is a floating point format, as the name suggests, so it should
be mapped to VK_FORMAT_R16_SFLOAT.
The comparable GBRPF16 and GBRAPF32 both already use SFLOAT.
The backwards compatibility wrappers for the deprecated
avpriv_slicethread_{create,execute}() dereference AVSliceThread, whose
definition only exists in the threaded branch of the file. A build
without pthreads/w32threads/os2threads therefore failed with
"dereferencing pointer to incomplete type 'AVSliceThread'".
Move the wrappers into the threaded branch and give the no-threads
branch stubs for the deprecated API matching those of create2/execute2.
Write ICC profiles as TIFF UNDEFINED data and align all out-of-line IFD values to a word boundary. Account for non-empty ICC side data while allocating the packet and calculate the image size in checked 64-bit arithmetic.
Signed-off-by: Wu Jianhua <toqsxw@outlook.com>
sizeof(slice_off) is the size of the uint32_t pointer, not of an entry.
Harmless in practice as it over-allocates on 64-bit, but wrong, and
under-allocates on targets where a pointer is smaller than uint32_t.
The tile count limit was only checked once on entry, but tileCount is
incremented by ff_vk_decode_add_slice() inside the loop, so a stream
carrying more than MAX_TILES tiles writes past the end of the
MAX_TILES-element tile_sizes array.
Check the limit on each iteration, and reject at MAX_TILES rather than
above it, since tileCount is used as the write index.
This fixes building for iOS after commit
c2802e520a.
On iOS, the fallback define kIOMasterPortDefault is never available.
The newer define kIOMainPortDefault is available since iOS 15.
However, if building with older iOS SDKs (iOS 15 SDK or older) then
IOKit is entirely missing. This caused compilation to succeed if
building with such older versions of the iOS SDK.
If building with a newer iOS SDK/Xcode but targeting an older version
of iOS, then IOKit is available, but compilation would fail as
kIOMasterPortDefault wasn't available.
Thus, this fixes compilation for iOS with iOS 16 SDKs and newer,
which was previously broken.
mov_flush_fragment() calls ffio_reset_dyn_buf() on mov->mdat_buf when
writing the initial moov. That buffer is allocated lazily on the first
packet, so it is still NULL when the trailer is written before anything
was muxed, and ffio_reset_dyn_buf() dereferences it.
627da1111c replaced ffio_free_dyn_buf(), which tolerates NULL, with
ffio_reset_dyn_buf(), which does not, turning this case into a crash.
This affects every fragmented mode that defers the initial moov to the
first flush.
Skip the block entirely when the buffer was never opened. Nothing
references the empty mdat it would otherwise write, as no samples were
muxed.
Reproduced with:
ffmpeg -f lavfi -i color=s=64x64 -frames:v 0 -movflags +frag_keyframe out.mp4
Fixes: 627da1111c ("libavformat/movenc: Uses dynamic buffers for fragmented chunks")
Signed-off-by: Ackanir <git.ackanir@gmail.com>
The MOV demuxer retains tmcd timecode flags in codec extradata, but copied
tracks never initialize MOVTrack.timecode_flags. mov_write_tmcd_tag()
therefore writes zero when reconstructing the sample description, causing
unchanged frame counters to be interpreted as non-drop-frame.
Initialize the complete flags field from sample-description extradata for
copied tmcd tracks. fate-copy-trac236 covers three drop-frame tmcd tracks.
Fixes issue #20841.
Reported-by: unsword01 <unsword01@noreply.code.ffmpeg.org>
Signed-off-by: Théo Valette <315040668+theovalette@users.noreply.github.com>
A status line carrying no HTTP version parsed as status code 0, which
check_http_code() then accepted as a success: the body of an error reply, or of
a reply from something that is not an HTTP server at all, was handed to the
demuxer. strtol() was equally lenient about the code itself, accepting a sign,
a count of digits other than three, and a value outside the range HTTP defines.
Parse the grammar RFC 9112 actually specifies: a case sensitive "HTTP/"
followed by DIGIT "." DIGIT, then exactly three digits on a whitespace
boundary, and reject a code outside the 100-599 range of RFC 9110 section 15.
A status line with no reason phrase at all, "HTTP/1.1 200", stays accepted; the
separator RFC 9112 requires there is missing often enough in the wild.
reason now points at the phrase itself rather than at the separator before it,
so check_http_code() no longer has to skip that separator on its own.
Worth noting: this invalidates several class of previously accepted
invalid http responses, include the ICY responses class from legacy
shoutcast servers.
ICY responses are planned on being re-introduced as opt-in via the new
libcurl HTTP stack.
Covers well formed status lines, a missing or non numeric status code, lines
that do not start with an HTTP version, and status lines whose version or code
does not respect the grammar of RFC 9112.
The reference output records the current behavior: every one of those malformed
lines is accepted, most of them as a 2xx success.
Implements planar intra prediction for 8 bit. x86 has no HEVC intra
prediction at all; aarch64 NEON already covers planar, DC and angular.
Every term except (size-1-x)*left[y] is linear in y, so for a group of
eight columns they collapse into one running accumulator whose increment
left[size]-top[x] is loop invariant. Running columns outermost keeps that
accumulator and its delta in registers, so a row costs one pmullw and two
paddw with no reload of top[]. 16x16 and 32x32 run two groups together,
sharing the left[y] broadcast and writing a row with one 16 byte store.
Words suffice throughout: each coefficient pair sums to size, so the
largest value is 2*32*255 + 32 = 16352. SSSE3 rather than SSE2 since
pshufb does the broadcasts and the byte to word widening.
checkasm --bench on a Core Ultra 7 155H:
hevc_pred_planar_4x4_8_c: 18.4
hevc_pred_planar_4x4_8_ssse3: 5.5 ( 3.35x)
hevc_pred_planar_8x8_8_c: 83.1
hevc_pred_planar_8x8_8_ssse3: 16.3 ( 5.10x)
hevc_pred_planar_16x16_8_c: 224.8
hevc_pred_planar_16x16_8_ssse3: 47.3 ( 4.75x)
hevc_pred_planar_32x32_8_c: 629.5
hevc_pred_planar_32x32_8_ssse3: 196.5 ( 3.20x)
Part of #23022.
Test av_frame_side_data_desc and av_frame_side_data_name on
valid and invalid types, av_frame_side_data_get and its _c
variant for present and absent types,
av_frame_side_data_remove (including MULTI types with
duplicates), av_frame_side_data_remove_by_props filtering by
AV_SIDE_DATA_PROP_*, av_frame_side_data_clone with and without
AV_FRAME_SIDE_DATA_FLAG_REPLACE (including the AVERROR(EEXIST)
and AVERROR(EINVAL) paths) and with AV_FRAME_SIDE_DATA_FLAG_UNIQUE,
av_frame_side_data_add in move mode, with
AV_FRAME_SIDE_DATA_FLAG_NEW_REF and with
AV_FRAME_SIDE_DATA_FLAG_REPLACE over an existing entry, and
av_frame_side_data_free.
Coverage for libavutil/side_data.c: 72.14% -> 86.43%
Numbers measured against a full FATE run. Remaining uncovered
lines are internal allocation-failure error paths requiring
injected failures mid-operation.
Signed-off-by: marcos ashton <marcosashiglesias@gmail.com>
Test av_dynamic_hdr_plus_alloc, av_dynamic_hdr_plus_create_side_data,
av_dynamic_hdr_plus_to_t35 in all three documented modes (size
query, caller-allocated, existing caller-owned buffer) including
the AVERROR_BUFFER_TOO_SMALL path, and av_dynamic_hdr_plus_from_t35
as a round-trip inverse.
Two HDR10+ fixtures are used. The first is a conforming
ST 2094-40:2020 ApplicationVersion 1 payload: one processing
window, the nine distribution maxrgb percentiles required by
section 8.5.4, and neither actual-peak-luminance item nor
ColorSaturationWeight, both excluded for that version by section
9.4. The second is marked in the source as deliberately
non-conforming and exists only to reach the optional
serializer/parser branches: a second processing window with its
geometry, both actual-peak-luminance matrices, per-window tone
mapping and color saturation. Every field either fixture
populates is decoded and printed, so a serializer or parser that
drops or corrupts a value changes the reference rather than
passing a flag-only check.
Also test that av_dynamic_hdr_plus_to_t35 ignores
application_version and always writes the CTA-861-H value, the
NULL and oversized error paths, and
av_dynamic_hdr_smpte2094_app5_alloc,
av_dynamic_hdr_smpte2094_app5_create_side_data and the App5
to_t35 / from_t35 round trip over four fixtures covering both
early-out paths (no adaptive tone map, reference-white tone
mapping) and both sides of the component mixing and gain curve
selectors: per-image parameters with component mixing type 3 and
explicit control point slopes, and common parameters shared from
alternate 0 with a mixing type other than 3 and the pchip slope.
The App5 to_t35 validation failures and the OOM paths via
av_max_alloc are covered as well.
Coverage for libavutil/hdr_dynamic_metadata.c: 31.02% -> 95.53%
The baseline comes from the HDR10+ sample tests on a full FATE
run (fate-hevc-hdr10-plus-metadata, fate-png-mdcv and the
Matroska/WebM HDR10+ remux tests).
Signed-off-by: marcos ashton <marcosashiglesias@gmail.com>
Test av_downmix_info_update_side_data() including first-call
allocation with zero-initialized defaults, field write/read-back,
second-call reuse (same pointer, preserved values), and the OOM
path via av_max_alloc.
Coverage for libavutil/downmix_info.c: 100.00% -> 100.00%
On a full FATE run the file is already covered by the AC-3
decoder tests (fate-ac3-4.0 and friends), which require
FATE_SAMPLES. This test exercises the API directly and keeps the
file covered on builds that run without samples.
Signed-off-by: marcos ashton <marcosashiglesias@gmail.com>
Test av_get_pix_fmt / av_get_pix_fmt_name round-trip across all
pixel formats, av_color_{range,primaries,transfer,space}_name,
av_chroma_location_name and av_alpha_mode_name round-trips
through their from_name() counterparts using a name-equivalence
check (two enum values may legitimately share the same name
string), av_chroma_location_enum_to_pos and pos_to_enum
round-trip across all valid chroma locations, and iterate every
pixel format printing av_get_padded_bits_per_pixel and
av_pix_fmt_count_planes.
Coverage for libavutil/pixdesc.c: 83.63% -> 90.03%
Numbers measured against a full FATE run. Remaining uncovered
lines are av_get_pix_fmt_string, the av_get_pix_fmt_loss /
av_find_best_pix_fmt_of_2 public wrappers and a few loss
branches, the palette branches of av_read_image_line and
av_write_image_line, and the *_ext name lookups in
av_color_{primaries,transfer}_from_name.
Signed-off-by: marcos ashton <marcosashiglesias@gmail.com>
I.e. not lines 9-16, then lines 1-8. No changes in benchmarks
here; it is mainly done to reduce differences with the 32bit code.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
