Both mestimate filters buffer one frame so they can estimate motion
against a future frame. At EOF, that frame remains in the next slot
and is discarded during uninitialization, making the output one frame
shorter than the input.
Add output request callbacks that drain the buffered frame when input
reaches EOF. Since no future reference exists while draining, emit only
backward motion vectors and size the side data accordingly.
Update the cropdetect FATE references for the newly emitted frame.
For all supported x86 ABIs, floating point values are returned
in xmm0, not in rax.
Reviewed-by: Niklas Haas <ffmpeg@haasn.dev>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Instead, we can just call the SIMD function a second time. This will reprocess
a few overlapping pixels, but that is safe since this function is read-only
and idempotent.
We only need to bail to the C function if the overall width is smaller than
a single SIMD vector.
checkasm:
- CPU: AMD Ryzen 9 9950X3D 16-Core Processor (00B40F40)
- Timing source: x86 (rdtsc)
- Bench duration: 100000 µs per function (462406519 cycles)
- Random seed: 555761032
Benchmark results:
name cycles (vs ref)
detect_alpha_8_full_c: 7884.5
detect_alpha_8_full_avx2: 439.9 (17.88x)
detect_alpha_8_full_avx512icl: 262.8 (30.00x)
detect_alpha_8_full_off_c: 7884.0
detect_alpha_8_full_off_avx2: 456.3 (17.27x)
detect_alpha_8_full_off_avx512icl: 267.3 (29.49x)
detect_alpha_8_limited_c: 13452.5
detect_alpha_8_limited_avx2: 1163.5 (11.56x)
detect_alpha_8_limited_avx512icl: 650.9 (20.67x)
detect_alpha_16_full_c: 3696.1
detect_alpha_16_full_avx2: 426.4 ( 8.67x)
detect_alpha_16_full_avx512icl: 262.6 (14.07x)
detect_alpha_16_full_off_c: 3698.7
detect_alpha_16_full_off_avx2: 454.7 ( 8.13x)
detect_alpha_16_full_off_avx512icl: 267.1 (13.85x)
detect_alpha_16_limited_c: 8215.9
detect_alpha_16_limited_avx2: 1288.4 ( 6.38x)
detect_alpha_16_limited_avx512icl: 691.3 (11.88x)
detect_range_8_c: 1411.0
detect_range_8_avx2: 281.6 ( 5.01x)
detect_range_8_avx512icl: 168.7 ( 8.36x)
detect_range_16_c: 1334.4
detect_range_16_avx2: 228.6 ( 5.83x)
detect_range_16_avx512icl: 139.7 ( 9.50x)
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
This partially addresses a a performance regression otherwise introduced by
f37b9a3f, which matters especially for 8-bit content where the default
threshold value rounds down to an offset of 0.
The numbers are still slower than before because the scalar tail handling
code is slower by ~200 cycles, since we benchmark across an unaligned
width.
checkasm:
- CPU: AMD Ryzen 9 9950X3D 16-Core Processor (00B40F40)
- Timing source: x86 (rdtsc)
- Bench duration: 100000 µs per function (460518890 cycles)
- Random seed: 2935750295
Benchmark results:
name cycles (vs ref)
detect_alpha_8_full_c: 7877.4
detect_alpha_8_full_avx2: 1063.4 ( 7.41x)
detect_alpha_8_full_avx512icl: 872.1 ( 9.03x)
detect_alpha_8_full_off_c: 7890.3
detect_alpha_8_full_off_avx2: 1085.5 ( 7.27x)
detect_alpha_8_full_off_avx512icl: 873.0 ( 9.04x)
detect_alpha_16_full_c: 3700.9
detect_alpha_16_full_avx2: 797.7 ( 4.64x)
detect_alpha_16_full_avx512icl: 612.1 ( 6.05x)
detect_alpha_16_full_off_c: 3701.8
detect_alpha_16_full_off_avx2: 801.2 ( 4.62x)
detect_alpha_16_full_off_avx512icl: 684.8 ( 5.41x)
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
I split this off from the commit adding the option to distinguish between the
change in code (which should be a no-op by default) and the change in behavior
(which is a deliberate deviation) when bisecting.
0.001 = 0.1% permits just one 10-bit code point of deviation, while
requiring tight bounds for 8-bit content.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
This is useful to filter false positives due to encoding noise in lossily
compressed sources.
The reason this is limited to 0.05 (= 5%) is twofold:
1. Higher values would mostly defeat the purpose of this filter, since a
deviation of >5% is enough to fully blur the distinction between JPEG and
MPEG range, let alone premultiplied and straight alpha. There's no useful
signal to extract with such a high tolerance for noise.
2. Higher values would overflow the `offset` value in the 8-bit SIMD. This
could also be solved by clamping, but the first point is the salient one
anyways.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
In particular, we also store s->mpeg_range explicitly due to the upcoming
commit, which will allow these values to deviates between the range detection
and alpha mode detection passes.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
This allows us to add a threshold for alpha comparisons to the full range
path as well.
The AArch64 changes were LLM-assisted, since I'm not familiar with that
architecture. However, the logic checks out to me.
Also adjusts the checkasm test to make sure there are no rounding differences
between the C and assembly implementations. Tested correct across 100k seeds
on x86.
checkasm:
- CPU: AMD Ryzen 9 9950X3D 16-Core Processor (00B40F40)
- Timing source: x86 (rdtsc)
- Bench duration: 100000 µs per function (432788090 cycles)
- Random seed: 4076092999
Benchmark results: (old)
name cycles (vs ref)
detect_alpha_8_full_c: 4670.7
detect_alpha_8_full_avx2: 856.6 ( 5.45x)
detect_alpha_8_full_avx512icl: 667.5 ( 7.00x)
detect_alpha_16_full_c: 2705.7
detect_alpha_16_full_avx2: 693.1 ( 3.90x)
detect_alpha_16_full_avx512icl: 546.4 ( 4.94x)
Benchmark results: (new)
name cycles (vs ref)
detect_alpha_8_full_c: 7988.5
detect_alpha_8_full_avx2: 1078.0 ( 7.41x)
detect_alpha_8_full_avx512icl: 876.7 ( 9.11x)
detect_alpha_16_full_c: 3734.7
detect_alpha_16_full_avx2: 814.6 ( 4.58x)
detect_alpha_16_full_avx512icl: 613.8 ( 6.08x)
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
The ONNX Runtime backend only ever runs one synchronous request, so
clamp nireq to 1 at load time (matching the actual allocation)
Signed-off-by: younengxiao <steven.xiao@amd.com>
full-scale transitions overflow or wrap integer sample formats. use saturating subtraction for the integer paths.
Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil@users.noreply.github.com>
converting a non-finite or too large sample count to int64_t is undefined. validate the value before conversion.
Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil@users.noreply.github.com>
the initial integer scan stores '.' as the suffix for values such as 1.5s, causing them to be interpreted as milliseconds. inspect the actual final suffix instead.
Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil@users.noreply.github.com>
This is trivial since the SAD result is already safe to 16 bits, and
the result is already normalized by the bit depth.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
This filter works by running the SAD function on the first plane, which
works for any (native endian) format that starts with an unshifted luma
plane. This notably includes formats like nv12, yuva420p, and grayscale
formats, as well as luma-alpha formats. The last addition is debatable,
but the precedent here is the existing support for RGBA, which also
dilutes scene scores by including the alpha component in the SAD
calculation). There were also some notable missing packed RGBA formats,
such as ARGB, as well as 16-bit variants.
This is an exhaustive listing as of the current set of formats.
Tested by looping over `format=<pixfmt>,scdet` and verifying that the
scene SAD score doesn't (meaningfully) change, except for expected
deviations (e.g. larger scores for full range vs limited range, and
lowered scores for packed alpha formats).
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
The warp calculation is currently done for every plane, but doesn't depend on
the plane contents, just the dimensions; so this is wasted work when plane
sizes don't vary (eg: RGB, 4:4:4 YUV).
This will not have a visible impact on performance at realistic interpolation
rates as the optical flow generation is the limiting factor until you are
doing something silly like 30->900fps.
But wasted work is wasted work, so best to stop doing it.
We have previously only supported some packed RGB and planar YUV, so the logic
for extracting the luma was not flexible enough to handle other scenarios.
To generalise this for planar RGB, we need to bind every colour plane to the
grayscale shader and derive luma from it.
Additionally, we add support packed 4:4:4 YUV formats.
In textureless regions the optical flow engine has no data to track and
its regularizer invents a smooth, often large, and internally
self-consistent flow field. Backward warping then traces a background
pixel along that invented flow onto a nearby moving object and pulls
object pixels into what should be empty background, leaving a ghost
outline at a distance from the object where nothing is actually moving.
Forward/backward flow consistency does not catch this because the
invented flow round-trips cleanly.
I have no idea if this is a common limitation in optical flow implementations
but it's definitely happening with the nvidia hardware, so this change
attempts to introduce a heuristic to work around it.
When the luma at a given location is sufficiently identical in both frames,
and the optical flow magnitude is sufficiently large, we will now assume that
the location is static and the flow is bogus. Genuine motion will usually lead
to luma differences, and low magnitude flow will be respected either way.
Again, this is all highly empirical, and the constants are justified purely by
by fiddling around with specific samples.
The grayscale inputs, flow images and optical flow session were a single shared
instance, so each source pair's optical flow execution had to wait for the
previous pair's interpolations to finish reading the flow images before it could
overwrite them, serialising the optical flow engine against the compute queue.
Double-buffer the per-pair resources across two slots (the optical flow session
bakes in its image bindings, so each slot owns its own session, grayscale and
flow images), indexed by the pair generation. One pair's optical flow execution
can then run while the previous slot's flow images are still being sampled by
the earlier pair's interpolations. The write-after-read fence on slot reuse is
tracked per slot (interp_done) instead of against the single shared instance.
As the optical flow engine is a single serial unit, more slots can't increase
throughput.
The grayscale, optical flow and interpolation stages are currently serialised
with host fence waits (ff_vk_exec_wait) after each of the the grayscale and
optical flow submissions.
We can remove these waits and manage everything with timeline semaphores. In
particular, we must correctly handle the case where more than one frame is
being interpolated per optical flow pair. In these situations, we have multiple
interpolations waiting on the optical flow generation, and then we have the
next optical flow generation waiting on all the previous interpolations.
This requires the ff_vk_exec_add_dep_signal_sem() we added in the previous
commit.
We also introduce a second context to the optical flow exec pool so that the
next command buffer can be recorded without a host-wait for the previous one
to retire - but of course, optical flow is serial so the queued one won't
actually execute until the previous one completes.
Replace the placeholder temporal blend in the interpolation shader with a
motion-compensated algorithm. This is finally the point where we are actually
using the optical flow information.
For each output pixel the source position on the f0 and f1 grids is recovered
by fixed-point Picard iteration before the samples are blended. The number of
iterations and the relaxation factor are established through empirical testing
on a couple of ad-hoc samples, representing highly uniform and non-uniform
motion respectively.
The approach here is hardly profound, and not meant to be. There are for sure
many more advanced algorithms for doing motion interpolation from optical flow,
and we can happily replace it with, or add, something more advanced once all
the machinery is in place.
Add the two options that tune the NV optical flow session:
* perf: performance level (slow/medium/fast), previously hard-coded to slow.
Trades motion-estimation quality for speed.
* grid: output grid size in pixels (auto/1/2/4/8), previously always the finest
grid the device supported. A coarser grid computes fewer flow vectors, so it
is faster and uses less memory at the cost of detail.
These settings are important as you can't handle high resolution input in
realtime at the highest quality settings. For example, 2160p24 movie content
needs reduced settings (eg: pref=medium,grid=2) for the filter to run
comfortably.
Add the Vulkan optical-flow machinery on top of the filter skeleton:
* a grayscale-extraction compute shader
* the VK_NV_optical_flow session and its persistent input/flow images
* the cross-queue (compute -> optical flow -> compute) submission with
binary-semaphore ordering
* an interpolation compute shader dispatched per output frame which blends each
new frame instead of repeating the nearest source frame
Notable characteristics:
* At this stage we are doing a simple proportional blend that doesn't actually
look at the optical flow information. That will happen in the next commit
* The optical flow is stored in a weird nvidia specific fixed point image
format. Yay?
* Optical flow is calculated based purely on Luma. For YUV format, this is
trivially obtained, but for RGB input, we should put some effort into getting
accurate Luma. But we only need relative Luma differences, so we don't have
to use the exactly correct colourspace conversion; I've just hard-coded
BT.709
* We're doing host-synchronisation for initial simplicity, but this will get
replaced in subsequent changes
* The accepted format set is deliberately narrow to begin with: planar and
semi-planar YUV, gray, and packed RGB whose sampled channels are already
in component order. Subsequent changes will expand the support set.
* We rely on VK_KHR_maintenance9 to enable implicit ownership transfers for
images between the compute and optical flow queue families.
This change introduces the basic Vulkan filter machinery, while preserving the
original vf_framerate timing logic. At this stage, the filter simply picks the
temporally closest input frame for the output.
Note that while input frames are not modified in anyway, they must be
deep-copied for output. This ensures that they match the output context, which
may differ from the input context in important way, and avoids extending the
lifetime of hardware decoded frames that are part of the decoder's DPB.
This requires exposing vkCmdCopyImage in the Vulkan function loader, and a
compute queue family and execution pool to record the copy on.
The filter depends on vulkan_1_4 for its Vulkan-Headers requirement; while this
is not required by any of the code in this commit, the final optical-flow
pipeline will need VK_KHR_maintenance9, which first appeared with Vulkan 1.4.
To avoid confusion, the requirement is asserted from the beginning.
The output timebase is derived, as in vf_fps / vf_framerate, by reducing the
source timebase against the requested frame rate. That reduction can collapse to
a zero time base - av_reduce() bounds its result, so the numerator can underflow
to zero, leaving 0/1 - when the source timebase shares no useful factors with
the requested rate, leaving an unusable output timebase that breaks the
downstream PTS rescaling.
Detect the degenerate result and fall back to the plain 1/fps timebase derived
directly from the requested rate, so config_output() always yields a valid
timebase.
This change enhances the `fps` parameter to support the `source_fps`
expressions from `vf_fps`, which are highly useful if you want to request a
multiple of the base frame rate.
The NTSC/PAL constants from `vf_fps` are not brought over. These are very
unlikely to be useful targets for a filter that is increasing the framerate of
a video.
This is the minimal level of renames required to get the new filter to function
as a clone of vf_framerate. The context type is inlined from the header, but
there are no actual functional changes.
The fruc_vulkan filter will re-use the frame timing logic from vf_framerate -
it works so we don't need to do something different. To maximise reviewability,
this commit simply copies the file into place, so that all the changes in the
subsequent commits can be more easily understood.
Use the generic weight-LUT path for Lanczos in automatic mode to provide
the full filter window and downscaling anti-aliasing. Keep the legacy
four-tap path available with use_filters=0.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
Pass param to libswscale when generating generic Lanczos filters,
allowing radii from 1 to 10. The fixed-function path remains unchanged.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
Store horizontal Lanczos results in floating-point textures so two-axis
filtering quantizes only after the final pass.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
Clamp generic-filter output to the logical source depth before restoring
its storage alignment. This prevents filter overshoot from setting
padding bits in formats such as P010 and P012.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
Generic filter coefficients depend on visible input dimensions, which
can change through per-frame crop metadata. Rebuild the filter set when
that geometry changes and synchronize before freeing active CUDA
resources.
Match av_frame_apply_cropping() for subsampled planes by flooring crop
origins while rounding visible extents up. This prevents odd left or top
crops from shifting chroma by one sample.
Clear applied cropping and size-dependent side data from the output.
Use visible dimensions when updating the sample aspect ratio.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
Group filter weights, the pass plan, and intermediate textures in a
replaceable filter set. Route generic kernel selection through the
active set and load both output directions so it owns all
geometry-dependent state.
Keep the requested use_filters value separate from the resolved mode.
Support a one-tap generic copy when no plane needs scaling.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
Plan luma and chroma passes from their sampling geometry. Allocate
intermediate storage only for planes needing horizontal scaling and
inherit untouched input textures.
Use virtual chroma extents to handle odd sizes and subsampling changes.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
Add 'lock' and 'eval' languages modes. With autodetection, whisper
reruns language detection on every queued chunk, an extra encoder pass
that can also drift to a different language mid-file.
Add two explicit values for the language option: 'eval' keeps the
per-chunk reevaluation, 'lock' caches the language detected by the
first chunk that produces transcribed text and reuses it for the
following chunks. Requiring actual text means a silent, musical or
[BLANK_AUDIO]-only opening chunk leaves detection to a later one.
'auto' remains an alias for 'eval' so existing behavior is unchanged.
the ir option may exceed nbirs because its static range is independent
of the configured number of impulse responses. init() only clamps
prev_selir, leaving selir able to index ctx->inputs out of range.
with the default nbirs=1, afir=ir=1 reads ctx->inputs[2] and crashes.
clamp selir like process_command() and add a fate test.
Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil@users.noreply.github.com>
