PCE is not compatible with all downstream sources. Some browsers, for example,
will refuse to play an AAC file with a PCE channel layout. So there is a
legitimate use case for encoding non-PCE files if required.
However, due to the historical work-around where FFmpeg produced out-of-spec
files that mislabelled 7.1 as 7.1(wide), we should gate this behind a new
option.
I considered tying it to -strict 1 (which also disables the equivalent
work-around in aacdec), but this may lead to users unknowingly opting into
this behavior due to enabling -strict 1 out of habit (or for other reasons).
This way, a user who really knows what they're doing can use
-af aformat=channel_layouts=7.1(wide) -aac_allow_71wide yes to force
producing a spec-compliant non-PCE AAC file.
I decided to keep the option undocumented for now, matching the existing
`aac_pce` option (also undocumented).
The channel IDs were taken by comparing the AAC bitstream channel order:
FC, FLC, FRC, FL, FR, BL, BR, LFE
Aginst the FFmpeg channel order for 7POINT1_WIDE_BACK:
0=FL 1=FR 2=FC 3=LFE 4=BL 5=BR 6=FLC 7=FRC
This gives us {2, 6, 7, 0, 1, 4, 5, 3}.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
When performing unscaled planar copies or format conversions on 16-bit
buffers in planarCopyWrapper, srcPtr2 and dstPtr2 are not guaranteed to
be 8-byte aligned (e.g. subsampled chroma planes with non-multiple-of-8
widths/strides or odd slice offsets).
AV_RN64A / AV_WN64A and AV_RN32A / AV_WN32A cast pointers to av_alias64
or av_alias32 which have __attribute__((aligned(8))) or aligned(4).
Dereferencing unaligned pointers with aligned attribute triggers Undefined
Behavior traps (SIGILL / UD1 instruction with Clang -fsanitize=alignment).
Replace them with AV_RN64 / AV_WN64 and AV_RN32 / AV_WN32.
Signed-off-by: yongdev <yongdev@google.com>
binka_read_packet() calls avio_read() into a buffer from av_new_packet()
and ignores the return value. packet_alloc() only zeroes the 64-byte
padding; the payload is left uninitialized.
pkt_size is avio_rl16() + 4 (max 65539). On a truncated file the demuxer
still returns 0, so av_read_frame delivers up to 64 KiB of uninitialized
heap per packet. ffmpeg -c copy writes those bytes into the output.
Use ffio_read_size() so a short read fails the packet with
AVERROR_INVALIDDATA. Same class as 29f513a (fsb) and 4b47405 (genh).
Signed-off-by: Paulo Cabral Sanz <paulosanz@poli.ufrj.br>
abs() cannot represent the magnitude of INT32_MIN. labs() has the same issue on platforms where long is 32 bits. use FFABSU for the full signed sample range.
Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil@users.noreply.github.com>
full-scale transitions overflow or wrap integer sample formats. use saturating subtraction for the integer paths.
Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil@users.noreply.github.com>
converting a non-finite or too large sample count to int64_t is undefined. validate the value before conversion.
Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil@users.noreply.github.com>
the initial integer scan stores '.' as the suffix for values such as 1.5s, causing them to be interpreted as milliseconds. inspect the actual final suffix instead.
Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil@users.noreply.github.com>
ISO/IEC 15444-1:2002 (E), Annex B.10.1 "Bit-stuffing routine":
> The last byte in the packet header shall not be an 0xFF value (thus the
> single zero bit stuffed after a byte with 0xFF must be included even if
> the 0xFF would otherwise have been the last byte).
This matches the behavior seen in the jpeg2000 decoder (jpeg2000_flush),
and the behavior of openjpeg.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
This is trivial since the SAD result is already safe to 16 bits, and
the result is already normalized by the bit depth.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
This filter works by running the SAD function on the first plane, which
works for any (native endian) format that starts with an unshifted luma
plane. This notably includes formats like nv12, yuva420p, and grayscale
formats, as well as luma-alpha formats. The last addition is debatable,
but the precedent here is the existing support for RGBA, which also
dilutes scene scores by including the alpha component in the SAD
calculation). There were also some notable missing packed RGBA formats,
such as ARGB, as well as 16-bit variants.
This is an exhaustive listing as of the current set of formats.
Tested by looping over `format=<pixfmt>,scdet` and verifying that the
scene SAD score doesn't (meaningfully) change, except for expected
deviations (e.g. larger scores for full range vs limited range, and
lowered scores for packed alpha formats).
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
Over HTTP, seeking to a trailing chunk can make avio_size() stale, so the PCM
duration guard fails and duration is N/A. Cache size before seeking.
Fixes#24122
Formats with the same dimensions may expose different frame rates. The
selection loop previously updated selected_format for every size match but
kept selected_range from an earlier format. This could apply a range to a
format that does not support it and make device configuration fail.
Only update selected_format together with its matching range. Track a
size-only match separately so the muxed-device fallback remains unchanged.
Reported-by: MW-de <mw-de@noreply.code.ffmpeg.org>
Suggested-by: MW-de <mw-de@noreply.code.ffmpeg.org>
Fixes: https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/22694
Signed-off-by: iSold Leo <s@qwqlog.com>
The warp calculation is currently done for every plane, but doesn't depend on
the plane contents, just the dimensions; so this is wasted work when plane
sizes don't vary (eg: RGB, 4:4:4 YUV).
This will not have a visible impact on performance at realistic interpolation
rates as the optical flow generation is the limiting factor until you are
doing something silly like 30->900fps.
But wasted work is wasted work, so best to stop doing it.
We have previously only supported some packed RGB and planar YUV, so the logic
for extracting the luma was not flexible enough to handle other scenarios.
To generalise this for planar RGB, we need to bind every colour plane to the
grayscale shader and derive luma from it.
Additionally, we add support packed 4:4:4 YUV formats.
In textureless regions the optical flow engine has no data to track and
its regularizer invents a smooth, often large, and internally
self-consistent flow field. Backward warping then traces a background
pixel along that invented flow onto a nearby moving object and pulls
object pixels into what should be empty background, leaving a ghost
outline at a distance from the object where nothing is actually moving.
Forward/backward flow consistency does not catch this because the
invented flow round-trips cleanly.
I have no idea if this is a common limitation in optical flow implementations
but it's definitely happening with the nvidia hardware, so this change
attempts to introduce a heuristic to work around it.
When the luma at a given location is sufficiently identical in both frames,
and the optical flow magnitude is sufficiently large, we will now assume that
the location is static and the flow is bogus. Genuine motion will usually lead
to luma differences, and low magnitude flow will be respected either way.
Again, this is all highly empirical, and the constants are justified purely by
by fiddling around with specific samples.
The grayscale inputs, flow images and optical flow session were a single shared
instance, so each source pair's optical flow execution had to wait for the
previous pair's interpolations to finish reading the flow images before it could
overwrite them, serialising the optical flow engine against the compute queue.
Double-buffer the per-pair resources across two slots (the optical flow session
bakes in its image bindings, so each slot owns its own session, grayscale and
flow images), indexed by the pair generation. One pair's optical flow execution
can then run while the previous slot's flow images are still being sampled by
the earlier pair's interpolations. The write-after-read fence on slot reuse is
tracked per slot (interp_done) instead of against the single shared instance.
As the optical flow engine is a single serial unit, more slots can't increase
throughput.
The grayscale, optical flow and interpolation stages are currently serialised
with host fence waits (ff_vk_exec_wait) after each of the the grayscale and
optical flow submissions.
We can remove these waits and manage everything with timeline semaphores. In
particular, we must correctly handle the case where more than one frame is
being interpolated per optical flow pair. In these situations, we have multiple
interpolations waiting on the optical flow generation, and then we have the
next optical flow generation waiting on all the previous interpolations.
This requires the ff_vk_exec_add_dep_signal_sem() we added in the previous
commit.
We also introduce a second context to the optical flow exec pool so that the
next command buffer can be recorded without a host-wait for the previous one
to retire - but of course, optical flow is serial so the queued one won't
actually execute until the previous one completes.
Add a helper to register a timeline-semaphore signal with an explicit value on
an execution context, mirroring ff_vk_exec_add_dep_wait_sem() on the wait side.
The fruc_vulkan filter will use this to chain timeline semaphores across its
pipelined cross-queue submissions, where an interpolation must signal a
specific (generation-keyed) value rather than the implicit binary signal.
Replace the placeholder temporal blend in the interpolation shader with a
motion-compensated algorithm. This is finally the point where we are actually
using the optical flow information.
For each output pixel the source position on the f0 and f1 grids is recovered
by fixed-point Picard iteration before the samples are blended. The number of
iterations and the relaxation factor are established through empirical testing
on a couple of ad-hoc samples, representing highly uniform and non-uniform
motion respectively.
The approach here is hardly profound, and not meant to be. There are for sure
many more advanced algorithms for doing motion interpolation from optical flow,
and we can happily replace it with, or add, something more advanced once all
the machinery is in place.
Add the two options that tune the NV optical flow session:
* perf: performance level (slow/medium/fast), previously hard-coded to slow.
Trades motion-estimation quality for speed.
* grid: output grid size in pixels (auto/1/2/4/8), previously always the finest
grid the device supported. A coarser grid computes fewer flow vectors, so it
is faster and uses less memory at the cost of detail.
These settings are important as you can't handle high resolution input in
realtime at the highest quality settings. For example, 2160p24 movie content
needs reduced settings (eg: pref=medium,grid=2) for the filter to run
comfortably.
Add the Vulkan optical-flow machinery on top of the filter skeleton:
* a grayscale-extraction compute shader
* the VK_NV_optical_flow session and its persistent input/flow images
* the cross-queue (compute -> optical flow -> compute) submission with
binary-semaphore ordering
* an interpolation compute shader dispatched per output frame which blends each
new frame instead of repeating the nearest source frame
Notable characteristics:
* At this stage we are doing a simple proportional blend that doesn't actually
look at the optical flow information. That will happen in the next commit
* The optical flow is stored in a weird nvidia specific fixed point image
format. Yay?
* Optical flow is calculated based purely on Luma. For YUV format, this is
trivially obtained, but for RGB input, we should put some effort into getting
accurate Luma. But we only need relative Luma differences, so we don't have
to use the exactly correct colourspace conversion; I've just hard-coded
BT.709
* We're doing host-synchronisation for initial simplicity, but this will get
replaced in subsequent changes
* The accepted format set is deliberately narrow to begin with: planar and
semi-planar YUV, gray, and packed RGB whose sampled channels are already
in component order. Subsequent changes will expand the support set.
* We rely on VK_KHR_maintenance9 to enable implicit ownership transfers for
images between the compute and optical flow queue families.
ff_vk_init_sampler() always used VK_SAMPLER_MIPMAP_MODE_LINEAR for samplers with
normalized coordinates. But it is possible to have an image view that uses
normalized coordinates while not supporting linear filtering (eg: R16G16_SINT).
The simple solution is to always use nearest mipmap filtering if the image
filter is nearest.
This change introduces the basic Vulkan filter machinery, while preserving the
original vf_framerate timing logic. At this stage, the filter simply picks the
temporally closest input frame for the output.
Note that while input frames are not modified in anyway, they must be
deep-copied for output. This ensures that they match the output context, which
may differ from the input context in important way, and avoids extending the
lifetime of hardware decoded frames that are part of the decoder's DPB.
This requires exposing vkCmdCopyImage in the Vulkan function loader, and a
compute queue family and execution pool to record the copy on.
The filter depends on vulkan_1_4 for its Vulkan-Headers requirement; while this
is not required by any of the code in this commit, the final optical-flow
pipeline will need VK_KHR_maintenance9, which first appeared with Vulkan 1.4.
To avoid confusion, the requirement is asserted from the beginning.
The output timebase is derived, as in vf_fps / vf_framerate, by reducing the
source timebase against the requested frame rate. That reduction can collapse to
a zero time base - av_reduce() bounds its result, so the numerator can underflow
to zero, leaving 0/1 - when the source timebase shares no useful factors with
the requested rate, leaving an unusable output timebase that breaks the
downstream PTS rescaling.
Detect the degenerate result and fall back to the plain 1/fps timebase derived
directly from the requested rate, so config_output() always yields a valid
timebase.
This change enhances the `fps` parameter to support the `source_fps`
expressions from `vf_fps`, which are highly useful if you want to request a
multiple of the base frame rate.
The NTSC/PAL constants from `vf_fps` are not brought over. These are very
unlikely to be useful targets for a filter that is increasing the framerate of
a video.
This is the minimal level of renames required to get the new filter to function
as a clone of vf_framerate. The context type is inlined from the header, but
there are no actual functional changes.
The fruc_vulkan filter will re-use the frame timing logic from vf_framerate -
it works so we don't need to do something different. To maximise reviewability,
this commit simply copies the file into place, so that all the changes in the
subsequent commits can be more easily understood.
Enable VK_NV_optical_flow if it is present, as an optional extension. The
extension was added in 1.3.230 which is older than the minimum required
header release, so we don't need #ifdef guards.
AV_PIX_FMT_GBRAPF16 is a floating point format, as the name suggests, so it should
be mapped to VK_FORMAT_R16_SFLOAT.
The comparable GBRPF16 and GBRAPF32 both already use SFLOAT.
The backwards compatibility wrappers for the deprecated
avpriv_slicethread_{create,execute}() dereference AVSliceThread, whose
definition only exists in the threaded branch of the file. A build
without pthreads/w32threads/os2threads therefore failed with
"dereferencing pointer to incomplete type 'AVSliceThread'".
Move the wrappers into the threaded branch and give the no-threads
branch stubs for the deprecated API matching those of create2/execute2.
Write ICC profiles as TIFF UNDEFINED data and align all out-of-line IFD values to a word boundary. Account for non-empty ICC side data while allocating the packet and calculate the image size in checked 64-bit arithmetic.
Signed-off-by: Wu Jianhua <toqsxw@outlook.com>
sizeof(slice_off) is the size of the uint32_t pointer, not of an entry.
Harmless in practice as it over-allocates on 64-bit, but wrong, and
under-allocates on targets where a pointer is smaller than uint32_t.
The tile count limit was only checked once on entry, but tileCount is
incremented by ff_vk_decode_add_slice() inside the loop, so a stream
carrying more than MAX_TILES tiles writes past the end of the
MAX_TILES-element tile_sizes array.
Check the limit on each iteration, and reject at MAX_TILES rather than
above it, since tileCount is used as the write index.
