This commit enables the DNN video filter to run entirely on the GPU
for the full D3D12 pipeline (decode, AI inference, and encode)
without ever copying frames back to system memory, making it
significantly faster on D3D12 hardware.
Benchmarked the new zero-copy D3D12/DirectML DNN pipeline against
the existing CPU round-trip pipeline on three major GPU vendors; up
to 10x throughput improvement was measured across every vendor.
Add dnn_onnx_d3d12, a helper that binds the DirectML execution
provider to a D3D12VA input frame's own device and does NV12
texture<->tensor conversion on the GPU (CopyTextureRegion + compute
shaders), avoiding a host round-trip.
dnn_processing negotiates this automatically for D3D12 NV12 input
with device=dml. Software-frame input retains the host I/O path.
D3D12 input requires successful DirectML negotiation; unsupported
hardware-input configurations fail during configuration. Currently
limited to single-channel FLOAT/NCHW models; chroma bypasses the
model and is copied/upscaled on the GPU. At runtime, DirectML.dll
and d3dcompiler_47.dll must be loadable from the application
directory or System32.
Example usage:
# Zero-copy pipeline
ffmpeg.exe -y -loglevel info \
-hwaccel d3d12va -hwaccel_output_format d3d12 \
-i input-1920x1080.mp4 \
-vf "dnn_processing=dnn_backend=onnx:device=dml:model=ecbsrnet_y_fused.onnx:input=input:output=output" \
-c:v h264_d3d12va -bf 0 -c:a copy \
output_2x_sr_zero_copy.mp4
# CPU round-trip pipeline
ffmpeg.exe -y -loglevel info \
-init_hw_device d3d12va=dev -filter_hw_device dev -hwaccel d3d12va \
-i input-1920x1080.mp4 \
-vf "dnn_processing=dnn_backend=onnx:device=dml:model=ecbsrnet_y_fused.onnx:input=input:output=output,hwupload" \
-c:v h264_d3d12va -bf 0 -c:a copy \
output_2x_sr_round_trip.mp4
Signed-off-by: younengxiao <steven.xiao@amd.com>
Appends to already serialized init info side data, growing it with
av_fast_realloc(), so a demuxer can collect pssh records in linear time
without depending on the side data format.
The o key lists the directory of the input, local or remote, in the
list. Enter or a click lists the chosen directory or plays the chosen
file instead of the input once it opened, so a file that is no media
leaves the input playing. Directories come first, hidden entries are
left out, and the popup of a file shows its size.
Assisted-by: Claude
A directory listing that blocks in a protocol could not be
interrupted; avio_open_dir2() passes an interrupt callback to the
protocol like avio_open2() does.
Assisted-by: Claude
The strips were rendered by the ass filter over a color source in a
filter graph, whose generic blending took most of the time. libass now
renders each strip and its glyphs are composited over the strip
background where they cover it; a row strip at 1080p takes 0.47 instead
of 1.29 ms. ffplay suggests libass instead of the ass filter and no
longer selects the color filter.
Assisted-by: Claude
The filter picked its output format in config_props(), after format
negotiation had already settled the output link. The frames it emitted
could therefore disagree with the negotiated link format, for example
when a downstream zscale had negotiated something else.
Switch to FILTER_QUERY_FUNC2() and decide the output format during
negotiation instead:
- with format=, the output is constrained to that single format, which
is now also validated in init() against the supported output list;
- otherwise, when no half-float candidate remains on the input, the
output list is the input list, so the filter passes the format
through and downstream constraints can still narrow the input;
- otherwise the output is restricted to float, and negotiation picks
GBRPF32 or GBRAPF32 to match the input's alpha.
Half-float input has to be promoted because swscale cannot read or
write half-float in its new backend, and its legacy code clamps
half-float to [0,1], which would destroy values outside that range.
Half-float is therefore no longer offered as an output format.
The default output depends on the input format, so the query waits for
the input list to be merged with the upstream one, which it detects by
comparing it against the input link's other end. The merged list need
not be a singleton. Waiting for a singleton would reject valid graphs,
because a filter such as zscale offers half-float and integer formats
on its output independently of its input, so that list never narrows
to one of the two.
While the query is delayed it publishes the input color space, color
range, alpha mode and chroma location lists, because a converter
inserted on the input requires them, but the matching output lists are
only published together with the output pixel formats, as a converter
inserted on the output would otherwise find them without a format list
and crash.
Finally, supply opaque input alpha when the output format has alpha and
the input format does not. Prepend an alpha-initialization matrix to the
resolved OCIO processor, allowing OCIO to optimize it with the user's
transform. This avoids a separate per-frame alpha fill and ensures that
transforms which use alpha in RGB calculations or modify alpha receive
the opaque value before processing and retain their transformed alpha.
Filling output alpha after processing would instead leave alpha-dependent
RGB incorrect and overwrite alpha produced by the user's transform.
Note, this includes Fairys fairy/pr21799-replace patch (Thank-you Fairy!).
Signed-off-by: Sam Richards <Sam.Richards@taurich.org>
Assisted-by: Fairy
The rx and tx threads no longer use pthread_cancel(), so gate the
circular buffer on HAVE_THREADS instead of HAVE_PTHREAD_CANCEL. This
also drops the w32threads override.
On targets that have threads but no pthread_cancel(), such as Android,
udp inputs now start a receive thread and allocate the default 7*4096
by 188 byte fifo, as on other platforms. Set fifo_size=0 to get the
previous behavior.
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
Resizing is done on the RGB side: YUV to RGB conversion converts at the
visible input size before scaling, RGB to YUV conversion converts after
scaling at the output size. Every scaling pass then works on a single
packed plane and reuses the generic RGB kernels, and the conversion
kernels are reused with a float RGB side.
The RGB intermediates hold normalized floats, so filter overshoot and
out of gamut colors are not clipped before the scaler rounds and clamps
once. As conversion and filtering are linear, the result matches
scaling on the YUV side, apart from rounding and the chroma upsampling
of the conversion kernels.
Resizing during conversion always uses the generic filters with the
selected interp_algo, with nearest using point sampling weights.
Conversion without resizing keeps using the direct conversion kernels.
Assisted-By: Claude Opus 5.5
Inject APP0..APP15 segments from a length-prefixed sidecar into each
MJPEG packet after SOI. A zero-length record passes the packet through.
An APP1-only payload retains the existing APP0; a payload starting with
APP0 replaces an APP0 located immediately after SOI.
Consume records in packet arrival order, with a 16 MiB record limit.
Missing or invalid records return an error. Remember sidecar read and
validation errors so later packets cannot consume a rejected record's
payload as metadata. Flushing neither rewinds the sidecar nor clears
this error; closing the context does not read ahead.
Count every consumed input packet in diagnostics, distinguish I/O errors
from truncated records, include the open error text for a sidecar that
cannot be opened, and document the sidecar and lifecycle contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Reinhard Nißl <rnissl@gmx.de>
Assisted-by: Fairy
"bytes A-B/*" means the complete length is unknown (RFC 9110 14.4), so
don't read it as a size of 0, and never derive the complete size from a
partial reply. Remember it once learned from a 200 Content-Length or a
"bytes A-B/N" total and reuse it for later "*" replies. If the complete
size was never learned, or the total is not a number, it stays unknown.
A 416 to a read or a seek past the end of a resource of unknown size
ends it with EOF and gives its complete length ("bytes */N"), unless it
states a length past the requested offset. That 416 is retried first
only if reconnect_on_http_error lists 416 by number, reconnect_at_eof
does not retry the end it marks, and an open past the end is retried as
configured. Keep the file size and the complete size when a request
fails. A seek that fails or is refused keeps the offset and the end of
the resource, and when reading goes on over the old connection, also
the range end, chunk state, connection mode and compression of the old
reply. A seek to the current offset makes a new request when there is
no connection.
Fixes#24210
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
This is a new library, separate from the existing fdk-aac (but
built from the same upstream repo), for encoding xHE-AAC.
This library has the same traits as fdk-aac; it's being distributed
from Fraunhofer to Android Open Source Project, and further
repackaged and distributed through opencore-amr [1] (and in a
personal repo on github at [2]). It is distributed under the
same license as fdk-aac.
This is an initial, barebones wrapper around the library - not
all features of the library are hooked up currently.
In particular, do note that the wrapper currently only handles
loudness in the live operating mode (single pass), where encoding
is done with a single loudness level set before encoding. This may
affect the dynamics of the input.
The preferred and intended mode of use for the fdk-xhe-aac encoder
is for it to be used in two pass mode, which handles the loudness
properly. The two pass operation mode is not set up in the
encoder wrapper yet, but may be added later.
[1] https://sourceforge.net/projects/opencore-amr/
[2] https://github.com/mstorsjo/fdk-aac
And export both functions. This is needed for a following change that will make
use of these functions from lavf.
Signed-off-by: James Almer <jamrial@gmail.com>
Like the fd protocol, but takes a raw Windows HANDLE instead of a CRT file descriptor.
On Windows, CRT file descriptors cannot be reliably passed to child processes, so
HANDLEs need to be used instead.
Signed-off-by: Felipe Araújo <felipdsa21@outlook.com>
Useful for example for scenarios like wanting to serialize only options that
are not marked as deprecated.
Signed-off-by: James Almer <jamrial@gmail.com>
After CFLAGS stopped being passed to the C++ compiler, supplying the
DeckLink SDK include path only through --extra-cflags still passed
configure but failed when building the DeckLink C++ sources.
Compile the DeckLinkAPI.h check as C++ and check the SDK version in both
languages. The C wrappers also include DeckLinkAPIVersion.h, so document
that both --extra-cflags and --extra-cxxflags need the SDK include path.
Fixes: 0ce413af9cFixes: #23072
Assisted-by: Fairy
-aac_rc cbr|abr (default cbr). ABR runs the nd-target quality solve with
the set-point owned by a slow rate servo, and joins VBR as a
quality-target mode in the psy model (fixed-reference PE reduction,
quality attack map, broadband mask floor).
The servo integrates the log rate error but applies it to the nd
set-point in rare, discrete, calm-gated steps. The set-point must be
quasi-static: any drift on content timescales moves lambda, and with it
band/PNS/scalefactor state, tracking sections in antiphase and
coarsening exactly what needs bits - measured 2x worse than a fixed
target at equal mean rate. Locally this mode IS fixed-target VBR; rate
honesty converges on the minutes scale.
Bootstrap. The set-point is seeded from the VBR calibration anchor (one
constant, NMR_VBR_ANCHOR: the two are required to agree, and moving only
one copy made ABR overshoot its rate target by up to 9%). After ~2s of
fast rate measurement off a representative window (an all-transient
head reads over-target and would coarsen the very content that needs
bits), an open-loop jump over the measured loop gain (~ -0.24 log2 rate
per target unit) corrects the seed's per-content error, applied at
1.2x - the estimator's own inverse warm-up factor, since the EMA is
seeded on target (a cold start must not read as a fake deficit) and so
recovers only ~82% of the error. One correction is not enough whatever
the gain estimate, because the 0.24 loop gain is only nominal: the boot
re-arms while the rate is still off, up to a bounded number of times,
re-measuring each time, and re-seeds the estimator on target rather than
to zero (a zero estimate pins the integrator at its clip via log2f(0)
and reads as a 100% deficit to the fill in the same frame). Its settle
window sits below the stepper's hold, or the stepper resets the shared
counter first and the re-arm never fires.
Every set-point correction is glided in at ~0.5 log2-units/s rather than
stepped: a one-frame noise-floor jump at a fixed stream time (the boot
lands at ~2s in every stream) is audible against revealing content even
at step sizes the corpus cannot see.
Bits-fill. Under the long-run target, an easy frame is topped up toward
the ask by moving lambda finer only - the quality target is a floor,
never traded away. It spends the rate the user asked for on sub-mask
margin the nd statistic cannot see (it clips at -4/band); without it,
stat-transparent content caps the rate below any ask. The fill runs from
frame one and fades continuously as the rate EMA converges (a boot-gated
fill flipped regimes at ~2s, an audible quality step), follows the psy
PE demand shape, and is slew-bounded, writing the lambda continuity
state from long frames only instead of free-diving up to 64x per frame.
The un-booted stream head behaves CBR-like until the servo takes over
(love.flac: >6k energy had swung 0.61-1.83x vs the source at frame rate
while the head ran 25% under the ask).
The reachable-range clamps do not track the calibration anchor: their
floors are set by the stat's -4/band sub-mask clip. A floor at -1.65
railed the servo on easy content (male_speech at a 128k ask pinned at
the clamp with a 10% standing deficit); -3.0 keeps 384k asks reachable
without the stall. Easy content can still fall short of very high asks
(main_theme: 196k at a 256k ask) - there is no padding. The coarse
clamp is a quality guard: set-points past it buy little rate for a lot
of quality (2.5 -> 6.0 at a 48k ask: -5% rate, +32% Zimtohrli), so asks
below about 56 kbps stereo land above the ask (24k/32k asks: ~40k).
The documentation points those rates at CBR.
Rate accuracy against the -b:a ask:
corpus bias mean|err| worst
IgorC-3 -1.3% 2.2% -11.8%
IgorC-1 +1.1% 2.0% +5.0%
32 random -0.3% 2.2% -10.0%
A 4.5-minute concatenation of the IgorC-3 samples lands at 128.2 kbps
on a 128k ask. castanets/velvet ABR beat CBR; ABR corpus 0.00060 at
126k.
-aac_rc is documented, and fate-aac-nmr-abr-encode pins the per-packet
sizes through the servo's boot.
-q:a with the NMR coder becomes a quality-domain VBR: bisect the pooled
lambda until the mean log2 dist/real-mask over coded bands (floored at
-4/band, so a flat sf-chain over-coding quiet bands cannot let a coarser
lambda read as a finer statistic) meets a target. The target is anchored
so -q:a 1 lands near 128 kbps stereo; higher q is finer, as with the
other coders, at -1.2 per q doubling (about a third more rate), clamped
at a reachable floor that q above about 6 reaches: the stat clips at
-4/band, targets below the clamp are asymptotically unreachable, the
bisect rails at the finest lambda and the frame exceeds the decoder
buffer. The anchor is calibrated with the broadband quality-mode mask
floor from earlier in the series; that floor measured -8.8% / -5.0% /
-0.9% Zimtohrli (ViSQOL +0.013 / +0.018 / +0.004) at matched rate on
three corpora, worst sample -34%, per-sample quality spread 2.41 ->
1.74.
The psy model runs its PE reduction at a fixed reference quality for
this coder (psy.unbounded_pe): thresholds keep their perceptual shaping
but stay independent of -q:a, which otherwise moves thr_real under the
coder's own statistic and inverts the quality ladder.
Frame classes the statistic mishandles:
- Short frames: the grouped-band statistic is inflated, so they bisect
against an offset target and never write solver state. A fixed +3
offset was a content-class policy in disguise - broadband beats
(velvet) need it, LF/mid tonal decays audibly do not and paid 1.7-2.8x
the CBR error through every post-hit decay. The offset is measured
per frame instead: evaluate the grouped stat at the long-anchor lambda
(continuity says true quality is on target there, the excess is this
content's inflation), weighted by the frame's HF energy share - the
stat reads inflated on both classes, audibility does not. Post-hit
decays 1.67x -> 1.13x CBR error (worst 2.82x -> 1.44x).
- Continuous shorts: the offset-bisect assumes isolated transients
between long anchors; on dense onsets the per-frame stat is bisect
noise and lambda careened across decades (0.16 -> 82 -> 2 -> 43), an
audible whoosh. Shorts hold to the CBR run slew, clamped after the
last solve (PASS 2 re-bisects on a fresh +-32x bracket and otherwise
rails at exactly lam*32); dives toward finer stay free.
- Operating points: only bisected long frames whose stat actually
landed within 0.5 of the target write slew/warm-start state. Cheap
frames (room tone, gaps) rail fine at no cost; the first frames of a
quiet channel rail coarse from no warm state - writing either builds
a slew prison the channel climbs out of one rung per frame (a muffled
stream head on fade-ins).
- Decoupled solo solves keep per-channel slew state: a shared lam_slew
clamped each channel against the OTHER channel's operating lambda and
landed the damage asymmetrically.
The window history the slew rules read (previous-frame short, frames
since the last short run) used to advance only inside the CBR reservoir
bookkeeping; it now advances once per frame in every rate-control mode,
so the run slew and the pooled short-TNS hysteresis see real history
here too. Measured neutral (IgorC-1 VBR q=1: -0.3% Zimtohrli).
Legality is absolute: a frame may never exceed the decoder buffer.
- The cap is the format limit regardless of -q:a. It used to scale with
the outer lambda, i.e. with -q:a itself, so the finest settings had
the smallest cap (531 bits/channel at q=0.1 under the old direction)
and lost bands where they wanted the most bits.
- Dense short frames at high rates can exceed 6144 bits/channel even at
the coarsest candidate grid (the 1e4 rail still emitted 11-13k
spectral bits); the legality path sheds the highest bands across all
window groups (act[] runs group by group) until the frame fits,
keeping the scalefactor-chain anchor.
- The bits a solve reports for the next frame's side-information
estimate count only the surviving bands, and PNS filters the active
list instead of rebuilding it from all bands. Counting shed bands
drove the estimate negative (one frame emitting 1256 bits counted
13825), and a negative estimate enlarged the next frame's cap: 13056
payload bits against a 12288 limit. A negative estimate can no longer
enlarge the cap.
- The outer loop no longer skips the size check under a constant
quality: an oversized frame is retried at a lower lambda, which
shrinks the coders' caps. This applies to every coder, and only to
frames that would otherwise be illegal.
A fine-quality encode test (fate-aac-nmr-vbr-encode) pins the per-packet
sizes; its largest frame is 11560 payload bits.
Encoder side: the bandwidth ladder takes the expected per-channel rate
from the quality ladder (q=1 ~ 64.5 kbps/ch, x1.27 per doubling), and
Qavg reports the coder's quality-mode slew state: the lambda of the last
long operating-point solve. The encoder documentation describes the
NMR meaning of -q:a.
CBR output is unchanged.
When the input duration rounds to zero in the output timebase, the
rounded EOF timestamp can equal next_pts before any frame has been
output. In that case, do not discard a buffered frame whose timestamp
matches next_pts.
Keep the selected EOF timestamp rounding, but allow this first-frame
exception to extend output by one output frame period. Preserve start_time
trimming by checking the last input frame and EOF before rounding. A frame
located at start_time remains eligible even when its duration is zero.
Document the first-frame exception and add FATE coverage for a short
single-frame input and trimming at start_time, including a frame spanning
the start and a zero-duration frame at the start.
Fixes ticket #7361.
Signed-off-by: Skanda Vyas <skandavyas20@gmail.com>
Assisted-by: Fairy
Add a third context model, selected with -context 2, using only the two
nearest gradients coarsely quantized, for 13 contexts per plane.
Every slice starts from fresh probability states, and with many slices
there is too little data per slice for the larger context sets to adapt.
On a 6464x4852 16-bit RGB film scan coded with 1024 slices, this codes
87.99 MB per frame, against 90.29 MB with -context 0 and 100.79 MB with
-context 1, while needing 832 bytes of probability state per slice
instead of 23 KB and 324 KB. The same holds on 10-bit and 8-bit content
at high slice counts; with few slices, -context 0 remains best.
The table is stored in the extradata like the existing ones, so decoders
need no changes.
This allows storing metadata in a separate directory, e.g. on a faster
drive, a tmpfs, or in a volume without quota limits.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
Take the output location from the link instead of only from the
out_chroma_loc option, so the filter performs the conversion the graph
negotiated rather than just relabelling the frame. The option still wins
when set, and it now also constrains the outgoing link. Input without a
chroma location is tagged but not converted, since libswscale takes it as
sited like the output.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
A '|'-separated list, or the inverse of one for the noformat variant, which
also excludes an unspecified location, since that keeps the location of the
input. This is how fftools tells the graph which chroma locations the
encoder accepts.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
Lets the caller read back the negotiated chroma location and constrain the
sink to a list of accepted ones.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
Lets the caller state the chroma siting of the frames it feeds in, which is
what the rest of the graph negotiates against. A tagged source constrains
the link to its own value, an untagged one places no constraint at all,
there being nothing to convert from.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
Lets an encoder state which chroma sample locations it can accept, so the
caller can convert to one.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
The colorspace, range and alpha_mode options are only read when the
formats are negotiated, so commands changing them were accepted without
any effect. Make them static like chroma_location, and document the
options that can not be changed with commands.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
The output link takes the chroma_location value, or any location when it
is not set, and the output frames are rendered to the negotiated one, or
keep the location of the input when none was negotiated, the same way as
for the color space, color range and alpha mode. The inputs accept any
location, and input without one is assumed to be sited like the output.
The option can no longer be changed with commands, since it only takes
effect when the formats are negotiated.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
Chroma sample location is the last of the video colorimetry properties
without a negotiated link value, so nothing downstream can ask a filter for
a particular siting and no encoder can constrain the graph. Add it alongside
the existing color space, color range and alpha mode negotiation.
Merge conflicts are resolved by inserting a scale filter, the same way as
for color space and color range.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
in_chroma_loc was assigned to the input frame unconditionally and it defaults
to AVCHROMA_LOC_UNSPECIFIED, so the location carried by the frame was
overwritten before libswscale ever saw it. AVFilterLink does not carry a
chroma location, the frame is the only carrier, so the value was lost and
libswscale fell back to center siting for everything.
The location was also destroyed for everyone downstream, since the output
frame takes it from the input through av_frame_copy_props(). Scaling a stream
tagged left produced an output tagged unspecified. That part is a regression
from 04ce01df0b.
Note that in_chroma_loc=unknown is an alias of auto and therefore no longer
forces center siting, pass in_chroma_loc=center for that.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
Enable LAME strict ISO buffer constraints when strict_std_compliance is at least FF_COMPLIANCE_STRICT, as suggested in #23733. Preserve the library defaults for lower compliance levels.
This lets users opt into the constraints used by lame --strictly-enforce-ISO without changing the requested sample rate or bitrate. Document the mapping.
Assisted-by: Fairy
This can be used to store two (or more, in principle) JPEG images in a single
Multiple Picture Format (MPF) file. This is a JPEG extension defined in
CIPA DC-007.
Currently, we only support gain maps, so the muxer as written could be made a
lot simpler by hard-coding that assumption. However, I anticipate a future need
to use this to store e.g. large thumbnails, multiple angles or other MPF picture
types, so I made some effort to generalize the code to follow the spec's design
more faithfully, rather than taking the lazy way out.
This muxer assumes each input is an already encoded JPEG stream. It's up
to users to determine how to produce that stream - in theory, it could even
be an externally encoded JPEG file that is passed through as-is, though the
fact that there's no current way to attach externally provided gain map
parameters onto an externally encoded JPEG image means that this functionality
is, in practice, essentially restricted to the base image encoding, with the
gain map having to be encoded using FFmpeg's own mjpeg muxer to ensure the
gain map parameters are included as an ISO21496 blob.
Since the MPF spec requires hard-coding offsets to later images as part of
the MPF header (relative to the start of that header), we use a two-pass
approach, where the first pass writes out the image using a null avio to
count the number of bytes, and the second pass writes the actual data.
This roughly matches the design used by the LCEVC muxer, among others.
The files output by this muxer pass `exiftool -validate`, though they are
not bit-exact with the output of libultrahdr because libultrahdr deviates
from the spec in a number of ways, all of which appear to be bugs in
libultrahdr to me:
- We tag 0x050000 ("Gain map image") rather than 0x00000 ("Undefined")
- We correctly mark image dependencies (libultrahdr leaves away this metadata)
- We correctly place the ISO21496 APP2 after the APP0 JFIF, while libultrahdr
places it as the first segment after the SOI
- We place the MP IFD index after the APP1 Exif attributes, but before any
other APPn segments, which is the order shown in the DC-007 spec (Figure 1),
while libultrahdr always places it right before the SOS segment.
Signed-off-by: Niklas Haas <git@haasn.dev>
