Remove `in_img_size[][]` from PushData.
Even though it is constant through a filter process, it occupies
128 bites of Push Constants. It may exceed limit of some hardware.
The information is passed via Specialization Constants.
In addition to `in_img_size`, number of planes is passed via
Specialization Constants, that was hard-coded with magic number.
`m_pi4` is for later enhancement.
Signed-off-by: TADANO Tokumei <aimingoff@pc.nifty.jp>
The output link accepts any location and the frames are converted to the
negotiated one. Format conversions tagged the output as top-left while the
chroma of each 2x2 block came from whichever of its four invocations
stored it last, and scaling without a format change scaled each plane on
its own grid, which only keeps center sited chroma in place. The chroma of
each block is now sampled once at its location, and the chroma planes are
resampled from the input location to the output one. Without a negotiated
location, subsampled input keeps its location and other input still gets
top-left, and input without a location is assumed to be sited like the
output.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
sizeX/sizeY were used as radius - 1, the normalisation ignored the
vertical radius and pixels outside the frame were read as black. Match
avgblur: use the radii, limited to half the chroma plane, and repeat the
edge pixels. Existing uses now blur one pixel more, as documented.
The shader runs per output pixel but mixed input and output coordinates:
cclock_flip left part of the output unwritten, clock came out mirrored
and cclock/clock_flip were off by one. Read the source pixel for each
output pixel instead, like the OpenCL filter.
The slides truncated the offset from the opposite end, landing one row
or column behind. The circles used the squared distance ratio. Every
edge now derives from the reversed progress the same way vf_xfade does,
and pixelize samples at integer coordinates.
Slides and wipes are now bit-exact. Fade, circles and pixelize remain
within 1 LSB, as the software filter truncates when storing.
The warp calculation is currently done for every plane, but doesn't depend on
the plane contents, just the dimensions; so this is wasted work when plane
sizes don't vary (eg: RGB, 4:4:4 YUV).
This will not have a visible impact on performance at realistic interpolation
rates as the optical flow generation is the limiting factor until you are
doing something silly like 30->900fps.
But wasted work is wasted work, so best to stop doing it.
We have previously only supported some packed RGB and planar YUV, so the logic
for extracting the luma was not flexible enough to handle other scenarios.
To generalise this for planar RGB, we need to bind every colour plane to the
grayscale shader and derive luma from it.
Additionally, we add support packed 4:4:4 YUV formats.
In textureless regions the optical flow engine has no data to track and
its regularizer invents a smooth, often large, and internally
self-consistent flow field. Backward warping then traces a background
pixel along that invented flow onto a nearby moving object and pulls
object pixels into what should be empty background, leaving a ghost
outline at a distance from the object where nothing is actually moving.
Forward/backward flow consistency does not catch this because the
invented flow round-trips cleanly.
I have no idea if this is a common limitation in optical flow implementations
but it's definitely happening with the nvidia hardware, so this change
attempts to introduce a heuristic to work around it.
When the luma at a given location is sufficiently identical in both frames,
and the optical flow magnitude is sufficiently large, we will now assume that
the location is static and the flow is bogus. Genuine motion will usually lead
to luma differences, and low magnitude flow will be respected either way.
Again, this is all highly empirical, and the constants are justified purely by
by fiddling around with specific samples.
Replace the placeholder temporal blend in the interpolation shader with a
motion-compensated algorithm. This is finally the point where we are actually
using the optical flow information.
For each output pixel the source position on the f0 and f1 grids is recovered
by fixed-point Picard iteration before the samples are blended. The number of
iterations and the relaxation factor are established through empirical testing
on a couple of ad-hoc samples, representing highly uniform and non-uniform
motion respectively.
The approach here is hardly profound, and not meant to be. There are for sure
many more advanced algorithms for doing motion interpolation from optical flow,
and we can happily replace it with, or add, something more advanced once all
the machinery is in place.
Add the Vulkan optical-flow machinery on top of the filter skeleton:
* a grayscale-extraction compute shader
* the VK_NV_optical_flow session and its persistent input/flow images
* the cross-queue (compute -> optical flow -> compute) submission with
binary-semaphore ordering
* an interpolation compute shader dispatched per output frame which blends each
new frame instead of repeating the nearest source frame
Notable characteristics:
* At this stage we are doing a simple proportional blend that doesn't actually
look at the optical flow information. That will happen in the next commit
* The optical flow is stored in a weird nvidia specific fixed point image
format. Yay?
* Optical flow is calculated based purely on Luma. For YUV format, this is
trivially obtained, but for RGB input, we should put some effort into getting
accurate Luma. But we only need relative Luma differences, so we don't have
to use the exactly correct colourspace conversion; I've just hard-coded
BT.709
* We're doing host-synchronisation for initial simplicity, but this will get
replaced in subsequent changes
* The accepted format set is deliberately narrow to begin with: planar and
semi-planar YUV, gray, and packed RGB whose sampled channels are already
in component order. Subsequent changes will expand the support set.
* We rely on VK_KHR_maintenance9 to enable implicit ownership transfers for
images between the compute and optical flow queue families.
This just adds a Vulkan compute-based 360-degree video conversion.
It implements a sufficient subset of the most popular 360-degree video formats.
Options such as rotation are dynamic and can be adjusted during runtime.
Some of the work was based on Paul B. Mahol's patch from 2020. There
were spots where the arithmetic conversion was incorrect.
texture() uses bilinear scaling; imageLoad() accesses the image directly.
The reason why texture() was used throughout Vulkan filters is that
back when they were written, they were targetting old Intel hardware,
which had a texel cache only for sampled images.
These days, GPUs have a generic cache that doesn't care what source it
gets populated with. Additionally, bypassing the sampling circuitry saves
us some performance.
Finally, all the old texture() code had an issue where unnormalized
coordinates were used, but an offset of 0.5 was not added, hence each
pixel ended up being interpolated. This fixes this.
