The warp calculation is currently done for every plane, but doesn't depend on
the plane contents, just the dimensions; so this is wasted work when plane
sizes don't vary (eg: RGB, 4:4:4 YUV).
This will not have a visible impact on performance at realistic interpolation
rates as the optical flow generation is the limiting factor until you are
doing something silly like 30->900fps.
But wasted work is wasted work, so best to stop doing it.
964d4786ec