If the filesize is known as a result of AVERROR_EOF on a block that ends
before the current seek position, this might end up negative. Error
out cleanly instead of aborting.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
flock() also locks against accesses by other threads of the same
process, unlike fcntl().
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
Instead, read to the output/temporary buffer (write_back path). This is to
lessen the impact of racing the write against other clients trying to race
the same pending block.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
Needed for the upcoming commit, but also more robust in general. The memory
waste is negligible.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
This value is matched to the typical seek latency in a reasonably capable
7200 rpm disk device, as well as the typical latency of an on-premise HTTP
request.
Note that this change should rarely have a significant effect, because
it only matters when using multiple concurrent processes, and one process
is somehow stuck in I/O (or died). Since we sleep in a loop for 1/16th of
the requested timeout value, this should only increase the effective read
latency by up to ~500 us on top of the actual underlying latency.
The alternative is hammering the same underlying resource with the exact
same requests at the exact same time (e.g. during init).
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
If this happens, something is almost surely wrong with the cache file
(e.g. mismatched source file), so it's much better to error out rather than
hit silent data corruption.
Sponsored-by: nxtedition AB
Signed-off-by: Niklas Haas <git@haasn.dev>
This will effectively disable the cache but allows the cache layer to verify
cached files against the original input file. Useful only for debugging
the shared cache protocol itself, as file corruption can already be caught by
the CRC check.
Decided to split this off from the previous commit in case we
ever want to revert it, since it does double the overhead of the spacemap
as well as adding extra overhead to both the read and write path.
Bump the cache version to 2 to reflect the changed disk format.
This adds a new protocol shared:URI which is distinct from the existing
`cache:` in that it is explicity designed to be thread-safe and cross-process,
enabling multiple ffmpeg processes (or multiple ffmpeg decoders within the same
process) to share a single cache file, for e.g. a remote HTTP stream. As such,
it uses a radically different internal design.
To facilitate zero-knowledge cross-process interoperability, the cache file
itself is just a memory-mapped representation of the underlying file data,
which has the side benefit that the resulting cache file will contain a
working copy of the streamed file (assuming the stream was read to
completion).
To keep track of which regions are cached and which are not, we use a
secondary file that contains a minimal header along with a static bytemap of
blocks within the file. This secondary file is also used to store metadata
such as the filesize, if known, as well as marking "failed" blocks.
Both files can grow dynamically in order to accommodate larger/growing files,
and can be atomically updated (through the use of shared space maps). I have
extensively checked the space map initalization and update code for race
conditions, and I believe the current design to be solid.
That said, it is the user's responsibility to some extent to ensure that the
same URI is not used for different streams, as we rely on the URI to uniquely
identify the cache files. That said, we use a cryptographic hash with
sufficient collision resistance to protect against possible abuse. The lack of
any implicit default on `-cache_dir` also means that `shared:` can't be enabled
via URL injection to possibly access random files on the disk (or intentionally
leak content from other streams with similar URIs, even if the cryptograhic
hash function is broken).
