mirror of
https://github.com/owncloud/ocis.git
synced 2025-04-18 23:44:07 +03:00
122 lines
6.2 KiB
Markdown
122 lines
6.2 KiB
Markdown
---
|
|
title: "Upload processing"
|
|
date: 2022-07-06T12:47:00+01:00
|
|
weight: 30
|
|
geekdocRepo: https://github.com/owncloud/ocis
|
|
geekdocEditPath: edit/master/docs/architecture
|
|
geekdocFilePath: upload-processing.md
|
|
---
|
|
|
|
Uploads are handled by a dedicated service that uses TUS.io for resumable uploads. When all bytes have been transferred the upload is finalized by making the file available in file listings and for download.
|
|
|
|
The finalization may be asynchronous when mandatory workflow steps are involved.
|
|
|
|
## Legacy PUT upload
|
|
|
|
{{<mermaid class="text-center">}}
|
|
|
|
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
|
|
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
|
|
%% edit this diagram by pasting it into eg. https://mermaid.live
|
|
sequenceDiagram
|
|
participant Client
|
|
participant ocdav
|
|
participant storageprovider
|
|
participant dataprovider
|
|
|
|
Client->>+ocdav: PUT /dav/spaces/{spaceid}/newfile.bin
|
|
ocdav->>+storageprovider: InitiateFileUpload
|
|
storageprovider-->>-ocdav: OK, Protocol simple, UploadEndpoint: /data, Token: {jwt}
|
|
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/simple/91cc9882-db71-4b37-b694-a522850fcee1
|
|
ocdav->>+dataprovider: PUT /data<br>X-Reva-Transfer: {jwt}
|
|
dataprovider-->>-ocdav: 201 Created
|
|
ocdav-->>-Client: 201 Created
|
|
|
|
{{</mermaid>}}
|
|
|
|
## TUS upload
|
|
|
|
{{<mermaid class="text-center">}}
|
|
|
|
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
|
|
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
|
|
%% edit this diagram by pasting it into eg. https://mermaid.live
|
|
sequenceDiagram
|
|
participant Client
|
|
participant ocdav
|
|
participant storageprovider
|
|
participant datagateway
|
|
participant dataprovider
|
|
|
|
|
|
Client->>+ocdav: POST /dav/spaces/{spaceid}<br>Upload-Metadata: {base64 encoded filename etc}<br>TUS-Resumable: 1.0.0
|
|
ocdav->>+storageprovider: InitiateFileUpload
|
|
storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt}
|
|
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160
|
|
ocdav-->>-Client: 201 Created<br>Location: /data/{jwt}<br>TUS-Resumable: 1.0.0
|
|
|
|
Client->>+datagateway: PATCH /data/{jwt}<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 0
|
|
Note over datagateway: unwrap the {jwt} target
|
|
datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160<br>X-Reva-Transfer: {jwt}
|
|
Note over dataprovider: storage driver<br>handles request
|
|
dataprovider-->>-datagateway: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
|
|
datagateway-->>-Client: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
|
|
|
|
{{</mermaid>}}
|
|
|
|
|
|
## TUS upload with async postprocessing
|
|
|
|
|
|
|
|
{{<mermaid class="text-center">}}
|
|
|
|
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
|
|
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
|
|
%% edit this diagram by pasting it into eg. https://mermaid.live
|
|
sequenceDiagram
|
|
participant Client
|
|
participant ocdav
|
|
participant storageprovider
|
|
participant datagateway
|
|
participant dataprovider
|
|
participant nats
|
|
|
|
|
|
Client->>+ocdav: POST /dav/spaces/{spaceid}<br>Upload-Metadata: {base64 encoded filename etc}<br>TUS-Resumable: 1.0.0
|
|
ocdav->>+storageprovider: InitiateFileUpload
|
|
storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt}
|
|
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160
|
|
ocdav-->>-Client: 201 Created<br>Location: /data/{jwt}<br>TUS-Resumable: 1.0.0
|
|
|
|
Client->>+datagateway: PATCH /data/{jwt}<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 0
|
|
|
|
Note over datagateway: unwrap the {jwt} target
|
|
datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160<br>X-Reva-Transfer: {jwt}
|
|
Note over dataprovider: storage driver<br>handles request
|
|
dataprovider-)nats: emit all-bytes-received event
|
|
nats-)processing: all-bytes-received({uploadid}) event
|
|
Note over dataprovider: TODO: A lot of time may pass here, we could use<br> the `Prefer: respond-async` header to return early<br>with a 202 Accepted status and a Location header<br>to a websocket endpoint
|
|
alt success
|
|
processing-)nats: emit processing-finished({uploadid}) event
|
|
nats-)dataprovider: processing-finished({uploadid}) event
|
|
dataprovider-->>-datagateway: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
|
|
datagateway-->>-Client: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
|
|
else failure
|
|
activate dataprovider
|
|
activate datagateway
|
|
processing-)nats: emit processing-aborted({uploadid}) event
|
|
nats-)dataprovider: processing-aborted({uploadid}) event
|
|
Note over dataprovider: FIXME: What HTTP status code should we report?<br>422 Unprocessable Content is just a proposal, see<br>https://httpwg.org/specs/rfc9110.html#status.422
|
|
dataprovider-->>-datagateway: 422 Unprocessable Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
|
|
datagateway-->>-Client: 422 Unprocessable Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
|
|
end
|
|
|
|
{{</mermaid>}}
|
|
|
|
|
|
## Async TUS upload with postprocessing
|
|
This might be a TUS extension or a misunderstanding on our side of what tus can do for us. Clients should send a `Prefer: respond-async` header to allow the server to return early when postprocessing might take longer. The PATCH requests can then return status `202 Accepted` and a `Location` header to a websocket that clients can use to track the processing / upload progress.
|
|
|
|
TODO there is a conflict with the TUS.io POST request with the creation extension, as that also returns a `Location` header which carries the upload URL. We would need another header to transport the websocket location. Maybe `Websocket-Location` or `Progress-Location`?
|