6.2 KiB
title, date, weight, geekdocRepo, geekdocEditPath, geekdocFilePath
| title | date | weight | geekdocRepo | geekdocEditPath | geekdocFilePath |
|---|---|---|---|---|---|
| Upload processing | 2022-07-06T12:47:00+01:00 | 30 | https://github.com/owncloud/ocis | edit/master/docs/architecture | upload-processing.md |
Uploads are handled by a dedicated service that uses TUS.io for resumable uploads. When all bytes have been transferred the upload is finalized by making the file available in file listings and for download.
The finalization may be asynchronous when mandatory workflow steps are involved.
Legacy PUT upload
{{}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% %% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 %% edit this diagram by pasting it into eg. https://mermaid.live sequenceDiagram participant Client participant ocdav participant storageprovider participant dataprovider
Client->>+ocdav: PUT /dav/spaces/{spaceid}/newfile.bin
ocdav->>+storageprovider: InitiateFileUpload
storageprovider-->>-ocdav: OK, Protocol simple, UploadEndpoint: /data, Token: {jwt}
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/simple/91cc9882-db71-4b37-b694-a522850fcee1
ocdav->>+dataprovider: PUT /data<br>X-Reva-Transfer: {jwt}
dataprovider-->>-ocdav: 201 Created
ocdav-->>-Client: 201 Created
{{}}
TUS upload
{{}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% %% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 %% edit this diagram by pasting it into eg. https://mermaid.live sequenceDiagram participant Client participant ocdav participant storageprovider participant datagateway participant dataprovider
Client->>+ocdav: POST /dav/spaces/{spaceid}<br>Upload-Metadata: {base64 encoded filename etc}<br>TUS-Resumable: 1.0.0
ocdav->>+storageprovider: InitiateFileUpload
storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt}
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160
ocdav-->>-Client: 201 Created<br>Location: /data/{jwt}<br>TUS-Resumable: 1.0.0
Client->>+datagateway: PATCH /data/{jwt}<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 0
Note over datagateway: unwrap the {jwt} target
datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160<br>X-Reva-Transfer: {jwt}
Note over dataprovider: storage driver<br>handles request
dataprovider-->>-datagateway: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
datagateway-->>-Client: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
{{}}
TUS upload with async postprocessing
{{}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% %% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 %% edit this diagram by pasting it into eg. https://mermaid.live sequenceDiagram participant Client participant ocdav participant storageprovider participant datagateway participant dataprovider participant nats
Client->>+ocdav: POST /dav/spaces/{spaceid}<br>Upload-Metadata: {base64 encoded filename etc}<br>TUS-Resumable: 1.0.0
ocdav->>+storageprovider: InitiateFileUpload
storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt}
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160
ocdav-->>-Client: 201 Created<br>Location: /data/{jwt}<br>TUS-Resumable: 1.0.0
Client->>+datagateway: PATCH /data/{jwt}<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 0
Note over datagateway: unwrap the {jwt} target
datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160<br>X-Reva-Transfer: {jwt}
Note over dataprovider: storage driver<br>handles request
dataprovider-)nats: emit all-bytes-received event
nats-)processing: all-bytes-received({uploadid}) event
Note over dataprovider: TODO: A lot of time may pass here, we could use<br> the `Prefer: respond-async` header to return early<br>with a 202 Accepted status and a Location header<br>to a websocket endpoint
alt success
processing-)nats: emit processing-finished({uploadid}) event
nats-)dataprovider: processing-finished({uploadid}) event
dataprovider-->>-datagateway: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
datagateway-->>-Client: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
else failure
activate dataprovider
activate datagateway
processing-)nats: emit processing-aborted({uploadid}) event
nats-)dataprovider: processing-aborted({uploadid}) event
Note over dataprovider: FIXME: What HTTP status code should we report?<br>422 Unprocessable Content is just a proposal, see<br>https://httpwg.org/specs/rfc9110.html#status.422
dataprovider-->>-datagateway: 422 Unprocessable Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
datagateway-->>-Client: 422 Unprocessable Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
end
{{}}
Async TUS upload with postprocessing
This might be a TUS extension or a misunderstanding on our side of what tus can do for us. Clients should send a Prefer: respond-async header to allow the server to return early when postprocessing might take longer. The PATCH requests can then return status 202 Accepted and a Location header to a websocket that clients can use to track the processing / upload progress.
TODO there is a conflict with the TUS.io POST request with the creation extension, as that also returns a Location header which carries the upload URL. We would need another header to transport the websocket location. Maybe Websocket-Location or Progress-Location?