mirror of
https://github.com/abraunegg/onedrive.git
synced 2026-10-24 15:19:18 +03:00
master
1817 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
abraunegg
|
abbb800775 |
Release files for 2.5.12 (#3919)
* Release files for 2.5.12v2.5.12 |
||
|
abraunegg
|
245bbaab36 |
Clarify Microsoft object sync limits (#3924)
Clarify Microsoft object sync limits |
||
mali
|
86272b1ba1 |
Escape regex metacharacters in 'skip_file' and 'skip_dir' patterns (#3922)
* Escape regex metacharacters in `skip_file` and `skip_dir` patterns
Previously, `wild2regex()` only escaped `.`, `+`, `/`, `(` and `)`. The characters `[`, `]`, `{`, `}`, `^` and `$` are valid in OneDrive file and directory names but were passed directly to the regex engine.
As a result, a rule such as `Photos [2023]` was interpreted as a character class, incorrectly matching `Photos 2` instead of the intended directory. Similarly, patterns such as `~$*` failed to match correctly, while unmatched `[` or `{` characters could trigger a `RegexException` during configuration processing.
Update `wild2regex()` to escape these characters correctly while preserving existing `*` and `?` wildcard behaviour.
Also correct sync-root-relative path handling for `skip_file` exclusions so that explicitly anchored file rules are consistently evaluated during local synchronisation.
* Expand E2E filtering validation in TC0011 and TC0012
Significantly expand the existing E2E filtering testcases to validate literal-character matching, wildcard behaviour, path anchoring and configuration safety across real OneDrive synchronisation operations.
- **TC0011 (`skip_file`):** Expand to 32 scenarios covering literal regex metacharacters, unmatched brackets and braces, wildcard combinations, case-insensitive matching, filename and sync-root-anchored path exclusions, configuration-change resync requirements, remote-content preservation and invalid-rule rejection.
- **TC0012 (`skip_dir`):** Expand to 38 scenarios covering literal regex metacharacters, strict and non-strict directory matching, sync-root-anchored paths, multi-component directory exclusions, wildcard combinations, configuration-change resync requirements, remote-content preservation and invalid-rule rejection.
- Add `sync_list` interaction scenarios that verify valid inclusion rules are not incorrectly classified as shadowed, while genuinely conflicting inclusion rules are rejected with explicit diagnostics.
- Validate filtering behaviour in both upload and download directions using independent client instances and remote-content verification.
- Update the configuration, usage and E2E documentation to reflect the corrected matching behaviour and expanded regression coverage.
The expanded filtering suites have been validated successfully against OneDrive Personal, Personal with a 15-character drive ID, OneDrive for Business and SharePoint configurations.
The filtering E2E coverage now comprises **70 scenarios per configuration**, with all 280 scenario executions passing.
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
|
||
|
abraunegg
|
2bf2f9f4d1 |
Clarify systemd usage (#3915)
Clarify systemd usage |
||
|
abraunegg
|
27af67a251 |
Update E2E cleanup process (#3918)
Update E2E cleanup process |
||
|
abraunegg
|
ec48e36944 |
Fix Microsoft Identity Broker automatic D-Bus activation for Intune SSO (#3916)
Fix an issue where the OneDrive client may incorrectly disable Intune SSO when Microsoft Identity Broker is installed and registered for D-Bus activation but is not already running. Previously, the client checked whether com.microsoft.identity.broker1 had an existing D-Bus owner. If the broker was inactive but available for automatic activation, this check could incorrectly conclude that the required broker capability was unavailable. This could cause OneDrive to fall back to the standard browser-based OAuth authentication workflow instead of using the configured Intune SSO integration. Changes Update src/intune.d to support automatic D-Bus activation of Microsoft Identity Broker. Allow the client to request broker activation when the broker is registered but not running. Preserve the existing Intune SSO configuration and authentication workflow. Retain compatibility with an already-running Microsoft Identity Broker. Preserve existing behaviour when the broker is unavailable or cannot be activated. The change is limited to the Intune SSO broker discovery and activation functionality. |
||
|
abraunegg
|
110fc80bbc |
Fix systemd user service installation paths on RHEL-based distributions (#3917)
## Summary
Remove legacy RHEL/CentOS systemd installation handling that incorrectly installs the `onedrive.service` user unit into the system-level service directory.
This behaviour originated from compatibility requirements for older RHEL/CentOS releases that are now end-of-life.
## Changes
- **Makefile.in**
- Install `onedrive.service` into `systemduserunitdir`.
- Install `onedrive@.service` into `systemdsystemunitdir`.
- Correct the corresponding uninstall rules.
- Remove the obsolete RHEL-specific installation exception.
- **contrib/spec/onedrive.spec.in**
- Remove the legacy `rhel_unitdir` conditional.
- Use `%{_userunitdir}` and `%{_unitdir}` for their respective service files.
- Align RPM file ownership with the corrected installation paths.
## Expected Behaviour
Modern RHEL, CentOS Stream and Fedora installations will place systemd units into their appropriate locations, allowing both system-level and user-level service management.
This change is limited to installation and packaging behaviour. No application runtime functionality is modified.
## Validation
Installation and packaging verification will be performed on RHEL 9.8 and RHEL 10.2, with regression coverage for Fedora and Ubuntu.
|
||
|
mali
|
da93547e99 |
Validate OAuth state on the local browser authentication callback (#3911)
* Validate OAuth state on the local browser authentication callback The loopback listener accepted the first authorisation code received on 127.0.0.1 without any state validation, so a forged request from a local process or web page during the authorisation window could sign the client in to an attacker-chosen account. Send a random 'state' value in the authorisation request and reject a callback that does not return it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Add urandom to the spelling allow list Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Validate OAuth state inside the local authentication listener Rework following review feedback: - Move secure random generation to util.d as generateCryptographicallySecureHexString(), reading /dev/urandom with no fallback to a non-cryptographic generator. - Generate the state only after a callback port has been found. - Validate the state inside the listener so that an invalid request is rejected (HTTP 400) and the listener keeps waiting for the genuine redirect within the existing overall timeout. - Reject missing, empty, incorrect or duplicate state, duplicate or ambiguous parameters, malformed percent-encoding and unexpected paths. - Bound each connection with a timeout so stalled connections cannot block. - Remove the inline unittest block. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Bound each callback request by a monotonic deadline and accept origin-form targets only - readRequestLine() now uses an absolute deadline (per-connection limit, capped by the overall authentication deadline) instead of a per-receive timeout, so a slow client cannot hold the listener. - Absolute-form request targets are rejected instead of being partially parsed; only the origin-form target a browser sends is accepted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Muhammet Ali Aydın <nlu@outlook.com.tr> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: abraunegg <alex.braunegg@gmail.com> |
||
|
abraunegg
|
e2977b4fd1 |
Create security.md (#3912)
* Create security.md * Update allow.txt |
||
|
abraunegg
|
7c9a1c8ebf |
Update Microsoft Intune validation (#3910)
Update Microsoft Intune validation |
||
|
abraunegg
|
47c9a3cabb |
Update sponsorship details (#3907)
Update sponsorship details |
||
|
abraunegg
|
c5fcb8e940 |
Refresh Docker build tooling and container documentation (#3906)
* Refresh Docker build tooling and container documentation ## Summary Refresh the Docker container build tooling and align the Docker and Podman documentation with the container bases already used by `master`. ## Changes - Update the Go builder used for gosu: - `1.25` → `1.27` - Update gosu across all container variants: - `1.17` → `1.19` - Parameterise the gosu version in the Debian Dockerfile. - Update Docker and Podman documentation to reflect: - Fedora 44 - Alpine Linux 3.24 - Debian 13 / Trixie * Update Dockerfile-debian |
||
|
abraunegg
|
ef80f38643 |
Display sponsorship section after successful --reauth (#3904)
## Summary Display the existing optional project sponsorship section following a successful graphical `--reauth` authentication flow. Previously, the sponsorship section was shown only after successful initial graphical authentication and was explicitly suppressed when `--reauth` was used. ## Change This is a deliberately minimal change. The existing graphical authentication completion page already displays the sponsorship section only when authentication succeeds. This PR changes the `displaySponsorship` gate so that the same section is also displayed following a successful explicit `--reauth`. No changes are made to the sponsorship page content or authentication flow itself. ## Behaviour After this change: - successful initial graphical authentication continues to display the sponsorship section; - successful graphical `--reauth` also displays the sponsorship section; - failed authentication does not display the sponsorship section; - terminal authentication output is unchanged; - `--auth-files` and `--auth-response` behaviour is unchanged; - normal synchronisation and monitor output is unchanged; - no additional sponsorship notifications or prompts are introduced. ## Rationale Users invoking `--reauth` are typically existing users who have already been using the client and need to refresh authentication, for example after credential or password changes. This provides an appropriate, low-frequency opportunity to present the existing optional sponsorship information without introducing additional runtime prompts or changing normal client operation. ## Validation The change is intentionally limited to the existing sponsorship display gate in `src/onedrive.d`. The existing success-only condition in the local graphical authentication response remains unchanged, ensuring the sponsorship section is only rendered after authentication has completed successfully. Manual validation will also be performed using a graphical desktop environment before merge. |
||
|
abraunegg
|
c5cba106ce |
Handle modified-upload HTTP 412 races and harden E2E validation (#3905)
* Handle modified-upload HTTP 412 races and harden E2E validation ## Summary Improve modified-file upload handling when Microsoft OneDrive changes an item's ETag between the client's metadata refresh and creation of an upload session, and harden the related E2E validation so genuine upload failures are detected at the phase in which they occur. This was identified through intermittent TC0029 failures across the E2E account matrix. ## Background TC0029 validates timestamp preservation when using: ```text --upload-only --local-first ``` The test performs a modified-file upload phase followed by a no-change convergence phase. During recent CI runs, Microsoft Graph occasionally returned: ```text HTTP 412 Precondition Failed ETag does not match current item's value ``` while creating an upload session for a modified file. The affected upload was reported as failed, but modified-file upload failures were not contributing to the final synchronisation failure state. The client therefore exited successfully and TC0029 continued into its no-change phase. On the subsequent execution, the client correctly detected that the failed file was still pending upload and retried it. TC0029 then incorrectly interpreted that retry as an unexpected upload during a nominal no-op phase. Debug reruns sometimes passed because the changed execution timing avoided the ETag race, while other reruns reproduced the failure against a different file. ## Application changes ### Bounded recovery from upload-session HTTP 412 When creation of a modified-file upload session receives HTTP 412: 1. Re-fetch the current remote DriveItem. 2. Verify that the remote item still represents the same file. 3. Verify that the remote content has not changed from the state inspected before the upload attempt. 4. If the remote content is unchanged, retry upload-session creation once using the refreshed ETag. 5. If the remote item or content has genuinely changed, do not retry and preserve the existing failure/conflict behaviour. This avoids blindly overwriting a legitimate concurrent remote modification while allowing recovery from a transient ETag-only race. The retry is deliberately limited to a single attempt. ### Correct modified-upload failure accounting Failed modified-file uploads now contribute to the normal upload failure accounting used to determine final synchronisation success. This ensures that an execution which contains a failed modified upload no longer reports a clean synchronisation result or exits successfully as though all requested transfers completed. The existing database state is retained after the failed upload so that the modification remains pending and can be safely retried during a subsequent synchronisation. ### Thread-safe failure collection Upload workers may operate concurrently. Updates to the shared upload-failure collection are therefore performed through a protected helper rather than directly mutating the shared array from parallel workers. Existing new-file upload failure recording also uses the same helper. ## E2E harness changes ### TC0029 TC0029 now explicitly detects a modified-upload failure during its modification phase. If phase 2 does not successfully upload all expected changes, the testcase fails at that phase rather than proceeding into the no-change validation and subsequently misclassifying a legitimate retry as an unexpected upload. This makes the failure report correspond to the actual point of failure. ### TC0058 Normalise the generated monitor configuration ordering so that: ```ini monitor_max_loop monitor_interval monitor_fullscan_frequency monitor_authoritative_sync ``` are written in the expected order. This is primarily harness hygiene. TC0058 currently uses production-safe cadence values, but writing `monitor_max_loop` first keeps the helper compatible with the reduced developer monitor limits introduced for bounded E2E monitor execution. ## Scope This change is deliberately limited to: - modified-file upload-session HTTP 412 handling; - modified-upload failure accounting; - thread-safe upload failure recording; - TC0029 failure detection; - TC0058 configuration ordering. |
||
|
abraunegg
|
98752fd185 |
Align documentation, CLI help and shell completions with current client behaviour (#3903)
Align the user-facing documentation, CLI help, sample configuration, man page and shell completion definitions with the behaviour currently implemented in `master`. This is a documentation and interface-consistency update only. It does not change synchronisation behaviour. ## Changes - Add `monitor_status` to the supplied example `config`, reflecting the existing default-enabled monitor runtime status functionality. - Update `monitor_fullscan_frequency` wording to consistently describe the **online full-scan true-up** cadence rather than implying a local filesystem scan. - Update the `--display-sync-status` summary text in: - application CLI help - `docs/usage.md` - the man page The wording now reflects the current read-only, bidirectional synchronisation-state assessment rather than the previous remote-only pending-change behaviour. - Update HTTP 429 documentation to reflect current logging behaviour: - successfully handled throttling is quiet during normal operation; - detailed 429 handling and retry information remains available in verbose/debug modes; - genuine unrecoverable failures continue to be reported. - Add the missing `--disable-time-check` option to Bash, Fish and Zsh shell completions. - Correct several shell-completion descriptions so they match the current CLI semantics, including: - `--monitor-fullscan-frequency` - `--get-file-link` - `--get-sharepoint-drive-id` ## Validation The supported command-line option set was compared directly against the options implemented in `src/config.d`. After these changes: - Bash completion exposes the complete current long-option set. - Fish completion exposes the complete current long-option set. - Zsh completion exposes the complete current long-option set. - All three completion implementations contain the same 68 supported long CLI options as the application. - The man page contains the complete current CLI option set. - Duplicate stale `--display-sync-status` wording has been removed. - Documentation describing HTTP 429 handling now matches the behaviour implemented in `master`. |
||
|
abraunegg
|
c4a1a41a0f |
Uplift E2E Testing - Phase 3 (#3902)
Uplift E2E Testing - Phase 3 |
||
|
abraunegg
|
977837deed |
Uplift E2E Testing - Phase 2 (#3889)
* Add PDF Uplift to E2E Harness * Add PNG/JPG Uplift to E2E Harness * Update E2E Document based on Phase 2 Uplift |
||
|
abraunegg
|
fe6004e1ba |
Suppress recoverable HTTP 429 retry messages during normal operation (#3900)
Summary This change improves the normal application UX when Microsoft Graph returns an HTTP 429 Too Many Requests response. HTTP 429 responses are retryable events and are normally handled automatically by the client using the Retry-After value supplied by Microsoft Graph. In many cases the request subsequently succeeds without any user intervention. Previously, these recoverable events were still displayed during normal application operation, for example: Handling a Microsoft Graph API HTTP 429 Response Code (Too Many Requests) - Internal Thread ID: ... Retrying the respective Microsoft Graph API call for Internal Thread ID: ... During large operations such as /delta enumeration, these messages can appear inline with normal progress output and give the impression that the synchronisation operation is encountering a significant problem, despite the retry completing successfully. Change When running with normal logging: suppress the Handling a Microsoft Graph API HTTP 429 Response Code... message; suppress the corresponding retry message triggered by that 429 response. When verbose or debug logging is enabled, the existing 429 diagnostic output remains available. Behaviour not changed This change does not alter: HTTP 429 detection; Retry-After processing; retry timing or backoff behaviour; retry limits; handling or reporting of retry exhaustion; final error reporting when an operation cannot recover; handling or logging of other HTTP or network errors. The change is intentionally limited to reducing normal-mode logging noise for successfully handled HTTP 429 throttling events. Expected result A recoverable Microsoft Graph throttling event becomes transparent during normal operation, while detailed diagnostic information remains available through verbose and debug logging. This provides a less alarming and more frictionless user experience without changing any underlying API retry behaviour. |
||
|
abraunegg
|
c703039af9 |
Update compiler details and associated documentation (#3899)
* Update compiler details and associated documentation |
||
Luan Vitor Simião Oliveira
|
e86d641c6e |
systemd: prevent hanging default target (#3874)
Fix systemd service ordering so the OneDrive 15-second startup grace period no longer delays `default.target` or `multi-user.target`. The existing startup delay and service behaviour are preserved, while explicitly restoring the required startup and shutdown dependencies when using `DefaultDependencies=no`. Validated on Ubuntu 24.04 with both `onedrive.service` and `onedrive@.service`, including real reboot, normal sync operation, restart/stop/start behaviour, and shutdown. |
||
|
abraunegg
|
6415cb4e56 |
Add monitor runtime status JSON for external observability (#3896)
This PR adds a new machine-readable runtime status capability for long-running `--monitor` operation.
When the client is running in monitor mode, it now maintains:
```text
<configuration directory>/monitor-status.json
```
alongside the existing configuration and application state files.
The status file provides a lightweight external view of the running monitor without requiring log parsing, additional Microsoft Graph requests, database inspection or execution of `--display-sync-status`.
### Runtime information exposed
The JSON status includes:
- application PID and per-process instance identifier
- configured operating mode
- configured monitor interval
- current monitor state
- application runtime
- monitor loop count
- current cycle start time
- last completed loop time
- whether the current process has completed its first successful synchronisation
- successful synchronisation cycle count
- last successful synchronisation time
- result and duration of the most recent cycle
- failed download count
- failed upload count
- failed Recycle Bin operation count
- Microsoft OneDrive service reachability
- current system-time validation state
All timestamps are emitted in UTC using an ISO-8601/RFC-3339-style representation so that monitoring systems can consume them consistently and convert them to local time when presenting them to users.
### Behaviour
Monitor runtime status is enabled by default:
```text
monitor_status = "true"
```
and may be disabled explicitly:
```text
monitor_status = "false"
```
The status file is:
- written with restrictive permissions
- updated using temporary-file + atomic rename semantics
- updated only at meaningful monitor lifecycle points rather than for individual file operations or notification events
- removed during a clean application shutdown
An abnormal termination such as `SIGKILL` or system power loss may leave the last status snapshot behind. A new monitor process immediately replaces any previous state with a new process instance and resets initial synchronisation readiness.
Status publication is deliberately non-fatal. Failure to write the runtime status file does not alter or interrupt normal synchronisation behaviour.
### Purpose
This provides a stable visibility layer for external tooling such as:
- Docker / Podman health and readiness checks
- systemd monitoring
- shell automation
- monitoring and observability platforms
- future graphical or management interfaces
In particular, consumers can determine whether a newly started monitor process has completed at least one successful synchronisation cycle without parsing normal application log output.
### Relationship to `--display-sync-status`
This capability does **not** replace or change the semantics of `--display-sync-status`.
The two interfaces answer different questions:
```text
monitor-status.json
-> What is the running monitor doing, and did its synchronisation
processing complete successfully?
--display-sync-status
-> Are the local filesystem and Microsoft OneDrive currently
converged?
```
This distinction is important for one-way operating modes.
For example, a `--download-only` monitor may successfully complete all permitted download processing while local-only files remain present.
In that situation:
```text
monitor-status.json
-> last cycle successful
--display-sync-status
-> NOT IN SYNC
```
Both results are correct: the configured monitor operation completed successfully, while the local and remote data states are not identical.
### I/O impact
The runtime status file is intentionally small and is not used as a high-frequency heartbeat.
It is updated during monitor lifecycle/state transitions such as:
- monitor startup
- synchronisation cycle start
- synchronisation cycle completion
- relevant condition changes
- orderly shutdown cleanup
It is not rewritten for individual uploads, downloads, inotify events or WebSocket notifications.
This keeps the additional filesystem I/O negligible relative to the database, filesystem and network activity already performed by normal monitor operation.
### Documentation
Documentation has been updated to cover:
- the new `monitor_status` configuration option
- runtime status file location and lifecycle
- JSON fields and timestamp conventions
- Docker and Podman observability use
- relationship to `--display-sync-status`
- application architecture integration
|
||
|
abraunegg
|
5691477a72 |
Fix Bug #3885: Reduce Full Scan database I/O and improve monitor-loop memory reclamation (#3888)
This pull request addresses excessive database I/O and process memory growth observed during large-account Full Scan and generated `/children` enumeration workloads. Testing and profiling were performed primarily against a fixed 101,494-object OneDrive dataset so that changes in database activity, allocation volume and retained process memory could be compared directly between implementations. ## Native Full Scan reconciliation Native tokenless `/delta` Full Scan True Up no longer uses the database `syncStatus` field as a temporary mark-and-sweep presence flag. Previously, the client: 1. materialised the applicable database subtree, 2. wrote existing records to `syncStatus = N`, 3. processed the authoritative online enumeration, 4. wrote observed records back to `syncStatus = Y`, and 5. treated records still marked `N` as absent online. For a no-change 101,494-object Full Scan this generated approximately 3.15 GiB of database write activity. The new implementation instead builds an in-memory presence candidate set containing only the IDs already present in the applicable database subtree. As authoritative `/delta` items are observed, their IDs are removed from that set. Any IDs remaining after a successfully completed enumeration are processed through the existing deletion/reconciliation logic. This makes native Full Scan presence reconciliation read-only for unchanged objects and avoids using persistent database state for temporary bookkeeping. Testing with the fixed 101,494-object dataset reduced no-change Full Scan database writes from approximately **3.15 GiB to 44 KiB**. ## Bounded native `/delta` processing Authoritative native `/delta` enumeration no longer requires all accepted JSON items to remain retained until the complete response has been downloaded. Accepted items are drained through the existing reconciliation path in bounded **500-item batches** while enumeration continues. The final partial batch is processed only after the authoritative enumeration has completed. This preserves the existing database transaction, filtering, move/rename, deletion and deferred-download behaviour while placing an explicit lifetime boundary around processed JSON pages and batches. A single strategic `GC.collect()` is performed after authoritative enumeration and reconciliation and before deferred download processing, allowing temporary enumeration state to be reclaimed before the next phase begins. ## Generated `/children` enumeration The generated current-state `/children` path has also been changed from account-sized accumulation to bounded processing. Previously, recursive `/children` traversal accumulated the complete remote object hierarchy before constructing and processing the generated response. On large accounts this created an unnecessarily large transient JSON working set. The new path preserves the existing depth-first traversal and generated-response reconciliation semantics, but submits accepted objects directly into the same bounded 500-item processing pipeline as enumeration progresses. The final partial batch is deliberately retained until traversal is complete so that absence reconciliation remains authoritative. Generated `/children` reconciliation continues to use the existing `syncStatus` semantics, but its initial database subtree marking no longer materialises full `Item` records for the entire tree. Instead, both generated subtree marking and native Full Scan presence enumeration use a minimal database projection containing only: - `id` - `type` This avoids constructing and duplicating full database `Item` state — names, tags, hashes, remote metadata and other fields — when only object identity and directory recursion are required. ## Database allocation reductions Several high-frequency database queries have been narrowed to return only the data actually required by their callers. Notable changes include: - subtree traversal uses `SELECT id, type` rather than full `Item` materialisation; - `idInLocalDatabase()` now performs a `SELECT 1 ... LIMIT 1` existence check; - `computePath()` retrieves only the seven fields required for path reconstruction rather than building a complete `Item`. Existing timestamp validation and relocation handling are retained. No database schema changes are introduced. ## Monitor-mode garbage collection and memory telemetry Monitor mode now performs: ```text GC.collect(); GC.minimize(); ``` after every completed monitor loop rather than using the previous 24-hour cleanup interval. `GC.collect()` and `GC.minimize()` are retained as separate operations because they address different aspects of the observed behaviour: reclaiming unreachable managed objects and returning unused managed heap capacity to the operating system. When memory telemetry is enabled, memory state is recorded both before and after this collection/minimisation boundary. The asynchronous logging subsystem now provides a drain barrier so that telemetry messages and logger-local buffers have been written and released before garbage collection is measured. This prevents the telemetry mechanism itself from changing the memory state being observed. ## Object and response lifetime cleanup The investigation also identified a number of long-lived references whose useful lifetime had already ended. The PR therefore explicitly releases temporary JSON responses, processed arrays and intermediate API state after their final use across sync, OneDrive API, webhook, Socket.IO, Intune and utility paths. CurlEngine pool handling has also been tightened: - popped pool slots are explicitly cleared; - stopped engines are destroyed instead of abandoned; - expired engines are deterministically destroyed after cleanup. These changes are intended to make ownership boundaries explicit and prevent stale references from extending the lifetime of otherwise reclaimable state. ## Memory validation The important measurement is retained memory after repeated large enumerations, not simply cumulative allocation volume. A native `/delta` control run using the bounded processing and strategic collection path completed **222 monitor loops over more than 22 hours**, including repeated authoritative Full Scans. After establishing its initial working-set high-water, post-minimisation memory remained bounded at approximately: ```text GC capacity: ~38.9 MiB RSS: ~77–78 MiB ``` The generated `/children` path was then tested with: ```text force_children_scan = true ``` so that every monitor loop enumerated the same fixed 101,494-object account through `/children`. An intermediate implementation that bounded JSON processing but still materialised the database subtree as full `Item` records continued to show retained-memory growth, reaching approximately: ```text GC capacity: ~169 MiB RSS: ~209 MiB ``` by Loop 14. After replacing that subtree materialisation with the minimal `id + type` traversal, the same workload remained around: ```text GC capacity: ~25.6 MiB RSS: ~64 MiB ``` through Loop 14, with no GC-arena staircase. At Loop 15, heap minimisation returned additional pages to the operating system, reducing the post-minimisation state to approximately: ```text GC capacity: ~18.4 MiB RSS: ~57 MiB ``` The long-duration `/children` soak is still being continued to confirm the same bounded behaviour over an overnight run. ## Functional validation The bounded generated-response changes have also been exercised through the existing E2E coverage. At the current PR revision: - Personal Account Shared Folder E2E tests are green. - Business Account Shared Folder E2E tests are green. The implementation preserves the existing filtering, reconciliation, deletion, move/rename, Shared Folder and deferred-download processing paths. ## Scope This PR changes the implementation of large authoritative enumeration and memory lifecycle management without changing the database schema or the externally visible synchronisation model. The primary goals are: - eliminate unnecessary Full Scan database writes; - avoid account-sized transient JSON and database object graphs; - process authoritative data using bounded working sets; - make temporary object lifetimes explicit; - reclaim managed heap capacity consistently during long-running monitor operation; and - preserve existing reconciliation semantics and E2E behaviour. |
||
|
abraunegg
|
2fc8decf89 |
Fix Bug #3884: Fix undefined dynamic-array mutation in getChildren() traversal (#3890)
This PR fixes Issue #3884 by replacing the recursive getChildren() implementation with an explicit iterative directory traversal. The previous implementation appended descendants to a dynamic array while iterating over that same array using foreach, resulting in unspecified behaviour under the D language specification. The replacement: Eliminates dynamic-array mutation during iteration. Preserves the existing descendant ordering and function interface. Avoids repeated recursive array concatenation and unnecessary copying. Eliminates recursive call-stack growth for deeply nested directory trees. The change is isolated to getChildren() in src/sync.d and is independent of PR #3888. Validation: Full E2E regression testing will be performed across all supported account types, including shared-folder scenarios. |
||
|
abraunegg
|
639506c41c |
Fix Bug #3897: Preserve configured drive_id when using national cloud endpoints (#3898)
Fix a regression where a configured `drive_id` could be overwritten during OneDrive API initialisation when using a Microsoft national cloud deployment.
This affects configurations combining:
```text
drive_id = "<configured drive>"
azure_ad_endpoint = "USL4|USL5|DE|CN"
```
The issue was reported against a GCC High / USL4 SharePoint document library configuration.
### Root Cause
During `OneDriveApi.initialise()`, the configured `drive_id` was applied before processing `azure_ad_endpoint`.
The initial sequence was effectively:
```text
apply configured drive_id
|
v
select national-cloud Graph endpoint
|
v
reset driveUrl/item URLs to /me/drive
```
For national-cloud configurations, endpoint selection subsequently rebuilt:
```text
driveUrl
itemByIdUrl
itemByPathUrl
```
using the applicable national-cloud Graph endpoint, but restored them to the default `/me/drive` paths.
This discarded the explicitly configured `drive_id`.
Because `/me/drive` can return valid information for the authenticated user's default OneDrive, startup discovery could then incorrectly establish that drive and its root as the client's synchronisation target instead of the configured SharePoint document library.
The same ordering issue applies to all currently supported non-global Graph endpoints:
- USL4
- USL5
- DE
- CN
### Change
Move the existing `drive_id` URL configuration so that it is applied **after** `azure_ad_endpoint` processing.
The resulting initialisation order is now:
```text
select configured Graph endpoint
|
v
apply configured drive_id
|
v
continue API initialisation
```
This ensures that the selected cloud endpoint and explicitly configured drive target are composed correctly.
For example:
```text
azure_ad_endpoint = "USL4"
drive_id = "<sharepoint-drive-id>"
```
now produces the intended drive URL:
```text
https://graph.microsoft.us/v1.0/drives/<sharepoint-drive-id>
```
rather than reverting to:
```text
https://graph.microsoft.us/v1.0/me/drive
```
### Scope
This is intentionally a minimal change.
It does not alter:
- authentication behaviour;
- tenant selection;
- national-cloud endpoint definitions;
- SharePoint discovery;
- synchronisation logic;
- `SyncEngine` behaviour; or
- handling of configurations without `drive_id`.
No national-cloud-specific workaround has been added. The existing `drive_id` handling has simply been moved to the correct point in API initialisation.
### Validation
The existing E2E suite provides regression coverage for normal configurations.
In particular, the SharePoint E2E workflow explicitly injects a `drive_id` and exercises the full client against a SharePoint document library. This will validate that normal commercial-cloud `drive_id` behaviour remains unchanged.
Personal and Business E2E testing provide additional control coverage where no explicit `drive_id` is configured.
Direct national-cloud validation requires access to an applicable Microsoft national-cloud tenant. The reporter of #3897 has access to a GCC High / USL4 environment and can validate this PR against the original reported configuration.
|
||
|
abraunegg
|
9adea3a04b |
Uplift E2E Testing - Phase 1 (#3878)
Completed the first phase of the E2E testing realism uplift in preparation for v2.5.12, replacing reliance on synthetic fixtures with realistic file content while retaining existing regression coverage. This phase delivers: * Extend applicable testcases with genuine, structurally valid XLSX files while preserving the existing TXT and binary fixtures. * Introduce small and large XLSX companions on either side of the 4 MiB upload threshold, exercising both simple-upload and session-upload paths. * Strengthen coverage of file and directory lifecycle operations, including creation, modification, deletion, renaming, moves, conflicts, safeBackup preservation, monitor mode and filtering. * Explicitly account for SharePoint documentLibrary enrichment, validating Microsoft's modified Office files without incorrectly assuming byte-for-byte identity after enrichment and re-download. * Add TC0080 for live monitor reconciliation of remotely renamed populated directories. * Add TC0081 to validate historical v2.5.5 database upgrades, schema integrity, backup restoration and recovery through `--resync`. * Improve CI diagnostics, shared-folder logging, test reporting and failed-test debug reruns. The expanded testing also exposed application defects that have been corrected as part of this work, including Microsoft Graph `resourceModified` conflicts during simple uploads, directory reconciliation edge cases and recovery from incompatible database schemas. **Validation:** The complete E2E suite has passed across all tested account configurations, including Personal, Personal with a 15-character drive ID, Business, SharePoint documentLibrary and shared-folder scenarios. This establishes the Phase 1 realism baseline for v2.5.12. Additional PDF and image-format coverage will follow in a subsequent phase, allowing the remaining pre-release effort to concentrate on large-account memory consumption, outstanding defects and long-running stability testing. |
||
|
abraunegg
|
960830cef7 |
Limit large-drive GUI warning to once per application run (#3887)
Summary Reduce repeated desktop notifications when a OneDrive account exceeds the recommended 300,000 object threshold. The warning condition itself is unchanged and continues to be evaluated and logged on every applicable full scan. Change Add a SyncEngine lifetime flag to track whether the user has already been notified. Send the GUI notification only on the first occurrence during the current application run. Continue logging the warning on subsequent detections. The notification state resets naturally when the application is restarted. This is a UX-only change and does not alter the object-count threshold, filtering behaviour, or synchronisation logic. |
||
|
abraunegg
|
e8100b323b |
Fix Bug #3871: Harden item database schema validation and require resync on structural mismatch (#3873)
This PR improves how the client handles an unexpected local item database schema.
Previously, `buildItem()` contained a hard assertion requiring every database result row to contain exactly 20 columns:
```text
The DB result must have 20 columns
```
If the local `items.sqlite3` database did not match the schema expected by the running client, this resulted in an `AssertError` and stack trace.
This PR replaces that behaviour with explicit database validation and a controlled `--resync` requirement.
## Changes
### Graceful handling of database schema mismatch
The hard assertion in `buildItem()` has been replaced with a controlled fatal error.
If a database row does not match the structure expected by the application, the client now:
* reports the expected and actual number of columns;
* explains that the local item database cannot safely be used;
* instructs the user to perform `--resync`;
* exits using the existing `EXIT_RESYNC_REQUIRED` exit code.
The client does **not** attempt to skip malformed database rows and continue synchronisation.
Continuing reconciliation when the local database structure cannot be trusted risks interpreting database data incorrectly or treating tracked items as missing.
### Centralised item database column definition
The positional layout of the `item` table is now represented by a single `ItemDatabaseColumn` enum.
The expected number of database columns is derived programmatically using `EnumMembers`, rather than being maintained as a hard-coded value.
This removes duplicated schema assumptions such as:
```text
20
[18]
[19]
```
from `buildItem()` and replaces numeric row indexes with named column positions.
This improves maintainability when the database schema changes in future.
### Startup physical schema validation
The client previously validated database compatibility using only:
```text
PRAGMA user_version
```
A database whose `user_version` matched the current application could therefore pass startup validation even if the physical `item` table did not match the schema expected by the client.
This PR adds startup validation using:
```sql
PRAGMA table_info(item);
```
The client now verifies that the physical table has:
* the expected number of columns;
* the expected column names;
* the expected column ordering / `cid` positions;
* no missing columns;
* no unexpected additional columns.
If any mismatch is detected, synchronisation is not allowed to proceed and the client exits with `EXIT_RESYNC_REQUIRED`.
This validation is performed once during database initialisation and therefore adds no per-item or per-sync reconciliation overhead.
### Minor cleanup
The database version already retrieved during initialisation is reused rather than calling `db.getVersion()` a second time.
## Rationale
The local item database is reconstructible synchronisation state, but its structure is fundamental to correct reconciliation.
If the client cannot prove that the physical database layout matches the structure expected by the running binary, the safest behaviour is:
```text
database mismatch
↓
stop synchronisation
↓
require --resync
```
rather than:
```text
database mismatch
↓
skip affected records
↓
continue reconciliation using incomplete or incorrectly interpreted state
```
The existing runtime row-width check remains as defence-in-depth, but structurally incompatible databases should now be detected during startup before normal synchronisation begins.
## Related
Addresses the unsafe failure mode exposed by Issue #3871.
This PR does not attempt to validate the proposed root-cause theory from that issue. It instead ensures that any incompatible or structurally unexpected local database is detected safely and handled through the existing `--resync` recovery path.
|
||
|
abraunegg
|
b647543dcb |
Add Unsupported platforms (#3877)
Add Unsupported platforms |
||
|
abraunegg
|
f785bd257c |
Fix Bug #3868: Fix false safeBackup on timestamp-preserving file replacements (#3869)
Fix modified-file reconciliation when a local file is replaced with different content while preserving an older modification timestamp.
This can occur through normal workflows such as:
* `cp -p`
* `rsync -t`
* restoring files from backup
* file managers that preserve source timestamps
Previously, if the replacement content differed but its local modification timestamp was older than the online timestamp, the client could incorrectly conclude that the online file was newer.
This caused the local replacement to be treated as a conflict even when the online object had not changed since the client's last synchronised state.
The resulting behaviour was:
```text
local content replaced with older preserved mtime
|
v
classified as "online is newer"
|
v
local replacement moved to safeBackup
|
v
safeBackup uploaded as a new file
|
v
existing online version downloaded back
```
This PR corrects that decision while retaining the existing data-preservation behaviour for genuine online changes.
## Root Cause
`performModifiedFileUpload()` already retrieves fresh metadata for the online DriveItem before deciding how to process a modified local file.
The existing conflict logic, however, primarily used:
```text
localModifiedTime < onlineModifiedTime
```
to determine that the online version should take precedence.
That assumption is not sufficient.
A local file can contain new content while legitimately retaining an older timestamp.
Importantly, the client also has another piece of information available at this point: the current online eTag.
Where the freshly retrieved online eTag is unchanged from the eTag stored in the local database, the online object has not changed relative to the client's synchronised baseline.
An older local timestamp alone therefore must not cause the local replacement to be classified as an online conflict.
## Change
The modified-file upload decision now determines whether the current online DriveItem can be proven unchanged from the database baseline.
The database baseline is trusted only when:
* the current online object corresponds to the same DriveItem identity represented by the database entry;
* the online item ID matches the expected target item;
* both the current online eTag and database eTag are present and non-empty;
* the current online eTag matches the database eTag; and
* the client is not operating under `--resync`.
When these conditions are satisfied:
```text
different local content
+ older local timestamp
+ unchanged online eTag
= local modification
```
The existing modified-file upload path is allowed to proceed.
Where the online eTag has changed, is unavailable, cannot be safely associated with the same DriveItem identity, or the client is running under `--resync`, the existing conservative conflict handling remains unchanged.
This intentionally keeps the fix narrowly focused on cases where the client can positively establish that the online object has not changed.
## `remoteItem` Handling
Shared/remote items require additional care because the top-level database item and the actual target DriveItem can represent different identities.
This PR does not attempt to redesign that behaviour.
Where the current online object cannot be safely correlated with the database eTag baseline, the existing data-preservation path remains in effect.
## E2E Test Coverage
The timestamp and replacement E2E coverage has been expanded to exercise actual Microsoft Office XLSX files rather than relying solely on synthetic text payloads.
This is intentional because XLSX files can exercise additional Microsoft Graph and SharePoint behaviour, including SharePoint enrichment of uploaded Office documents.
The existing XLSX implementation previously contained within TC0021 has been centralised into the E2E framework and TC0021 now consumes that common implementation.
The shared XLSX helper supports:
* creation of valid OOXML XLSX files;
* validation of generated/downloaded workbooks;
* mutation of existing workbooks while preserving other ZIP package members; and
* generation of files on both sides of the simple/session upload boundary.
This allows subsequent mutations to operate on the actual workbook returned by Microsoft, including any service-side package changes.
### TC0021
Existing resumable-transfer XLSX testing now uses the common XLSX framework implementation.
The existing large-file/session-upload behaviour remains unchanged.
### TC0029
`--local-first --upload-only` timestamp-preservation testing now uses a real XLSX document.
### TC0036
Overwrite/replace coverage has been expanded substantially.
#### OR-0001 — newer local replacement
A synchronised XLSX is replaced with different XLSX content having a newer local timestamp.
Expected result:
* normal modified-file upload;
* replacement becomes the canonical online content.
#### OR-0002 — older timestamp, simple upload
A synchronised XLSX is replaced using a timestamp-preserving copy operation with:
* different content;
* an older local modification timestamp; and
* no independent online change.
Expected result:
* online eTag still matches the database baseline;
* replacement is processed as a modified file;
* no erroneous safeBackup is created;
* canonical online content becomes the local replacement.
This directly covers the reported failure condition.
#### OR-0003 — older timestamp, session upload
The same scenario as OR-0002 is performed using an XLSX larger than the simple-upload threshold.
Expected result:
* session upload is used;
* modified upload succeeds;
* no erroneous safeBackup is created.
This confirms that the reconciliation fix operates correctly before upload-method selection.
#### OR-0004 — genuine online conflict
A separate mutator client is created using the established E2E multi-client model:
```text
clone tracked configuration/database/local state
|
v
mutator runs --monitor --upload-only
|
v
mutator changes XLSX online
|
v
remote modification independently verified
|
v
stale client receives different local replacement
with older preserved timestamp
```
Expected result:
* current online eTag differs from the stale client's database baseline;
* existing conflict handling remains active;
* the local replacement is preserved through safeBackup;
* the independently changed online version remains authoritative.
This is the safety control demonstrating that the fix does not weaken protection against genuine online changes.
### TC0037
mtime-only local-change handling now uses XLSX files across:
* simple upload-sized files;
* automatic session-upload-sized files;
* forced-session small files; and
* forced-session large files.
The downloaded Microsoft-returned workbook is used as the settled content baseline before timestamp-only mutations are performed.
### TC0051
Monitor-mode mtime-only testing now uses XLSX content and confirms that a timestamp-only local filesystem event does not result in an unnecessary upload or remote content change.
## Scope
This PR does not change the general timestamp authority policy introduced by earlier timestamp work.
It does not change:
* Microsoft timestamp application after successful uploads;
* simple versus session upload selection;
* `--local-first` conflict policy;
* `--resync` semantics;
* genuine online-change conflict preservation; or
* shared/remote item identity handling beyond refusing to trust an ambiguous eTag baseline.
The change is deliberately limited to preventing an older preserved local timestamp from being treated as proof of an online conflict when the current online DriveItem can be positively demonstrated to be unchanged from the client's database baseline.
|
||
|
abraunegg
|
a9a3904c86 |
Update architecture documentation (#3870)
* Update architecture documentation |
||
|
abraunegg
|
6965af2b0b |
Add project sponsorship visibility to initial graphical authentication (#3867)
## Summary This PR adds restrained GitHub Sponsors visibility to the OneDrive Client for Linux while deliberately avoiding recurring or intrusive sponsorship messaging. Following a successful **initial graphical authentication**, the browser completion page now includes an optional invitation to help sustain the project through GitHub Sponsors. The sponsorship message is not displayed during `--reauth`, authentication failures, application startup, monitor startup, normal synchronisation activity, terminal output or desktop notifications. ## Changes - Enhances the graphical authentication completion page with a clearer responsive layout and application identity. - Displays the sponsorship section only after successful initial graphical authentication. - Does not display sponsorship content during explicit `--reauth`. - Adds an attributable GitHub Sponsors link using GitHub Sponsors campaign metadata. - Adds passive sponsorship information to the project README and graphical-authentication documentation. - Adds browser response protections including `Referrer-Policy: no-referrer`, `Cache-Control: no-store`, a restrictive Content Security Policy and `X-Content-Type-Options: nosniff`. - Does not add application telemetry, click tracking or any new configuration option. ## Sponsorship Behaviour The graphical authentication page displays: > Help sustain OneDrive Client for Linux OneDrive Client for Linux is free, open-source software that is independently developed and maintained. If it is useful to you or your organisation, you can help sustain ongoing development, testing and maintenance through GitHub Sponsors. The user must explicitly select **Sponsor on GitHub** before any connection to GitHub is made. GitHub Sponsors campaign metadata is used only to attribute completed sponsorships to the relevant project surface. ## Privacy The application does not report whether the sponsorship message was displayed or selected. No application telemetry or analytics have been added. The local OAuth completion response explicitly prevents the Microsoft OAuth callback URL from being supplied as a browser referrer when following the external sponsorship link. |
||
|
abraunegg
|
4e69d7022d |
Fix Bug #3865: Fix graphical authentication reporting success before OAuth token redemption completes (#3866)
Fix graphical browser-based authentication reporting success before OAuth token redemption and credential persistence have actually completed.
Previously, the local loopback authentication listener treated receipt of a Microsoft OAuth authorisation code as successful authentication and immediately returned an `Authentication complete` page to the browser.
However, receipt of the authorisation code is only an intermediate stage of the authentication flow. The code must still be redeemed against Microsoft's token endpoint, the returned authentication data processed, and the required refresh token successfully persisted locally.
This meant that the browser could report:
```text
Authentication complete
```
even if the subsequent token redemption or credential persistence failed.
This change corrects that lifecycle/state-reporting behaviour.
## Changes
The graphical authentication flow now keeps the browser connection open until the normal OneDrive authentication code has completed the remaining OAuth transaction.
The flow is now effectively:
```text
Microsoft redirects to localhost with ?code=...
|
v
Local authentication listener parses callback
|
v
Authorisation code passed to existing authentication flow
|
v
OAuth token redemption
|
v
Authentication response processing
|
v
Refresh token persistence
|
v
Final authentication result returned to local listener
|
+--> success: Authentication complete
|
+--> failure: Authentication failed
```
The existing separation of responsibilities is retained:
* `localAuth.d` continues to manage the local loopback listener, browser callback parsing and browser HTTP response.
* `onedrive.d` continues to manage OAuth token redemption, authentication response processing and credential persistence.
No OAuth/token handling has been moved into the local authentication module.
## Authentication failure handling
For graphical authentication, token acquisition can now return a failure result to the local browser authentication flow rather than terminating the process before the browser can be given an accurate result.
This allows the browser to report authentication failure when, for example:
* the returned authorisation code is rejected during token redemption;
* the token endpoint returns an unsuccessful response;
* the returned authentication response cannot be processed into usable authentication state;
* required authentication credentials cannot be persisted locally.
Refresh-token persistence failures are also now propagated as authentication failure for this flow rather than being logged without affecting the final graphical authentication result.
## Behavioural scope
The change is deliberately limited to graphical browser authentication.
Existing behaviour is preserved for:
* manual/paste-the-redirect-URI authentication;
* normal refresh-token based access-token renewal;
* device authentication;
* Intune authentication;
* other non-GUI authentication paths.
Graphical `--reauth` uses the same corrected completion semantics as initial graphical authentication.
## Expected Result
The graphical browser authentication success page is now displayed only after the OneDrive Client for Linux has genuinely completed:
1. receipt of the Microsoft authorisation code;
2. successful OAuth token redemption;
3. authentication response processing;
4. required authentication-state creation;
5. successful refresh-token persistence.
If any required stage fails, the browser must not display `Authentication complete`.
## Validation
GUI validation will cover:
* successful initial graphical authentication;
* Microsoft callback cancellation/error;
* authorisation code received but rejected during token redemption;
* refresh-token persistence failure;
* graphical `--reauth`;
* manual/headless authentication regression testing;
* existing token refresh;
* graphical authentication using the `BROWSER` environment variable.
This change is intentionally narrow and does not include unrelated authentication refactoring or sponsorship functionality.
|
||
|
abraunegg
|
9b1b0d5d03 |
Fix Bug #3862: Fix upload transient retry coverage and fragment retry logging (#3864)
### Summary Improve upload transient-error handling for both simple uploads and session uploads by extending the existing common HTTP retry handling and correcting misleading session-upload logging. This change is intentionally narrow and does not introduce any new retry mechanism. ### Changes The common OneDrive API error handler now treats the following server-side responses consistently as transient: ```text 500 Internal Server Error 502 Bad Gateway 503 Service Unavailable 504 Gateway Timeout ``` Previously, `503` and `504` were retried while `500` and `502` were returned immediately. Because both simple uploads and session fragment uploads use the same common API wrapper, this provides consistent transient handling across both upload methods. The HTTP status description handling has also been completed for: ```text 408 Request Timeout 502 Bad Gateway ``` ### Session upload logging `performSessionFileUpload()` historically contained: ```text Retrying fragment upload ``` before determining whether an actual fragment retry would occur. Repository history shows that this retry logic has existed for many years and has served several different recovery paths, so no functional retry behaviour has been removed or reworked. Instead, the logging statement has simply been moved to immediately before the actual `uploadFragment()` retry call. This ensures that: ```text Retrying fragment upload ``` is only emitted when the client is genuinely about to retry a fragment. ### Scope This change deliberately does not modify: * upload-session replacement handling for HTTP 403 or 404 * the existing HTTP 416 upload-session behaviour * upload offsets or `nextExpectedRanges` processing * upload-session persistence or resume handling * fragment retry architecture * retry-count or backoff architecture * any synthetic HTTP failure or fault-injection mechanisms Ordinary transient HTTP retries remain owned by the common OneDrive API error handler, while upload-session-specific recovery remains within the session upload state machine. ### Validation * Changes compile successfully against the current master codebase. * Existing simple and session upload paths continue to use the established common HTTP/API retry architecture. * The `sync.d` change is logging-only and does not alter operational retry behaviour. |
||
|
abraunegg
|
e17643a7b2 |
Fix Bug #3861: Session resume does not verify that the local source file is unchanged (#3863)
Interrupted session uploads can be resumed from persisted `session_upload.*` state.
Before this change, the client validated that the persisted Microsoft upload session was still usable, but did not verify that the local file being resumed was still the same local file revision that originally created the upload session.
This creates a data-integrity risk.
For example:
```text
Local revision A
Fragment 1 -> uploaded
Fragment 2 -> uploaded
Fragment 3 -> uploaded
Client interrupted
```
If the local file is then modified before the client restarts, the existing Microsoft upload session can still advertise the next expected range beginning at fragment 4.
The resume worker would then read the remaining data from the current local file:
```text
Remote object
Fragment 1 -> revision A
Fragment 2 -> revision A
Fragment 3 -> revision A
Fragment 4 -> revision B
Fragment 5 -> revision B
...
```
A file-size change may subsequently invalidate the Microsoft upload session, but this does not protect against a same-sized local modification.
For example:
```text
Revision A = 100 MiB
Revision B = 100 MiB
contents differ
size remains unchanged
```
In this situation, the existing upload session could previously be resumed using fragments from two different local revisions.
## Resolution
Persist local source identity with every upload session:
```text
sourceFileSize
sourceFileMtime
```
`sourceFileMtime` retains the full filesystem timestamp precision available to the client and is captured independently of the timestamp normalisation used when sending metadata to Microsoft Graph.
When an interrupted upload session is discovered, the client now compares the current local source against the persisted source state before allowing the Microsoft upload session to be reused.
If either the file size or modification timestamp differs:
```text
local source changed
↓
existing upload session is unsafe
↓
best-effort cancel Microsoft upload session
↓
remove stale session_upload.* state
↓
allow normal reconciliation to process the current file
↓
fresh upload begins from byte zero
```
The source is also revalidated immediately before the resume worker begins uploading fragments, reducing the race window between initial session validation and actual upload execution.
Persisted upload-session files created by older versions of the client do not contain the new source identity fields. These sessions are intentionally treated as unsafe to resume and are discarded so that a fresh upload can occur.
The upload-session replacement handling introduced for #3859 / PR #3860 has also been updated so that replacement sessions retain the source identity associated with the replacement transfer.
## E2E Regression Coverage
TC0021 has been extended with:
```text
RT-0005 - changed local source invalidates an interrupted upload session
```
The scenario uses a genuine Microsoft resumable upload session and does not use synthetic API failures or fault injection.
RT-0005:
1. Generates deterministic 100 MiB local revision A.
2. Starts a genuine multi-fragment session upload.
3. Allows multiple fragments to be successfully uploaded.
4. Interrupts the client while a real `session_upload.*` file and Microsoft upload session remain active.
5. Confirms the persisted upload session has progressed beyond byte zero.
6. Replaces the local source with deterministic revision B.
7. Keeps revision B exactly the same size as revision A.
8. Ensures the local modification timestamp changes.
9. Restarts the client.
10. Verifies that the previous upload session is not reused.
11. Allows normal reconciliation to start a fresh upload.
12. Independently downloads the completed remote object.
13. Verifies that the downloaded SHA-256 exactly matches revision B and does not match revision A.
Existing TC0021 scenarios remain intact, including normal resumable upload/download behaviour and upload-session replacement handling.
## Validation
Full E2E testing completed successfully.
### Business Account Testing
**79** Test Cases Run
**79** Test Cases Passed
**0** Test Cases Failed
### SharePoint Account Testing
**79** Test Cases Run
**79** Test Cases Passed
**0** Test Cases Failed
### Personal Account Testing
**79** Test Cases Run
**79** Test Cases Passed
**0** Test Cases Failed
TC0021, including the new RT-0005 regression scenario, passed successfully.
This change prevents fragments belonging to different local file revisions from being combined through reuse of a persisted resumable upload session.
|
||
|
abraunegg
|
1b95d9dcdd |
Fix Bug #3859: Fix upload session replacement after 404/403 recovery (#3860)
## Summary Fixes upload-session recovery when an existing resumable upload session becomes invalid and Microsoft requires a replacement session to be created. Previously, after a `404 itemNotFound` / `The upload session was not found` response, the client created a replacement upload session and used it for the immediate retry, but the original upload-session state remained authoritative. The next fragment could therefore be sent back to the superseded session URL, causing another failure and, in SharePoint scenarios, a subsequent `409 nameAlreadyExists`. The same stale-session-state problem also affected the existing `403` expired temporary-authentication recovery path. ## Change When a replacement upload session is created, the client now: * adopts the replacement `uploadUrl` * adopts the replacement `expirationDateTime` * adopts the replacement `nextExpectedRanges` * preserves the required client-side upload metadata * recalculates the upload offset from the replacement session * persists the replacement session as a coherent authoritative state * resumes through the normal fragment upload loop This ensures that all subsequent fragments continue using the replacement session rather than returning to the invalid session URL. The recovery also correctly restarts from the offset advertised by the replacement session, which is normally `0-` for a newly-created upload session. ## Validation The failure was reproduced repeatedly against a SharePoint Online document library using a genuine multi-fragment XLSX file. Observed sequence before recovery: ```text Session A Fragment 1 -> 404 "The upload session was not found" Create Session B ``` With the fix applied: ```text Session A Fragment 1 -> 404 Create Session B Adopt Session B Fragment 1 -> Session B Fragment 2 -> Session B Upload completes ``` Repeated manual validation confirmed: * exact SharePoint `404 itemNotFound` condition reproduced * replacement session created and adopted * upload restarted at the replacement session offset * all subsequent fragments remained on the replacement session * no `409 nameAlreadyExists` * no safeBackup created * modified file upload completed successfully * overall synchronisation completed successfully ## E2E coverage TC0021 (`Resumable transfer recovery validation`) has been extended with a new RT-0004 scenario covering modified multi-fragment XLSX upload-session replacement. The test: * generates a valid large XLSX workbook programmatically at runtime * uploads it normally * modifies the resulting tracked workbook in place * performs a second multi-fragment session upload * validates replacement-session adoption when SharePoint returns the upload-session 404 * verifies that subsequent fragment PUTs remain on the replacement session GUID * fails on `nameAlreadyExists`, safeBackup creation, stale upload-session state, or incomplete upload * independently downloads and validates the final modified workbook |
||
|
abraunegg
|
e9c9f9f8ea |
Update docs - usage.md and manpage- #3858s (#3858)
Update docs - usage.md and manpage- #3858 |
||
|
abraunegg
|
ac876dfc1e |
Improve HTTP 404 messaging during file downloads (#3857)
Improve how an HTTP 404 (`itemNotFound`) response from Microsoft OneDrive is presented when an item disappears before it can be downloaded. During normal reconciliation, an item may be queued for download and subsequently be moved or deleted online before the download request is performed. Microsoft Graph then returns HTTP 404 because the previously identified item is no longer available. Previously, `syncEngine.downloadFileItem()` passed this response to the generic Microsoft OneDrive API error formatter, resulting in output such as: ```text id="h1mbz8" ERROR: Microsoft OneDrive API returned an error with the following message: Error Message: HTTP request returned status code 404 (Not Found) Error Reason: The resource could not be found. Error Code: itemNotFound Calling Function: syncEngine.downloadFileItem() ``` While the download attempt has failed, this can be a normal reconciliation race and the generic API error output can incorrectly suggest that user intervention is required. ## Change When `syncEngine.downloadFileItem()` receives HTTP 404, replace the generic Microsoft OneDrive API error block with a concise informational message: ```text id="xnyx8i" The online item is no longer available at the time of download; continuing reconciliation: <path> ``` This is intentionally a presentation-only change. The existing download failure and reconciliation behaviour remains unchanged, including: * The download is still marked as failed. * Existing `fileDownloadFailures` handling remains unchanged. * Existing `syncFailures` handling remains unchanged. * Existing database handling remains unchanged. * Subsequent Microsoft OneDrive delta and reconciliation processing remains unchanged. * Existing handling of all other HTTP errors remains unchanged. ## Scope This change is intentionally limited to HTTP 404 handling within `syncEngine.downloadFileItem()`. It does not globally suppress HTTP 404 responses or alter the application's existing download, database, retry, or reconciliation semantics. |
||
|
abraunegg
|
feae8a0835 |
Quiet expected HTTP 404 errors during timestamp reconciliation (#3856)
Avoid presenting an expected Microsoft OneDrive API HTTP 404 (`itemNotFound`) response as an application error when updating an item's remote modified timestamp. During reconciliation, `uploadLastModifiedTime()` may attempt to update the timestamp of an item that existed when it was evaluated but has subsequently been moved or deleted online by another client. In this situation, Microsoft Graph returns HTTP 404 because the item is no longer available at the original item ID. This is a normal reconciliation race and does not require user intervention. Previously, the 404 was passed to `displayOneDriveErrorMessage()`, producing output such as: ```text ERROR: Microsoft OneDrive API returned an error with the following message: Error Message: HTTP request returned status code 404 (Not Found) Error Reason: The resource could not be found. Error Code: itemNotFound Calling Function: syncEngine.uploadLastModifiedTime() ``` This can incorrectly suggest that the client has encountered an actionable synchronisation error. ## Change When `uploadLastModifiedTime()` receives HTTP 404: * Do not display the generic Microsoft OneDrive API error block. * Treat the timestamp update as no longer applicable. * Emit an informational message only when debug logging is enabled. * Leave the current state for normal reconciliation processing. Handling of HTTP 409, HTTP 412, transient HTTP errors, and all other Microsoft OneDrive API errors remains unchanged. ## Scope This change is intentionally limited to `syncEngine.uploadLastModifiedTime()`. It does not introduce global suppression of HTTP 404 responses and does not alter HTTP 404 handling for file downloads or other Microsoft OneDrive API operations. |
||
|
abraunegg
|
d5ba7a12a2 |
Fix Bug #3769: honour 'check_nosync' / .nosync for directories already in the sync database (#3846)
## Summary This PR fixes #3769 so that a local `.nosync` file continues to act as a local synchronisation exclusion boundary when it is added to a directory that has **already been synchronised and is already represented in the local `items.sqlite3` database**. Previously, `check_nosync` worked correctly when `.nosync` existed before the directory was first synchronised, but adding `.nosync` later could result in incorrect reconciliation behaviour. In particular, content already known to the client could be treated as unwanted or no longer in sync, while newly-created local content beneath the `.nosync` directory could still be discovered and uploaded. The intended behaviour is that `.nosync` is a **local exclusion marker**. It must not be interpreted as a request to remove existing content from Microsoft OneDrive. --- ## Problem Consider an already-synchronised directory: ```text Local Microsoft OneDrive Project/ Project/ ├── existing-a.txt ├── existing-a.txt └── existing-b.txt └── existing-b.txt ``` The directory and its files are already present in the local sync database. If `.nosync` is then added locally: ```text Project/ ├── .nosync ├── existing-a.txt ├── existing-b.txt └── local-only-after-nosync.txt ``` the expected behaviour is: * existing local content remains unchanged; * existing Microsoft OneDrive content remains unchanged by this client; * `.nosync` is not uploaded; * new local content beneath `.nosync` remains local-only; * new or modified online content beneath `.nosync` is not applied locally; * local and online state are intentionally allowed to diverge while `.nosync` exists. Prior to this fix, the application could instead: 1. detect `.nosync` while processing an existing online item; 2. classify that item using the generic `unwanted` path; 3. allow existing database entries beneath the excluded directory to become reconciliation/deletion candidates; 4. remove previously-synchronised local files; and 5. subsequently walk the same local directory and upload newly-created local files despite `.nosync` being present. This meant `.nosync` was not reliably honoured once the directory was already known to the sync database. --- ## Fix The application now treats `.nosync` as a dedicated **local synchronisation boundary**, rather than allowing it to collapse into generic unwanted-item semantics. The updated handling ensures that paths protected by `.nosync` are excluded consistently across the relevant synchronisation paths. ### Online → Local For Microsoft OneDrive items whose resolved local path is protected by `.nosync`: * online items are ignored rather than treated as unwanted; * newer online content is not downloaded over retained local content; * remote-only content is not created locally; * online deletion processing does not remove retained local content; * existing database identity is preserved; * excluded database objects are not interpreted as absent online during reconciliation. ### Local → Online For local paths protected by `.nosync`: * local filesystem traversal stops at the `.nosync` boundary; * directories already represented in the database do not bypass the `.nosync` check; * newly-created local files are not uploaded; * modifications to existing local files are not propagated online; * local deletions beneath the excluded subtree do not become online deletion requests; * directory creation and upload actions re-check the `.nosync` boundary before performing the remote operation. The existing behaviour of other client-side filtering mechanisms such as `skip_dir`, `skip_file` and `sync_list` is unchanged. --- ## E2E regression coverage TC0016 has been expanded from its original pre-existing `.nosync` test into two scenarios. ### NS-0001 — `.nosync` present before initial synchronisation Preserves the existing regression coverage and confirms that: * allowed content synchronises normally; * a directory containing `.nosync` is excluded; * the `.nosync` marker itself is not uploaded; * files beneath the excluded directory are not uploaded. ### NS-0002 — `.nosync` added after the directory is already synchronised This scenario specifically reproduces #3769. The test: 1. creates and synchronises an initial directory tree; 2. establishes independent client database state for the subject client; 3. adds `.nosync` to the already-synchronised directory; 4. creates a new local-only file beneath that directory; 5. uses a separate OneDrive client running in monitor/upload-only mode to: * modify an existing online file; * create a new online-only file; * modify a control file outside the `.nosync` subtree; 6. independently verifies those remote changes before the subject client runs; 7. performs normal reconciliation using the subject client's existing database state; 8. verifies that: * existing local files beneath `.nosync` remain unchanged; * the local-only file remains local-only; * the remote-only file is not downloaded; * the remotely modified file does not overwrite the retained local version; * existing online files remain online; * the local-only file and `.nosync` marker are not uploaded; * the control file outside `.nosync` reconciles normally, proving that remote reconciliation was active. This validates that `.nosync` is selectively honoured as a synchronisation boundary rather than preventing the client from processing unrelated changes. --- ## Validation Focused TC0016 testing was performed across: * Personal Account * Personal Account with 15-character `driveId` * Business Account * SharePoint Account Both TC0016 scenarios passed across all account types after the application fix. The complete E2E suite was then enabled and completed successfully: ### SharePoint Account * **79** Test Cases Run * **79** Test Cases Passed * **0** Test Cases Failed ### Business Account * **79** Test Cases Run * **79** Test Cases Passed * **0** Test Cases Failed ### Personal Account - 15 Character `driveId` * **79** Test Cases Run * **79** Test Cases Passed * **0** Test Cases Failed ### Personal Account * **79** Test Cases Run * **79** Test Cases Passed * **0** Test Cases Failed ### Business Account Shared Folder * **5** Test Cases Run * **5** Test Cases Passed * **0** Test Cases Failed ### Personal Account Shared Folder * **3** Test Cases Run * **3** Test Cases Passed * **0** Test Cases Failed --- ## Result With `check_nosync = "true"`, adding `.nosync` to an already-synchronised directory now means: ```text .nosync | +-- Local -> Microsoft OneDrive: ignore | +-- Microsoft OneDrive -> Local: ignore | +-- Preserve existing local content | +-- Preserve existing online content ``` The local and online copies may intentionally diverge while the local `.nosync` marker is present. This restores the intended semantics of `.nosync`: **exclude the local directory from synchronisation without treating the existing local or online data as deleted or unwanted.** |
||
|
abraunegg
|
6e544bbde5 |
Fix Bug #3835: Directory move reconciliation can trigger repeated HTTP 412 precondition failures (#3849)
This pull request addresses repeated HTTP `412 Precondition Failed` responses observed during sustained multi-client directory move and rename reconciliation.
The failure was caused by the client attempting an online move using state captured earlier in the reconciliation process. Under concurrent activity, another client or Microsoft OneDrive could modify or move the same DriveItem before the request was issued.
This meant the locally stored:
```text
eTag
name
parentId
```
could already be stale when the move request reached Microsoft Graph.
The move would then be submitted with an obsolete precondition and Microsoft correctly rejected it with:
```text
HTTP 412 Precondition Failed
```
Although the previously observed cases recovered successfully, repeatedly submitting stale move requests is unnecessary and could make reconciliation behaviour less deterministic under heavy concurrent activity.
## Change
Immediately before performing an online move, the client now requests the current DriveItem state directly from Microsoft OneDrive using the item's Drive ID and item ID.
The current online state is then validated against the source state on which the local move decision was based.
The move is only attempted when the remote item still represents the expected source object and location:
```text
DriveItem ID matches
name matches
parent ID matches
item has not been deleted
```
If the online state no longer matches the local database source state, the move is deferred and normal reconciliation is allowed to process the newer remote state.
This prevents the client from attempting to move an object based on a source state that is already obsolete.
## Current eTag
Where Microsoft returns an eTag for the refreshed DriveItem, that current online eTag is used for the move request instead of the potentially stale value stored in the local database.
The sequence is therefore now:
```text
detect local move
|
v
load database source state
|
v
request current DriveItem from Microsoft Graph
|
+--> item no longer represents expected source state
| |
| +--> defer move for normal reconciliation
|
v
use current online eTag
|
v
perform online move
```
If Microsoft unexpectedly omits the eTag from the refreshed DriveItem response, the existing database eTag remains available as the fallback.
## Failure handling
Failure to obtain the current DriveItem state does not result in the client proceeding with potentially stale information.
The move is abandoned for that reconciliation pass and the normal synchronisation process is allowed to retry once current online state can be obtained.
Existing Microsoft Graph retry handling for transient responses such as:
```text
408
429
503
504
```
remains unchanged.
## Diagnostic logging
Additional debug logging has been added for cases where:
```text
the current online item no longer matches the expected source state
the current online eTag differs from the database eTag
Microsoft returns no eTag and the database value must be used as fallback
```
These diagnostics are debug-only and do not add noise to normal application output.
## End-to-End Validation
The existing end-to-end test suites have successfully completed against this change, including:
```text
Business Account Testing
79 / 79 passed
SharePoint Account Testing
79 / 79 passed
Personal Account Testing
79 / 79 passed
Personal Account - 15 Character driveId
79 / 79 passed
Business Shared Folder Testing
5 / 5 passed
Personal Shared Folder Testing
3 / 3 passed
```
The branch was subsequently brought forward to the current master baseline, including the completed fix for #3853, before final soak validation.
## Four-Client Business Account Soak
Final validation was performed using:
```text
onedrive v2.5.11-35-g0ec7e96
```
against a Business account with four simultaneous clients:
```text
Ubuntu
Fedora
FreeBSD
OpenBSD
```
Each client operated with its own independent high-churn generator repeatedly performing:
```text
create
modify
move
modify
delete
```
operations.
The soak ran continuously for approximately:
```text
25 hours
```
and produced approximately:
```text
2,386 generated directories
2,368 completed generated directory move transitions
```
across the four clients.
### Issue #3835 result
```text
Directory move / reconciliation HTTP 412 failures: 0
```
No recurrence of the Issue #3835 failure was observed.
One unrelated HTTP `412` occurred during the entire soak while performing a file timestamp-only correction under concurrent activity.
That condition was explicitly handled by the existing timestamp reconciliation path and the client immediately reconverged successfully.
It was not associated with a directory move and was not a recurrence of Issue #3835.
### Additional concurrency behaviour
The soak also generated normal high-concurrency Microsoft Graph conditions including:
```text
404 itemNotFound responses
429 throttling responses
503 transient service responses
objects disappearing while reconciliation was in progress
transient download metadata inconsistencies
```
The clients continued operating and reconciling normally throughout these conditions.
No persistent reconciliation failure or resulting drift associated with directory moves was observed.
## Result
Directory moves are no longer issued solely from potentially stale database state.
The client now validates the current remote DriveItem immediately before the move and uses the latest available Microsoft eTag when constructing the request.
Where concurrent activity has already changed the object or its location, the move is deferred rather than being attempted with obsolete preconditions.
Extended four-client Business-account testing materially exceeded the historical reproduction conditions for Issue #3835 without reproducing the directory move HTTP `412` failure.
|
||
|
abraunegg
|
1a7ffb8269 |
Fix Bug #3853: Concurrent downloads of the same DriveItem race on shared .partial file and can report false download failures (#3855)
This pull request fixes a download concurrency race where multiple parallel download workers targeting the same canonical file could simultaneously operate on the same deterministic:
```text
<canonical-path>.partial
```
staging file.
Under sustained multi-client synchronisation activity this could result in one worker successfully promoting the shared `.partial` file while another worker subsequently attempted to inspect or operate on the same staging pathname and reported:
```text
ERROR: Download failed (file not present after download)
```
More importantly, multiple workers could have the same staging file open at the same time, creating a genuine download-integrity risk rather than merely producing a false failure message.
The issue was reproduced during four-client soak testing across:
```text
Ubuntu
Fedora
FreeBSD
OpenBSD
```
and was observed with both Personal and Business accounts.
## Root Cause
The download lifecycle had gradually moved away from the ownership model used prior to the transactional replacement changes.
The effective download flow had become:
```text
sync.d
|
+--> downloadById(... deferFinalRename=true)
|
+--> onedrive.d / curlEngine.d
|
+--> write <canonical>.partial
|
+--> return without final promotion
|
+--> inspect <canonical>.partial
|
+--> perform integrity validation
|
+--> perform safeBackup / replacement decisions
|
+--> safeRename(
<canonical>.partial,
<canonical>
)
```
This meant that:
* the transport layer created and wrote the `.partial` file;
* `sync.d` later became responsible for inspecting and promoting it;
* multiple download workers could independently believe they owned the same deterministic staging pathname.
If two workers targeted the same canonical file, both could open:
```text
file.data.partial
```
at the same time.
One worker could then successfully rename that object to:
```text
file.data
```
while another worker was still processing the same staging object.
This explains both the false:
```text
file not present after download
```
failure and the more serious possibility of concurrent writes to a staging inode that had already become the canonical file.
## Changes
### Restore download staging ownership to the download subsystem
The `.partial` lifecycle is once again exclusively owned by:
```text
onedrive.d
->
curlEngine.d
```
`sync.d` no longer:
* constructs the `.partial` pathname;
* checks for the existence of `.partial`;
* removes `.partial`;
* renames `.partial` to the canonical pathname.
The `deferFinalRename` download behaviour has been removed.
A successful `downloadById()` now once again means that the download subsystem has completed the full staging transaction and produced the canonical file.
### Serialize downloads targeting the same canonical path
A per-canonical-path download transaction lock has been added inside `onedrive.d`.
This ensures that two parallel reconciliation workers cannot simultaneously:
```text
open
write
resume
inspect
remove
or promote
```
the same deterministic `.partial` staging file.
This protection is deliberately implemented inside the download subsystem rather than in `sync.d`.
This change does **not** deduplicate the download queue.
If duplicate download requests are scheduled, they may still both execute, but they cannot concurrently own the same staging transaction.
Investigation of why duplicate effective download candidates can be scheduled remains separate from this fix.
## Download integrity boundary
The existing download-integrity policy remains in `sync.d`.
Because `onedrive.d` exclusively owns the private staging file, it generates neutral facts describing the completed transfer:
```text
actual file size
streamed QuickXorHash, when available
generated QuickXorHash, when required
generated SHA256 hash, when required
```
Those values are supplied back to `sync.d` for evaluation.
`onedrive.d` does **not** decide whether the downloaded content matches Microsoft Graph metadata.
The responsibility boundary is therefore:
```text
curlEngine.d
|
| HTTP transfer
| streamed QuickXorHash
v
onedrive.d
|
| owns private .partial staging object
| generates required local size/hash facts
v
sync.d
|
| compares downloaded facts with Graph metadata
| performs integrity policy
| performs SharePoint/AIP handling
| performs HEIC data-loss handling
| evaluates --disable-download-validation
| performs safeBackup preservation decisions
v
onedrive.d
|
| accepted download
| final canonical-state recheck
| .partial -> canonical
v
sync.d
|
| authoritative Microsoft timestamp
| database reconciliation
| xattr handling
| monitor expected-effect notification
```
This retains the established distinction between:
```text
onedrive.d:
"What was downloaded?"
sync.d:
"Is what was downloaded valid for this Graph DriveItem?"
```
## Data-preservation behaviour retained
The transactional safeBackup protections introduced by earlier work remain in place.
Before an existing canonical file is replaced:
* `sync.d` evaluates whether the local canonical object contains data that must be preserved;
* any required safeBackup is created before replacement;
* `onedrive.d` snapshots and rechecks the canonical file around the inspection callback;
* if the canonical pathname or its content changes during the download/commit preparation window, the staged replacement is rejected rather than overwriting new local activity.
This retains the important guarantee that a validated online replacement must not silently overwrite unique local data.
## HTTP download safety retained
The HTTP-response protections introduced previously are also retained.
An unsuccessful or non-content-bearing HTTP response cannot be promoted into the canonical pathname.
The download subsystem only reaches canonical promotion after the HTTP transfer has been accepted.
## Existing download validation behaviour retained
The existing `sync.d` behaviour remains responsible for:
* streamed QuickXorHash versus generated QuickXorHash selection;
* SHA256 fallback;
* expected versus actual size comparison;
* expected versus actual hash comparison;
* treating hash mismatch as the definitive integrity failure;
* advisory size-mismatch handling;
* HEIC data-loss diagnostics;
* SharePoint download-validation workaround messaging;
* Azure Information Protection / SharePoint enrichment handling;
* `--disable-download-validation`;
* database handling following a failed download.
The existing explanatory comments documenting these behaviours have also been retained.
## Timestamp and reconciliation behaviour retained
The authoritative timestamp changes remain unchanged.
After successful canonical promotion, `sync.d` applies the Microsoft-provided:
```text
fileSystemInfo.lastModifiedDateTime
```
to the downloaded canonical file.
No fabricated local timestamp is introduced.
The monitor/drift expected-effect handling also remains intact:
```text
downloadById() succeeds
->
canonical file exists
->
downloadTransferCompleted = true
->
notifyExpectedLocalFileDownload(...)
```
## Validation
### End-to-end testing
The earlier implementation of this fix successfully completed the existing E2E suites, including:
```text
Business Account Testing
79 / 79 passed
SharePoint Account Testing
79 / 79 passed
Personal Account Testing
79 / 79 passed
Personal Account - 15 Character driveId
79 / 79 passed
Business Shared Folder Testing
5 / 5 passed
Personal Shared Folder Testing
3 / 3 passed
```
The final code revision also passed the Linux build and smoke-test workflows.
### Four-client soak testing
The final implementation:
```text
v2.5.11-34-g86ebe45
```
was tested simultaneously on:
```text
Ubuntu
Fedora
FreeBSD
OpenBSD
```
using a Business account with:
```text
force_session_upload = "true"
```
and four independent high-churn generators performing repeated:
```text
create
modify
move
modify
delete
```
operations.
The soak ran for approximately:
```text
3 hours 47 minutes
```
and completed approximately:
```text
33,313 successful downloads
```
across the four clients.
Results:
```text
Download failed (file not present after download) = 0
download promotion failures = 0
canonical-change race warnings = 0
canonical verification failures = 0
```
For comparison, the pre-fix control run reproduced the Issue #3853 failure:
```text
~17 minutes:
failure already present on 3 of 4 clients
~33 minutes:
failure present on all 4 clients
~3 hours 16 minutes:
25 "file not present after download" failures
```
The fixed implementation therefore materially exceeded the historical reproduction window without reproducing the race.
A small number of integrity rejections were observed during forced session-upload churn where Microsoft Graph temporarily reported:
```text
size = 0
quickXorHash = AAAAAAAAAAAAAAAAAAAAAAAAAAA=
```
for newly uploaded non-zero files.
In each case the staging download was correctly rejected by `sync.d`, was not promoted by `onedrive.d`, and succeeded on the subsequent retry once current Graph metadata became available.
This also validates the restored integrity-inspection boundary.
## Result
The download transaction is now self-contained again:
```text
request download
->
serialize canonical-path transaction
->
write private .partial
->
validate HTTP response
->
generate staging facts
->
sync.d performs integrity/preservation inspection
->
recheck canonical state
->
promote .partial -> canonical
->
return success
```
The sync layer no longer owns or manipulates the transport staging pathname, and concurrent workers can no longer share ownership of the same deterministic `.partial` transaction.
|
||
Thomas Drake-Brockman
|
368bb572fd |
Add notify_monitor_start option to silence monitor-start notification (#3779)
Addresses discussion #3109 Running in `--monitor` mode sends a GUI notification on every startup confirming that filesystem monitoring has begun. There was previously no way to silence just this notification short of disabling all notifications via `disable_notifications`. * Add `notify_monitor_start` config option (default `true`, preserving existing behaviour) that gates only the monitor-start GUI notification * Document the option in `config`, `docs/application-config-options.md` and `docs/usage.md` To disable just this notification, add this to your cofig: ``` notify_monitor_start = "false" ``` Checked to compile with LDC v1.20.1 (Debian) and LDC 1.42.0 (Fedora). |
||
|
abraunegg
|
c2804ef7db |
Fix Bug #3850: 'skip_dir' full path rule (leading '/') also excludes same-named directories at any depth (#3851)
## Summary Fixes #3850, where a `skip_dir` full-path rule using a leading `/` could incorrectly exclude same-named directories at deeper levels. For example: ```text skip_dir = "/bin" ``` is intended to exclude only: ```text [sync_dir]/bin ``` but could also exclude: ```text [sync_dir]/some/deeper/path/bin ``` when processing remote `/delta` data. ## Root Cause When processing a remote directory, the client can calculate two paths for `skip_dir` evaluation: * a simplified path, which may degrade to only the directory name when `parentReference.name` is not supplied by the Microsoft OneDrive API * a complete sync-root-relative path derived from database state or `parentReference.path` The existing logic evaluated the simplified path first and only evaluated the complete path when the simplified path did not match. For a rule such as: ```text skip_dir = "/bin" ``` a remotely discovered nested directory could therefore be reduced to: ```text bin ``` before matching. The existing path normalisation performed by `isDirNameExcluded()` then generates both relative and rooted matching candidates, allowing the degraded `bin` value to match `/bin` before the actual full path was considered. This also meant that `skip_dir_strict_match = "true"` could not prevent the incorrect exclusion. ## Resolution Where a complete sync-root-relative path is available, the client now treats that path as authoritative for `skip_dir` evaluation. The simplified path is used only when a complete path genuinely cannot be constructed. This change has been applied to both affected remote directory filtering paths in `src/sync.d`. The existing `skip_dir` matching implementation itself is unchanged. This preserves existing behaviour for: * non-strict directory-name matching * strict full-path matching * leading `/` sync-root anchoring * wildcard matching * case-insensitive matching * multiple configured `skip_dir` rules In particular, case-insensitive matching remains intentional and unchanged. ## E2E Test Coverage TC0012 has been significantly expanded to validate `skip_dir` semantics in both upload and download directions. Coverage now includes: * unanchored non-strict directory-name matching * strict explicit multi-segment path matching * leading `/` single-segment sync-root anchoring with strict matching disabled * leading `/` single-segment sync-root anchoring with strict matching enabled * rooted multi-segment paths containing spaces and trailing `/` * `*` and `?` wildcard matching * case-insensitive matching * multiple and pipe-separated `skip_dir` rules * rooted wildcard rules * local-to-remote and remote-to-local filtering behaviour The regression scenarios specifically confirm that a root-anchored directory is excluded while the same directory name at a deeper path remains synchronised. ### TC0061 adjustment The full E2E run also exposed that TC0061 had historically depended on the same simplified-path behaviour corrected by this PR. TC0061 configured: ```text skip_dir = "Pictures/Archive" skip_dir_strict_match = "true" ``` while the actual test path was: ```text [sync_dir]/ZZ_E2E_TC0061_<run>/Pictures/Archive ``` The previous simplified-path-first behaviour caused this testcase to pass because the complete sync-root-relative path was never evaluated. TC0061 has therefore been corrected so its strict `skip_dir` rule describes the actual full path relative to `sync_dir`. No production behaviour was relaxed to accommodate this testcase. ## Validation Validation completed successfully: * Original Issue #3850 behaviour reproduced before the fix * Expanded TC0012 regression coverage passes all scenarios * Corrected TC0061 passes using the complete strict path * Full 79-case E2E suite passes * Original issue reporter has independently validated the fix against the environment that reproduced the defect This preserves the documented distinction between directory-name matching and explicit full-path matching while ensuring remote directory filtering uses the most authoritative path information available. |
||
|
abraunegg
|
9e074dd310 |
Implement FR #3787: Use different browser with --reauth and friends (#3848)
## Overview This pull request implements the feature request raised in #3787 to allow a user to explicitly select which browser is used for interactive Microsoft authentication. The current GUI authentication path launches the Microsoft authorisation URL using `xdg-open`. This means the desktop environment's configured default browser is normally selected. In environments where a user intentionally uses a different browser for Microsoft 365 authentication, setting the standard `BROWSER` environment variable does not reliably override that desktop preference because `xdg-open` may successfully select the desktop-configured browser before consulting `BROWSER`. For example, a user may use Firefox as their normal desktop browser while using Microsoft Edge exclusively for Microsoft 365 authentication and SSO. This change allows that user to run: ```bash BROWSER=/usr/bin/microsoft-edge-stable onedrive --reauth ``` and have the Microsoft authentication URL opened directly using Microsoft Edge. ## Changes The browser launch logic in `src/localAuth.d` has been updated so that: 1. If the `BROWSER` environment variable is set, the client first attempts to launch the configured executable directly with the Microsoft authorisation URL. 2. If the configured browser cannot be launched, the failure is written to the debug log and the client falls back to the existing `xdg-open` behaviour. 3. If `BROWSER` is not set, behaviour remains unchanged and `xdg-open` is used as before. 4. If browser launch ultimately fails, the existing authentication fallback behaviour remains unchanged. Both an explicit executable path: ```bash BROWSER=/usr/bin/microsoft-edge-stable onedrive --reauth ``` and an executable available through `PATH`: ```bash BROWSER=microsoft-edge-stable onedrive --reauth ``` are supported. ## Design Considerations This change deliberately does not attempt to automatically detect Microsoft Edge or introduce a Microsoft Edge-specific configuration option. `BROWSER` is treated as a user-selected browser executable, making the implementation browser-neutral and allowing the same mechanism to be used with Firefox, Chromium, Chrome, Edge or another browser. The value is passed directly to `spawnProcess()` as an executable rather than being executed through a shell. This keeps the authentication path deterministic and avoids introducing shell command parsing into OAuth browser launching. The existing behaviour remains the fallback path, so an invalid or stale `BROWSER` value does not prevent authentication from proceeding through the normal desktop browser. ## Behaviour With no override configured: ```bash unset BROWSER onedrive --reauth ``` the existing desktop/default-browser behaviour is retained. With an explicit browser configured: ```bash BROWSER=/usr/bin/microsoft-edge-stable onedrive --reauth ``` the selected browser is used for Microsoft authentication. If the configured browser does not exist or cannot be started: ```bash BROWSER=/does/not/exist onedrive --reauth ``` the client falls back to `xdg-open`. ## Scope This is intentionally a laser-focused change. There are no changes to: * OAuth request or response handling * local loopback authentication * GUI session detection * authentication callback processing * device authentication * headless or SSH authentication behaviour * application configuration options * Microsoft Edge detection * the existing manual authentication fallback Only the browser-selection step of the existing GUI authentication workflow is changed. |
||
|
abraunegg
|
1188caab0f |
Fix Bug #3784: Handle local Recycle Bin move failures without deleting the database record (#3845)
When processing an online deletion with `use_recycle_bin=true`, the client previously removed the item's database record **before** attempting to move the corresponding local file or directory into the configured Recycle Bin. If the subsequent `rename()` failed — for example when `sync_dir` and `recycle_bin_path` were located on different filesystems and `rename()` returned `EXDEV` — the local item remained inside `sync_dir` but its database identity had already been removed. A subsequent local scan could then classify the surviving object as new content and upload it again, effectively recreating content that had already been deleted online. This change makes the local Recycle Bin operation and database update behave transactionally from the client's perspective. ## Changes * Change `movePathToRecycleBin()` to return an explicit success/failure result. * Only remove the OneDrive database identity after the local Recycle Bin move succeeds. * Retain the database identity when the Recycle Bin move fails. * Prevent `.trashinfo` metadata from being created when the corresponding filesystem move did not succeed. * Record failed Recycle Bin moves as synchronisation failures and report the affected paths. * Preserve the previous `/delta` checkpoint after a failed move so the online deletion remains eligible for subsequent reconciliation. * Stop processing the current deletion batch after a failed parent move so descendant objects and their database identities remain intact. * Preserve the existing successful Recycle Bin and hard-delete behaviour. ## E2E validation TC0057 has been extended with: `S05_online_delete_recycle_bin_move_failure` The scenario uses the existing multi-client E2E architecture: 1. Seed a directory and nested file using the primary client. 2. Clone the established client configuration/database state and corresponding local tree. 3. Run the cloned client using `--monitor --upload-only`. 4. Delete the populated directory locally on the monitor client, producing a genuine online deletion while leaving the primary client on its previous state. 5. Configure the primary client's Recycle Bin on `/dev/shm` and verify that `sync_dir` and the Recycle Bin reside on different filesystem device IDs. 6. Process the online deletion, forcing the local `rename()` operation to fail with the cross-filesystem failure condition. 7. Verify that: * the local directory and nested file remain present; * no Recycle Bin payload is created; * no misleading `.trashinfo` entry is created; * the Recycle Bin failure is reported. 8. Run the primary client again with uploads enabled. 9. Use an independent client to verify that the retained local object was **not re-uploaded to OneDrive**. The new TC0057 scenario is passing. This directly validates the failure mode reported in #3784 and protects against the deleted-online → failed-local-trash → unintended re-upload sequence. |
||
|
abraunegg
|
bec9b20c0c |
Fix Bug #3806: Complete --display-sync-status redesign with documentation and E2E regression coverage (#3844)
This PR completes the redesign of `--display-sync-status` to provide a meaningful, read-only assessment of whether the configured OneDrive sync scope is currently in sync. Previously, `--display-sync-status` was primarily based on pending Microsoft Graph `/delta` data. This meant it could identify some pending remote changes, but it did not provide a complete picture of synchronisation state and could miss important conditions such as local-only changes, zero-byte files, directory changes, moves or renames, and deletion tombstones. The redesigned implementation now evaluates synchronisation state in **both directions**: ```text Microsoft OneDrive -> Local filesystem Local filesystem -> Microsoft OneDrive ``` and reports an overall result of: * `IN SYNC` * `NOT IN SYNC` * `INDETERMINATE` This provides a safe way for users, monitoring systems and diagnostic workflows to answer: > Is this client currently in sync? without performing a sync or destroying/rebuilding application state. --- ## What has changed ### Remote state assessment The Microsoft OneDrive → Local filesystem assessment now identifies pending remote changes including: * new files * new directories * modified files * moved or renamed items * deleted items * zero-byte files * approximate download volume Microsoft Graph deletion tombstones are explicitly recognised and handled rather than being incorrectly processed as malformed live objects. The existing stored delta cursor is used only as the starting point for the assessment and is **not advanced or modified** by `--display-sync-status`. ### Local state assessment The Local filesystem → Microsoft OneDrive assessment now compares the configured local filesystem against the client's stored sync database and identifies: * new local files * new local directories * modified local files * timestamp-only differences * locally deleted or missing items * file/directory type changes * files requiring upload * directories requiring creation online * remote deletions that would be requested * approximate upload volume The implementation follows the same database hierarchy and relative-path semantics used by normal synchronisation processing. ### Overall sync state The command now produces a clear overall state. `IN SYNC` means that no pending local or remote changes were detected for the configured sync scope. `NOT IN SYNC` means that one or more pending local or remote changes were positively identified. `INDETERMINATE` is used where the command cannot safely prove that the configured scope is fully assessed. The implementation deliberately prefers `INDETERMINATE` over incorrectly reporting `IN SYNC` where remote traversal completeness cannot be guaranteed. --- ## Read-only behaviour `--display-sync-status` remains a query-only operation. The command does **not**: * upload files * download files * create or remove local files or directories * modify the live sync database * advance stored Microsoft Graph delta cursors * perform reconciliation The command operates against the existing copied/read-only application state used for no-sync operations. The command also now establishes the configured `sync_dir` as its working directory before local filesystem assessment, consistent with normal sync and monitor operation. The existing `.nosync` / `--check-nomount` protection is also applied so that an unavailable mounted filesystem cannot be incorrectly interpreted as mass local deletion. --- ## Scope and filtering The status assessment honours the configured sync scope and applicable client-side filtering, including: * `--single-directory` * `sync_list` * `skip_file` * `skip_dir` * `skip_dotfiles` * `skip_size` * root-file handling * upload-only and download-only operating modes * `--no-remote-delete` Where configured operation requires a remote traversal mechanism that cannot be safely reproduced by the read-only status query, the result is reported as `INDETERMINATE` rather than returning a false clean result. --- ## Transfer estimates Pending upload and download data is now displayed using a human-readable value while retaining the exact byte count. For example: ```text Approximate download data: 14.15 MB (14838177 bytes) Approximate upload data: 2.09 KB (2144 bytes) ``` Values smaller than 1 KiB are displayed directly in bytes: ```text Approximate upload data: 90 bytes ``` This avoids misleading output such as `0 KB` for small but non-zero files while retaining deterministic values useful for diagnostics and testing. --- ## Example output A clean client now reports: ```text Synchronization status for the configured sync scope: Microsoft OneDrive -> Local filesystem No pending remote changes detected. Local filesystem -> Microsoft OneDrive No pending local changes detected. Overall status: IN SYNC No pending local or remote changes were detected for the configured sync scope. ``` A client with changes in both directions may report: ```text Microsoft OneDrive -> Local filesystem Pending remote items: 1 New files: 1 Files requiring download: 1 Local filesystem -> Microsoft OneDrive Pending local items: 1 Modified local files: 1 Files requiring upload: 1 Overall status: NOT IN SYNC ``` When remote changes are also pending, locally inferred actions are reported as the current point-in-time assessment and may subsequently be reconciled or superseded when normal synchronisation processing occurs. --- ## End-to-end regression coverage A new **TC0079** end-to-end test case has been added for the redesigned `--display-sync-status` behaviour. TC0079 exercises the status command through multiple lifecycle states, including: * established clean state → `IN SYNC` * deterministic incomplete assessment → `INDETERMINATE` * local-only pending changes * same-size local content modification * timestamp-only local differences * new local files and directories * local deletion * filtered local content * remote-only pending changes * remote deletion tombstones * remote rename * remote modification * zero-byte remote files * new remote files and directories * filtered remote content * mixed local and remote pending changes * repeated status queries proving that the command does not consume or advance the live delta cursor TC0079 is registered in the standard E2E suite and therefore runs across the supported Personal, Personal 15-character `driveId`, Business and SharePoint account workflows. --- ## Documentation Documentation has been updated to describe the redesigned capability and its intended use. Updates include: * `docs/usage.md` * explains why `--display-sync-status` should be used when a user wants to determine sync health without performing a sync or `--resync` * documents the bidirectional assessment and overall states * `docs/application-config-options.md` * expands the CLI option documentation and behaviour * `docs/end_to_end_testing.md` * documents TC0079 and its coverage The documentation also makes clear that `NOT IN SYNC` is a reported synchronisation state rather than a command execution failure; consumers should evaluate the reported `Overall status` rather than treating the process exit code as the synchronisation result. --- ## Result This change turns `--display-sync-status` from a limited remote-change check into a substantially more useful synchronisation-health diagnostic while preserving the most important property of the command: **it determines sync state without performing a sync or modifying the client's existing synchronisation state.** |
||
James C. Owens
|
6285f62fe4 |
Fix Bug #3822: Preserve the online copy when a local rename produces a name that cannot be used online (#3833)
* Preserve the online copy when a local rename produces an unusable name uploadMoveItem() used a single 'unwanted' flag for two different rejections. A new path filtered out by the user's configuration is genuinely no longer wanted online, but a new path that violates the Microsoft naming rules or is not validly encoded has not left the sync scope at all. It is still inside sync_dir, it just cannot be named online. Both cases fell through to the same else branch, which removes the online copy. For an unusable name that discards the only synced copy of data that still exists locally, and no later scan can restore it because the local name is still invalid. Track the two reasons separately. An unusable name now skips the move and leaves the online copy and its database entry untouched, so the data stays protected until the name is corrected locally. Also load the database record for the old path before removing it online in the genuinely out of scope case. 'oldItem' is only populated by the successful move path, so that delete was previously issued with a default constructed Item and could not identify the item to remove. * Stop tracking the old path when a rename produces an unusable name Preserving the online copy at the point the rename is handled is not enough on its own. The database still records the item at its old path, and nothing exists at that path any more, so processLocalDatabaseItem() reads it as a local deletion on the next sync pass and removes the online copy after all. Remove the database record for the old path instead, along with the records for any children if the item was a directory, since those describe paths that have also gone. The online data is left untouched and is picked up again by a full enumeration. * Add e2e test case tc0077: rename a synced file to an unusable name tc0025 covers files created with a name the Microsoft rules reject, and the existing rename cases cover renames between valid names. Nothing covers the intersection: a file already present online being renamed locally to a name that cannot be represented online. In that situation the item has not left the sync scope. It is still inside sync_dir, it simply cannot be named online, so the online copy is the only synced copy of data that still exists locally. No later scan can restore it if it is removed, because the local name remains unusable. The test asserts that no remote delete is attempted, and separately that the online copy still exists afterwards. The second assertion holds even with the misclassification present, because oldItem is only populated inside the !unwanted branch and the resulting delete is issued against a default constructed Item that the API rejects. It is asserted so that correcting that in isolation cannot silently turn this into data loss. * Remove trailing whitespace introduced by the rename fix Two blank lines inside the new branches were tab indented, and one pre-existing blank line had been given trailing tabs. src/sync.d has 2141 blank lines and only four of them carry whitespace, so these were anomalies rather than house style. * Adopt the reworked fix and test case from PR #3843 Replaces the two-boolean classification with an explicit LocalMoveTargetDisposition enum, so an unhandled disposition fails closed rather than falling through to the removal branch. Stale database detachment moves into detachUnrepresentableMoveFromDatabase(), which relies on the item table's ON DELETE CASCADE for the tracked subtree, handles shared-folder shortcuts and orphaned shared-drive roots, verifies each removal and refuses to continue if the safe state cannot be established. Suppresses the naming-validation helper's own notification for this path so the move emits exactly one terminal notification, after the database has reached the preservation state. The test case gains a seeded baseline verified by manifest and content, a directory rename alongside the file rename, and reconciliation plus convergence passes before the final independent verification. * Document e2e test case tc0078 in end_to_end_testing.md Describes the coverage gap between tc0025 (items created with an invalid name) and tc0044/tc0046 (renames between valid names), and lists the file rename, directory rename, reconciliation and convergence phases separately so the subtree detachment and the deferred-deletion path are both visible. --------- Co-authored-by: James Owens <jamesowens001@gmail.com> Co-authored-by: abraunegg <alex.braunegg@gmail.com> |
||
|
abraunegg
|
ba2fe60098 |
Fix Bug #3834: Fix personal account timestamp assignment (#3839)
## Summary This PR completes the timestamp-assignment correction work associated with #3834. The change ensures that timestamp authority is handled correctly after uploads, with different behaviour depending on the upload mechanism: - **Simple uploads:** Microsoft assigns the resulting `fileSystemInfo.lastModifiedDateTime`, so the timestamp returned by Microsoft becomes authoritative for the local object after a successful upload. - **Personal account session uploads:** the originating local filesystem timestamp is already supplied when the upload session is created, so the completed upload response can be saved directly without issuing a redundant timestamp PATCH. - **Business and SharePoint session-upload handling remains unchanged**, including the existing handling for Microsoft SharePoint enrichment behaviour. - **`--upload-only` semantics are preserved:** the local filesystem object is not modified after upload. ## Problem Simple uploads do not carry the originating local filesystem timestamp in the same way as upload-session uploads. After a successful simple upload, Microsoft assigns the resulting `fileSystemInfo.lastModifiedDateTime`. Continuing to treat the pre-upload local timestamp as authoritative can leave the local filesystem, local database and Microsoft OneDrive metadata describing different timestamps for the same object. For Personal account session uploads, the inverse applies: the local filesystem timestamp has already been supplied as part of the upload-session workflow. Performing an additional timestamp PATCH after a successful session upload is therefore unnecessary and can alter metadata that is already correct. These differences need to be handled explicitly according to the upload mechanism actually used. ## Changes ### Simple uploads Track whether an upload actually used the simple-upload path. Following a successful upload and successful integrity validation: - save the upload response returned by Microsoft; - use the Microsoft-returned `fileSystemInfo.lastModifiedDateTime` as the authoritative timestamp for the local object; - update the local filesystem timestamp accordingly during normal bidirectional operation; - do not modify the local filesystem timestamp when operating with `--upload-only`. This handling applies consistently to both new-file and modified-file simple uploads. ### Personal account session uploads For successful Personal account upload-session operations: - retain the filesystem timestamp supplied when the upload session was created; - save the completed upload response directly; - do not perform the previous additional timestamp PATCH. Existing Personal account integrity-failure handling is retained. ### Business / SharePoint Existing Business and SharePoint session-upload behaviour is intentionally retained. This includes the existing handling for files modified by Microsoft's SharePoint enrichment behaviour described in: https://github.com/OneDrive/onedrive-api-docs/issues/935 The purpose of this PR is not to redesign that handling. ## Behavioural outcome The intended timestamp authority model is now: ```text Simple upload local file | v upload | v Microsoft assigns fileSystemInfo.lastModifiedDateTime | v Microsoft timestamp becomes authoritative | v local filesystem + database reconcile to that value ``` For Personal session uploads: ```text Upload session local filesystem timestamp | v supplied during upload-session creation | v upload completes | v returned metadata saved directly | v no redundant timestamp PATCH ``` This prevents timestamp disagreement after upload from causing an unchanged object to subsequently appear modified and avoids timestamp-driven upload feedback. ## Validation ### GitHub Actions E2E Full E2E testing completed successfully on the final PR branch. #### SharePoint Account - **77** Test Cases Run - **77** Test Cases Passed - **0** Test Cases Failed #### Business Account - **77** Test Cases Run - **77** Test Cases Passed - **0** Test Cases Failed #### Personal Account - 15 Character `driveId` - **77** Test Cases Run - **77** Test Cases Passed - **0** Test Cases Failed #### Personal Account - **77** Test Cases Run - **77** Test Cases Passed - **0** Test Cases Failed #### Business Account Shared Folder - **5** Test Cases Run - **5** Test Cases Passed - **0** Test Cases Failed #### Personal Account Shared Folder - **3** Test Cases Run - **3** Test Cases Passed - **0** Test Cases Failed The existing E2E suite contains targeted timestamp and upload-path validation, including: - `--local-first --upload-only` timestamp preservation; - direct/simple-upload mtime handling; - automatic upload-session mtime handling; - forced upload-session mtime handling; - mtime-only local changes; - monitor-mode timestamp stability; - new and modified file uploads; - upload-session workflows; - atomic-save and completed-file hand-off workflows. ### Focused Personal Account Validation Additional manual validation was performed using: ```text onedrive v2.5.11-28-g3fcc159 ``` The following normal bidirectional monitor workflows were exercised: | Scenario | Result | | --- | --- | | New file -> simple upload | PASS | | Modified file -> simple upload | PASS | | New file -> upload session | PASS | | Modified file -> upload session | PASS | For every tested workflow: - the expected upload occurred once; - the resulting Microsoft metadata was processed; - subsequent reconciliation reported the file as unchanged; - no timestamp-driven repeat upload occurred. Typical post-upload reconciliation: ```text Processing: <filename> The file has not changed ``` followed by an empty local upload scan and no further upload of the object. ## Scope This PR is deliberately limited to correcting timestamp assignment and post-upload timestamp authority. It does not attempt to redesign Business or SharePoint enrichment handling or introduce unrelated synchronization behaviour changes. |
||
|
abraunegg
|
84e49d8720 |
Fix Bug #3840: Monitor mode can miss completed files created via temporary-file hard-link finalisation on Linux (#3841)
On Linux, some applications can finalise a completed temporary file without generating the normal `IN_MOVED_FROM` / `IN_MOVED_TO` sequence expected for a rename. This was reproduced with WinSCP, where the kernel instead reports: ```text IN_CREATE <temporary file> IN_CLOSE_WRITE <temporary file> IN_CREATE <final file> IN_DELETE <temporary file> ``` The existing Linux monitor logic intentionally does not treat a regular-file `IN_CREATE` event as immediately actionable, as the file may still be actively written. As a result, the `IN_CLOSE_WRITE` event makes only the temporary pathname actionable, that pending change is subsequently cancelled when the temporary pathname is deleted, and no actionable event remains for the surviving final pathname. The final file is therefore not immediately uploaded and is only discovered by the next normal monitor synchronisation, potentially delaying the upload by up to the configured `monitor_interval`. This PR adds a deliberately narrow, Linux-only, capture-local state machine to recognise this specific completed-file hand-off sequence. When the complete sequence is observed on the same inotify watch: ```text CREATE source CLOSE_WRITE source CREATE destination DELETE source ``` the destination pathname inherits the already-observed write-completion state and is queued as an actionable local change. The implementation deliberately: * does **not** make generic Linux `IN_CREATE` events actionable; * does **not** special-case WinSCP or the `.filepart` filename suffix; * does **not** reinterpret genuine `IN_MOVED_FROM` / `IN_MOVED_TO` events; * abandons the hand-off candidate when the sequence becomes incomplete or ambiguous; * preserves expected-event suppression and existing move/delete coalescing; * keeps the state local to a single inotify capture/drain; * leaves FreeBSD and OpenBSD monitor behaviour unchanged; * does not modify the existing monitor drift/reconciliation handling; * does not modify `sync.d` or any remote synchronisation logic. Manual reproduction using the original WinSCP workflow confirms that the final file is now detected and uploaded immediately through the monitor event path, without requiring a manual `touch` or waiting for the next 300-second synchronisation cycle. Subsequent monitor cycles recognise the uploaded file as unchanged, with no duplicate upload or reconciliation loop. |
||
|
abraunegg
|
305ec0839f |
Fix Bug #3816: Fix OAuth token exposure in --debug-https logging (#3838)
Prevent `--debug-https` from exposing OAuth access and refresh tokens in diagnostic output. The existing HTTPS debug path filtered the `Authorization` header from the application's request-header dump, but OAuth credentials could still be exposed through several other paths: * libcurl verbose output could print the `Authorization` header directly * OAuth token request bodies could expose `refresh_token` * OAuth token endpoint responses could expose both `access_token` and `refresh_token` * exception and response diagnostics could stringify these values unchanged Add a central sanitisation layer in `curlEngine.d` that redacts OAuth credentials only when producing diagnostic output. The sanitiser covers: * `Authorization` and `Proxy-Authorization` headers * HTTP/2-style libcurl `[authorization: ...]` trace output * form-encoded `access_token` and `refresh_token` values * JSON `access_token` and `refresh_token` fields Replace the default libcurl verbose handler with a sanitised debug callback so useful HTTPS connection, TLS, protocol and header diagnostics remain available without exposing credentials. Raw HTTP/TLS payload callbacks are not emitted by the libcurl debug callback. Application-level request and response diagnostics remain available through `CurlResponse`, with OAuth secrets redacted before logging. Add regression tests covering HTTP/1.1 headers, HTTP/2 diagnostic formatting, form-encoded refresh tokens and OAuth JSON responses. This change does not alter OAuth authentication, token renewal, token storage, Graph API behaviour or the explicit `--print-access-token` functionality. |