mirror of
https://github.com/abraunegg/onedrive.git
synced 2026-10-24 15:19:18 +03:00
master
455 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
abraunegg
|
245bbaab36 |
Clarify Microsoft object sync limits (#3924)
Clarify Microsoft object sync limits |
||
mali
|
86272b1ba1 |
Escape regex metacharacters in 'skip_file' and 'skip_dir' patterns (#3922)
* Escape regex metacharacters in `skip_file` and `skip_dir` patterns
Previously, `wild2regex()` only escaped `.`, `+`, `/`, `(` and `)`. The characters `[`, `]`, `{`, `}`, `^` and `$` are valid in OneDrive file and directory names but were passed directly to the regex engine.
As a result, a rule such as `Photos [2023]` was interpreted as a character class, incorrectly matching `Photos 2` instead of the intended directory. Similarly, patterns such as `~$*` failed to match correctly, while unmatched `[` or `{` characters could trigger a `RegexException` during configuration processing.
Update `wild2regex()` to escape these characters correctly while preserving existing `*` and `?` wildcard behaviour.
Also correct sync-root-relative path handling for `skip_file` exclusions so that explicitly anchored file rules are consistently evaluated during local synchronisation.
* Expand E2E filtering validation in TC0011 and TC0012
Significantly expand the existing E2E filtering testcases to validate literal-character matching, wildcard behaviour, path anchoring and configuration safety across real OneDrive synchronisation operations.
- **TC0011 (`skip_file`):** Expand to 32 scenarios covering literal regex metacharacters, unmatched brackets and braces, wildcard combinations, case-insensitive matching, filename and sync-root-anchored path exclusions, configuration-change resync requirements, remote-content preservation and invalid-rule rejection.
- **TC0012 (`skip_dir`):** Expand to 38 scenarios covering literal regex metacharacters, strict and non-strict directory matching, sync-root-anchored paths, multi-component directory exclusions, wildcard combinations, configuration-change resync requirements, remote-content preservation and invalid-rule rejection.
- Add `sync_list` interaction scenarios that verify valid inclusion rules are not incorrectly classified as shadowed, while genuinely conflicting inclusion rules are rejected with explicit diagnostics.
- Validate filtering behaviour in both upload and download directions using independent client instances and remote-content verification.
- Update the configuration, usage and E2E documentation to reflect the corrected matching behaviour and expanded regression coverage.
The expanded filtering suites have been validated successfully against OneDrive Personal, Personal with a 15-character drive ID, OneDrive for Business and SharePoint configurations.
The filtering E2E coverage now comprises **70 scenarios per configuration**, with all 280 scenario executions passing.
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
|
||
|
abraunegg
|
2bf2f9f4d1 |
Clarify systemd usage (#3915)
Clarify systemd usage |
||
|
mali
|
da93547e99 |
Validate OAuth state on the local browser authentication callback (#3911)
* Validate OAuth state on the local browser authentication callback The loopback listener accepted the first authorisation code received on 127.0.0.1 without any state validation, so a forged request from a local process or web page during the authorisation window could sign the client in to an attacker-chosen account. Send a random 'state' value in the authorisation request and reject a callback that does not return it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Add urandom to the spelling allow list Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Validate OAuth state inside the local authentication listener Rework following review feedback: - Move secure random generation to util.d as generateCryptographicallySecureHexString(), reading /dev/urandom with no fallback to a non-cryptographic generator. - Generate the state only after a callback port has been found. - Validate the state inside the listener so that an invalid request is rejected (HTTP 400) and the listener keeps waiting for the genuine redirect within the existing overall timeout. - Reject missing, empty, incorrect or duplicate state, duplicate or ambiguous parameters, malformed percent-encoding and unexpected paths. - Bound each connection with a timeout so stalled connections cannot block. - Remove the inline unittest block. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Bound each callback request by a monotonic deadline and accept origin-form targets only - readRequestLine() now uses an absolute deadline (per-connection limit, capped by the overall authentication deadline) instead of a per-receive timeout, so a slow client cannot hold the listener. - Absolute-form request targets are rejected instead of being partially parsed; only the origin-form target a browser sends is accepted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Muhammet Ali Aydın <nlu@outlook.com.tr> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: abraunegg <alex.braunegg@gmail.com> |
||
|
abraunegg
|
7c9a1c8ebf |
Update Microsoft Intune validation (#3910)
Update Microsoft Intune validation |
||
|
abraunegg
|
47c9a3cabb |
Update sponsorship details (#3907)
Update sponsorship details |
||
|
abraunegg
|
c5fcb8e940 |
Refresh Docker build tooling and container documentation (#3906)
* Refresh Docker build tooling and container documentation ## Summary Refresh the Docker container build tooling and align the Docker and Podman documentation with the container bases already used by `master`. ## Changes - Update the Go builder used for gosu: - `1.25` → `1.27` - Update gosu across all container variants: - `1.17` → `1.19` - Parameterise the gosu version in the Debian Dockerfile. - Update Docker and Podman documentation to reflect: - Fedora 44 - Alpine Linux 3.24 - Debian 13 / Trixie * Update Dockerfile-debian |
||
|
abraunegg
|
98752fd185 |
Align documentation, CLI help and shell completions with current client behaviour (#3903)
Align the user-facing documentation, CLI help, sample configuration, man page and shell completion definitions with the behaviour currently implemented in `master`. This is a documentation and interface-consistency update only. It does not change synchronisation behaviour. ## Changes - Add `monitor_status` to the supplied example `config`, reflecting the existing default-enabled monitor runtime status functionality. - Update `monitor_fullscan_frequency` wording to consistently describe the **online full-scan true-up** cadence rather than implying a local filesystem scan. - Update the `--display-sync-status` summary text in: - application CLI help - `docs/usage.md` - the man page The wording now reflects the current read-only, bidirectional synchronisation-state assessment rather than the previous remote-only pending-change behaviour. - Update HTTP 429 documentation to reflect current logging behaviour: - successfully handled throttling is quiet during normal operation; - detailed 429 handling and retry information remains available in verbose/debug modes; - genuine unrecoverable failures continue to be reported. - Add the missing `--disable-time-check` option to Bash, Fish and Zsh shell completions. - Correct several shell-completion descriptions so they match the current CLI semantics, including: - `--monitor-fullscan-frequency` - `--get-file-link` - `--get-sharepoint-drive-id` ## Validation The supported command-line option set was compared directly against the options implemented in `src/config.d`. After these changes: - Bash completion exposes the complete current long-option set. - Fish completion exposes the complete current long-option set. - Zsh completion exposes the complete current long-option set. - All three completion implementations contain the same 68 supported long CLI options as the application. - The man page contains the complete current CLI option set. - Duplicate stale `--display-sync-status` wording has been removed. - Documentation describing HTTP 429 handling now matches the behaviour implemented in `master`. |
||
|
abraunegg
|
c4a1a41a0f |
Uplift E2E Testing - Phase 3 (#3902)
Uplift E2E Testing - Phase 3 |
||
|
abraunegg
|
977837deed |
Uplift E2E Testing - Phase 2 (#3889)
* Add PDF Uplift to E2E Harness * Add PNG/JPG Uplift to E2E Harness * Update E2E Document based on Phase 2 Uplift |
||
|
abraunegg
|
c703039af9 |
Update compiler details and associated documentation (#3899)
* Update compiler details and associated documentation |
||
|
abraunegg
|
6415cb4e56 |
Add monitor runtime status JSON for external observability (#3896)
This PR adds a new machine-readable runtime status capability for long-running `--monitor` operation.
When the client is running in monitor mode, it now maintains:
```text
<configuration directory>/monitor-status.json
```
alongside the existing configuration and application state files.
The status file provides a lightweight external view of the running monitor without requiring log parsing, additional Microsoft Graph requests, database inspection or execution of `--display-sync-status`.
### Runtime information exposed
The JSON status includes:
- application PID and per-process instance identifier
- configured operating mode
- configured monitor interval
- current monitor state
- application runtime
- monitor loop count
- current cycle start time
- last completed loop time
- whether the current process has completed its first successful synchronisation
- successful synchronisation cycle count
- last successful synchronisation time
- result and duration of the most recent cycle
- failed download count
- failed upload count
- failed Recycle Bin operation count
- Microsoft OneDrive service reachability
- current system-time validation state
All timestamps are emitted in UTC using an ISO-8601/RFC-3339-style representation so that monitoring systems can consume them consistently and convert them to local time when presenting them to users.
### Behaviour
Monitor runtime status is enabled by default:
```text
monitor_status = "true"
```
and may be disabled explicitly:
```text
monitor_status = "false"
```
The status file is:
- written with restrictive permissions
- updated using temporary-file + atomic rename semantics
- updated only at meaningful monitor lifecycle points rather than for individual file operations or notification events
- removed during a clean application shutdown
An abnormal termination such as `SIGKILL` or system power loss may leave the last status snapshot behind. A new monitor process immediately replaces any previous state with a new process instance and resets initial synchronisation readiness.
Status publication is deliberately non-fatal. Failure to write the runtime status file does not alter or interrupt normal synchronisation behaviour.
### Purpose
This provides a stable visibility layer for external tooling such as:
- Docker / Podman health and readiness checks
- systemd monitoring
- shell automation
- monitoring and observability platforms
- future graphical or management interfaces
In particular, consumers can determine whether a newly started monitor process has completed at least one successful synchronisation cycle without parsing normal application log output.
### Relationship to `--display-sync-status`
This capability does **not** replace or change the semantics of `--display-sync-status`.
The two interfaces answer different questions:
```text
monitor-status.json
-> What is the running monitor doing, and did its synchronisation
processing complete successfully?
--display-sync-status
-> Are the local filesystem and Microsoft OneDrive currently
converged?
```
This distinction is important for one-way operating modes.
For example, a `--download-only` monitor may successfully complete all permitted download processing while local-only files remain present.
In that situation:
```text
monitor-status.json
-> last cycle successful
--display-sync-status
-> NOT IN SYNC
```
Both results are correct: the configured monitor operation completed successfully, while the local and remote data states are not identical.
### I/O impact
The runtime status file is intentionally small and is not used as a high-frequency heartbeat.
It is updated during monitor lifecycle/state transitions such as:
- monitor startup
- synchronisation cycle start
- synchronisation cycle completion
- relevant condition changes
- orderly shutdown cleanup
It is not rewritten for individual uploads, downloads, inotify events or WebSocket notifications.
This keeps the additional filesystem I/O negligible relative to the database, filesystem and network activity already performed by normal monitor operation.
### Documentation
Documentation has been updated to cover:
- the new `monitor_status` configuration option
- runtime status file location and lifecycle
- JSON fields and timestamp conventions
- Docker and Podman observability use
- relationship to `--display-sync-status`
- application architecture integration
|
||
|
abraunegg
|
5691477a72 |
Fix Bug #3885: Reduce Full Scan database I/O and improve monitor-loop memory reclamation (#3888)
This pull request addresses excessive database I/O and process memory growth observed during large-account Full Scan and generated `/children` enumeration workloads. Testing and profiling were performed primarily against a fixed 101,494-object OneDrive dataset so that changes in database activity, allocation volume and retained process memory could be compared directly between implementations. ## Native Full Scan reconciliation Native tokenless `/delta` Full Scan True Up no longer uses the database `syncStatus` field as a temporary mark-and-sweep presence flag. Previously, the client: 1. materialised the applicable database subtree, 2. wrote existing records to `syncStatus = N`, 3. processed the authoritative online enumeration, 4. wrote observed records back to `syncStatus = Y`, and 5. treated records still marked `N` as absent online. For a no-change 101,494-object Full Scan this generated approximately 3.15 GiB of database write activity. The new implementation instead builds an in-memory presence candidate set containing only the IDs already present in the applicable database subtree. As authoritative `/delta` items are observed, their IDs are removed from that set. Any IDs remaining after a successfully completed enumeration are processed through the existing deletion/reconciliation logic. This makes native Full Scan presence reconciliation read-only for unchanged objects and avoids using persistent database state for temporary bookkeeping. Testing with the fixed 101,494-object dataset reduced no-change Full Scan database writes from approximately **3.15 GiB to 44 KiB**. ## Bounded native `/delta` processing Authoritative native `/delta` enumeration no longer requires all accepted JSON items to remain retained until the complete response has been downloaded. Accepted items are drained through the existing reconciliation path in bounded **500-item batches** while enumeration continues. The final partial batch is processed only after the authoritative enumeration has completed. This preserves the existing database transaction, filtering, move/rename, deletion and deferred-download behaviour while placing an explicit lifetime boundary around processed JSON pages and batches. A single strategic `GC.collect()` is performed after authoritative enumeration and reconciliation and before deferred download processing, allowing temporary enumeration state to be reclaimed before the next phase begins. ## Generated `/children` enumeration The generated current-state `/children` path has also been changed from account-sized accumulation to bounded processing. Previously, recursive `/children` traversal accumulated the complete remote object hierarchy before constructing and processing the generated response. On large accounts this created an unnecessarily large transient JSON working set. The new path preserves the existing depth-first traversal and generated-response reconciliation semantics, but submits accepted objects directly into the same bounded 500-item processing pipeline as enumeration progresses. The final partial batch is deliberately retained until traversal is complete so that absence reconciliation remains authoritative. Generated `/children` reconciliation continues to use the existing `syncStatus` semantics, but its initial database subtree marking no longer materialises full `Item` records for the entire tree. Instead, both generated subtree marking and native Full Scan presence enumeration use a minimal database projection containing only: - `id` - `type` This avoids constructing and duplicating full database `Item` state — names, tags, hashes, remote metadata and other fields — when only object identity and directory recursion are required. ## Database allocation reductions Several high-frequency database queries have been narrowed to return only the data actually required by their callers. Notable changes include: - subtree traversal uses `SELECT id, type` rather than full `Item` materialisation; - `idInLocalDatabase()` now performs a `SELECT 1 ... LIMIT 1` existence check; - `computePath()` retrieves only the seven fields required for path reconstruction rather than building a complete `Item`. Existing timestamp validation and relocation handling are retained. No database schema changes are introduced. ## Monitor-mode garbage collection and memory telemetry Monitor mode now performs: ```text GC.collect(); GC.minimize(); ``` after every completed monitor loop rather than using the previous 24-hour cleanup interval. `GC.collect()` and `GC.minimize()` are retained as separate operations because they address different aspects of the observed behaviour: reclaiming unreachable managed objects and returning unused managed heap capacity to the operating system. When memory telemetry is enabled, memory state is recorded both before and after this collection/minimisation boundary. The asynchronous logging subsystem now provides a drain barrier so that telemetry messages and logger-local buffers have been written and released before garbage collection is measured. This prevents the telemetry mechanism itself from changing the memory state being observed. ## Object and response lifetime cleanup The investigation also identified a number of long-lived references whose useful lifetime had already ended. The PR therefore explicitly releases temporary JSON responses, processed arrays and intermediate API state after their final use across sync, OneDrive API, webhook, Socket.IO, Intune and utility paths. CurlEngine pool handling has also been tightened: - popped pool slots are explicitly cleared; - stopped engines are destroyed instead of abandoned; - expired engines are deterministically destroyed after cleanup. These changes are intended to make ownership boundaries explicit and prevent stale references from extending the lifetime of otherwise reclaimable state. ## Memory validation The important measurement is retained memory after repeated large enumerations, not simply cumulative allocation volume. A native `/delta` control run using the bounded processing and strategic collection path completed **222 monitor loops over more than 22 hours**, including repeated authoritative Full Scans. After establishing its initial working-set high-water, post-minimisation memory remained bounded at approximately: ```text GC capacity: ~38.9 MiB RSS: ~77–78 MiB ``` The generated `/children` path was then tested with: ```text force_children_scan = true ``` so that every monitor loop enumerated the same fixed 101,494-object account through `/children`. An intermediate implementation that bounded JSON processing but still materialised the database subtree as full `Item` records continued to show retained-memory growth, reaching approximately: ```text GC capacity: ~169 MiB RSS: ~209 MiB ``` by Loop 14. After replacing that subtree materialisation with the minimal `id + type` traversal, the same workload remained around: ```text GC capacity: ~25.6 MiB RSS: ~64 MiB ``` through Loop 14, with no GC-arena staircase. At Loop 15, heap minimisation returned additional pages to the operating system, reducing the post-minimisation state to approximately: ```text GC capacity: ~18.4 MiB RSS: ~57 MiB ``` The long-duration `/children` soak is still being continued to confirm the same bounded behaviour over an overnight run. ## Functional validation The bounded generated-response changes have also been exercised through the existing E2E coverage. At the current PR revision: - Personal Account Shared Folder E2E tests are green. - Business Account Shared Folder E2E tests are green. The implementation preserves the existing filtering, reconciliation, deletion, move/rename, Shared Folder and deferred-download processing paths. ## Scope This PR changes the implementation of large authoritative enumeration and memory lifecycle management without changing the database schema or the externally visible synchronisation model. The primary goals are: - eliminate unnecessary Full Scan database writes; - avoid account-sized transient JSON and database object graphs; - process authoritative data using bounded working sets; - make temporary object lifetimes explicit; - reclaim managed heap capacity consistently during long-running monitor operation; and - preserve existing reconciliation semantics and E2E behaviour. |
||
|
abraunegg
|
9adea3a04b |
Uplift E2E Testing - Phase 1 (#3878)
Completed the first phase of the E2E testing realism uplift in preparation for v2.5.12, replacing reliance on synthetic fixtures with realistic file content while retaining existing regression coverage. This phase delivers: * Extend applicable testcases with genuine, structurally valid XLSX files while preserving the existing TXT and binary fixtures. * Introduce small and large XLSX companions on either side of the 4 MiB upload threshold, exercising both simple-upload and session-upload paths. * Strengthen coverage of file and directory lifecycle operations, including creation, modification, deletion, renaming, moves, conflicts, safeBackup preservation, monitor mode and filtering. * Explicitly account for SharePoint documentLibrary enrichment, validating Microsoft's modified Office files without incorrectly assuming byte-for-byte identity after enrichment and re-download. * Add TC0080 for live monitor reconciliation of remotely renamed populated directories. * Add TC0081 to validate historical v2.5.5 database upgrades, schema integrity, backup restoration and recovery through `--resync`. * Improve CI diagnostics, shared-folder logging, test reporting and failed-test debug reruns. The expanded testing also exposed application defects that have been corrected as part of this work, including Microsoft Graph `resourceModified` conflicts during simple uploads, directory reconciliation edge cases and recovery from incompatible database schemas. **Validation:** The complete E2E suite has passed across all tested account configurations, including Personal, Personal with a 15-character drive ID, Business, SharePoint documentLibrary and shared-folder scenarios. This establishes the Phase 1 realism baseline for v2.5.12. Additional PDF and image-format coverage will follow in a subsequent phase, allowing the remaining pre-release effort to concentrate on large-account memory consumption, outstanding defects and long-running stability testing. |
||
|
abraunegg
|
f785bd257c |
Fix Bug #3868: Fix false safeBackup on timestamp-preserving file replacements (#3869)
Fix modified-file reconciliation when a local file is replaced with different content while preserving an older modification timestamp.
This can occur through normal workflows such as:
* `cp -p`
* `rsync -t`
* restoring files from backup
* file managers that preserve source timestamps
Previously, if the replacement content differed but its local modification timestamp was older than the online timestamp, the client could incorrectly conclude that the online file was newer.
This caused the local replacement to be treated as a conflict even when the online object had not changed since the client's last synchronised state.
The resulting behaviour was:
```text
local content replaced with older preserved mtime
|
v
classified as "online is newer"
|
v
local replacement moved to safeBackup
|
v
safeBackup uploaded as a new file
|
v
existing online version downloaded back
```
This PR corrects that decision while retaining the existing data-preservation behaviour for genuine online changes.
## Root Cause
`performModifiedFileUpload()` already retrieves fresh metadata for the online DriveItem before deciding how to process a modified local file.
The existing conflict logic, however, primarily used:
```text
localModifiedTime < onlineModifiedTime
```
to determine that the online version should take precedence.
That assumption is not sufficient.
A local file can contain new content while legitimately retaining an older timestamp.
Importantly, the client also has another piece of information available at this point: the current online eTag.
Where the freshly retrieved online eTag is unchanged from the eTag stored in the local database, the online object has not changed relative to the client's synchronised baseline.
An older local timestamp alone therefore must not cause the local replacement to be classified as an online conflict.
## Change
The modified-file upload decision now determines whether the current online DriveItem can be proven unchanged from the database baseline.
The database baseline is trusted only when:
* the current online object corresponds to the same DriveItem identity represented by the database entry;
* the online item ID matches the expected target item;
* both the current online eTag and database eTag are present and non-empty;
* the current online eTag matches the database eTag; and
* the client is not operating under `--resync`.
When these conditions are satisfied:
```text
different local content
+ older local timestamp
+ unchanged online eTag
= local modification
```
The existing modified-file upload path is allowed to proceed.
Where the online eTag has changed, is unavailable, cannot be safely associated with the same DriveItem identity, or the client is running under `--resync`, the existing conservative conflict handling remains unchanged.
This intentionally keeps the fix narrowly focused on cases where the client can positively establish that the online object has not changed.
## `remoteItem` Handling
Shared/remote items require additional care because the top-level database item and the actual target DriveItem can represent different identities.
This PR does not attempt to redesign that behaviour.
Where the current online object cannot be safely correlated with the database eTag baseline, the existing data-preservation path remains in effect.
## E2E Test Coverage
The timestamp and replacement E2E coverage has been expanded to exercise actual Microsoft Office XLSX files rather than relying solely on synthetic text payloads.
This is intentional because XLSX files can exercise additional Microsoft Graph and SharePoint behaviour, including SharePoint enrichment of uploaded Office documents.
The existing XLSX implementation previously contained within TC0021 has been centralised into the E2E framework and TC0021 now consumes that common implementation.
The shared XLSX helper supports:
* creation of valid OOXML XLSX files;
* validation of generated/downloaded workbooks;
* mutation of existing workbooks while preserving other ZIP package members; and
* generation of files on both sides of the simple/session upload boundary.
This allows subsequent mutations to operate on the actual workbook returned by Microsoft, including any service-side package changes.
### TC0021
Existing resumable-transfer XLSX testing now uses the common XLSX framework implementation.
The existing large-file/session-upload behaviour remains unchanged.
### TC0029
`--local-first --upload-only` timestamp-preservation testing now uses a real XLSX document.
### TC0036
Overwrite/replace coverage has been expanded substantially.
#### OR-0001 — newer local replacement
A synchronised XLSX is replaced with different XLSX content having a newer local timestamp.
Expected result:
* normal modified-file upload;
* replacement becomes the canonical online content.
#### OR-0002 — older timestamp, simple upload
A synchronised XLSX is replaced using a timestamp-preserving copy operation with:
* different content;
* an older local modification timestamp; and
* no independent online change.
Expected result:
* online eTag still matches the database baseline;
* replacement is processed as a modified file;
* no erroneous safeBackup is created;
* canonical online content becomes the local replacement.
This directly covers the reported failure condition.
#### OR-0003 — older timestamp, session upload
The same scenario as OR-0002 is performed using an XLSX larger than the simple-upload threshold.
Expected result:
* session upload is used;
* modified upload succeeds;
* no erroneous safeBackup is created.
This confirms that the reconciliation fix operates correctly before upload-method selection.
#### OR-0004 — genuine online conflict
A separate mutator client is created using the established E2E multi-client model:
```text
clone tracked configuration/database/local state
|
v
mutator runs --monitor --upload-only
|
v
mutator changes XLSX online
|
v
remote modification independently verified
|
v
stale client receives different local replacement
with older preserved timestamp
```
Expected result:
* current online eTag differs from the stale client's database baseline;
* existing conflict handling remains active;
* the local replacement is preserved through safeBackup;
* the independently changed online version remains authoritative.
This is the safety control demonstrating that the fix does not weaken protection against genuine online changes.
### TC0037
mtime-only local-change handling now uses XLSX files across:
* simple upload-sized files;
* automatic session-upload-sized files;
* forced-session small files; and
* forced-session large files.
The downloaded Microsoft-returned workbook is used as the settled content baseline before timestamp-only mutations are performed.
### TC0051
Monitor-mode mtime-only testing now uses XLSX content and confirms that a timestamp-only local filesystem event does not result in an unnecessary upload or remote content change.
## Scope
This PR does not change the general timestamp authority policy introduced by earlier timestamp work.
It does not change:
* Microsoft timestamp application after successful uploads;
* simple versus session upload selection;
* `--local-first` conflict policy;
* `--resync` semantics;
* genuine online-change conflict preservation; or
* shared/remote item identity handling beyond refusing to trust an ambiguous eTag baseline.
The change is deliberately limited to preventing an older preserved local timestamp from being treated as proof of an online conflict when the current online DriveItem can be positively demonstrated to be unchanged from the client's database baseline.
|
||
|
abraunegg
|
a9a3904c86 |
Update architecture documentation (#3870)
* Update architecture documentation |
||
|
abraunegg
|
6965af2b0b |
Add project sponsorship visibility to initial graphical authentication (#3867)
## Summary This PR adds restrained GitHub Sponsors visibility to the OneDrive Client for Linux while deliberately avoiding recurring or intrusive sponsorship messaging. Following a successful **initial graphical authentication**, the browser completion page now includes an optional invitation to help sustain the project through GitHub Sponsors. The sponsorship message is not displayed during `--reauth`, authentication failures, application startup, monitor startup, normal synchronisation activity, terminal output or desktop notifications. ## Changes - Enhances the graphical authentication completion page with a clearer responsive layout and application identity. - Displays the sponsorship section only after successful initial graphical authentication. - Does not display sponsorship content during explicit `--reauth`. - Adds an attributable GitHub Sponsors link using GitHub Sponsors campaign metadata. - Adds passive sponsorship information to the project README and graphical-authentication documentation. - Adds browser response protections including `Referrer-Policy: no-referrer`, `Cache-Control: no-store`, a restrictive Content Security Policy and `X-Content-Type-Options: nosniff`. - Does not add application telemetry, click tracking or any new configuration option. ## Sponsorship Behaviour The graphical authentication page displays: > Help sustain OneDrive Client for Linux OneDrive Client for Linux is free, open-source software that is independently developed and maintained. If it is useful to you or your organisation, you can help sustain ongoing development, testing and maintenance through GitHub Sponsors. The user must explicitly select **Sponsor on GitHub** before any connection to GitHub is made. GitHub Sponsors campaign metadata is used only to attribute completed sponsorships to the relevant project surface. ## Privacy The application does not report whether the sponsorship message was displayed or selected. No application telemetry or analytics have been added. The local OAuth completion response explicitly prevents the Microsoft OAuth callback URL from being supplied as a browser referrer when following the external sponsorship link. |
||
|
abraunegg
|
e17643a7b2 |
Fix Bug #3861: Session resume does not verify that the local source file is unchanged (#3863)
Interrupted session uploads can be resumed from persisted `session_upload.*` state.
Before this change, the client validated that the persisted Microsoft upload session was still usable, but did not verify that the local file being resumed was still the same local file revision that originally created the upload session.
This creates a data-integrity risk.
For example:
```text
Local revision A
Fragment 1 -> uploaded
Fragment 2 -> uploaded
Fragment 3 -> uploaded
Client interrupted
```
If the local file is then modified before the client restarts, the existing Microsoft upload session can still advertise the next expected range beginning at fragment 4.
The resume worker would then read the remaining data from the current local file:
```text
Remote object
Fragment 1 -> revision A
Fragment 2 -> revision A
Fragment 3 -> revision A
Fragment 4 -> revision B
Fragment 5 -> revision B
...
```
A file-size change may subsequently invalidate the Microsoft upload session, but this does not protect against a same-sized local modification.
For example:
```text
Revision A = 100 MiB
Revision B = 100 MiB
contents differ
size remains unchanged
```
In this situation, the existing upload session could previously be resumed using fragments from two different local revisions.
## Resolution
Persist local source identity with every upload session:
```text
sourceFileSize
sourceFileMtime
```
`sourceFileMtime` retains the full filesystem timestamp precision available to the client and is captured independently of the timestamp normalisation used when sending metadata to Microsoft Graph.
When an interrupted upload session is discovered, the client now compares the current local source against the persisted source state before allowing the Microsoft upload session to be reused.
If either the file size or modification timestamp differs:
```text
local source changed
↓
existing upload session is unsafe
↓
best-effort cancel Microsoft upload session
↓
remove stale session_upload.* state
↓
allow normal reconciliation to process the current file
↓
fresh upload begins from byte zero
```
The source is also revalidated immediately before the resume worker begins uploading fragments, reducing the race window between initial session validation and actual upload execution.
Persisted upload-session files created by older versions of the client do not contain the new source identity fields. These sessions are intentionally treated as unsafe to resume and are discarded so that a fresh upload can occur.
The upload-session replacement handling introduced for #3859 / PR #3860 has also been updated so that replacement sessions retain the source identity associated with the replacement transfer.
## E2E Regression Coverage
TC0021 has been extended with:
```text
RT-0005 - changed local source invalidates an interrupted upload session
```
The scenario uses a genuine Microsoft resumable upload session and does not use synthetic API failures or fault injection.
RT-0005:
1. Generates deterministic 100 MiB local revision A.
2. Starts a genuine multi-fragment session upload.
3. Allows multiple fragments to be successfully uploaded.
4. Interrupts the client while a real `session_upload.*` file and Microsoft upload session remain active.
5. Confirms the persisted upload session has progressed beyond byte zero.
6. Replaces the local source with deterministic revision B.
7. Keeps revision B exactly the same size as revision A.
8. Ensures the local modification timestamp changes.
9. Restarts the client.
10. Verifies that the previous upload session is not reused.
11. Allows normal reconciliation to start a fresh upload.
12. Independently downloads the completed remote object.
13. Verifies that the downloaded SHA-256 exactly matches revision B and does not match revision A.
Existing TC0021 scenarios remain intact, including normal resumable upload/download behaviour and upload-session replacement handling.
## Validation
Full E2E testing completed successfully.
### Business Account Testing
**79** Test Cases Run
**79** Test Cases Passed
**0** Test Cases Failed
### SharePoint Account Testing
**79** Test Cases Run
**79** Test Cases Passed
**0** Test Cases Failed
### Personal Account Testing
**79** Test Cases Run
**79** Test Cases Passed
**0** Test Cases Failed
TC0021, including the new RT-0005 regression scenario, passed successfully.
This change prevents fragments belonging to different local file revisions from being combined through reuse of a persisted resumable upload session.
|
||
|
abraunegg
|
1b95d9dcdd |
Fix Bug #3859: Fix upload session replacement after 404/403 recovery (#3860)
## Summary Fixes upload-session recovery when an existing resumable upload session becomes invalid and Microsoft requires a replacement session to be created. Previously, after a `404 itemNotFound` / `The upload session was not found` response, the client created a replacement upload session and used it for the immediate retry, but the original upload-session state remained authoritative. The next fragment could therefore be sent back to the superseded session URL, causing another failure and, in SharePoint scenarios, a subsequent `409 nameAlreadyExists`. The same stale-session-state problem also affected the existing `403` expired temporary-authentication recovery path. ## Change When a replacement upload session is created, the client now: * adopts the replacement `uploadUrl` * adopts the replacement `expirationDateTime` * adopts the replacement `nextExpectedRanges` * preserves the required client-side upload metadata * recalculates the upload offset from the replacement session * persists the replacement session as a coherent authoritative state * resumes through the normal fragment upload loop This ensures that all subsequent fragments continue using the replacement session rather than returning to the invalid session URL. The recovery also correctly restarts from the offset advertised by the replacement session, which is normally `0-` for a newly-created upload session. ## Validation The failure was reproduced repeatedly against a SharePoint Online document library using a genuine multi-fragment XLSX file. Observed sequence before recovery: ```text Session A Fragment 1 -> 404 "The upload session was not found" Create Session B ``` With the fix applied: ```text Session A Fragment 1 -> 404 Create Session B Adopt Session B Fragment 1 -> Session B Fragment 2 -> Session B Upload completes ``` Repeated manual validation confirmed: * exact SharePoint `404 itemNotFound` condition reproduced * replacement session created and adopted * upload restarted at the replacement session offset * all subsequent fragments remained on the replacement session * no `409 nameAlreadyExists` * no safeBackup created * modified file upload completed successfully * overall synchronisation completed successfully ## E2E coverage TC0021 (`Resumable transfer recovery validation`) has been extended with a new RT-0004 scenario covering modified multi-fragment XLSX upload-session replacement. The test: * generates a valid large XLSX workbook programmatically at runtime * uploads it normally * modifies the resulting tracked workbook in place * performs a second multi-fragment session upload * validates replacement-session adoption when SharePoint returns the upload-session 404 * verifies that subsequent fragment PUTs remain on the replacement session GUID * fails on `nameAlreadyExists`, safeBackup creation, stale upload-session state, or incomplete upload * independently downloads and validates the final modified workbook |
||
|
abraunegg
|
e9c9f9f8ea |
Update docs - usage.md and manpage- #3858s (#3858)
Update docs - usage.md and manpage- #3858 |
||
Thomas Drake-Brockman
|
368bb572fd |
Add notify_monitor_start option to silence monitor-start notification (#3779)
Addresses discussion #3109 Running in `--monitor` mode sends a GUI notification on every startup confirming that filesystem monitoring has begun. There was previously no way to silence just this notification short of disabling all notifications via `disable_notifications`. * Add `notify_monitor_start` config option (default `true`, preserving existing behaviour) that gates only the monitor-start GUI notification * Document the option in `config`, `docs/application-config-options.md` and `docs/usage.md` To disable just this notification, add this to your cofig: ``` notify_monitor_start = "false" ``` Checked to compile with LDC v1.20.1 (Debian) and LDC 1.42.0 (Fedora). |
||
|
abraunegg
|
c2804ef7db |
Fix Bug #3850: 'skip_dir' full path rule (leading '/') also excludes same-named directories at any depth (#3851)
## Summary Fixes #3850, where a `skip_dir` full-path rule using a leading `/` could incorrectly exclude same-named directories at deeper levels. For example: ```text skip_dir = "/bin" ``` is intended to exclude only: ```text [sync_dir]/bin ``` but could also exclude: ```text [sync_dir]/some/deeper/path/bin ``` when processing remote `/delta` data. ## Root Cause When processing a remote directory, the client can calculate two paths for `skip_dir` evaluation: * a simplified path, which may degrade to only the directory name when `parentReference.name` is not supplied by the Microsoft OneDrive API * a complete sync-root-relative path derived from database state or `parentReference.path` The existing logic evaluated the simplified path first and only evaluated the complete path when the simplified path did not match. For a rule such as: ```text skip_dir = "/bin" ``` a remotely discovered nested directory could therefore be reduced to: ```text bin ``` before matching. The existing path normalisation performed by `isDirNameExcluded()` then generates both relative and rooted matching candidates, allowing the degraded `bin` value to match `/bin` before the actual full path was considered. This also meant that `skip_dir_strict_match = "true"` could not prevent the incorrect exclusion. ## Resolution Where a complete sync-root-relative path is available, the client now treats that path as authoritative for `skip_dir` evaluation. The simplified path is used only when a complete path genuinely cannot be constructed. This change has been applied to both affected remote directory filtering paths in `src/sync.d`. The existing `skip_dir` matching implementation itself is unchanged. This preserves existing behaviour for: * non-strict directory-name matching * strict full-path matching * leading `/` sync-root anchoring * wildcard matching * case-insensitive matching * multiple configured `skip_dir` rules In particular, case-insensitive matching remains intentional and unchanged. ## E2E Test Coverage TC0012 has been significantly expanded to validate `skip_dir` semantics in both upload and download directions. Coverage now includes: * unanchored non-strict directory-name matching * strict explicit multi-segment path matching * leading `/` single-segment sync-root anchoring with strict matching disabled * leading `/` single-segment sync-root anchoring with strict matching enabled * rooted multi-segment paths containing spaces and trailing `/` * `*` and `?` wildcard matching * case-insensitive matching * multiple and pipe-separated `skip_dir` rules * rooted wildcard rules * local-to-remote and remote-to-local filtering behaviour The regression scenarios specifically confirm that a root-anchored directory is excluded while the same directory name at a deeper path remains synchronised. ### TC0061 adjustment The full E2E run also exposed that TC0061 had historically depended on the same simplified-path behaviour corrected by this PR. TC0061 configured: ```text skip_dir = "Pictures/Archive" skip_dir_strict_match = "true" ``` while the actual test path was: ```text [sync_dir]/ZZ_E2E_TC0061_<run>/Pictures/Archive ``` The previous simplified-path-first behaviour caused this testcase to pass because the complete sync-root-relative path was never evaluated. TC0061 has therefore been corrected so its strict `skip_dir` rule describes the actual full path relative to `sync_dir`. No production behaviour was relaxed to accommodate this testcase. ## Validation Validation completed successfully: * Original Issue #3850 behaviour reproduced before the fix * Expanded TC0012 regression coverage passes all scenarios * Corrected TC0061 passes using the complete strict path * Full 79-case E2E suite passes * Original issue reporter has independently validated the fix against the environment that reproduced the defect This preserves the documented distinction between directory-name matching and explicit full-path matching while ensuring remote directory filtering uses the most authoritative path information available. |
||
|
abraunegg
|
9e074dd310 |
Implement FR #3787: Use different browser with --reauth and friends (#3848)
## Overview This pull request implements the feature request raised in #3787 to allow a user to explicitly select which browser is used for interactive Microsoft authentication. The current GUI authentication path launches the Microsoft authorisation URL using `xdg-open`. This means the desktop environment's configured default browser is normally selected. In environments where a user intentionally uses a different browser for Microsoft 365 authentication, setting the standard `BROWSER` environment variable does not reliably override that desktop preference because `xdg-open` may successfully select the desktop-configured browser before consulting `BROWSER`. For example, a user may use Firefox as their normal desktop browser while using Microsoft Edge exclusively for Microsoft 365 authentication and SSO. This change allows that user to run: ```bash BROWSER=/usr/bin/microsoft-edge-stable onedrive --reauth ``` and have the Microsoft authentication URL opened directly using Microsoft Edge. ## Changes The browser launch logic in `src/localAuth.d` has been updated so that: 1. If the `BROWSER` environment variable is set, the client first attempts to launch the configured executable directly with the Microsoft authorisation URL. 2. If the configured browser cannot be launched, the failure is written to the debug log and the client falls back to the existing `xdg-open` behaviour. 3. If `BROWSER` is not set, behaviour remains unchanged and `xdg-open` is used as before. 4. If browser launch ultimately fails, the existing authentication fallback behaviour remains unchanged. Both an explicit executable path: ```bash BROWSER=/usr/bin/microsoft-edge-stable onedrive --reauth ``` and an executable available through `PATH`: ```bash BROWSER=microsoft-edge-stable onedrive --reauth ``` are supported. ## Design Considerations This change deliberately does not attempt to automatically detect Microsoft Edge or introduce a Microsoft Edge-specific configuration option. `BROWSER` is treated as a user-selected browser executable, making the implementation browser-neutral and allowing the same mechanism to be used with Firefox, Chromium, Chrome, Edge or another browser. The value is passed directly to `spawnProcess()` as an executable rather than being executed through a shell. This keeps the authentication path deterministic and avoids introducing shell command parsing into OAuth browser launching. The existing behaviour remains the fallback path, so an invalid or stale `BROWSER` value does not prevent authentication from proceeding through the normal desktop browser. ## Behaviour With no override configured: ```bash unset BROWSER onedrive --reauth ``` the existing desktop/default-browser behaviour is retained. With an explicit browser configured: ```bash BROWSER=/usr/bin/microsoft-edge-stable onedrive --reauth ``` the selected browser is used for Microsoft authentication. If the configured browser does not exist or cannot be started: ```bash BROWSER=/does/not/exist onedrive --reauth ``` the client falls back to `xdg-open`. ## Scope This is intentionally a laser-focused change. There are no changes to: * OAuth request or response handling * local loopback authentication * GUI session detection * authentication callback processing * device authentication * headless or SSH authentication behaviour * application configuration options * Microsoft Edge detection * the existing manual authentication fallback Only the browser-selection step of the existing GUI authentication workflow is changed. |
||
|
abraunegg
|
1188caab0f |
Fix Bug #3784: Handle local Recycle Bin move failures without deleting the database record (#3845)
When processing an online deletion with `use_recycle_bin=true`, the client previously removed the item's database record **before** attempting to move the corresponding local file or directory into the configured Recycle Bin. If the subsequent `rename()` failed — for example when `sync_dir` and `recycle_bin_path` were located on different filesystems and `rename()` returned `EXDEV` — the local item remained inside `sync_dir` but its database identity had already been removed. A subsequent local scan could then classify the surviving object as new content and upload it again, effectively recreating content that had already been deleted online. This change makes the local Recycle Bin operation and database update behave transactionally from the client's perspective. ## Changes * Change `movePathToRecycleBin()` to return an explicit success/failure result. * Only remove the OneDrive database identity after the local Recycle Bin move succeeds. * Retain the database identity when the Recycle Bin move fails. * Prevent `.trashinfo` metadata from being created when the corresponding filesystem move did not succeed. * Record failed Recycle Bin moves as synchronisation failures and report the affected paths. * Preserve the previous `/delta` checkpoint after a failed move so the online deletion remains eligible for subsequent reconciliation. * Stop processing the current deletion batch after a failed parent move so descendant objects and their database identities remain intact. * Preserve the existing successful Recycle Bin and hard-delete behaviour. ## E2E validation TC0057 has been extended with: `S05_online_delete_recycle_bin_move_failure` The scenario uses the existing multi-client E2E architecture: 1. Seed a directory and nested file using the primary client. 2. Clone the established client configuration/database state and corresponding local tree. 3. Run the cloned client using `--monitor --upload-only`. 4. Delete the populated directory locally on the monitor client, producing a genuine online deletion while leaving the primary client on its previous state. 5. Configure the primary client's Recycle Bin on `/dev/shm` and verify that `sync_dir` and the Recycle Bin reside on different filesystem device IDs. 6. Process the online deletion, forcing the local `rename()` operation to fail with the cross-filesystem failure condition. 7. Verify that: * the local directory and nested file remain present; * no Recycle Bin payload is created; * no misleading `.trashinfo` entry is created; * the Recycle Bin failure is reported. 8. Run the primary client again with uploads enabled. 9. Use an independent client to verify that the retained local object was **not re-uploaded to OneDrive**. The new TC0057 scenario is passing. This directly validates the failure mode reported in #3784 and protects against the deleted-online → failed-local-trash → unintended re-upload sequence. |
||
|
abraunegg
|
bec9b20c0c |
Fix Bug #3806: Complete --display-sync-status redesign with documentation and E2E regression coverage (#3844)
This PR completes the redesign of `--display-sync-status` to provide a meaningful, read-only assessment of whether the configured OneDrive sync scope is currently in sync. Previously, `--display-sync-status` was primarily based on pending Microsoft Graph `/delta` data. This meant it could identify some pending remote changes, but it did not provide a complete picture of synchronisation state and could miss important conditions such as local-only changes, zero-byte files, directory changes, moves or renames, and deletion tombstones. The redesigned implementation now evaluates synchronisation state in **both directions**: ```text Microsoft OneDrive -> Local filesystem Local filesystem -> Microsoft OneDrive ``` and reports an overall result of: * `IN SYNC` * `NOT IN SYNC` * `INDETERMINATE` This provides a safe way for users, monitoring systems and diagnostic workflows to answer: > Is this client currently in sync? without performing a sync or destroying/rebuilding application state. --- ## What has changed ### Remote state assessment The Microsoft OneDrive → Local filesystem assessment now identifies pending remote changes including: * new files * new directories * modified files * moved or renamed items * deleted items * zero-byte files * approximate download volume Microsoft Graph deletion tombstones are explicitly recognised and handled rather than being incorrectly processed as malformed live objects. The existing stored delta cursor is used only as the starting point for the assessment and is **not advanced or modified** by `--display-sync-status`. ### Local state assessment The Local filesystem → Microsoft OneDrive assessment now compares the configured local filesystem against the client's stored sync database and identifies: * new local files * new local directories * modified local files * timestamp-only differences * locally deleted or missing items * file/directory type changes * files requiring upload * directories requiring creation online * remote deletions that would be requested * approximate upload volume The implementation follows the same database hierarchy and relative-path semantics used by normal synchronisation processing. ### Overall sync state The command now produces a clear overall state. `IN SYNC` means that no pending local or remote changes were detected for the configured sync scope. `NOT IN SYNC` means that one or more pending local or remote changes were positively identified. `INDETERMINATE` is used where the command cannot safely prove that the configured scope is fully assessed. The implementation deliberately prefers `INDETERMINATE` over incorrectly reporting `IN SYNC` where remote traversal completeness cannot be guaranteed. --- ## Read-only behaviour `--display-sync-status` remains a query-only operation. The command does **not**: * upload files * download files * create or remove local files or directories * modify the live sync database * advance stored Microsoft Graph delta cursors * perform reconciliation The command operates against the existing copied/read-only application state used for no-sync operations. The command also now establishes the configured `sync_dir` as its working directory before local filesystem assessment, consistent with normal sync and monitor operation. The existing `.nosync` / `--check-nomount` protection is also applied so that an unavailable mounted filesystem cannot be incorrectly interpreted as mass local deletion. --- ## Scope and filtering The status assessment honours the configured sync scope and applicable client-side filtering, including: * `--single-directory` * `sync_list` * `skip_file` * `skip_dir` * `skip_dotfiles` * `skip_size` * root-file handling * upload-only and download-only operating modes * `--no-remote-delete` Where configured operation requires a remote traversal mechanism that cannot be safely reproduced by the read-only status query, the result is reported as `INDETERMINATE` rather than returning a false clean result. --- ## Transfer estimates Pending upload and download data is now displayed using a human-readable value while retaining the exact byte count. For example: ```text Approximate download data: 14.15 MB (14838177 bytes) Approximate upload data: 2.09 KB (2144 bytes) ``` Values smaller than 1 KiB are displayed directly in bytes: ```text Approximate upload data: 90 bytes ``` This avoids misleading output such as `0 KB` for small but non-zero files while retaining deterministic values useful for diagnostics and testing. --- ## Example output A clean client now reports: ```text Synchronization status for the configured sync scope: Microsoft OneDrive -> Local filesystem No pending remote changes detected. Local filesystem -> Microsoft OneDrive No pending local changes detected. Overall status: IN SYNC No pending local or remote changes were detected for the configured sync scope. ``` A client with changes in both directions may report: ```text Microsoft OneDrive -> Local filesystem Pending remote items: 1 New files: 1 Files requiring download: 1 Local filesystem -> Microsoft OneDrive Pending local items: 1 Modified local files: 1 Files requiring upload: 1 Overall status: NOT IN SYNC ``` When remote changes are also pending, locally inferred actions are reported as the current point-in-time assessment and may subsequently be reconciled or superseded when normal synchronisation processing occurs. --- ## End-to-end regression coverage A new **TC0079** end-to-end test case has been added for the redesigned `--display-sync-status` behaviour. TC0079 exercises the status command through multiple lifecycle states, including: * established clean state → `IN SYNC` * deterministic incomplete assessment → `INDETERMINATE` * local-only pending changes * same-size local content modification * timestamp-only local differences * new local files and directories * local deletion * filtered local content * remote-only pending changes * remote deletion tombstones * remote rename * remote modification * zero-byte remote files * new remote files and directories * filtered remote content * mixed local and remote pending changes * repeated status queries proving that the command does not consume or advance the live delta cursor TC0079 is registered in the standard E2E suite and therefore runs across the supported Personal, Personal 15-character `driveId`, Business and SharePoint account workflows. --- ## Documentation Documentation has been updated to describe the redesigned capability and its intended use. Updates include: * `docs/usage.md` * explains why `--display-sync-status` should be used when a user wants to determine sync health without performing a sync or `--resync` * documents the bidirectional assessment and overall states * `docs/application-config-options.md` * expands the CLI option documentation and behaviour * `docs/end_to_end_testing.md` * documents TC0079 and its coverage The documentation also makes clear that `NOT IN SYNC` is a reported synchronisation state rather than a command execution failure; consumers should evaluate the reported `Overall status` rather than treating the process exit code as the synchronisation result. --- ## Result This change turns `--display-sync-status` from a limited remote-change check into a substantially more useful synchronisation-health diagnostic while preserving the most important property of the command: **it determines sync state without performing a sync or modifying the client's existing synchronisation state.** |
||
James C. Owens
|
6285f62fe4 |
Fix Bug #3822: Preserve the online copy when a local rename produces a name that cannot be used online (#3833)
* Preserve the online copy when a local rename produces an unusable name uploadMoveItem() used a single 'unwanted' flag for two different rejections. A new path filtered out by the user's configuration is genuinely no longer wanted online, but a new path that violates the Microsoft naming rules or is not validly encoded has not left the sync scope at all. It is still inside sync_dir, it just cannot be named online. Both cases fell through to the same else branch, which removes the online copy. For an unusable name that discards the only synced copy of data that still exists locally, and no later scan can restore it because the local name is still invalid. Track the two reasons separately. An unusable name now skips the move and leaves the online copy and its database entry untouched, so the data stays protected until the name is corrected locally. Also load the database record for the old path before removing it online in the genuinely out of scope case. 'oldItem' is only populated by the successful move path, so that delete was previously issued with a default constructed Item and could not identify the item to remove. * Stop tracking the old path when a rename produces an unusable name Preserving the online copy at the point the rename is handled is not enough on its own. The database still records the item at its old path, and nothing exists at that path any more, so processLocalDatabaseItem() reads it as a local deletion on the next sync pass and removes the online copy after all. Remove the database record for the old path instead, along with the records for any children if the item was a directory, since those describe paths that have also gone. The online data is left untouched and is picked up again by a full enumeration. * Add e2e test case tc0077: rename a synced file to an unusable name tc0025 covers files created with a name the Microsoft rules reject, and the existing rename cases cover renames between valid names. Nothing covers the intersection: a file already present online being renamed locally to a name that cannot be represented online. In that situation the item has not left the sync scope. It is still inside sync_dir, it simply cannot be named online, so the online copy is the only synced copy of data that still exists locally. No later scan can restore it if it is removed, because the local name remains unusable. The test asserts that no remote delete is attempted, and separately that the online copy still exists afterwards. The second assertion holds even with the misclassification present, because oldItem is only populated inside the !unwanted branch and the resulting delete is issued against a default constructed Item that the API rejects. It is asserted so that correcting that in isolation cannot silently turn this into data loss. * Remove trailing whitespace introduced by the rename fix Two blank lines inside the new branches were tab indented, and one pre-existing blank line had been given trailing tabs. src/sync.d has 2141 blank lines and only four of them carry whitespace, so these were anomalies rather than house style. * Adopt the reworked fix and test case from PR #3843 Replaces the two-boolean classification with an explicit LocalMoveTargetDisposition enum, so an unhandled disposition fails closed rather than falling through to the removal branch. Stale database detachment moves into detachUnrepresentableMoveFromDatabase(), which relies on the item table's ON DELETE CASCADE for the tracked subtree, handles shared-folder shortcuts and orphaned shared-drive roots, verifies each removal and refuses to continue if the safe state cannot be established. Suppresses the naming-validation helper's own notification for this path so the move emits exactly one terminal notification, after the database has reached the preservation state. The test case gains a seeded baseline verified by manifest and content, a directory rename alongside the file rename, and reconciliation plus convergence passes before the final independent verification. * Document e2e test case tc0078 in end_to_end_testing.md Describes the coverage gap between tc0025 (items created with an invalid name) and tc0044/tc0046 (renames between valid names), and lists the file rename, directory rename, reconciliation and convergence phases separately so the subtree detachment and the deferred-deletion path are both visible. --------- Co-authored-by: James Owens <jamesowens001@gmail.com> Co-authored-by: abraunegg <alex.braunegg@gmail.com> |
||
|
abraunegg
|
84e49d8720 |
Fix Bug #3840: Monitor mode can miss completed files created via temporary-file hard-link finalisation on Linux (#3841)
On Linux, some applications can finalise a completed temporary file without generating the normal `IN_MOVED_FROM` / `IN_MOVED_TO` sequence expected for a rename. This was reproduced with WinSCP, where the kernel instead reports: ```text IN_CREATE <temporary file> IN_CLOSE_WRITE <temporary file> IN_CREATE <final file> IN_DELETE <temporary file> ``` The existing Linux monitor logic intentionally does not treat a regular-file `IN_CREATE` event as immediately actionable, as the file may still be actively written. As a result, the `IN_CLOSE_WRITE` event makes only the temporary pathname actionable, that pending change is subsequently cancelled when the temporary pathname is deleted, and no actionable event remains for the surviving final pathname. The final file is therefore not immediately uploaded and is only discovered by the next normal monitor synchronisation, potentially delaying the upload by up to the configured `monitor_interval`. This PR adds a deliberately narrow, Linux-only, capture-local state machine to recognise this specific completed-file hand-off sequence. When the complete sequence is observed on the same inotify watch: ```text CREATE source CLOSE_WRITE source CREATE destination DELETE source ``` the destination pathname inherits the already-observed write-completion state and is queued as an actionable local change. The implementation deliberately: * does **not** make generic Linux `IN_CREATE` events actionable; * does **not** special-case WinSCP or the `.filepart` filename suffix; * does **not** reinterpret genuine `IN_MOVED_FROM` / `IN_MOVED_TO` events; * abandons the hand-off candidate when the sequence becomes incomplete or ambiguous; * preserves expected-event suppression and existing move/delete coalescing; * keeps the state local to a single inotify capture/drain; * leaves FreeBSD and OpenBSD monitor behaviour unchanged; * does not modify the existing monitor drift/reconciliation handling; * does not modify `sync.d` or any remote synchronisation logic. Manual reproduction using the original WinSCP workflow confirms that the final file is now detected and uploaded immediately through the monitor event path, without requiring a manual `touch` or waiting for the next 300-second synchronisation cycle. Subsequent monitor cycles recognise the uploaded file as unchanged, with no duplicate upload or reconciliation loop. |
||
|
abraunegg
|
e286a56e61 |
Fix Bug #3834: Validate timestamp authority and detect time differences between host platform and Microsoft OneDrive (#3836)
## Summary This PR implements the first stage of #3834 by introducing a system-time authority and safety gate before timestamp-sensitive OneDrive synchronisation is allowed to proceed. The client now validates the host system clock against time returned by the Microsoft service and maintains a process-wide time-authority state that can be used to determine whether synchronisation is safe. This PR intentionally **does not change how file or directory timestamps are applied during synchronisation**. The broader audit and correction of timestamp application logic described in #3834 will be handled in a separate follow-up PR. ## System Time Validation The client uses Microsoft service time as the external reference and accounts for HTTP round-trip time and measurement uncertainty when calculating effective clock skew. The current policy is: * **0–15 seconds:** system time considered acceptable * **>15 seconds to 120 seconds:** warning is generated, synchronisation continues * **>120 seconds:** potentially unsafe; a second independent Microsoft time observation is performed before the condition is treated as blocking A single anomalous observation therefore cannot immediately prevent synchronisation. When significant drift is confirmed, the client enters `TIME_DRIFT_BLOCKING` and timestamp-sensitive synchronisation is not permitted until system time has been corrected and successfully revalidated. System time is periodically revalidated while the application is running, and significant wall-clock discontinuities trigger revalidation of the previously established authority state. ## Behaviour ### `--sync` If unsafe system clock drift is confirmed before synchronisation begins: * synchronisation does not start * the user is advised to correct the local system clock or time synchronisation service * the application exits with failure If system time becomes unsafe during synchronisation, further timestamp-sensitive processing is stopped and the application exits with failure. ### `--monitor` If unsafe clock drift is present when monitor mode starts: * the monitor process remains running * synchronisation and event processing do not begin * system time is periodically revalidated * normal monitor startup automatically continues once system time is corrected If system time becomes unsafe while monitor mode is already running: * timestamp-sensitive synchronisation is suspended * the monitor process remains running * local filesystem observations may still be retained for later processing * WebSocket/Webhook-triggered reconciliation is prevented while the safety gate is closed * processing automatically resumes once system time has been corrected and revalidated Important blocking and recovery state changes are emitted through normal application logging and notification paths and therefore do not require `--verbose`. ## Configuration A `disable_time_check` configuration option / `--disable-time-check` command-line option is provided for users who explicitly need to bypass system-time validation. ## E2E Coverage A new **TC0077 — Timestamp authority safety gating validation** test case has been added with three scenarios: * **TA-0001:** `--monitor` starts with unsafe clock drift, remains suspended, and automatically starts synchronisation after time is corrected * **TA-0002:** an already-running `--monitor` instance develops unsafe clock drift, suspends synchronisation, retains pending local work, and resumes processing after time recovery * **TA-0003:** `--sync` starts with unsafe clock drift, performs no synchronisation and exits with failure * **TA-0004:** unsafe clock drift with `--disable-time-check` validates that the explicit time-validation override bypasses the clock safety gate, permits normal synchronisation to proceed despite otherwise blocking clock drift, and confirms through independent remote verification that local content is successfully propagated to OneDrive The E2E implementation uses process-local clock manipulation so the GitHub Actions runner, filesystem timestamps and Microsoft-side timestamps are not modified by the test. ## Scope This PR establishes **whether the local system clock can safely participate in timestamp-sensitive synchronisation decisions**. It does **not** yet alter: * remote-to-local timestamp application * local-to-remote timestamp propagation * file or directory timestamp semantics * move/rename timestamp handling * timestamp-only reconciliation * `lastModifiedDateTime` update behaviour * `--local-first` timestamp authority semantics Those areas will be reviewed separately once this timestamp-authority foundation has been merged. |
||
|
abraunegg
|
81971a9c30 |
Fix Bug #3825: Fix safeBackup replacement handling to preserve canonical files transactionally (#3826)
This PR fixes a serious data-preservation flaw in the `safeBackup` conflict-resolution workflow and strengthens several related reconciliation paths where unique local file content could otherwise be lost or incorrectly classified. When an existing local file needed to be preserved before being replaced by a newer or conflicting Microsoft OneDrive version, several code paths could rename the original canonical file to a `-safeBackup-` filename **before the replacement operation had successfully completed**. If the subsequent replacement failed, the client could be left with only the `-safeBackup-` file and no file at the original canonical pathname. This PR changes replacement-style safeBackup handling so that preservation and replacement are performed transactionally: > **The existing canonical file remains present until a complete, validated replacement is ready to be committed and any required preservation has succeeded.** The implementation has also been hardened following dedicated E2E regression testing, which exposed additional reconciliation paths where the same data-preservation rules were not being applied consistently. |
||
|
abraunegg
|
70d0c18de8 |
Fix Bug #3775: Fix monitor drift through observation-based reconciliation (#3794)
Reconstruct the filesystem monitor and reconciliation behaviour using the accepted rebuilt implementation as the functional baseline, replacing the superseded PR #3754 drift-handling changes. Harden inotify watch management, directory move/delete handling, stale and overflow event recovery, filesystem worker shutdown, and local-file race handling so monitor state remains consistent with the local filesystem. Preserve the validated Full Scan and WebSocket reconciliation behaviour and correct `sync_list` handling for remote directory moves so existing database and local state is retained until normal move reconciliation can complete. Add and update E2E coverage, including TC0065 for `sync_list` remote directory move/rename reconciliation. |
||
|
abraunegg
|
1adf12ba6e |
Fix Bug #3803: Correct Recycle Bin collision filename formatting (#3807)
When moving a local item to the configured Recycle Bin, the client generates a numbered filename if an item with the same name already exists. The existing collision filename construction adds an additional `.` before the result of D's `extension()` function. Because `extension()` already includes the leading dot, a filename such as: `file.data` can incorrectly become: `file.2..data` instead of: `file.2.data` The corresponding `.trashinfo` filename consequently inherits the same malformed filename. For files without an extension, the existing construction can similarly produce an unnecessary trailing dot, such as `file.2.`. ## Solution Remove the additional literal `.` before the extension when: * checking whether a numbered collision filename already exists; and * constructing the final numbered Recycle Bin filename. The collision suffix remains separated from the basename by a single dot, while the extension returned by `extension()` is appended directly. For example: ```text file.data file.2.data file.3.data file.4.data ``` with corresponding metadata such as: ```text file.2.data.trashinfo ``` Files without an extension are handled as: ```text file file.2 file.3 ``` ## Scope This is an isolated Recycle Bin filename-formatting correction. It does not change: * Recycle Bin move behaviour; * deletion or reconciliation processing; * database handling; * collision counter behaviour; or * `.trashinfo` metadata generation other than using the corrected recycled filename. This fix is independent of the monitor/reconciliation work in PR #3794 and the separate Recycle Bin failure-handling issue #3784. |
||
|
abraunegg
|
778cc05d24 |
Fix Bug #3782: Change mandatory resync exit status from 126 to 78 (#3799)
The client currently exits with status `126` when it detects a condition that requires the user to perform a `--resync`. Exit status `126` is conventionally used by shells to indicate that a command was found but could not be executed. This does not accurately represent the OneDrive client's behaviour: the application has executed successfully, but has detected a persistent configuration or state condition that requires user intervention before normal synchronisation can continue. Change the mandatory resync exit status from `126` to `78`. Exit status `78` corresponds to the traditional `EX_CONFIG` value and more accurately represents the condition being reported. It also avoids using the shell-reserved `126` exit status. This change: * changes `EXIT_RESYNC_REQUIRED` from `126` to `78`; * updates both supplied systemd service templates to use `RestartPreventExitStatus=78`; * preserves the existing systemd behaviour where a mandatory resync prevents automatic service restart while leaving the service in a failed state requiring operator action; * improves the resync-required error message to make it clear that `--resync` must be appended to the normal invocation, for example: * `onedrive --sync --resync` * `onedrive --monitor --resync` * documents the new exit status and the compatibility impact for scripts, monitoring systems and custom service definitions that currently test for exit status `126`. No synchronisation or resync processing behaviour is otherwise changed. Existing automation that explicitly checks for exit status `126` will need to be updated to check for exit status `78` instead. |
||
|
abraunegg
|
e3a0494316 |
Add support for OpenBSD (#3765)
Add OpenBSD as a supported platform, including build-system integration, runtime portability fixes, improved crash diagnostics, and long-running monitor-mode validation. Build and platform support: * Detect the `egdc` compiler supplied with OpenBSD 7.9 and handle its GCC-style version output. * Harden compiler-version comparison when version components are absent. * Add the required `libexecinfo` linkage for OpenBSD GDC builds while preserving DMD compatibility. * Document supported OpenBSD releases, required packages, compiler availability, and build instructions. * Document that the OpenBSD 7.8 package repository does not provide a supported D compiler configuration. Runtime and diagnostic improvements: * Add OpenBSD-specific fatal signal handling using `SA_SIGINFO`. * Report the fault address, trap information, interrupted register context, and a native stack trace for segmentation faults. * Re-raise fatal signals after diagnostic output so normal core-dump handling is preserved. * Make database row data independently owned before extracting fields and parse stored timestamps safely before constructing database items. * Add coarse, time-gated garbage collection for long-running monitor processes, running no more than once every 24 hours after sync processing has completed. Microsoft Graph response hardening: * Validate Graph collection responses before accessing or iterating their `value` arrays. * Validate required item, parent-reference, shared-folder, search, upload, directory-creation, and WebSocket subscription fields before accessing them. * Harden path construction and remote item processing against missing or unexpectedly typed JSON properties. * Strengthen shared JSON helper functions to verify object structure and expected value types before accessing strings, integers, objects, or arrays. * Treat incomplete upload responses as failures rather than persisting invalid item data. * Fail safely through the existing skip, retry, return, or failure paths when malformed responses are received. * Correct the generated fake folder response used by test and dry-run processing. Validation completed: * All GitHub Actions E2E account suites passed, including Personal, Business, SharePoint, shared-folder, and 15-character drive-ID testing. * OpenBSD monitor mode completed more than 24 hours of continuous filesystem churn without a crash or processing failure. * The scheduled 24-hour garbage-collection path executed successfully. * Normal WebSocket delta processing, local deletion propagation, remote directory creation, and file uploads continued successfully after the scheduled cleanup. * Resident memory remained stable during the long-running OpenBSD monitor test. |
||
|
abraunegg
|
3bd7a4ba17 |
Fix Bug #3770: Fix that a broken symbolic link can crash the client on resume upload (#3774)
* Apply technical fix to close bug, validated by E2E testing results |
||
|
abraunegg
|
0b7299adb1 |
Release files for 2.5.11 (#3764)
* Add the missing `remove_source_folders` entry to `--display-config` output. * Release files for 2.5.11 |
||
|
abraunegg
|
fcf369c2b6 |
Update docs clarify monitor operations (#3760)
* Clarify 'monitor' configuration options |
||
|
abraunegg
|
0efee85980 |
Implement GUI-based OAuth callback handling using local loopback listener (#3750)
This change improves the user authentication experience when running the client in a graphical desktop environment. Previously, browser-based authentication relied on the user manually copying and pasting the final redirect URL from the browser back into the client. Due to Microsoft authentication workflow changes and automatic browser redirection behaviour, this process has become increasingly unreliable and has generated ongoing user support requests. **Changes included:** - Added a dedicated local authentication listener implementation (localAuth.d) - Automatically detects when the client is running in a GUI session - Opens the Microsoft authentication URL in the user's default browser - Uses a local loopback HTTP listener to capture the OAuth authorisation response directly - Eliminates the need for manual copy-and-paste of the redirect URL when using a desktop environment - Returns a simple browser success/failure page after authentication completes - Uses a fixed loopback callback endpoint of: http://127.0.0.1:53100/ - Falls back to the existing nativeclient copy-and-paste authentication workflow when: - no GUI environment is detected - the local listener cannot be started - browser launch fails - local callback authentication cannot be completed This change is fully backwards compatible for headless, SSH and server deployments while providing a significantly improved authentication experience for desktop users. |
||
|
abraunegg
|
77938303d4 |
Fix single-tenant authentication redirect URI handling and add administrator consent support (#3745)
Fix authentication URL generation when `azure_tenant_id` is configured and the default OneDrive Client for Linux application registration is being used. Changes include: * Fix redirect URI generation for single-tenant authentication when using the default application registration * Retain the registered Microsoft Identity Platform native-client redirect URI: `https://login.microsoftonline.com/common/oauth2/nativeclient` * Add the `--display-admin-consent-url` CLI option * Generate tenant-specific Microsoft Entra ID administrator consent URLs * Improve documentation for `azure_tenant_id` * Add documentation covering Microsoft Entra ID administrator consent workflows and tenant-wide application approval These changes resolve Microsoft Entra ID authentication failures reporting `AADSTS50011` when using `azure_tenant_id` with the default application registration and provide a documented workflow for administrator consent in managed Microsoft 365 and Microsoft Entra ID environments. |
||
|
abraunegg
|
0bcc08bd51 |
Implement FR #3738: Implement capability to display the Admin Consent URL via --display-admin-consent-url (#3739)
* Add CLI option to generate an Admin Consent URL * Document new capability |
||
|
abraunegg
|
c7cf522ac9 |
Implement FR #3649: Add --mirror-local-state to ensure online state is a mirror of local state (#3742)
* Add --mirror-local-state to ensure online state is a mirror of local state * Add tc0064 to test new functionality |
||
|
abraunegg
|
24114d5457 |
Fix Bug #3740: Improve filesystem error handling during downloads (#3741)
* Improve filesystem error handling during downloads * Add try block for saveToPath set attributes |
||
|
abraunegg
|
5280df6984 |
Fix Bug #3736: Fix that Transfer Metrics duration includes non-transfer processing time (#3737)
* Fix that Transfer Metrics duration includes non-transfer processing time * Add test case and details |
||
|
abraunegg
|
31ffde7112 |
Fix Bug #3731: Fix that application cannot query SharePoint Shared Library - 'onedrive' application is already running (#3734)
* Fix that application cannot query SharePoint Shared Library - 'onedrive' application is already running * Add E2E Test Case bsftc0005 for this scenario * Add test to documentation |
||
|
abraunegg
|
d64ec0efe2 |
Fix Bug #3728: Fix that Microsoft Graph API 'sharedWithMe' has been officially deprecated impacting --sync-shared-files functionality (#3729)
* Fix that Microsoft Graph API 'sharedWithMe' has been officially deprecated impacting --sync-shared-files functionality |
||
|
abraunegg
|
15f7dde6cf |
Add tc0061 E2E validation for remote move into skip_dir reconciliation (#3723)
Adds new E2E test case tc0061 validating reconciliation behaviour when files are remotely moved from a synced path into a path excluded via skip_dir. The test simulates a second synced endpoint performing file moves while running in --monitor mode, preserving realistic online move semantics. The original Linux client then reconciles the remote changes using its existing items.sqlite3 state and active skip_dir configuration. Validation confirms: * moved files are removed locally from the original synced path * skipped destination content is not downloaded locally * unrelated files remaining in the source directory are preserved * moved files remain correctly online in the skipped destination |
||
|
abraunegg
|
90b1ccef6e |
Update E2E OneDrive Business Shared Folder Tests (#3722)
* Update E2E OneDrive Business Shared Folder Tests |
||
|
abraunegg
|
d438eed871 |
Update ubuntu-package-install.md (#3721)
* Remove Ubuntu 25.04 as supported platform |
||
|
abraunegg
|
7c984c900d |
Implement stream hashing to support 'on-demand' development work (#3653)
* Implement stream hashing to support 'on-demand' development work |
||
|
abraunegg
|
4475943d79 |
Fix Bug #3672: Improve handling of local inotify IN_MOVED_* events (#3720)
Improve handling of local inotify IN_MOVED_* events in monitor mode to better preserve file and directory move operations within the configured sync_dir. This change refines monitor action coalescing in monitor.d to suppress stale descendant delete events when a parent path move has already been detected. This prevents certain local move operations from being incorrectly interpreted as delete/re-upload sequences during monitor processing. Validated locally with: File moves between directories Nested directory moves Sequential move operations within the same sync tree The updated logic now consistently performs remote move operations instead of unnecessary delete/re-upload behaviour where the source and destination remain within the same synchronised hierarchy. |
||
|
abraunegg
|
692f74408b |
Add tc0059 to specifically test websocket event monitoring and handling (#3719)
* Add tc0059 to specifically test websocket event monitoring and handling |