Verify every MTP step as a tree of fixed capacity (4 or 8 rows): a draft chain plus runner-up siblings, with fixed depths 2-5 as single-child trees. Attention merges the verify rows into the committed prefix with a per-row ancestor mask, the DeltaNet recurrence runs the tree program through the deferred control vector, and the accepted path is committed in place. The auto mode reallocates the draft rows between depth and siblings from conditional rank hits and measured step cost.
* Load the optional local agent through comfy-agent's v1 host
Add --enable-agent. When set, ComfyUI discovers the "v1" entry point in
the comfyui.agent group, creates the host, lets it mount its own routes
on the aiohttp app during route registration, starts it once the server
is listening and stops it on shutdown. The host's state is published as
the `agent` server feature flag and an `agent_status` websocket message.
Any failure in the package (missing, outdated protocol, a raising
entry point, register or start) disables the agent and is reported
through the flag; it never stops ComfyUI from starting.
* Pin comfy-agent in agent_requirements.txt to its published version
* Keep a failing agent from escaping startup or leaving a stale flag
A host whose status() raises SystemExit no longer ends ComfyUI startup
when load() publishes the flag, and a register() failure now republishes
the agent flag as failed instead of leaving it at "starting".
* Disable an agent host whose status() exits
load() reads status() to publish the flag. A host that raises SystemExit
there is now disabled, so register() and start() skip it, matching the
"failed" flag it reports.
* Run startup callbacks from a list of hooks
The browser launch and the agent start each append a named hook, and
call_on_start runs them in order, instead of the agent block wrapping
whatever call_on_start was before it.
* Start the agent from a named call_on_start, like the browser launch
* Renew partner-node sign-in tokens in API nodes
Read exp from tokens with the comfy-partner-node audience, renew them
shortly before expiry or after a 401 from comfy-api, keep the renewed
token in memory per session id, share one in-flight renewal per session,
and resend a 401-rejected request once. Workspace tokens and API keys
are unchanged.
---------
Signed-off-by: bigcat88 <bigcat88@icloud.com>
Co-authored-by: Alexander Piskun <bigcat88@icloud.com>
Co-authored-by: Alexander Piskun <13381981+bigcat88@users.noreply.github.com>
* Revert "Disable the output assets scanning. (#16926)"
This reverts commit 3a52aae7ca.
* Turn the assets system off unless --enable-assets is given
--enable-assets turns assets on again and is no longer deprecated.
--disable-assets still turns them off, and wins when both are given.
The 503 from /api/assets and the warning shown when another ComfyUI
holds the database no longer say this one was started with
--disable-assets.
* Test that the startup scan covers the output folder
* Test that running a prompt rescans the output folder
* Say --disable-assets must be left out too, and that hashing needs --enable-assets; skip the rescan test on a checkout with extra model paths
* Stop at startup when --enable-asset-hashing is given with assets off
* Warn and ignore --enable-asset-hashing when assets are off instead of stopping startup
* Test that hashing with assets on logs no ignored-hashing warning
* Name --disable-assets in the ignored-hashing warning; test the output rescan's phase, hashing and disabled guard
---------
Co-authored-by: Alexis Rolland <alexisrolland@hotmail.com>
* fix(assets): date scanned assets by their file's mtime
The scanner stamped every file it found with the scan time, so a library
catalogued on its first scan listed newest-first in reverse walk order.
Records the scanner creates now take the file's mtime (capped at now) as
created_at. Migration 0009 redates existing scanned records the same way,
only ever moving a record earlier. Generated outputs and uploads keep their
registration time.
* test(assets): pass created_at through the seeder's create_record stub
* docs(assets): state what the mtime cap guarantees
* test(assets): bound the cursor walk, probe just outside the migration window; note why 0009 inlines its conversion
* fix(assets): cap a future mtime at the file's ctime too
* fix(assets): use the ctime only for a future mtime
* test(assets): check the ctime's now cap directly; say what the ctime is per platform
* test(assets): drop an unused import
* test(assets): a future mtime with a pre-1970 ctime is dated now
* fix(assets): fall back to now when the ctime is before 1970
* Turn the assets system on by default; add --disable-assets
Assets are on unless ComfyUI is started with --disable-assets; --enable-assets
is still accepted, does nothing and logs one deprecation line.
Startup failures that already stopped ComfyUI with assets on now log
"ASSETS_STARTUP_FAILED: <kind>" and a message naming the database file and
the fix (another ComfyUI or program holding it, corrupt, written by a newer
ComfyUI, not writable, a file in the way, an unsupported --database-url),
each ending with --disable-assets. Missing database packages stop startup
with the install command instead of quietly turning assets off.
A read-only database that needs an upgrade is reported before it's opened
or backed up, so no read-only copies are left behind. A % in the install or
database path works, and the default database URL is built by SQLAlchemy.
* Read the stored revision through the WAL after an unclean shutdown; point a bad --database-url back to the default
* Build the suggested --database-url with SQLAlchemy; mention programs holding the file; log the full error at DEBUG
The suggested URL is quoted the same way as the default one, so a %20 or ?
in the path opens the intended file. A file held open by another program
(SQLite reports it as unable to open) gets that in the advice. An
unrecognised startup failure logs its traceback at DEBUG, as its message
says, except for an unsupported --database-url, whose error repeats the URL.
* Read through the WAL only when its -shm is there too, so the check never creates one
* Quote only what SQLAlchemy reads as URL syntax; report a corrupt read-only database as corrupt
A suggested --database-url keeps a plain path as typed, so pasting it into a
.bat file can't expand a quoted ':' or '\' as a batch argument. With
SQLAlchemy 2.1+, % ? and # are still quoted so the URL opens that file.
The pre-upgrade revision read lets a corrupt database raise, so it is
reported as corrupt rather than as not writable.
* Stub the revision read in the legacy-copy lock test, whose copied file isn't a real database
* Drop the absolute --database-url example; build the default URL with SQLAlchemy again
The failure messages go back to the generic --database-url example, so the
helper and the version-dependent quoting that kept its pasted form plain
are gone; the default database URL is built by URL.create again. Also tests
a read-only -wal left by a run that didn't shut down.
* Remove the pre-upgrade read-only check
A read-only database that needs an upgrade fails in the upgrade again, with
the not-writable message. The check and its tests move to a follow-up.
* feat: add CameraAngle node with camera_info and shot prompt outputs
* fix: carry CameraAngle zoom into camera_info.zoom
* fix: orbit CameraAngle around the world origin
* fix: rename CameraAngle display name to Compose Camera Angle Prompt
---------
Co-authored-by: Alexis Rolland <alexisrolland@hotmail.com>
* fix(assets): copy uploads into place when the destination is on another volume
os.replace cannot rename across volumes (EXDEV; WinError 17 on Windows), so
an upload into an input/output folder on a different drive than the temp
directory failed. On EXDEV, copy to a hidden staging file beside the
destination and rename that into place, so a partial copy is never visible
under the final name; the staging file is removed on any failure. The
copy's own stat is recorded, since a copy can change the mtime.
* fix(assets): don't fail cross-volume uploads on metadata, verify the source
shutil.copy2 also copies permission bits, which raises on filesystems that
cannot store them (FAT/exFAT on Linux), failing the upload after the bytes
were copied. Copy the bytes, then copy the mode best-effort.
The copy's stat is what gets recorded, so check that the source still
matches the stat hashing verified once the copy is done; otherwise bytes
rewritten after hashing would be published under the old hash.
* test(assets): use NTFS-representable mtimes in cross-volume upload tests
NTFS stores mtimes at 100 ns, so a sub-microsecond source mtime was
truncated and a +1 ns rewrite left the mtime unchanged on Windows.
* fix(assets): fail fast when the destination denies writes
tempfile.mkstemp on Windows retries a PermissionError for as long as
os.access reports the directory writable, which an ACL deny does not
change, so an upload into a write-denied folder hung the server. Create
the staging file directly with O_EXCL, retrying only on a name
collision.
* fix(assets): report a source changed during a cross-volume copy as unstable
Raise UploadUnstableError, as the hash-time check does, so the API returns
UPLOAD_UNSTABLE instead of a generic internal error.
* refactor(assets): reduce the cross-volume upload move to shutil.move
Same-volume uploads stay a single os.replace. On EXDEV, fall back to
shutil.move and record the destination's stat, removing a truncated copy
if the move fails. Drop the staging file, mode copy and source re-check
along with their tests.
* fix(assets): stage cross-volume copies beside the destination
Copying straight onto the hash-named path overwrote an existing file in
place, so a copy that failed partway (e.g. a full disk) left a good asset
truncated. shutil.move into a uniquely named .part sibling and os.replace
it into place, removing the .part on failure.
* refactor(assets): reduce the cross-volume upload fallback to a plain copy
On EXDEV, copy the upload into place with shutil.copyfile, skipping the
copy when the hash-named file is already there at the same size, and
record the stat of the file at the destination. Drop the staging file
and its tests.
* docs(assets): note why the placed file's stat is recorded
* fix(assets): compare content, not size, before keeping a cross-volume destination
Keep an existing hash-named file only when its own hash matches the upload;
otherwise remove the entry first, so the copy never writes through a link
or over other bytes. Tests now pin the recorded mtime, the replacement of
mismatched or truncated files, links, and non-EXDEV errors.
* test(assets): cover a dangling symlink at a cross-volume destination
* fix(assets): replace linked destinations and clean up a failed cross-volume copy
Keep an existing hash-named file only when it is a regular, single-link
file with matching bytes, so a symlink or hardlink left by a dedupe tool
is replaced as a same-volume rename would. Remove the partial file if
the copy fails, so a full disk doesn't stay full.
* refactor(assets): reduce the cross-volume fallback to a bare copy
On EXDEV, copy the upload onto the hash-named path with shutil.copyfile and
record the placed file's stat. Drop the content-hash keep check, the link
checks and the partial-file cleanup; their cases are listed as known
limitations on the PR.
* test(assets): pin that a failed cross-volume copy fails the upload and registers nothing
* refactor(assets): drop the unused verified stat binding in uploads
* fix(assets): pause the fast scan during its folder walk and file stats
The startup fast scan only checked for a pause between phases, so a prompt
that started during the walk or the per-file stats waited for the whole
library to be listed and stat'ed. Check the pause gate before each directory
and each file instead.
* test(assets): pin that a cancelled walk stops, and loosen the paused-time margin
* test(assets): pin the walk stop in the seeder cancel test and the admission stop path
* test(assets): wait for the scan to park before timing the pause
* docs(assets): say why the admission check sleeps between its two stats
---------
Co-authored-by: guill <jacob.e.segal@gmail.com>
* fix(assets): resume the background scan once the prompt queue is empty
The prompt worker pauses the scan before every prompt and resumed it only
on the GC tick, which waits 10 s from the last collection. With a prompt
every few seconds the scan barely ran, and a large first scan could take
far longer than the prompts themselves. Resume it as soon as a finished
prompt leaves nothing queued; a queued backlog still keeps it paused.
* test(assets): model the prompt queue's running count in the prompt-worker fake
The fake defaulted to an empty queue, so the idle resume made the
error-path test pass without the handler's resume, and its constant
count could not tell whether the idle check ran after task_done.
* fix(assets): insert scanned files in batches of 200 with a pause between them
The fast scan wrote 500 files per write transaction and took the lock back
within milliseconds for the next batch. A foreground write (an upload's
registration, an output being recorded after a prompt) waits in SQLite's
busy handler, which retries after sleeps of up to 100 ms, so it almost
never saw the lock free and failed with "database is locked" after the
5 s busy timeout, while /upload/image held the event loop the whole time.
The scan now inserts 200 files per transaction and, while more follow,
sleeps for SQLite's longest busy-handler sleep plus a Windows timer tick
(0.12 s). Each batch is still one call to insert_asset_specs with its
existing error handling, and pause and cancel are checked between batches
as before.
* perf(db): sync the WAL at checkpoints instead of on every commit
With synchronous=FULL, every commit fsyncs the WAL while it still holds
the write lock, about 100 ms on a hard drive, which a waiting foreground
write has to sit through. In WAL mode, synchronous=NORMAL syncs at
checkpoints instead. An app crash loses nothing; a power loss or OS crash
can roll back commits the OS had not yet flushed, typically the last tens
of seconds, and cannot corrupt the database. Behind WAL_SYNCHRONOUS so it
can be set back to "FULL" on its own, and applied only once WAL mode is
confirmed, so a database in rollback-journal mode is left as it was.
* fix(assets): batch the prune's and the offline marking's writes
The startup prune, POST /api/assets/prune and the fast scan's marking step
each held the SQLite write lock for their whole loop, so foreground output
registration failed with "database is locked" during a large one. They now
write in short batches, wait while a prompt runs between batches, and the
prune endpoint runs off the event loop.
* fix(assets): start the queued scan after a standalone prune, and recheck listing rows after a pause
A prompt that ends while POST /api/assets/prune runs queues its output rescan;
the prune now starts it when it finishes, as a scan does. The output-listing
rescan takes its batch gate before reading the live rows, so a pause during the
walk makes the marking re-stat what it retires. A cancel that arrives after the
last batch no longer reports a finished prune as cancelled.
* refactor(assets): drop the pause rechecks and the cancellable standalone prune
Batching the writes is what keeps the lock short; the layers on top of it
guarded edge cases that heal on the next scan. Batches now just commit, sleep
about as long as they held the lock, and between batches honour the scan's
pause/cancel checkpoint. The standalone prune is batched but not pausable, so
it needs no cancel status or pending-scan handling, and the API contract is
unchanged apart from running off the event loop.
* fix(assets): start the scan queued behind a standalone prune; skip the last batch's yield
POST /api/assets/prune now runs off the event loop, so a prompt can finish
while it runs and queue its output rescan; the prune starts it when it ends,
as a scan does. The batch loop checks for a stop before every batch and no
longer sleeps after the last one.
* test(assets): compare the set-mark paths in their stored, absolute form
create_content stores os.path.abspath(path), which carries a drive letter on
Windows, so the expected list must be built the same way.
* fix(assets): a seed request during an API prune waits for it instead of 409
The prune now runs off the event loop, so POST /api/assets/seed can arrive
while it holds the seeder; start() fails and the route answered 409, which a
client reads as "a scan is already coming". A prune emits no scan events, so
the refresh was lost. The route now waits the prune out and starts the scan,
as it effectively did when the prune blocked the loop.
* fix(assets): a cancel or shutdown stops a standalone prune between batches
The API prune runs on a worker thread that interpreter exit joins, so a
shutdown that only flagged it left Ctrl-C waiting for the whole prune. It now
stops at the next batch once cancelled, and shutdown waits for that. A seed
request also retries start() once after any failure, covering a prune that
ends between the failed start and the check.
* fix(assets): report a cancelled API prune as cancelled, not completed
A cancel now stops a standalone prune between batches, so its response can
carry a partial count; say so with status "cancelled" rather than presenting
it as a finished prune.
* fix(assets): a cancelled standalone prune leaves a queued scan queued
Shutdown cancels the prune; starting the scan a prompt had queued from the
prune's finalizer would run it on into teardown after shutdown returned. It
now stays queued for the next scan's finalizer.
* test(assets): assert the cancelled prune's outcome in the test thread
pytest.raises inside the worker thread only produced a warning when the
exception was missing, so the test could not fail on it.
* fix(assets): wait for a prune on the loop, and close shutdown gaps around it
A seed request during an API prune now polls on the event loop instead of
holding an executor thread for the prune's length, and retries while a prune
holds the seeder. Shutdown marks the seeder so a prune that has not started
yet does not, both of its waits share one deadline, and the prune's idle flag
is set even if its cleanup raises.
* perf(assets): let the partial live-path index serve every live-row-at-path lookup
SQLite will not use the partial index uq_asset_contents_path_live (WHERE
is_missing = 0) for a predicate written is_missing IS 0, which is what
is_(False) renders. Each "live row at this path" lookup therefore scanned
asset_contents. register_executed_output and the seeder's hash-on recovery
check ran that scan under the write lock, so the lock window grew with the
catalog. Compare with == false() instead; the column is NOT NULL, so the
result is the same.
* test(assets): match the pre-3.36 SQLite wording of a table scan too
* test(assets): stage the upload where the upload route does, so the move stays on one drive
* fix(assets): don't take the database lock when assets are off
An assets-off ComfyUI no longer initialises the database or takes its lock, so it
can't block a later assets-on start on the same install. If another process holds
the lock, it prints a startup warning saying a future version will refuse to
start the second instance, and keeps running.
* fix(assets): never let the lock check stop startup; tighten the lock tests
* fix(assets): keep the lock check's release inside its guard
* fix(assets): reword the shared-database startup warning
seeder.scan_completed reported only elapsed_ms, which is wall-clock time
and includes every pause taken while prompts ran. It now also carries:
- cpu_ms: the scan thread's CPU time (time.thread_time()).
- paused_ms: time blocked at the pause gate, summed across pauses.
- dirs_listed_count: directories the input/output walk or the output
rescan's folder listing listed.
- files_statted_count: os.stat calls on files in the scan's per-file
loops (reference sync, the listing check, discovery, admission, seed,
watch list and enrich).
Scan failure events carried only the exception class, so every SQLite
failure read as OperationalError. seeder.scan_failed, batch_insert_failed
and scanner.{fast_scan,temp_sync,mark_missing,stat,watch_stat,watch_seed}
_failed now also carry error_kind, one of a closed set:
expression_tree_too_large, too_many_variables, database_locked,
disk_full, disk_io, unable_to_open, database_corrupt, permission_denied,
file_locked, read_only, other.
It is a closed enum rather than a scrubbed message because str() of a
SQLAlchemy error includes the statement and its bound parameters, which
are file paths. Classification reads SQLite's result code where the
driver exposes it (Python 3.11+), then SQLite's fixed message on the
driver exception (exc.orig), then errno and, on Windows, winerror.
Co-authored-by: guill <jacob.e.segal@gmail.com>
* fix(assets): batch prefix filters so scans work with many model folders
Every scan prefix (one per model folder, including each extra_model_paths
base) adds terms to a single OR, and SQLite rejects an expression tree
deeper than 1000. From about 500 prefixes every models scan failed with
"Expression tree is too large" and the catalog stayed empty.
Run the prefix filter in batches of at most 200 prefixes and merge the
results, deduping rows that nested or overlapping prefixes put in more than
one batch. With 200 prefixes or fewer the statement is unchanged. The
enrichment candidate query merges each batch's keyset page, which yields
the same page a single statement would.
* fix(assets): filter enrich candidates in one pass above one prefix batch
Paging each prefix batch separately made every page scan to the end of
the table for any batch with few matches, which is quadratic in catalog
size. Above one batch, read the candidates in id order once and apply the
same prefix test in Python, stopping at the page limit. At 200 prefixes
or fewer the statement is unchanged.
Run the many-prefix tests under a 999 bound-variable cap too, the limit
of SQLite before 3.32, which each 200-prefix batch stays under.
* fix(assets): yield the GIL while filtering enrich candidates in Python
Above one prefix batch the enrich candidates are filtered in a Python loop,
which could hold the GIL across many rows outside the prefixes. Yield it per
row as the other scan loops do, so the UI stays responsive.
* Revert "fix(assets): yield the GIL while filtering enrich candidates in Python"
Measured, it bought nothing: with a 60k-asset catalogue at 450 and 2000
prefixes, p99 lateness of a 1 ms sleeper thread is the same within noise
either way, because the Python work between 500-row fetches is short and
sqlite3 releases the GIL during each fetch. The yield made the loop 30-90%
slower.
* test(assets): import db at module level in the many-prefix scan test
* fix(assets): recover records when a drive comes back with hashing off
A scan that runs while a drive is offline marks every row on it missing. With
hashing off (the default), nothing could recover those rows when the drive
returned, so the scan created new records and the user's names, tags, metadata
and job links stayed on the orphaned originals.
- With hashing off, a file that reappears at a path no live row occupies
recovers the missing row there whose size and mtime match exactly. If several
match, the newest recovers. Hashing on is unchanged.
- A reference stat that fails with an I/O error other than not-found leaves the
row live, as the output listing rescan already does.
- seeder.marked_missing is emitted per root when a fast scan marks rows missing,
and scan_completed carries missing_marked_count and recovered_count.
* fix(assets): read recovery candidates before the write transaction; count recoveries only once committed
* test(assets): a pruned model folder recovers its records when it is registered again
* fix(assets): never recover a missing row whose records were all deleted
* docs(assets): scope the multiple-match rule to hashing on
* docs(assets): only new content splits a same-path edit
* test(assets): scan_completed carries nonzero missing and recovered counts
* perf(assets): let the partial live-path index serve the hashing-off recovery check
Written as IS 0, the check that no live row occupies the path scanned
asset_contents once per recovered file, inside the batch's write transaction.
At 50k rows that stretched each batch's lock window to about 2.5s and made
concurrent output registrations time out.
* ci: run the checks against master
* test(assets): import folder_paths at module level
folder_paths.get_filename_list raises when a folder it cached earlier has gone
away (or on an OSError other than not-found), and collect_models_files let that
abort the whole asset scan, so nothing in models, input or output was
catalogued. Fall back to a fresh listing of the category, which skips a folder
that's gone; if that raises too, skip the category with a warning and scan the
rest.
With --enable-assets, the scans that keep the asset database in sync could starve the web server's event loop and burn CPU on large libraries. Three changes:
- First scan yields the GIL. The loop that derives names and tags for new files now sleeps briefly every 2 ms of work, so the event loop keeps running. At 51k files, time to a usable UI on the first run drops from 4.7 s to 2.0 s (1.5 s with assets off); the first scan takes about 3% longer. (was #16546)
- Output rescans list folders instead of stat'ing every file. The post-prompt output rescan compares directory listings against the database instead of stat'ing each catalogued file. That per-file stat was 70–95% of the rescan's cost: ~5.8 s per rescan at 200k outputs on local SSD, ~57 s at 10k on NFS. Any row the listing can't vouch for gets an individual stat before it is retired, and only "file not found" retires it: case-insensitive or normalizing filesystems, hidden folders, folders that fail to list, symlink aliases, and permission or I/O errors keep their rows. The one behaviour left to the next full scan is a file overwritten in place under the same name. (was #16599)
- Rescan loops pause. The listing and comparison loops pause every 10 ms of work, so prompts start and the UI stays responsive during a long rescan. At 200k outputs, without this a prompt could wait ~1 s to start. This costs 7–15% more rescan time. (was #16600)
Assets-only: none of this runs with assets off.
ComfyUI failed to start when the assets database packages (sqlalchemy, alembic, blake3) were missing, e.g. in a venv not re-synced after requirements.txt changed. That happened with assets off too, although that mode never needs them.
- The asset imports in lifecycle.py, manager.py and server.py are guarded by db.py's existing dependencies_available() check. Without the packages, NoAssets registers no asset routes and temp cleanup still runs.
- --enable-assets without the packages logs which packages are missing, prints the install command, and continues with assets disabled.
- /view resolves blake3: filenames only when assets are enabled. Trade-off: if assets are turned off on an install whose saved workflows hold blake3: widget values, those previews return 404. Running such workflows already fails, since execution never resolves hashes.
* fix(db): wait briefly for the database lock at startup
A relaunch can start while the previous process is still exiting and
holding the lock. Wait up to 5 seconds for it to be released before
treating the database as in use, and log how long the wait took. The
lock is never taken from a holder.
* fix(db): log the lock wait at info without guessing its cause
* fix(db): say when startup waits for the database lock
Log once before the wait instead of after it, so a lock that stays held
explains the pause before the error. Document the wait in the docstring,
and make the wait test release the lock only after the wait has started.
