WC_LINUXKM_SVR_FORCE_DYNAMIC_AUDITING overrides and an svr_dynamic_auditing
runtime gate, conditional (when available) on IS_INTEL_AVX1(cpuflags), to avoid
false positive "inhibited-save instrumentation was not exercised" error at
startup.
Also add !defined(WC_SVR_USE_NATIVE_REG_BUFS) as a predicate for implicit
WC_LINUXKM_SVR_DYNAMIC_AUDITING -- no fallbacks to C occur with
WC_SVR_USE_NATIVE_REG_BUFS even if WC_C_DYNAMIC_FALLBACK is configured.
The 3.17-to-4.14 composition of the NMI-safe epoch time accessors is
spelled with READ_ONCE()/WRITE_ONCE() on the cached real-time offset,
and those arrived in 3.19 (include/linux/compiler.h), so 3.17 and 3.18
did not build. ACCESS_ONCE() stands in below 3.19.
wolfcrypt/src/rng_bank.c:
* wc_rng_bank_fini(): a root pinned by an external (caller-spawned)
child, or held, is now refused before commitment, with the bank fully
intact -- STILL_REFERENCED_E / BUSY_E, as for a member. Previously a
root free failure was only tallied, the bank's refcount and flags were
destroyed, and the retry of wc_rng_bank_free() deallocated the bank
under the child's RBGC_parent. The pre-commit check counts member
pins (released by the member frees) and refuses on any pin beyond
them, regardless of the root's status: a root torn down by a failed
reinit is NOT_INIT with its lock word live and its pins outstanding,
and was previously skipped. The post-commit root free now restores
the refcount and returns on BUSY_E / STILL_REFERENCED_E, as the member
loop does, so a child that arrives after the pre-commit check still
leaves a retryable bank.
* wc_rng_bank_reinit_rng(): WC_RNG_INIT_FLAG_PRESERVE_LOCK is added
only when WC_RNG_HAVE_LOCK, as wc_rng_bank_inst_reinit() already does.
_InitRng() refuses the flag with NOT_COMPILED_IN otherwise, so in a
bank built without the lock facility wc_rng_bank_root_rng_reinit()
tore the root down and could not reinstantiate it; the inst-op gate is
the exclusion in that arm.
wolfcrypt/src/random.c:
* _InitRng(): on an RDRAND-capable CPU with HAVE_INTEL_RDRAND, the DRBG
bypass now refuses caller-supplied seed material with
WRONG_TYPE_OBJECT_E, the instance left uninstantiated. Previously
wc_InitRngNonce_UserSeed() and wc_RNG_global_fallback_init_user_seed()
returned success with the seed silently dropped and the instance
relabeled stratum 0, generating from RDRAND -- a primary-class
instance under a user-seeded name. Instantiating the DRBG anyway is
not an option: wc_RNG_GenerateBlock() reads RDRAND whenever the CPU
has it, DRBG or not. wc_InitRngFixedSeed()'s post-hoc backstop
(which stays, for CUSTOM_RAND_GENERATE_BLOCK) returns the same code.
doc/dox_comments/header_files/random.h: wc_InitRngNonce_UserSeed()
documents the refusal; the previous text described a bypass
"HAVE_INTEL_RDRAND without HAVE_HASHDRBG", which cannot reach the
function (WC_RNG_HAVE_RBGC requires HAVE_HASHDRBG), while the reachable
case went undocumented.
wolfcrypt/test/test.c: rng_lifecycle_test() gains the bank-teardown
scenario (external child of the root; refused fini leaves the bank
generating; held root; torn-down pinned root; convergence once the child
is released) and a lock-less bank root reinit. The latter fails against
the previous rng_bank.c.
linuxkm/module_hooks.c: in the pre-3.17 NMI-context time accessors, read
jiffies after the calibration latch rather than at entry: a calibration
published by another CPU between the two can carry a later tick, and
the unsigned difference wrapped.
linuxkm/linuxkm_wc_port.h: in_task() shim for kernels before 4.10, where
it first appears (include/linux/preempt.h); wc_linuxkm_can_long_loop()
used it unconditionally.
Fixes for RNG invalidation handling, WC_RNG_BANK_HAVE_ROOT_RNG gating, and
CONFIG_PREEMPT_RT preemption balance:
* Pivot on WC_RNG_BANK_HAVE_ROOT_RNG for the entropy daemon's root_rng
setup/teardown, the root stats dump, and the invalidation pre/post checks,
replacing hand-spelled feature disjunctions -- glue now follows root_rng
member existence exactly, including the WC_RNG_HAVE_LOCK-only configs the
old spellings missed, and root-less configs now compile the affected
functions cleanly.
* Wipe staged output on invalidation in the user-facing paths:
wc_get_random_bytes_user() and wc_extract_crng_user() now ForceZero the
staged block on EAGAIN before the post-check. Bytes generated before an
invalidation was noticed may be live in a forked VM; wc__get_random_bytes()
already wiped, the user paths did not.
* Rework invalidation retry in wc__get_random_bytes(): retire the ad-hoc
1-second jiffies window in favor of rng_invalidation_post_check() as the
single bounding authority -- EAGAIN retries, anything else is terminal
(-ETIMEDOUT/-EBUSY/-EINTR from the post-check) -- matching the user-facing
paths.
* WC_DRBG_NEXT_STIR_LEN -> WC_RNG_NEXT_STIR_LEN ripple in
wc_mix_pool_bytes(); root stats label "wc_entropyd root" -> "default bank
root" (the stats are the bank's root's, daemon or no daemon).
* wc_lkm_LockMutex(): latch wc_linuxkm_can_block() once at entry (the
caller's context is the disposition that matters, and the function itself
toggles preemption), and on CONFIG_PREEMPT_RT kernels skip the
post-trylock preempt_enable() -- RT spin_trylock() does not disable
preemption, so re-enabling underflowed the preempt count.
Fix error-trace machinery and add missing WC_NO_ERR_TRACE()s: errno-domain
comparisons in the PK and SHA glue and module_hooks (ENOMEM, EAGAIN, EINTR,
EBADMSG, and wolfCrypt INTERRUPTED_E/NOT_READY_E sites) now use the
-WC_NO_ERR_TRACE(E*) idiom, with .wolfssl_known_macro_extras updated to match.
Change default kernel "mutex" (spinlock) behavior: atomic-context callers now
never spin -- they trylock and fail fast -- and task-context lockers re-enable
preemption (and with it, migration) immediately after acquisition, rebalancing
the preempt count just before release. The legacy behavior -- task-context
holders promoted to non-preemptible for the duration of the hold, permitting
atomic-context callers (except NMI) to spin-wait -- remains available via a new
WC_LINUXKM_SPIN_IN_ATOMIC macro, which is #error-incompatible with
CONFIG_PREEMPT_RT; the mutex state field (.irq_flags vs .preempt_reenabled)
follows the selected regime, and wc_lkm_UnlockMutex() moves out of line (with a
pie redirect table entry).
Add pre- and post-checks bracketing DRBG operations against the invalidation
recovery window (keyed off last_invalidation_at, stamped at invalidation). The
pre-check works as a waiting room, keeping callers from generating a CAS scan
storm during the invalidation recovery interval (during which there are no
usable RNGs system-wide). In the post-check, -EAGAIN passes through for another
lap, a window overrun returns -ETIMEDOUT, atomic-context callers get one-shot
-EBUSY (blockability is checked before signal state, because in hardirq context
`current` is an unrelated task), pending signals return -EINTR, and otherwise
the caller relaxes and retries. Both checks have stubs so reduced
configurations build, and in-window messaging is ratelimited.
Add emergency reinitialization of the entropy daemon's root_rng: when the
recovery sweep finds the root invalidated, the daemon retries
wc_RNG_DRBG_Reseed_Now() with an uncredited timestamp nonce while the failure
remains retryable (BUSY_E, NOT_READY_E, ENTROPY_RT_E, ENTROPY_APT_E), relaxing
between attempts; for the default bank the reseed runs with preemption disabled
to maximize its chances. If the reseed fails terminally, the daemon makes one
last-ditch recovery via wc_FreeRng() and wc_rng_bank_root_rng_init() (or
wc_InitRng() for a standalone root); a successful reinit always clears the
invalidated state. If that too fails, the daemon exits, leaving bank instances
to self-reseed. Outcomes are reported at severities differentiated by default
vs non-default bank (pr_notice/pr_info success, pr_emerg/pr_err failure). The
daemon also now supports a standalone malloc'd root_rng where no bank features
are built, instantiated at startup and freed on exit.
Also:
* wc_linuxkm_drbg_generate_tfm(): retry loop around the invalidation window
replaces the single-shot call; output buffer is zeroized on terminal error.
* wc__get_random_bytes(): honors the recovery window, and zeroizes its
staging buffer both before retry re-entry and on the error exit. An
invalidation may be peculiar to this VM and leave the bytes live in another
VM, so they are wiped when abandoned, as possible CSP elsewhere.
* wc_linuxkm_entropy_daemon(): the registered-leaf recovery pass is now
additionally gated on WC_LINUXKM_HAVE_RNG_REGISTRY, matching its actual
dependency; seed health test failures (ENTROPY_RT_E/ENTROPY_APT_E) on bank
instances are now reported unconditionally at pr_warn_ratelimited severity
("entropy source suspect"), rather than pr_err only under WC_VERBOSE_RNG;
the WC_RNG_DEBUG_STATS dump at daemon exit is removed (was redundant).
* wolfcrypt/test/test.c: SVR access-counter checks now additionally gated on
new WC_SVR_HAVE_FLAGS, matching the configurations that actually maintain
the counters.
opening, leaving just the existing batch opening inside the main crypt loop, to
avoid unnecessary atomic-context allocations in the preamble;
linuxkm/module_hooks.c:
* add wc_linuxkm_deinstall(), which idempotently deregisters all algorithms and
sysfs nodes, and call it from libwolfssl_cleanup() and wolfssl_exit();
* refactor in_optest atomic flag (local to FIPS_optest_trig_common()) to
file-scoped in_fips_test, and use it as a mutex for both
FIPS_rerun_self_test_handler() and FIPS_optest_trig_common().
* always call wolfCrypt_Init() directly, even in TLS builds, to assure
mutex-free scheduler context during allocations;
* convert sysfs nodes from struct kobj_attribute to struct module_attribute,
matching the module_sysfs_ops dispatch on THIS_MODULE->mkobj.kobj, for
compatibility with CONFIG_CFI_CLANG (kCFI) kernels.
* add libwolfssl_inited flag variable and use it for proper error-path cleanup;
* call wc_linuxkm_allocate_svr_states() and wc_linuxkm_free_svr_states() explicitly in FIPS builds (avoid atomic-context synchronous call to wc_linuxkm_allocate_svr_states() via fipEntry());
* add missing WOLFSSL_FIPS_DEV_NO_POST gates;
* in wc_lkm_LockMutex(), return BUSY_E immediately if in_nmi().
* add WOLFSSL_DEBUG_TRACE_ERROR_CODES coverage for all system error codes used by module_hooks.c and its includes.
* implement double-run wolfCrypt_IntegrityTest_fips() and wc_RunAllCast_fips() in WC_C_DYNAMIC_FALLBACK configurations, once with acceleration on (leveraging fallback instrumentation to fail the module if any fallbacks occur), and once with it forced off using DISABLE_VECTOR_REGISTERS().
native vector register save buffers, serving hardirq and softirq contexts
where kernel_fpu_begin() is unusable.
previously these contexts were refused (WC_ACCEL_INHIBIT_E): hardirq
outright, softirq whenever !may_use_simd() (pre-6.15, a softirq
interrupting a foreign kernel_fpu section). an exact save/compute/restore
cycle is invisible to the interrupted context and to kernel FPU
bookkeeping, and run-to-completion freezes the blocking condition beneath
the section, so both classes are natively serviceable.
- per-CPU {softirq, hardirq} state, each {depth, pin_preempt, save_area};
xsave64/xrstor64 with RFBM 0xe7 & XCR0 (fxsave64 fallback), plain XSAVE
only; feature detection by direct CPUID/XGETBV, independent of kernel
cpufeature machinery.
- recursion is a per-(CPU, class) depth count -- save on the 0->1 edge,
restore on 1->0; softirq sections hold preempt_disable() (load-bearing
on PREEMPT_RT).
- hardirq serves plain saves; softirq serves plain and _MAYBE_INHIBIT
(its legacy short-circuit is bypassed when native is ready); _INHIBIT,
hardirq _MAYBE, and NMI keep refusal semantics; task context is
untouched.
- wc_svr_native_init() activates in place from
wc_linuxkm_allocate_svr_states() in any context but NMI -- the FIPS
pre-Init lazy allocation arrives inside the outermost save's own
bracket, and wolfCrypt_Init() has been observed with irqs disabled. a
self-bracketed register round-trip selftest gates wc_svr_native_ready;
on any failure the long-standing refusal semantics remain fully in
force.
- wc_linuxkm_svr_native_is_ready(): runtime readiness accessor for policy
decisions elsewhere (e.g. the FIPS<v7 default-bank NO_VECTOR_OPS pin).
- WOLFSSL_LINUXKM_VERBOSE_DEBUG: split the softirq WC_ACCEL_INHIBIT_E
counter into per-class counters (softirq, hardirq, NMI, other),
reported at module shutdown.
in non-VERBOSE builds with WC_SVR_DONT_USE_NATIVE_REG_BUFS (or in non-SVR
builds), generated object code is byte-identical to the previous revision.
and AM_CONDITIONAL BUILD_FIPS_NO_POST. Refactor FIPS dev/ready version setup:
hoist ENABLED_FIPS_DEV / ENABLED_FIPS_READY to set -DWOLFSSL_FIPS_DEV /
-DWOLFSSL_FIPS_READY centrally, and switch the FIPS AS_CASE arms from
`test "$FIPS_VERSION" != "dev"` to `test "$ENABLED_FIPS_DEV" != "yes"` so the
dev semantics extend to v5-dev/v6-dev/lean-aesgcm-dev.
src/include.am: under !BUILD_FIPS_NO_POST, drop fips.c / fips_test.c /
wolfcrypt_first.c / wolfcrypt_last.c from the build (dev-no-post uses no fips
repo content).
wolfssl/wolfcrypt/settings.h, wolfssl/wolfcrypt/wc_compat.h: under
WOLFSSL_FIPS_DEV_NO_POST, squat WOLF_CRYPT_FIPS_H to inhibit fips.h, and change
the FIPS_READY/DEV version block guard to `!defined(HAVE_FIPS_VERSION)`
(required so an externally supplied version is not clobbered).
wolfssl/wolfcrypt/fips_test.h: add WOLFSSL_FIPS_DEV_NO_POST stub block
(fipsCastStatus_get, the PRIVATE_KEY macros) so master builds without the fips
repo.
linuxkm/linuxkm_wc_port.h, linuxkm/module_hooks.c: accommodate
WOLFSSL_FIPS_DEV_NO_POST (guard verifyCore / CAST / fencepost paths that the
fips repo would otherwise provide; force WC_USE_PIE_FENCEPOSTS_FOR_FIPS).
wolfcrypt/test/test.c: in hmac_sha256_test(), don't expect HMAC_KAT_FIPS_E in
WOLFSSL_FIPS_DEV_NO_POST builds.
wolfssl/wolfcrypt/types.h: add stub macro for
wolfCrypt_SetPrivateKeyReadEnable_fips() when WOLFSSL_FIPS_DEV_NO_POST.
* set up WC_LINUXKM_HAVE_SELFTEST and WC_LINUXKM_HAVE_SELFTEST_FULL early.
* replace the forced-downgrade from FIPS 186-5 to 186-4 for SHA-1 support, with a #error if configuration is incompatible.
linuxkm/linuxkm_wc_port.h, linuxkm/lkcapi_glue.c, linuxkm/module_exports.c.template, linuxkm/module_hooks.c, wolfcrypt/src/random.c: consolidate into the warning-masked span of linuxkm_wc_port.h, the #includes for kernel headers linux/fips.h, linux/vmalloc.h, and linux/random.h.
linuxkm/: when logging PTR_ERR(), cast it to int, and use "%d" as the format. Globally, `#define PTR_ERR(x) ((int)PTR_ERR(x))` in linuxkm_wc_port.h to fix clang warnings on kernel headers.
linuxkm/lkcapi_aes_glue.c: add casts in linuxkm_test_aesgcm() to mollify clang.
linuxkm/linuxkm_wc_port.h, linuxkm/module_hooks.c:
* add __clang__ compat code to allow including clang stdatomic.h while masking out kernel-incompatible __CLANG_STDINT_H.
* add clang-specific suppressions for kernel headers (-Wshorten-64-to-32, -Wframe-address).
linuxkm/lkcapi_sha_glue.c:
* in wc__get_random_bytes(), add bounds-checking for len.
* in wc_extract_crng_user(), fix type conflicts.
wolfssl/wolfcrypt/wc_port.h and wolfssl/wolfcrypt/types.h:
* move the old-FIPS compatibility mapping from INLINE to WC_INLINE from types.h to wc_port.h.
* activate stdatomic.h for clang kernel module builds.
linuxkm/Kbuild:
* add clang-specific flags.
* add gcc gate around gcc-specific flags.
* allow override value for MAX_STACK_FRAME_SIZE.
wolfcrypt/src/asn.c: add casts in GetFormattedTime_ex() to mollify clang build of linuxkm.
fix F-3291: Copy-paste error in linuxkm_test_aesgcm error message uses WOLFKM_AESCBC_DRIVER
fix F-1431: AES-GCM RFC4106 SetKey Uses memcpy Instead of XMEMCPY for Nonce Copy
(note, for F-1431, changed all relevant memset() and memcpy() calls in linuxkm/ to XMEMSET() and XMEMCPY() respectively.)
* add WOLFSSL_API attribute to wc_linuxkm_sig_ignore_begin(), wc_linuxkm_sig_ignore_end(), wc_linuxkm_check_for_intr_signals(), and wc_linuxkm_relax_long_loop().
* fix WC_CONTAINERIZE_THIS macro wrappers for wc_linuxkm_sig_ignore_begin() and wc_linuxkm_sig_ignore_end() (stray semicolons).
linuxkm/linuxkm_wc_port.h, linuxkm/lkcapi_sha_glue.c, linuxkm/module_hooks.c: add wc_linuxkm_can_block(), and refactor ad hoc `preempt_count() != 0` checks for sleep safety as calls to wc_linuxkm_can_block().
linuxkm/module_hooks.c: fix wc_linuxkm_malloc_usable_size() implementation for kvmalloc() compatibility.
linuxkm/Makefile, linuxkm/linuxkm-fips-hash-wrapper.sh, linuxkm/linuxkm_memory.c: refactor coreKey extraction to use ELF tools rather than WOLFCRYPT_FIPS_CORE_DYNAMIC_HASH_VALUE and user_settings.h.
linuxkm/module_hooks.c: add stack measurement for wc_RunAllCast_fips().
tests/api/test_slhdsa.c: frivolous initialization to work around a false positive -Wmaybe-uninitialized in slhdsa_der_roundtrip_one().
wolfcrypt/src/wc_slhdsa.c, wolfssl/wolfcrypt/wc_slhdsa.h:
* refactor lifecycle management for SHA-2 objects to fix a leak via wc_SlhDsaKey_CheckKey().
* add support for WC_SLHDSA_NO_ASM.
* add WOLFSSL_SLHDSA_VERIFY_ONLY gates around prototypes, to get compile-time failures for misuse.
wolfcrypt/test/test.c:
* clean up myFipsCb() and restore usability of TEST_ALWAYS_RUN_TO_END with bad FIPS hash (useful test coverage).
* add wc_RunAllCast_fips() to wolfcrypt_test().
* when WOLFSSL_KERNEL_MODE or BENCH_EMBEDDED, force on WOLFSSL_SLHDSA_VERIFY_ONLY unless WOLFSSL_SLHDSA_FORCE_FULL_TESTS is defined.
wolfssl/wolfcrypt/settings.h:
* add WC_MLKEM_NO_ASM to WOLFSSL_LINUXKM section to work around asm bug.
* remove clause in WOLFSSL_KERNEL_MODE section that forced on WOLFSSL_SLHDSA_VERIFY_ONLY.
linuxkm/Makefile: update the GENERATE_RELOC_TAB recipe to generate both wc_linuxkm_pie_text_reloc_tab[] and wc_linuxkm_pie_rodata_reloc_tab.
linuxkm/linuxkm-fips-hash-wrapper.sh: add handling for wc_linuxkm_pie_rodata_reloc_tab.
linuxkm/linuxkm-fips-hash.c: add handling for rodata_reloc_tab.*.
linuxkm/linuxkm_memory.c:
* refactor find_reloc_tab_offset() to be segment-agnostic and tolerate empty reloc tabs.
* refactor wc_reloc_normalize_segment():
* to be segment-agnostic,
* identify the src segment dynamically,
* return BAD_FUNC_ARG where previously returning literal -1,
* use seg_in_out_len arg to accommodate size skew between input and output (not currently used), and
* rename working vars for better mnemonicitude.
* update wc_fips_generate_hash() to
* handle seg_map->rodata_reloc_tab,
* use new calling convention for wc_reloc_normalize_segment(), and
* add wc_reloc_normalize_segment() loop for .rodata_wolfcrypt.
linuxkm/linuxkm_memory.h and linuxkm/linuxkm_wc_port.h: rename WOLFSSL_TEXT_SEGMENT_CANONICALIZER* to WOLFSSL_SEGMENT_CANONICALIZER*, with backward-compat provisions.
linuxkm/module_hooks.c:
* add wc_linuxkm_normalize_relocations_noresize() backward-compat wrapper.
* wolfssl_init(): add .rodata_wolfcrypt relocation handling alongside existing .text_wolfcrypt handling, and update for new wc_reloc_normalize_segment() calling convention.
* add seg_map.rodata_reloc_tab initialization.
* update wc_linuxkm_normalize_relocations() to be segment-agnostic and use new wc_reloc_normalize_segment() calling convention.
also, minor fixes:
linuxkm/lkcapi_aes_glue.c: in linuxkm_test_aescfb(), call the appropriate aes_cfb_test(), not aes_cfb_test().
linuxkm/lkcapi_glue.c: fix bugprone-macro-parentheses in REGISTER_ALG_OPTIONAL().
linuxkm/module_hooks.c: in updateFipsHash(), add suppression for clang-diagnostic-cast-function-type-strict warnings around actually-safe function pointer casts in call to wc_fips_generate_hash().
fixes and workarounds for clang-tidy complaints:
* clang-diagnostic-unknown-warning-option
* bugprone-sizeof-expression
* clang-diagnostic-error "address argument to atomic operation must be a pointer to a trivially-copyable type"
* bugprone-macro-parentheses
* clang-diagnostic-unused-but-set-variable
* readability-redundant-declaration
