database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2026-10-23 18:49:29 +03:00
Code Activity

MDEV-41115 Add RISC-V Zbc accelerated CRC32C implementation

Browse Source 
Add an accelerated CRC32C path for RISC-V 64-bit using the carry-less
multiply (clmul/clmulh) extension of the Zbc extension. The same
instructions also accelerate the ISO 3309 CRC-32 (Ethernet, zlib) that
my_checksum() uses, so both polynomials are served by a single core,
the way crc32_avx512() already serves both on x86:
  - mysys/CMakeLists.txt: detect the extension by compiling the same
    inline assembler that the implementation uses, add
    -DHAVE_RISCV_ZBC and build crc32/crc32c_riscv_zbc.cc with
    -march=rv64gc_zbc_zbb. Only that compilation unit is built for the
    extension; the detection and dispatch in crc32/crc32c_riscv.cc use
    the normal settings, so that no code that may run on a CPU without
    the extension can be compiled for it (MDEV-24745).
  - mysys/crc32/crc32c.cc: on RISC-V, my_crc32c is an indirect function
    (ifunc) whose resolver selects crc32c_riscv_zbc() or the existing
    crc32c_slow() at load time. The other architectures keep the
    existing *_available() dispatch unchanged.
  - mysys/crc32/crc32c_riscv_zbc.cc (new): 128-bit 4-way parallel
    folding + Barrett reduction using clmul/clmulh inline asm, with a
    bitwise fallback for short inputs. The polynomial-dependent
    constants (fold factors, Barrett constants, reflected polynomial)
    live in a constexpr rv_crc32_tab passed to the one core function,
    so the same code serves both polynomials:
      * crc32c_riscv_zbc(): CRC-32C, Castagnoli 0x1EDC6F41
      * crc32_riscv_zbc():  ISO 3309, 0x04C11DB7
  - mysys/crc32/crc32c_riscv.cc (new): the runtime Zbc probe,
    rv_zbc_supported(). It uses the riscv_hwprobe system call and
    nothing else; a /proc/cpuinfo probe cannot be used, because the
    function is also called from the ifunc resolver and that runs before
    the C library is initialised.
  - mysys/crc32ieee.cc: likewise, my_checksum is dispatched by an ifunc
    on RISC-V, its resolver selecting crc32_riscv_zbc() or the existing
    my_crc32_zlib(), instead of the runtime init_crc32() used elsewhere.

On RISC-V the dynamic linker calls a resolver as
(hwcap, __riscv_hwprobe, reserved). rv_zbc_supported() uses the
__riscv_hwprobe pointer passed there when it is available; a C library
older than glibc 2.40 passes NULL in that position, and the system call
is then made directly. A resolver cannot report "extension absent" by
returning NULL, as that crashes at the first call, so it returns a real
fallback; crc32c_slow() and my_crc32_zlib() are both linked
unconditionally and so are the natural choice.

Correctness: RFC 3720 + official unittest/mysys/crc32-t.c (36/36; 18
of its cases exercise my_checksum(), i.e. ISO 3309). my_checksum() was
also cross-checked against zlib crc32() over an exhaustive sweep of
lengths 0..4096 x 16 input alignments plus non-zero seeds: 0
mismatches.

Performance on Spacemit X100 (k3), vs inherited slicing-by-4
crc32c_slow (official slow path, same my_crc32c entry):

    len      slow (MB/s)   Zbc (MB/s)   speedup
    128 B        56            477         8.58x
    256 B        55            920        16.59x
    1 KiB        55           3125        56.50x
    4 KiB        55           7216       130.21x
    64 KiB       55          11893       214.68x

ISO 3309 (my_checksum, 64 MiB buffer on the same machine): 0.65 GB/s
on the generic zlib path before, about 8 GB/s with the Zbc core after
(~12x), on par with my_crc32c.

No behavior change on non-riscv64 or Zbc-less RISC-V builds: the code
is compiled only when the extension is available, and each entry point
selects the accelerated implementation only when the probe reports Zbc.

Assisted-by: YuanSheng:DeepSeek-V4-Flash
Co-authored-by: Xiaofei Gong <gongxiaofei24@iscas.ac.cn>
Signed-off-by: Jiakai Xu <xujiakai2025@iscas.ac.cn>
This commit is contained in:
Jiakai Xu
2026-09-14 02:59:36 +00:00
committed by Marko Mäkelä
parent 531d2e2114
commit 45717ef4c7
5 changed files with 396 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
mysys/CMakeLists.txt
View File

@@ -150,6 +150,43 @@ ELSEIF(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64|AARCH64")
SET(CMAKE_REQUIRED_FLAGS "${SAVE_CMAKE_REQUIRED_FLAGS}")
ENDIF()
ENDIF()
ELSEIF(CMAKE_SYSTEM_PROCESSOR MATCHES "riscv64|RISCV64")
IF(CMAKE_CXX_COMPILER_ID MATCHES "Clang|GNU")
include(CheckCXXSourceCompiles)
SET(SAVE_CMAKE_REQUIRED_FLAGS "${CMAKE_REQUIRED_FLAGS}")
SET(CMAKE_REQUIRED_FLAGS "${CMAKE_REQUIRED_FLAGS} -march=rv64gc_zbc_zbb")
# Check for the Zbc extension by compiling the same inline assembler that
# the implementation uses. This also verifies that the compiler/assembler
# accepts the -march value.
CHECK_CXX_SOURCE_COMPILES("
#include <stdint.h>
uint64_t clmul(uint64_t a, uint64_t b) {
uint64_t r;
__asm__ (\"clmul %0, %1, %2\" : \"=r\"(r) : \"r\"(a), \"r\"(b));
return r;
}
uint64_t clmulh(uint64_t a, uint64_t b) {
uint64_t r;
__asm__ (\"clmulh %0, %1, %2\" : \"=r\"(r) : \"r\"(a), \"r\"(b));
return r;
}
int main() { return (int) (clmul(1, 2) ^ clmulh(1, 2)); }
" HAVE_RISCV_ZBC)
SET(CMAKE_REQUIRED_FLAGS "${SAVE_CMAKE_REQUIRED_FLAGS}")
IF(HAVE_RISCV_ZBC)
ADD_DEFINITIONS(-DHAVE_RISCV_ZBC)
# The ISA extension is enabled only for the core, so that the detection
# and dispatch code in crc32c_riscv.cc cannot be compiled for it.
SET(MYSYS_SOURCES ${MYSYS_SOURCES}
crc32/crc32c_riscv.cc crc32/crc32c_riscv_zbc.cc)
SET_SOURCE_FILES_PROPERTIES(crc32/crc32c_riscv_zbc.cc PROPERTIES
COMPILE_FLAGS "-march=rv64gc_zbc_zbb")
ENDIF()
ENDIF()
ENDIF()
IF(CMAKE_SYSTEM_PROCESSOR MATCHES "ppc64|powerpc64" OR CMAKE_SYSTEM_NAME MATCHES AIX)

43
mysys/crc32/crc32c.cc
View File

@@ -489,11 +489,39 @@ static int arch_ppc_probe(void) {
#if defined(HAVE_ARMV8_CRC)
extern "C" my_crc32_t crc32c_aarch64_available(void);
extern "C" const char *crc32c_aarch64_impl(my_crc32_t);
#elif defined HAVE_RISCV_ZBC
extern "C" unsigned crc32c_riscv_zbc(unsigned, const void *, size_t);
extern "C" int rv_zbc_supported(void *);
extern "C" const char *crc32c_riscv_impl(my_crc32_t);
#elif defined __i386__||defined __x86_64__||defined _M_X64||defined _M_IX86
extern "C" my_crc32_t crc32c_x86_available(void);
extern "C" const char *crc32c_x86_impl(my_crc32_t);
#endif
#if defined HAVE_RISCV_ZBC
static my_crc32_t crc32c_riscv_choose(void *hwprobe)
{
return rv_zbc_supported(hwprobe) ? crc32c_riscv_zbc : crc32c_slow;
}
/* The RISC-V resolver. Unlike the other architectures, the implementation is
selected by an indirect function instead of at the first call, because the
target operating systems of RISC-V (Linux and FreeBSD) support that.
The dynamic linker calls this at load time, before main(), so it may only
run code that is safe there. rv_zbc_supported() is: it uses the
riscv_hwprobe system call and nothing else.
The resolver must never return NULL. Whereas the *_available() functions
used elsewhere return NULL to mean "the extension is not present", an ifunc
resolver that returns NULL crashes at the first call. crc32c_slow() is
therefore returned instead; it is linked unconditionally anyway. */
extern "C" { static my_crc32_t rv_crc32c_resolver(unsigned long long,
void *hwprobe, void *)
{
return crc32c_riscv_choose(hwprobe);
} }
#else
static inline my_crc32_t Choose_Extend()
{
#if defined HAVE_POWER8 && defined HAS_ALTIVEC
@@ -510,10 +538,18 @@ static inline my_crc32_t Choose_Extend()
}
static const my_crc32_t ChosenExtend= Choose_Extend();
#endif
extern "C" const char *my_crc32c_implementation()
{
#if defined HAVE_POWER8 && defined HAS_ALTIVEC
#if defined HAVE_RISCV_ZBC
/* The address of an ifunc symbol is the address of the selected
implementation, so it cannot be compared against a specific one from
here. The probe is repeated instead; that is what the resolver uses to
make the same choice. */
if (const char *ret= crc32c_riscv_impl(crc32c_riscv_choose(NULL)))
return ret;
#elif defined HAVE_POWER8 && defined HAS_ALTIVEC
if (ChosenExtend == crc32c_ppc)
return "Using POWER8 crc32 instructions";
#elif defined HAVE_ARMV8_CRC
@@ -528,7 +564,12 @@ extern "C" const char *my_crc32c_implementation()
} // namespace crc32c
} // namespace mysys_namespace
#if defined HAVE_RISCV_ZBC
extern "C" uint32 my_crc32c(uint32, const void *, size_t)
__attribute__((ifunc("rv_crc32c_resolver")));
#else
extern "C" uint32 my_crc32c(uint32 crc, const void *buf, size_t size)
{
return mysys_namespace::crc32c::ChosenExtend(crc,buf, size);
}
#endif

86
mysys/crc32/crc32c_riscv.cc Normal file
View File

@@ -0,0 +1,86 @@
/* RISC-V CRC-32 runtime detection.
This file is compiled with the normal settings. The Zbc-accelerated core is
in crc32c_riscv_zbc.cc, the only compilation unit for which the ISA extension
is enabled. That way rv_zbc_supported(), which may run on a CPU that does not
implement Zbc, cannot be compiled for the extension; enabling it for a whole
compilation unit was the cause of a past problem (MDEV-24745).
rv_zbc_supported() is also called from an indirect function (ifunc) resolver.
A resolver runs at load time, before libc has been initialised, so it must not
use anything that depends on that: fopen() on /proc/cpuinfo, as an earlier
version of this probe did, faults there. Only the riscv_hwprobe system call is
used instead. Since glibc 2.40 the dynamic linker passes a pointer to
__riscv_hwprobe() as an argument to a RISC-V resolver; on older versions that
argument is NULL and the system call is made directly.
*/
#include <stddef.h>
#include <unistd.h>
#include <sys/syscall.h>
#ifdef __has_include
# if __has_include(<sys/hwprobe.h>)
# include <sys/hwprobe.h>
# define HAVE_SYS_HWPROBE_H 1
# endif
#endif
#ifndef HAVE_SYS_HWPROBE_H
/* Fallback for a C library that does not provide <sys/hwprobe.h> (glibc
before 2.40). The structure and the constants are copied from the Linux UAPI
header <asm/hwprobe.h>. */
struct riscv_hwprobe { long long key; unsigned long long value; };
# define RISCV_HWPROBE_KEY_IMA_EXT_0 4
#endif
#ifndef RISCV_HWPROBE_EXT_ZBC
# define RISCV_HWPROBE_EXT_ZBC (1ULL << 7)
#endif
#ifndef SYS_riscv_hwprobe
# ifdef __NR_riscv_hwprobe
# define SYS_riscv_hwprobe __NR_riscv_hwprobe
# else
# define SYS_riscv_hwprobe 258 /* riscv_hwprobe, if <sys/syscall.h> is old */
# endif
#endif
typedef unsigned (*my_crc32_t)(unsigned, const void *, size_t);
/* Whether the Zbc extension is available. Safe to call from an ifunc
resolver: it does not touch any C library state that is not yet
initialised.
hwprobe is the __riscv_hwprobe pointer that the dynamic linker passes to a
resolver (glibc 2.40 and later), or NULL on an older C library. */
extern "C" int rv_zbc_supported(void *hwprobe)
{
struct riscv_hwprobe p;
p.key= RISCV_HWPROBE_KEY_IMA_EXT_0;
p.value= 0;
#ifdef HAVE_SYS_HWPROBE_H
if (hwprobe != NULL)
{
unsigned long long value= 0;
if (__riscv_hwprobe_one(reinterpret_cast<__riscv_hwprobe_t>(hwprobe),
RISCV_HWPROBE_KEY_IMA_EXT_0, &value) != 0)
return 0;
return (value & RISCV_HWPROBE_EXT_ZBC) != 0;
}
#else
(void) hwprobe;
#endif
if (syscall(SYS_riscv_hwprobe, &p, (size_t) 1, (size_t) 0, NULL, 0) != 0)
return 0;
return (p.value & RISCV_HWPROBE_EXT_ZBC) != 0;
}
extern "C" unsigned crc32c_riscv_zbc(unsigned, const void *, size_t);
extern "C" const char *crc32c_riscv_impl(my_crc32_t c)
{
if (c == crc32c_riscv_zbc)
return "Using RISC-V Zbc carry-less multiply instructions";
return NULL;
}

209
mysys/crc32/crc32c_riscv_zbc.cc Normal file
View File

@@ -0,0 +1,209 @@
/* RISC-V Zbc accelerated CRC-32 core.
Zbc scalar path: clmul/clmulh inline asm + 128-bit 4-way parallel folding +
Barrett reduction (adapted from apache/brpc#3312).
This is the only compilation unit for which the ISA extension is enabled:
it is compiled with -march=rv64gc_zbc_zbb (see mysys/CMakeLists.txt), so that
the compiler may emit the carry-less multiplication instructions. The runtime
detection and dispatch live in crc32c_riscv.cc, which is compiled with the
normal settings, so that no function that must run on a CPU without Zbc can
be compiled for it (see MDEV-24745).
Two polynomials share this core: CRC-32C (Castagnoli, my_crc32c()) and the
ISO 3309 CRC-32 (my_checksum()). They differ only in their constants, which
are passed in as a table, mirroring how the x86 crc32_avx512() accepts a
const crc32_tab & parameter.
*/
#include <stdint.h>
#include <stddef.h>
#include <string.h>
#include "assume_aligned.h"
namespace mysys_namespace {
namespace crc32c {
/* Constants of one CRC-32 polynomial, in bit-reflected form.
See the use sites in rv_crc32_impl() for what each field is. */
struct rv_crc32_tab {
alignas(16) uint32_t fold[4]; /* x^(64*i+64) mod P for i=1..4 */
uint64_t const0; /* x^64 mod P */
uint64_t const1; /* x^96 mod P */
uint64_t quo; /* floor(x^64 / P) */
uint64_t poly; /* P(x) true LE full */
uint32_t bitwise_poly; /* reflected polynomial, small-chunk path */
};
// RISC-V Zbc carry-less multiplication inline helpers
static inline uint64_t rv_clmul(uint64_t a, uint64_t b) {
uint64_t result;
__asm__ volatile ("clmul %0, %1, %2" : "=r"(result) : "r"(a), "r"(b));
return result;
}
static inline uint64_t rv_clmulh(uint64_t a, uint64_t b) {
uint64_t result;
__asm__ volatile ("clmulh %0, %1, %2" : "=r"(result) : "r"(a), "r"(b));
return result;
}
// Bitwise CRC fallback for small chunks
static inline uint32_t rv_crc32_bitwise(uint32_t crc, const uint8_t* buf,
size_t len, uint32_t poly) {
uint32_t c = crc;
for (size_t i = 0; i < len; ++i) {
c ^= buf[i];
for (int k = 0; k < 8; ++k) {
c = (c >> 1) ^ ((c & 1) ? poly : 0);
}
}
return c;
}
// Fold the 128-bit CRC state (lo:hi) with fold constants k0/k1, then XOR in
// the value (v0:v1). The latter is either the next chunk of input data or
// another folded state to be combined.
static inline void rv_fold_pair(uint64_t* lo, uint64_t* hi,
uint64_t k0, uint64_t k1,
uint64_t v0, uint64_t v1) {
uint64_t l = rv_clmul(*lo, k0) ^ rv_clmul(*hi, k1);
uint64_t h = rv_clmulh(*lo, k0) ^ rv_clmulh(*hi, k1);
*lo = l ^ v0;
*hi = h ^ v1;
}
/* CRC-32C (Castagnoli), polynomial 0x1EDC6F41 */
static constexpr rv_crc32_tab rv_crc32c_tab= {
{ 0x740eef02, 0x9e4addf8, 0xf20c0dfe, 0x493c7d27 },
0x00000000dd45aab8ULL, 0x00000000493c7d27ULL,
0x0000000dea713f1ULL, 0x0000000105ec76f1ULL,
0x82F63B78
};
/* ISO 3309 CRC-32 (Ethernet, zlib), polynomial 0x04C11DB7 */
static constexpr rv_crc32_tab rv_crc32_iso3309_tab= {
{ 0x8f352d95, 0x1d9513d7, 0xae689191, 0xccaa009e },
0x00000000b8bc6765ULL, 0x00000000ccaa009eULL,
0x0000001f7011641ULL, 0x0000001db710641ULL,
0xEDB88320
};
constexpr uint64_t RV_CRC32_MASK32= 0x00000000FFFFFFFFULL;
// Hardware-accelerated CRC-32 using RISC-V Zbc carry-less multiplication.
// Processes data in 64-byte chunks with 128-bit folding, then Barrett reduces.
static uint32_t rv_crc32_impl(uint32_t crc, const char* buf, size_t len,
const rv_crc32_tab &tab) {
// Convert external CRC to internal register state
crc ^= 0xFFFFFFFF;
const uint8_t* p = reinterpret_cast<const uint8_t*>(buf);
size_t n = len;
// Small data: use bitwise fallback
if (n < 64) {
return rv_crc32_bitwise(crc, p, n, tab.bitwise_poly) ^ 0xFFFFFFFF;
}
// Align to 16-byte boundary
if (uintptr_t mis= reinterpret_cast<uintptr_t>(p) & 0xF) {
size_t pre = 16 - mis;
if (pre > n) pre = n;
crc = rv_crc32_bitwise(crc, p, pre, tab.bitwise_poly);
p += pre;
n -= pre;
if (n < 64) {
return rv_crc32_bitwise(crc, p, n, tab.bitwise_poly) ^ 0xFFFFFFFF;
}
}
// Load first 64 bytes and XOR CRC into the first 8 bytes.
// p is 16-byte aligned here, hence memcpy_aligned<8> is safe.
uint64_t x0, x1, y0, y1, z0, z1, w0, w1;
memcpy_aligned<8>(&x0, p + 0, 8);
memcpy_aligned<8>(&x1, p + 8, 8);
memcpy_aligned<8>(&y0, p + 16, 8);
memcpy_aligned<8>(&y1, p + 24, 8);
memcpy_aligned<8>(&z0, p + 32, 8);
memcpy_aligned<8>(&z1, p + 40, 8);
memcpy_aligned<8>(&w0, p + 48, 8);
memcpy_aligned<8>(&w1, p + 56, 8);
x0 ^= (uint64_t)crc;
p += 64;
n -= 64;
const uint64_t k1 = tab.fold[0];
const uint64_t k2 = tab.fold[1];
const uint64_t k3 = tab.fold[2];
const uint64_t k4 = tab.fold[3];
// Main loop: fold 64 bytes per iteration using 128-bit folding
// p stays 16-byte aligned across iterations
while (n >= 64) {
uint64_t d0, d1;
memcpy_aligned<8>(&d0, p + 0, 8);
memcpy_aligned<8>(&d1, p + 8, 8);
rv_fold_pair(&x0, &x1, k1, k2, d0, d1);
memcpy_aligned<8>(&d0, p + 16, 8);
memcpy_aligned<8>(&d1, p + 24, 8);
rv_fold_pair(&y0, &y1, k1, k2, d0, d1);
memcpy_aligned<8>(&d0, p + 32, 8);
memcpy_aligned<8>(&d1, p + 40, 8);
rv_fold_pair(&z0, &z1, k1, k2, d0, d1);
memcpy_aligned<8>(&d0, p + 48, 8);
memcpy_aligned<8>(&d1, p + 56, 8);
rv_fold_pair(&w0, &w1, k1, k2, d0, d1);
p += 64;
n -= 64;
}
// Reduce 4x128-bit to 1x128-bit
rv_fold_pair(&x0, &x1, k3, k4, y0, y1);
rv_fold_pair(&x0, &x1, k3, k4, z0, z1);
rv_fold_pair(&x0, &x1, k3, k4, w0, w1);
// Barrett reduction: 128-bit -> 32-bit CRC
uint64_t t4 = rv_clmul(x0, tab.const1);
uint64_t t3 = rv_clmulh(x0, tab.const1);
uint64_t t1 = x1 ^ t4;
t4 = t1 & RV_CRC32_MASK32;
t1 >>= 32;
uint64_t t0 = rv_clmul(t4, tab.const0);
t3 = (t3 << 32) ^ t1 ^ t0;
t4 = t3 & RV_CRC32_MASK32;
t4 = rv_clmul(t4, tab.quo);
t4 &= RV_CRC32_MASK32;
t4 = rv_clmul(t4, tab.poly);
t4 ^= t3;
uint32_t c = (uint32_t)((t4 >> 32) & RV_CRC32_MASK32);
// Handle remaining bytes
if (n) {
c = rv_crc32_bitwise(c, p, n, tab.bitwise_poly);
}
// Convert internal register state to external CRC
return c ^ 0xFFFFFFFF;
}
} // namespace crc32c
} // namespace mysys_namespace
extern "C" unsigned crc32c_riscv_zbc(unsigned crc, const void *buf,
size_t size)
{
return mysys_namespace::crc32c::rv_crc32_impl(crc,
(const char *) buf, size,
mysys_namespace::crc32c::rv_crc32c_tab);
}
extern "C" unsigned crc32_riscv_zbc(unsigned crc, const void *buf,
size_t size)
{
return mysys_namespace::crc32c::rv_crc32_impl(crc,
(const char *) buf, size,
mysys_namespace::crc32c::rv_crc32_iso3309_tab);
}

25
mysys/crc32ieee.cc
View File

@@ -33,9 +33,30 @@ extern "C" my_crc32_t crc32_pclmul_enabled();
#elif defined HAVE_ARMV8_CRC
extern "C" int crc32_aarch64_available();
extern "C" unsigned int crc32_aarch64(unsigned int, const void *, size_t);
#elif defined HAVE_RISCV_ZBC
extern "C" int rv_zbc_supported(void *);
extern "C" unsigned int crc32_riscv_zbc(unsigned int, const void *, size_t);
#endif
#ifdef __powerpc64__
# error "my_checksum() is defined in mysys/crc32/crc32_ppc64.c"
#endif
#if defined HAVE_RISCV_ZBC
/* my_checksum() is dispatched by an indirect function on RISC-V, mirroring
my_crc32c() in crc32c.cc. That file documents the two constraints on the
resolver: it runs at load time, so it may only call code that is safe
there, and it must not return NULL. */
extern "C" { static my_crc32_t rv_my_checksum_resolver(unsigned long long,
void *hwprobe,
void *)
{
return rv_zbc_supported(hwprobe) ? crc32_riscv_zbc : my_crc32_zlib;
} }
extern "C" uint32 my_checksum(uint32, const void *, size_t)
__attribute__((ifunc("rv_my_checksum_resolver")));
#else
static my_crc32_t init_crc32()
{
#if defined _M_IX86 || defined _M_X64 || defined __i386__ || defined __x86_64__
@@ -50,11 +71,9 @@ static my_crc32_t init_crc32()
static const my_crc32_t my_checksum_func= init_crc32();
#ifdef __powerpc64__
# error "my_checksum() is defined in mysys/crc32/crc32_ppc64.c"
#endif
extern "C"
uint32 my_checksum(uint32 crc, const void *data, size_t len)
{
return my_checksum_func(crc, data, len);
}
#endif