1
0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-10-30 07:25:34 +03:00
Commit Graph

6837 Commits

Author SHA1 Message Date
aleksei.bukhalov
2218807c9d chore(ci): disable build with gcc-toolset 2025-10-17 13:11:59 +02:00
Serguey Zefirov
bb631dcffb feat(JSON,data_type): MCOL-6197 - support for JSON type
This patch does exactly this, it implements support for JSON in DDL.

Right now we use server's check for JSON validity on INSERT. We do not implement
JSON validity check during updates, it is postponed for later work.
2025-10-17 11:48:46 +03:00
Sergey Zefirov
5c834ac7cd fix(CEJ, segfault): MCOL-6198 - segfault during crossengine join
* fix(CEJ, segfault): MCOL-6198 - segfault during crossengine join

The patch moves joiners' initialization to a place after all possible
allocations of smallSideRGs vector so pointer to it's data does not
change anymore. This makes crash to cease.

An appropriate test is added to bugfixes suite.

* Change to test

* Another dangling pointer

* A change to test

* A change to test
2025-10-15 14:54:35 +01:00
julienfritsch44
0eeb86fc20 chore(docs): Update QueryAccelerator.md (#3811)
Small changes
2025-10-10 13:38:13 +01:00
Aleksei Antipovskii
3cf5792779 fix(build): fix compilation issue with 11.08-server 2025-10-09 19:01:45 +04:00
drrtuy
99392ed87f fix(cmapi,systemd): pre-save locks cleanup has been moved into mcs-savebrm.py (#3794) 2025-10-07 12:31:23 +01:00
drrtuy
7842d8ab4b chore(tests): disabled an unstable test to pass MTR @Jenkins for ES 11.8. 2025-10-06 12:01:50 +01:00
drrtuy
6ab6694af5 chore(): Bump VERSION up to 25.10.0-1 2025-10-03 10:54:51 +01:00
Leonid Fedorov
f002c5abc1 MCOL-5843: fix(extents): print corrent N/A for temporal extents 2025-10-02 22:15:01 +04:00
Leonid Fedorov
2740cbec7b fix(plugin,leak): One more memory leak in plugin (#3805) 2025-10-02 11:44:51 +01:00
Leonid Fedorov
6225f6bbac MCOL-6163: Debian:13 added to CI, MTR is not ignored for server 11.8 anymore, sanitizers are temporary disabled 2025-10-02 04:32:57 +04:00
mariadb-AlanMologorsky
f8678ff51f fix(cs_package_manager): RHEL install
Do not download MariaDB-server for RHEL installation
Fix yum repo by adding module_hotfixes = 1
2025-10-02 04:09:51 +04:00
mariadb-AlanMologorsky
c86586c228 feat(cmapi,failover): MCOL-6006 Disable failover when shared storage not detected
- Add SharedStorageMonitor thread to periodically verify shared storage:
  * Writes a temp file to the shared location and validates MD5 from all nodes.
  * Skips nodes with unstable recent heartbeats; retries once; defers decision if any node is unreachable.
  * Updates a cluster-wide stateful flag (shared_storage_on) only on conclusive checks.
- New CMAPI endpoints:
  * PUT /cmapi/{ver}/cluster/check-shared-storage — orchestrates cross-node checks.
  * GET /cmapi/{ver}/node/check-shared-file — validates a given file’s MD5 on a node.
  * PUT /cmapi/{ver}/node/stateful-config — fast path to distribute stateful config updates.
- Introduce in-memory stateful config (AppStatefulConfig) with versioned flags (term/seq) and shared_storage_on flag:
  * Broadcast via helpers.broadcast_stateful_config and enhanced broadcast_new_config.
  * Config PUT is now validated with Pydantic models; supports stateful-only updates and set_mode requests.
- Failover behavior:
  * NodeMonitor keeps failover inactive when shared_storage_on is false or cluster size < 3.
  * Rebalancing DBRoots becomes a no-op when shared storage is OFF (safety guard).
- mcl status improvements: per-node 'state' (online/offline), better timeouts and error reporting.
- Routing/wiring: add dispatcher routes for new endpoints; add ClusterModeEnum.
- Tests: cover shared-storage monitor (unreachable nodes, HB-based skipping), node manipulation with shared storage ON/OFF, and server/config flows.
- Dependencies: add pydantic; minor cleanups and logging.
2025-10-01 21:10:34 +04:00
Leonid Fedorov
111d169e40 fix(crash): Debug build of columnstore crashed on asserts with 11.8 (#3800)
due this wonderful code

  DBUG_ASSERT(!comment || !comment[0] || comment[strlen(comment)-1] != '.');
  DBUG_ASSERT(!comment || !comment[0] || comment[strlen(comment)-1] != ' ');

So I had to fix the comments
2025-09-30 18:54:17 +04:00
mariadb-AlanMologorsky
a76e153a1d feat(upgrade): MCOL-6028 upgrade MVP with repo-managed flow, prechecks, and other enhancements
Implements the initial upgrade capability across CMAPI and the CLI, including
repository setup, package operations, environment prechecks, and coordinated
cluster steps with progress reporting.

Details:
- CMAPI upgrade manager:
  - Add `cmapi/cmapi_server/managers/upgrade/` modules:
    - `repo.py`, `packages.py`, `preinstall.py`, `upgrade.py`, `utils.py` and `__init__.py`
  - Extend endpoints and routing to expose upgrade operations and status:
    - `cmapi_server/controllers/{endpoints.py, dispatcher.py, api_clients.py}`
    - `cmapi_server/managers/{application.py, process.py}`
    - Add improved constants and helpers for upgrade flow
- Backup/restore and safety:
  - Add `cmapi_server/managers/backup_restore.py`
  - Fix pre-upgrade backup regressions (due to `mcs_backup_manager.sh 3.17 changes`)
  - Improve cluster version validation; add `ignore_missmatch` override
- CLI enhancements:
  - Progress UI and richer feedback (`mcs_cluster_tool/tools_commands.py`, `README.md`, `mcs.1`)
  - Add steps to start MDB and start MCS during/after upgrade
  - Improved error surfacing for version validation
- Platform and packaging:
  - Ubuntu and Rocky Linux support
  - RHEL/DNF dry-run support
  - Distro detection and platform-dependent logic hardened
  - Logging improvements
- Updater service:
  - Add `cmapi/updater/cmapi_updater.service.template` and `cmapi_updater.sh` to make CMAPI update itself
- Docs:
  - Update mcs cli README and mcs.1 man file
  - Add `cmapi/updater/README.md`
2025-09-30 18:48:32 +04:00
aleksei.bukhalov
9e1f5db0a0 MCOL-5250 add test 2025-09-30 18:01:30 +04:00
Timofey Turenko
84c145590c fix regex in mcs2_create_table_valid_names test 2025-09-30 17:51:00 +04:00
Timofey Turenko
ba2dfeedcf Merge pull request #3798 from mariadb-corporation/disable_not_compatible_with_11.8
temporary disable tests incompatible with 11.8
2025-09-29 15:30:34 +03:00
Leonid Fedorov
be91a85621 Merge branch 'stable-23.10' into disable_not_compatible_with_11.8 2025-09-27 00:12:22 +04:00
mariadb-AlanMologorsky
0753aac849 fix(cmapi): Fix error in docker cluster: "is not currently part of the cluster" 2025-09-26 21:55:10 +04:00
Alexander Presnyakov
bc4c7e27d9 MCOL-5806: added ability to start node in read-only mode
* feat(cmapi): add read_only param for API add node endpoint
* style(cmapi): fixes for string length and quotes

Add dbroots of other nodes to the read-only node

On every node change adjust dbroots in the read-only nodes

Fix logging (trace level) in tests

Remove ExeMgr from constants

Fix tests

Manually remove read-only node from ReadOnlyNodes on node removal (because nodes are only deactivated)

Review fixes (mostly switching to StrEnum analog before py3.11, also changes in ruff config)

Read-only nodes are now called read replica consistently

Don't write hostname into IP fields of the config like PMSx/IPAddr, pmx_WriteEngineServer/IPAddr

We calculate ReadReplicas by finding PMs without WriteEngineServer

In _replace_localhost, replace local IP addrs with resolved IP addrs and local hostnames -- with the resolved hostnames.
ModuleHostName/ModuleIPAddr is kept intact.

Keep only IPv4 in ActiveNodes/DesiredNodes/InactiveNodes

feat: add mock DNS resolution builder for testing hostname/IP mappings

* Fix _add_node_to_PMS: if node is already in PMS, save it to existing items to not miss it during the reconstruction of the list
* Make tests independent from CWD

Fixed for _add_Module_entries

Fixed node removal and tests

Fixes for node manipulation tests
2025-09-26 21:55:10 +04:00
Leonid Fedorov
e0a3d58d21 replace map::contains with map::count for gcc 8.5 compatibility
Fix build
2025-09-26 21:55:10 +04:00
Serguey Zefirov
a9f23729a9 feat(MCOL-6082) Cluster with read-only nodes should correctly work with DML
This patch changes logic from counting all nodes to counting only
read-write nodes when messaging about DML operations.

feat(MCOL-6082): Multiple readers of dbroots using OamCache logic

This patch introduces centralized logic of selecting what dbroot is
accessible in PrimProc on what node. The logic is in OamCache for time
being and can be moved later.

Fix build
2025-09-26 21:55:10 +04:00
Timofey Turenko
1616b32101 fix mcs2_create_table_valid_names test 2025-09-26 21:35:12 +04:00
Timofey Turenko
fd3c385373 Fix 'illegal mix of collation' in the Columnxtore MTR 2025-09-26 21:35:12 +04:00
Timofey Turenko
8b8a6df5e3 Make MTR tests compatible with 11.8 default collation settings 2025-09-26 21:35:12 +04:00
Timofey Turenko
acfb0e6035 temporary disable tests incompatible with 11.8 2025-09-26 17:07:10 +03:00
Timofey Turenko
33693686b2 disable unstabe tests from MCOL-6165 2025-09-26 17:21:19 +04:00
drrtuy
a8909a27a9 fix(shmem,brm,scripts): mcs-savebrm.py wrapper now cleans shmem locks before calling save_brm. mcs-shmem-locks now has --reset-all flag 2025-09-24 22:52:11 +04:00
mariadb-KristinaPavlova
b01be6ae2d fix: MCOL-5778 sum return null value due to nullif function 2025-09-24 19:21:25 +04:00
mariadb-KristinaPavlova
af6a716079 fix: MCOL-6100 large primloc logs generated after upgrade 2025-09-24 19:20:55 +04:00
Allen Herrera
e75f86058d update mcs_backup_manager_v3.18 - dbrm_restore - fixed issue with it trying to apply locks while database is offline resulting in a failed restore 2025-09-22 16:39:39 +04:00
drrtuy
ffce2e3e85 chore(binaries): remove redundant reset_locks. Use mcs-shmem-locks instead. 2025-09-22 16:39:12 +04:00
Alexey Antipovsky
cfa9a7ff2c feat(optimizer): MCOL-5250 rewrite queries with DISTINCT (#3666)
* feat(optimizer): MCOL-5250 rewrite queries with DISTINCT

... as aggregated queries.
So query
```
SELECT DISTINCT <cols list>
FROM <from list>
WHERE <where clause>
HAVING <having clause>
ORDER BY <orderby list>
LIMIT <limit>
```
will become
```
SELECT *
FROM
  (
    SELECT <cols list>
    FROM <from list>
    WHERE <where clause>
    HAVING <having clause>
  ) a
GROUP BY 1,2,3,...,N
ORDER BY <orderby list>
LIMIT limit
```

* move ORDER BY to the outer query

* fix test

* reuse cloneWORecursiveSelects() in clone()

* fix subselect columns processing
2025-09-22 16:16:37 +04:00
Aleksei Antipovskii
736ec81e4d Add 11.8-enterprise 2025-09-19 12:01:49 +04:00
Aleksei Antipovskii
5ffc688c26 fix(plugin): MCOL-6029 use the right field from the rows_stats struct for inserts 2025-09-19 12:01:49 +04:00
mariadb-AlanMologorsky
e8b9adda8f fix(config): MCOL-5508 Fixed missing DBRMUnresponsiveTimeout tag in some of .xml config files. 2025-09-18 17:16:40 +03:00
Timofey Turenko
74659192bf Make mcs11_show_db_table, mcs12_alter_table, mcs1_create_table_data_types compatible tests with 11.4 2025-09-18 18:11:40 +04:00
Timofey Turenko
5d3a95b927 fix version check in mysql-test/columnstore/include/disable_before_10.9.inc 2025-09-18 18:11:20 +04:00
Alexander Presniakov
0fadd0bd9c Merge pull request #3776 from mariadb-corporation/feat/MCOL-6094-savebrm-clean-shmem-locks
Shmem locks: debug loggin telling that no locks remain
2025-09-18 10:36:36 -03:00
Alexander Presnyakov
4422cf04b2 Just a debug logging telling that no locks remain 2025-09-18 13:05:16 +00:00
Alexander Presniakov
7d4028729b Merge pull request #3765 from mariadb-corporation/MCOL-6159-fix-active-nodes-membership-check
MCOL-6159: Add forward/reverse DNS validation when adding nodes by hostname
2025-09-17 22:43:17 -03:00
Alexander Presnyakov
a357566db7 Use is_only_loopback_hostname instead of LOCALHOSTS 2025-09-18 01:10:49 +00:00
Alexander Presnyakov
85690a0de1 refactor: standardize error handling with new context managers for CMAPI exceptions
We do it for the API to return error details.
2025-09-17 17:58:51 +00:00
Alexander Presnyakov
4a1b3b9355 feat: add forward/reverse DNS validation when adding nodes by hostname 2025-09-17 17:58:51 +00:00
aleksei.bukhalov
14c1bd872d MCOL-5756 add tests 2025-09-17 12:56:51 +02:00
drrtuy
88b441278f chore(QA,tests): fixed MTR tests to return resut for TPC-H q20. 2025-09-15 20:22:10 +01:00
drrtuy
9cbb160130 feat(joblist): descend into SelectFilter even if it contains correlated sub. 2025-09-15 20:22:10 +01:00
Rucha Deodhar
9fe37d5919 MDEV-32854: Make JSON_DEPTH_LIMIT unlimited
This patch is the columnstore-part of the task. Columnstore wanted to have
previous 32 depth, so this patch aims at keeping the compatibility.
2025-09-14 17:16:17 +04:00
drrtuy
9a2ebebaf9 Merge pull request #3749 from mariadb-corporation/feat/MCOL-6072-parallel-scan-4-CES-4
feat(rbo,rules,QA): QA rule refactoring
2025-09-12 17:11:32 +01:00