1
0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-09-16 19:42:06 +03:00
Commit Graph

69 Commits

Author SHA1 Message Date
Alexander Presnyakov
8fadbf2819 Fix downloading libmariadb-dev when --with-dev is passed 2025-09-10 18:58:11 +00:00
Alexander Presnyakov
151903cd18 Tracing fixes:
Don't log trace_params in tracing logger, because it already has all this data
Don't print span attrs, it can contain lots of headers
Save small part of the response into the span, if the response was a JSON string
Added JSON logging of trace details into a separate file (to not spam the main log with machine readable stuff)
Record part of the response into the span
Set duration attribute in server spans
Log 404 errors
Colorize the traces (each span slightly changes the color of the parent span)
Improve trace visualization with duration formatting and notes for request/response pairs
2025-09-10 17:06:12 +00:00
mariadb-AlanMologorsky
194e7ccbdc fix(timeout): MCOL-5508 make node unresponse timeout configurable
Read DBRM unresponsive timeout from Columnstore.xml DBRMUnresponsiveTimeout Tag instead of hardcoded 300 sec value.
2025-09-10 19:19:40 +04:00
Alexander Presnyakov
38d7f6a51c Added an option to cs_package manager to enable downloading\installing dev packages 2025-09-10 09:06:20 +04:00
Alexander Presnyakov
55241e6f0e Fixed import problem 2025-09-09 17:23:20 +04:00
Alexander Presnyakov
bd1575d34a Handle waiting locks 2025-09-09 17:23:20 +04:00
Alexander Presnyakov
cff5244e35 Use the same mcs-shmem-locks tool that we used to inspect locks state to unlock them 2025-09-09 17:23:20 +04:00
Alexander Presnyakov
384dd56a61 Moved locks cleanup into cmapi 2025-09-09 17:23:20 +04:00
Alexander Presnyakov
48e14ed5e5 MCOL-6094: reset shmem locks before stopping workernode 2025-09-09 17:23:20 +04:00
Alexander Presnyakov
873eeb172c CLI command to enable/disable Sentry logging 2025-09-03 20:32:03 +04:00
Alexander Presnyakov
113d6cc41b Disabled sentry backend debug prints
Added couple of debug prints for host readiness
2025-09-03 20:32:03 +04:00
Alexander Presnyakov
08d46a9e70 Pass git revision from the build process -> cmapi - Sentry (to be able to understand which revision caused some error) 2025-09-03 20:32:03 +04:00
Alexander Presnyakov
3ca5a2e6bb Fixed Sentry transaction continuation 2025-09-03 20:32:03 +04:00
Alexander Presnyakov
9821f14d0f Enable loggers that were not explicitly mentioned in cmapi_logger.conf 2025-09-03 20:32:03 +04:00
Alexander Presnyakov
9b98c5c20a Created a separate package for tracing-related stuff
Added mirroring of spans into Sentry
Tracer is a facade that redirects actions to tracing backends
2025-09-03 20:32:03 +04:00
Alexander Presnyakov
a0b4bcd1ce Basic request tracer
Tracing requests

Custom log factory adds all trace values as one log record parameter (it will be empty if trace values are empty, like in MainThread where there are no incoming requests)
2025-09-03 20:32:03 +04:00
Leonid Fedorov
15944923bd chore(cmapi runtime): add Network as a prerequisite to start CMAPI service 2025-08-28 03:45:46 +04:00
Leonid Fedorov
46922f8d6a chore(build, cmapi): fallback of packaging to os-release if lsb-core is missing 2025-08-28 03:45:46 +04:00
Alexander Presnyakov
5939aa5f55 Added support for Sentry in cmapi server
Support distributed request tracing

-Direct dependencies now in requirements[-dev].in, pip-compile generates full requirement[-dev].txt from them
2025-08-26 14:21:33 +00:00
Allen Herrera
f0db98fbba 3.16 added retry rsync logic for backups, 3.17 improved dbrm backups with save_brm, write poll checks and write locks 2025-08-26 16:49:58 +04:00
Alexander Presnyakov
19c4914194 feat/MCOL-6074: CMAPI integrational tests that work with real hosts (using columnstore-ansible-aws for cluster creation/provisioning) 2025-08-12 13:08:31 +00:00
Alexander Presnyakov
58040871bd Fixed "raise None" that happened on signal interception
Also fixed couple of (possibly) unbound variables
2025-08-11 16:48:19 +00:00
Alexander Presnyakov
f8a57b4412 BRM journal is always singular, do not apply suffix logic to it 2025-07-25 14:54:59 +01:00
mariadb-AlanMologorsky
b6a5c1d71f fix(cmapi): MCOL-5913 cmapi write local address 127.0.1.1 instead real ip address
* feat(cmapi): NetworkManager class for some ip hostname opoerations.
 * fix(cmapi): Use NetworkManager class to resolve ip and hostname in node_manipulation.add_node function
 * fix(cmapi): Minor docstring and formatting fixes
2025-07-08 15:57:28 +03:00
mariadb-AlanMologorsky
5d0b3a88f3 fix(cmapi): MCOL-5899 confusing cluster_mode status when columnstore offline. 2025-06-19 19:22:44 +04:00
aleksei.bukhalov
3de8af2fa4 chore(ci): remove sudo usage from systemctl calls in cmapi tests 2025-06-03 22:30:22 +02:00
Leonid Fedorov
6db2dc668f stubs and cmake formatting 2025-05-20 18:22:59 +04:00
mariadb-AlanMologorsky
2c0367ea2b feat(mcs): MCOL-5300 review/finetune log collection tools.
* chore(mcs, scripts): extra/columnstore_review.sh with scripts/columnstore_review.sh with 1.4.13 version
* feat(mcs): add review command to the Tools section. It's the wrapper for columnstore_review.sh
* feat(mcs): add review command implementation to tools.py file + constansts.py
* chore(mcs): add separator argument to cook_sh_arg function
* docs(mcs): updated README.md and mcs.1 man file
2025-05-02 16:12:14 +03:00
Allen Herrera
b731c0a326 chore(backup): spellcheck mcs_backup_manager.sh adjusting lots of variable names, functions and help text (#3515) 2025-04-21 18:27:04 +01:00
Allen Herrera
441327ed84 chore(cs_package_manager,backup): update cs_package_manager.sh to v 3.10 (#3498)
* update cs_package_manager.sh to v 3.10 - added download maxscale and check maxscale

* adjust cs_package_manager.sh rename ci variables
2025-04-21 15:26:12 +01:00
Alexander Presnyakov
e34f7d2530 Fix thread arg errors (app instead of sequence) 2025-04-18 18:52:42 +01:00
mariadb-AlanMologorsky
927eb4b2bd feat(cmapi, mcs): MCOL-5941 Improve mcs cli backup\restore wrapper.
* fix(mcs, wrapper): default is None for --backup-location, --backup-destination, --storage, --parallel, --highavilability, --skip-save-brm, --skip-polls, --skip-locks, --skip-mariadb-backup, --skip-bucket-data, --name-backup, --quiet, --no-verify-ssl, --poll-interval, --poll-max-wait, --retention-days, -scp, -bb, -url, -f, -m, -aro, -li and most of arguments in backup_commands.py and restore_commands.py
* fix(mcs, helpers): cook_sh_arg parser function now detects None as a value
* fix(mcs, wrapper): list -> li in typer command argument name for both backup_commands.py and restore_commands.py
* docs(mcs, wrapper): --parralel arg help message was edited to simple and readable one
* fix(mcs, wrapper): removing -no- prefixed flag variants for all bool arguments in backup_commands.py and restore_commands.py
2025-04-18 18:51:59 +01:00
mariadb-AlanMologorsky
7c27d803ba feat(cmapi, mcs): MCOL-5525 Single start command for Single Node.
* feat(mcs): new mcs bootstrap-single-node command added into tools pane
2025-04-15 04:57:12 +03:00
mariadb-AlanMologorsky
fde9155d18 chore(cmapi, deps): MCOL-5905 Bump versions of CMAPI dependencies. 2025-04-08 15:48:47 +03:00
mariadb-AlanMologorsky
d17fc7d9be Review fixes. 2025-04-08 14:56:06 +03:00
mariadb-AlanMologorsky
aa57a7684c feat(cmapi): MCOL-5019: review fixes.
[fix] CEJPasswordHandler class methods to use directory for cskeys file
[fix] CEJPasswordHandler.encrypt_password to return password in hex format
[fix] CEJPasswordHandler key_length
[fix] CEJPasswordHandler os.urandom call typo
[upd] mcs cli README.md and man page
[upd] mcs cli README_DEV.md
[fix] mcs_cluster_tool/decorators.py to handle typer.Exit exception
[add] various docstrings
2025-04-08 14:56:06 +03:00
mariadb-AlanMologorsky
215e4eea4d feat(cmapi): MCOL-5019: distributing cskeys secrets file, move cskeys and cspasswd functions to mcs cli.
[add] distribute .secrets file to all nodes while adding a new node
[add] encrypt_password, generate_secrets_data, save_secrets to CEJPasswordHandler
[add] tools section to mcs cli tool
[add] mcs_cluster_tool/tools_commands.py file with cskeys and cspasswd commands
[add] cskeys and cspasswd commands to tools section of mcs cli
[mv] backup/restore commands to tools section mcs cli
[fix] minor imports ordering
[fix] constants
2025-04-08 14:56:06 +03:00
mariadb-AlanMologorsky
10dec6ea94 fix(cmapi): MCOL-5962 Fix rollback in mcs cli and passing errors during broadcasting config
* fix(cmapi): ClusterControllerClient request error handling
* fix(cmapi): broadcast_new_config request error handling
* style(cmapi): logs
2025-04-07 14:34:13 +03:00
Allen Herrera
f04b189476 feat(backup): update to mcs_backup_manager to v3.14 adding integer checks on inputs (#3465) 2025-04-04 21:28:43 +01:00
mariadb-AlanMologorsky
55b09a71fc fix(cmapi): MCOL-5638 Improve Client Error Messaging and async broadcasting config.
* fix(cmapi): broadcast_new_config function using aiohttp and improved errors handling mechanics
* fix(cmapi): requirements updating aiohttp version and it's dependencies
* fix(cmapi, deps): unused imports
* fix(cmapi, tests): test_broadcast_new_config
2025-04-01 17:27:22 +03:00
Allen Herrera
11dd3f0422 feat(cs-package-manager): added multinode install via --nodes flag & dev upgrade along with some help texts (#3279)
Co-authored-by: Allen Herrera <allen@AllensBlackMBP.attlocal.net>
2025-04-01 18:03:07 +04:00
mariadb-AlanMologorsky
2f25c47a57 fix(cmapi): MCOL-5133 remove has_active_nodes handler for cluster shutdown because TransactionManager has advance variant of this check 2025-04-01 18:03:07 +04:00
mariadb-AlanMologorsky
0e5842abcc fix(cmapi tests): MCOL-5133 test mode set without nodes now returns adequate message 2025-04-01 18:03:07 +04:00
mariadb-AlanMologorsky
8a4f857df4 fix(cmapi): MCOL-5133 maintainance flag typo. 2025-04-01 18:03:07 +04:00
mariadb-AlanMologorsky
27e3b8d808 feat(cmapi): MCOL-5133: Stage3 stand alone cli tool.
MAJOR: Some logic inside node remove changed significantly using active
    nodes list from Columnstore.xml to broadcast config after remove.

 [fix] TransactionManager passsing  extra, remove and optional nodes arguments to start_transaction function
 [fix] commit and rollback methods of TransactionManager adding nodes argument
 [fix] TransactionManager using success_txn_nodes inside
 [fix] remove node logic to use Transaction manager
 [fix] cluster set api key call using totp on a top level cli call
 [add] missed docstrings
 [fix] cluster shutdown timeout for next release
2025-04-01 18:03:07 +04:00
mariadb-AlanMologorsky
c40e4ba00d feat(cmapi): MCOL-5133: Stage2 stand alone cli tool.
[fix] client class methods
[fix] Transaction management at endpoint handling level
[fix] add_node and set_mode methods to use TransactionManager
2025-04-01 18:03:07 +04:00
Alan Mologorsky
a60a5288d8 feat(cmapi): MCOL-5133: Stage1 to stand alone cli tool. (#3378)
[add] cluster api client class
[fix] cli cluster_app using cluster api client
[add] ClusterAction Enum
[add] toggle_cluster_state function to reduce code duplication
2025-04-01 18:03:07 +04:00
Alan Mologorsky
a00711cc16 fix!(cmapi): MCOL-5454: Self-signed certificate autorenew. (#3213)
[add] managers/certificate.py with CertificateManger class
[mv] creating self-signed certificate logic into CertificateManger class
[add] renew and days_before_expire methods to CertificateManger class
[mv] several certificate dependent constants to managers/certificate.py
[add] CherryPy BackgroundTask to invoke certificate check hourly (3600 secs)
[fix] tests
[fix] bug with txn timer clean (clean_txn_by_timeout, worker and invoking of it)
2025-04-01 18:03:07 +04:00
Alexander Presnyakov
de7a4a2d7e feat(mcs): add -v option to enable logging to console
refactor(ci): switch S3 uploading to AWS CLI for logging S3 link

docs: update README with to reflect changes in Drone pipeline

feat(env): add activate script for portable Python

fix(logging): log RequestException response body
2025-03-31 16:54:46 +00:00
mariadb-AlanMologorsky
91e82f4d15 fix(mcs): MCOL-5618: change list option to --list and refactor.
fix(mcs):  list option to -li/--list for mcs backup, restore and dbrm_restore commands
fix(mcs): add missed option "aro"
fix(mcs): resort options for backup and dbrm_backup to keep same ordering as in original scrypt
fix(mcs docs): updated README.md + mcs.1
2025-03-28 13:02:19 +03:00