* feat(cmapi): add read_only param for API add node endpoint
* style(cmapi): fixes for string length and quotes
Add dbroots of other nodes to the read-only node
On every node change adjust dbroots in the read-only nodes
Fix logging (trace level) in tests
Remove ExeMgr from constants
Fix tests
Manually remove read-only node from ReadOnlyNodes on node removal (because nodes are only deactivated)
Review fixes (mostly switching to StrEnum analog before py3.11, also changes in ruff config)
Read-only nodes are now called read replica consistently
Don't write hostname into IP fields of the config like PMSx/IPAddr, pmx_WriteEngineServer/IPAddr
We calculate ReadReplicas by finding PMs without WriteEngineServer
In _replace_localhost, replace local IP addrs with resolved IP addrs and local hostnames -- with the resolved hostnames.
ModuleHostName/ModuleIPAddr is kept intact.
Keep only IPv4 in ActiveNodes/DesiredNodes/InactiveNodes
feat: add mock DNS resolution builder for testing hostname/IP mappings
* Fix _add_node_to_PMS: if node is already in PMS, save it to existing items to not miss it during the reconstruction of the list
* Make tests independent from CWD
Fixed for _add_Module_entries
Fixed node removal and tests
Fixes for node manipulation tests
* chore(mcs, scripts): extra/columnstore_review.sh with scripts/columnstore_review.sh with 1.4.13 version
* feat(mcs): add review command to the Tools section. It's the wrapper for columnstore_review.sh
* feat(mcs): add review command implementation to tools.py file + constansts.py
* chore(mcs): add separator argument to cook_sh_arg function
* docs(mcs): updated README.md and mcs.1 man file
* fix(mcs, wrapper): default is None for --backup-location, --backup-destination, --storage, --parallel, --highavilability, --skip-save-brm, --skip-polls, --skip-locks, --skip-mariadb-backup, --skip-bucket-data, --name-backup, --quiet, --no-verify-ssl, --poll-interval, --poll-max-wait, --retention-days, -scp, -bb, -url, -f, -m, -aro, -li and most of arguments in backup_commands.py and restore_commands.py
* fix(mcs, helpers): cook_sh_arg parser function now detects None as a value
* fix(mcs, wrapper): list -> li in typer command argument name for both backup_commands.py and restore_commands.py
* docs(mcs, wrapper): --parralel arg help message was edited to simple and readable one
* fix(mcs, wrapper): removing -no- prefixed flag variants for all bool arguments in backup_commands.py and restore_commands.py
[fix] CEJPasswordHandler class methods to use directory for cskeys file
[fix] CEJPasswordHandler.encrypt_password to return password in hex format
[fix] CEJPasswordHandler key_length
[fix] CEJPasswordHandler os.urandom call typo
[upd] mcs cli README.md and man page
[upd] mcs cli README_DEV.md
[fix] mcs_cluster_tool/decorators.py to handle typer.Exit exception
[add] various docstrings
[add] distribute .secrets file to all nodes while adding a new node
[add] encrypt_password, generate_secrets_data, save_secrets to CEJPasswordHandler
[add] tools section to mcs cli tool
[add] mcs_cluster_tool/tools_commands.py file with cskeys and cspasswd commands
[add] cskeys and cspasswd commands to tools section of mcs cli
[mv] backup/restore commands to tools section mcs cli
[fix] minor imports ordering
[fix] constants
MAJOR: Some logic inside node remove changed significantly using active
nodes list from Columnstore.xml to broadcast config after remove.
[fix] TransactionManager passsing extra, remove and optional nodes arguments to start_transaction function
[fix] commit and rollback methods of TransactionManager adding nodes argument
[fix] TransactionManager using success_txn_nodes inside
[fix] remove node logic to use Transaction manager
[fix] cluster set api key call using totp on a top level cli call
[add] missed docstrings
[fix] cluster shutdown timeout for next release
[add] cluster api client class
[fix] cli cluster_app using cluster api client
[add] ClusterAction Enum
[add] toggle_cluster_state function to reduce code duplication
refactor(ci): switch S3 uploading to AWS CLI for logging S3 link
docs: update README with to reflect changes in Drone pipeline
feat(env): add activate script for portable Python
fix(logging): log RequestException response body
fix(mcs): list option to -li/--list for mcs backup, restore and dbrm_restore commands
fix(mcs): add missed option "aro"
fix(mcs): resort options for backup and dbrm_backup to keep same ordering as in original scrypt
fix(mcs docs): updated README.md + mcs.1
[add] list option but in argument notation "list" for backup, restore, dbrm_restore
[fix] help messages for --config-file
[fix] -p/--path > -bl/--backup-location option for dbrm_backup and dbrm_restore
[fix] -d/--directory > '-l', '--load' option for dbrm_restore
[add] now all 'cluster' subcommand commands working without using 'cluster' subcommand itself
[fix] 'cluster' command hidding its help message
[add] explicit names for commands
[add] help-all command to list all nested help messages in man page view
[add] rich colored tabled help
[add] table panels for cluster commands
[upd] typer dependency version and some extra indirect dependencies
[add] README.md for mcs cli tool with all command reference
[add] mcs.1 man page
[add] README_DEV.md for mcs cli tool
[fix] cmapi/CMakeLists.txt to install mcs.1 man page
* feat(package_manager): improved cs_package_manager functionality for better UX
- add apt support for community and enterprise install
- add enterprise_staging install
- filter check results down to os/arch
- add --token flag for enterprise (#3002)
* Update cs package manager dev ubuntu (#3060)
* cs_package_manager.sh - fix ubuntu dev installs
* cs_package_manager.sh - remove some useless comments
* debian/ubuntu dev install to use drone repo + remove wget dependancy
* + exits if repo files missing
* auto install aws cli for dev builds
* removed commented section from cs_package_manager
* feat(tooling): add rhel support in cs_package_manager.sh (#3105)
* added dbrm_restore for LocalStorage & S3
* small comment cleanup
* resolved PR comments for cs_package_manager.sh & mcs_backup_manager.sh
* added dbrm_restore for LocalStorage & S3
* Added backup retention flag -r and incremental auto_most_recent option to mcs_backup_manager.sh
* resolved PR comments for cs_package_manager.sh & mcs_backup_manager.sh
* add comment to auto_select_most_recent_backup_for_incremental in mcs_backup_manager.sh
* feat(cmapi,cli): MCOL-5618: Add backup, restore, backup-dbrm, restore-dbrm commands for mcs cli tool. (#3130)
* MCOL-5618: Add backup, restore, backup-dbrm, restore-dbrm commands for mcs cli tool.
[add] mcs_cluster_tool/helpers.py file with cook_sh_arg function inside
[add] MCS_BACKUP_MANAGER_SH to mcs_cluster_tool/constants.py
[add] backup and restore commands to "mcs" Typer APP
* MCOL-5618: Move mcs_backup_manager.sh to cmapi/scripts.
* MCOL-5618: Install mcs_backup_manager.sh with CMAPI package.
* MCOL-5618: Testing fix + script minor fixes.
- [fix] wrong cooking of .sh invoke string
- [fix] restore_dbrm -> dbrm_restore, backup_dbrm -> dbrm_backup to use same naming in both python wrapper and mcs_backup_manager.sh
- [fix] mathod of checking cmapi online or not in mcs_backup_manager.sh
* MCOL-5618: Minor improvements to sync with PR 3144.
[add] -ns option for dbrm_restore.
* MCOL-5618: Minor fixes to sync after #3147
[add] -r option for backup command
* MCOL-5618: Minor fixes to sync after PR #1398.
[add] -ssm argument for dbrm backup
[add] -sdbk -ssm arguments for dbrm restore
[add] silenece annoying pylint warning unused argument for bachup_commands.py and restore_commnads.py
* update to the latest mcs_backup_manager + cs_package_manager
* Intermediate commit to move changes from old filepath to the new one.
* remove old file
* MCOL-5618: Fix packaging.
[mv] extra/cs_package_manager.sh -> cmapi/scripts/cs_package_manager.sh to pack and deliver the script with cmapi package
[add] cs_package_manager.sh to cmapi package
[fix] download paths for mcs_backup_manager.sh and cs_package_manager.sh
* MCOL-5618: Fix after testing
[add] -P to restore help in mcs_backup_restore.sh
[fix] -f argument default value in both backup and restore wrapper commands
[fix] -i argument behaviour and type + default value
[fix] -P argument position and help for both backup and restore wrapper commands
[fix] disable some pylint warning for both backup and restore wrapper files
---------
Co-authored-by: Allen Herrera <82840027+mariadb-AllenHerrera@users.noreply.github.com>
Co-authored-by: Allen Herrera <allen.herrera@mariadb.com>
Co-authored-by: Allen Herrera <allen@Allens-Black-MacBook-Pro.local>
Co-authored-by: Leonid Fedorov <79837786+mariadb-LeonidFedorov@users.noreply.github.com>
* MCOL-5594: Interactive "mcs cluster stop" command for CMAPI. (#3024)
* MCOL-5594: Interactive "mcs cluster stop" command for CMAPI.
[add] NodeProcessController class to handle Node operations
[add] two endpoints: stop_dmlproc (PUT) and is_process_running (GET)
[add] NodeProcessController.put_stop_dmlproc method to separately stop DMLProc on primary Node
[add] NodeProcessController.get_process_running method to check if specified process running or not
[add] build_url function to helpers.py. It needed to build urls with query_params
[add] MCSProcessManager.gracefully_stop_dmlproc method
[add] MCSProcessManager.is_service_running method as a top level wrapper to the same method in dispatcher
[fix] MCSProcessManager.stop by using new gracefully_stop_dmlproc
[add] interactive option and mode to mcs cluster stop command
[fix] requirements.txt with typer version to 0.9.0 where supports various of features including "Annotated"
[fix] requirements.txt click version (8.1.3 -> 8.1.7) and typing-extensions (4.3.0 -> 4.8.0). This is dependencies for typer package.
[fix] multiple minor formatting, docstrings and comments
* MCOL-5594: Add new CMAPI transaction manager.
- [add] TransactionManager ContextDecorator to manage transactions in less code and in one place
- [add] TransactionManager to cli cluster stop command and to API cluster shutdown command
- [fix] id -> txn_id in ClusterHandler class
- [fix] ClusterHandler.shutdown class to use inside existing transaction
- [add] docstrings in multiple places
* MCOL-5594: Review fixes.
* fix(cmapi): MCOL-5594: Fix after testing.. (#3150)
- [fix] wrong invoke already imported constants and function from signal lib
This task is a next logical part of MCOL-5470 that implements AppManager and API endpoint
to check if CMAPI ready or not.
- [add] cmapi_app.py with is-ready command implementation
- [fix] add cmapi is-ready command to main typer app
