Moved the check for transaction state before certification step
into separate method abort_or_interrupted() which will check the state
and adjust state and client_state error status accordingly.
Moved the check for abort_or_interrupted() to happen before
the state is changed to certifying and write set data is appended.
This makes the check atomic and reduces the probability of race
conditions. After this check we rely on provider side transaction
state management and error reporting until the certification step
is over.
Change to public API: Pass client_state mutex wrappend in unique_lock
object to client_service::interrupted() call. This way the DBMS side
has a control to the lock object in case it needs to unlock it
temporarily. The underlying mutex will always be locked when the lock
object is passed via interrupted() call.
Other: Allow server_state change from donor to connected. This may
happen if the joiner crashes during SST and the provider reports
it before the DBMS side SST mechanism detects the error.
The streaming_rollback() was not called from transaction::bf_abort()
if a streaming transaction was BF aborted in executing state. This
was because the condition to enter streaming_rollback() checked
if the transaction state is executing. However, the transaction
state had already been changed to must_abort before.
As a fix, save the state to local variable when entering
bf_abort() and check against saved state in condition to enter
streaming_rollback().
Added unit test for the correct behavior in the case when streaming
transaction is BF aborted in executing state.
Silenced warning about failing to replicate rollback fragment.
This is pretty normal condition and may happen during shutdown/
configuration changes when SR transactions are rolled back.
If the application uses caching for client sessions, the
client_state object may be reused. This will cause the opened
client session to have unexpected value for sync_wait_gtid and
last_written_gtid.
In order to work around the problem, reset sync_wait_gtid and
last_written_gtid in client_state::open().
A high priority transaction T2 which calls commit_order_enter() with
client_state mutex locked may cause a deadlock if an another high
priority transaction T1 ordered before tries to access the transaction
T2 state.
As a fix, make sure that commit_order_enter() is never called with
client_state mutex locked.
Fragment removal for SR transaction is done in transaction context
to make it atomic. However, this means that if the BF abort arrives
during fragment removal, the transaction may be rolled back inside
client_service::remove_fragments(), and client_state::after_prepare()
is left in aborted state. This case was not handled in the
post condition checks of after_prepare().
Fixed assertion in after_prepare() and implemented unit test.
Earlier versions of cluster software may not support storing
the view info into stable storage. In order to work around this
during rolling upgrade, skip sanity checks for recovered view
if the view state_id ID is undefined.
The empty implementation of log_dummy_write_set() in dbsim
high priority service implementation left unreleased commit
order critical section behind whenever remote write set failed
certification. Added calls to do empty commit to release the
critical section.
Other:
Implemented ostream operator<< for wsrep:🧵:id, and added
printout of owning thread into transaction debug output.
Flag init_initialized_ must be checked before changing the
state to s_initializing in on_primary_view() in order to avoid
race between main thread and applier thread. Otherwise it is
possible that main thread gains control after setting state
to initializing and changes the flag init_initialized_ to true
before the check is done in on_primary_view().
Convert streaming client to applier only if the server is not
in disconnected state. In disconnected state the appliers map
is supposed to be empty and will be reconstructed from fragment
storage when the server is connected back to cluster.
Introduced server_service recover_streaming_appliers() interface
call which will be called in total order whenever streaming appliers
must be recovered. The call comes with two overloads, one which
can be called from client context (e.g. after SST has been received)
and the other from high priority context (e.g. view event handling).
The client context overload should be eventually be deprecated once
there is a mechanism to make provider signal that it has joined to
the cluster and will start applying events.
* Implemented encryption callback and enc_set_key
* Added pure virtual functions for encryption functionality
* Set enc key if provider was not loaded on time
In general the position where the storage recovers after a SST
cannot be known untile the recovery process is over. This in turn
means that the position cannot be known when the server_state
sst_received() method is called. Worked around the problem by
introducing get_position() method into server service which
can be used to get the position from stable storage after SST
has completed and the state has been recovered.
Certification for commit fragments was changed to happen in
before_prepare in order to make GTID available for storage
engines in prepare phase.
Removed unused file src/wsrep-lib_test.cpp
* Adds method wsrep::transaction::streaming_step() so that there is a
single place where streaming context unit counter is udpated.
The method also checks that some data has been generated before
attempting fragment replication.
* Emit a warning if there is an attempt to replicate a fragment and
there is no data to replicate.
Instead of handling error case at the beginning, execute the middle
of method body in case of success, leaving only single call to
provider().sst_received() at the end.
Intruduced server_state::interrupt_state_waiters() to interrupt
all waiters inside server_state::wait_until_state(). This mechanism
is needed when an error is encountered during state change processing
and waiting threads may need to be interrupted to check and handle
the error condition.
Made server_state::wait_until_state() to throw exception if the
wait was interrupted and the new server state is either disconnecting
or disconnected, which usually indicates error condition.
Init first join crashed in
server: s1 unallowed state transition: joined -> joined
This was due to missing state check for state in on_primary_view()
before changing to joined state. Added appropriate check.
Implemented unit tests for simple IST scenarios.
Replaced all references to provider_ in server_state methods to
provider() call which is virtual and can be overridden by test classes.
Provider pointer may not be initialized during unit tests yet.
Transition joiner - disconnecting may happen when the joiner failed
to receive SST succesfully. Because the system is at undefined state
at this point, skip most of the processing in sst_received()
and return control to caller after notifying the provider about
failure.
Transition from server_state connected state to disconnecting must
be allowed to deal with errors during server startup.
Added SST first test cases for server_state transitions:
* Successful join via SST
* Error in connect state
* Error in joiner state
Use iterators for scanning members vector in order to avoid
issues with integer signedness and range checks. The vector is
usually rather small and not in hot codepath, so performance
is here not an issue.
Added unit test for member_index() method.
Provider desync may return an error if the provider cannot communicate
with rest of the cluster. However, this is acceptable for example
if the node has dropped from primary view. Instead of returning
error immediately after failed desync(), attempt to pause the provider
regardless of the error. If pause operation fails, error is returned.
In order to avoid resync in resume_and_resync() in the case desync
failed in desync_and_pause(), new member variable desynced_on_pause_
was introduced to decide whether to resync or not in resume_and_resync().
This variable is protected by pause()/resume() calls since they do
not allow concurrent pause/resume operations.
