perl
The problem here was the method how MTR gets its unique thread ids.
Prior to this patch, the method to do it was to maintain a global
table of pid,mtr_unique_id) pairs. The table was backed by a text
file. The table was cleaned up one in a while and dead processes leaking
unique_ids were determined with with kill(0) or with scripting tasklist
on Windows.
This method is flawed specifically on native Windows Perl. fork() is
implemented with starting a new thread, give it a syntetic negative PID
(threadID*(-1)), until this thread creates a new process with exec()
However, neither tasklist nor any other native Windows tool can cope with
negative perl PIDs. This lead to incorrect determination of dead process
and reusing already used mtr_unique_id.
The patch introduces alternative portable method of solving unique-id
problem. When a process needs a unique id in range [min...max], it just
starts to open files named min, min+1,...max in a loop . After file is
opened, we do non-blocking flock(). When flock() succeeds, process has
allocated the ID. When process dies, file is unlocked . Checks for zombies
are not necessary.
Since the change would create a co-existence problems with older version
of MTR, because of different way to calculate IDs, the default ID range
is changed from 250-299 to 300-349.
Another fix that was necessary enable --parallel option was to serialize
spawn() calls on Windows. specifically, IO redirects needed to be protected.
This patch also fixes hanging CRTL-C (as described in Bug #38629) for the
"new" MTR. The fix was already in 6.0 and is now downported.
Fix mysql-test-run.pl to not terminate early when warnings in error logs are detected during
server shutdown. Instead, give a nice summary report at the end of the failures.
Fix code to make 100% sure no failures will go undetected.
Revert earlier wrong change.
Fix race with port allocation semaphore file permissions.
Adjust testsuite to copy with new PBXT engine now in the tree. The PBXT engine causes an
extra table to appear in the INFORMATION_SCHEMA. This causes different output for a few
test cases.
dbug/dbug.c:
If DbugParse() is called multiple times, the stack->keywords for the
top stack frame could be overwritten without being freed, causing a
memory leak reported by Valgrind.
include/my_global.h:
Add useful macro for different values when Valgrind (HAVE_purify) and not.
mysql-test/extra/rpl_tests/rpl_auto_increment.test:
Omit pbxt variables from show variables output.
mysql-test/include/have_pbxt.inc:
Add facility to disable test if PBXT engine is not available.
mysql-test/lib/mtr_report.pm:
Give a nice summary report at the end of tests of any warnings seen in logs during
server shutdowns.
mysql-test/lib/mtr_unique.pm:
Move chmod 777 to greatly reduce the risk of leaving the port semaphore file unaccessible
bu other users.
mysql-test/mysql-test-run.pl:
Don't abort in case of warnings detected, instead give a nice summary report.
Fix code to make 100% sure no failures will go undetected.
Revert earlier wrong change when master disconnects early.
mysql-test/r/information_schema.result:
Omit PBXT INFORMATION_SCHEMA table from output.
Move part of test to information_schema_all_engines.
mysql-test/r/information_schema_all_engines.result:
New file for information_schema tests that depend on which engines are available.
mysql-test/r/information_schema_db.result:
Move part of test to information_schema_all_engines.
mysql-test/r/innodb-autoinc.result:
Omit pbxt variables from show variables output.
mysql-test/r/mysqlshow.result:
Move part of test to information_schema_all_engines.
mysql-test/suite/rpl/r/rpl_auto_increment.result:
Omit pbxt variables from show variables output.
mysql-test/t/information_schema.test:
Omit PBXT INFORMATION_SCHEMA table from output.
Move part of test to information_schema_all_engines.
mysql-test/t/information_schema_all_engines.test:
New file for information_schema tests that depend on which engines are available.
mysql-test/t/information_schema_db.test:
Move part of test to information_schema_all_engines.
mysql-test/t/innodb-autoinc.test:
Omit pbxt variables from show variables output.
mysql-test/t/mysqlshow.test:
Move part of test to information_schema_all_engines.
mysql-test/valgrind.supp:
Add variant suppression (different system library versions).
Add suppression for problem with inet_ntoa().
sql/mysqld.cc:
Fix missing DBUG_RETURN.
Fix uninitialised thd->connect_utime, likely introduced by pool_of_threads.
sql/set_var.cc:
Fix one-byte buffer overflow in several places.
Fix unsafe use of String::c_ptr() of stack-allocated String buffer.
sql/sql_select.cc:
Silence valgrind warning due to GCC bug.
sql/sql_string.h:
Document potential problem with String::c_ptr() and String() constructor with caller-supplied buffer.
storage/archive/azio.c:
Silence Valgrind false warning for libz.
- Some tests need to modify the server(s) so much that a total restart of all servers are
necessary after test. Make it possible for a test to signal it want mtr.pl to restart
all servers.
When the thread executing a DDL was killed after finished its
execution but before writing the binlog event, the error code in
the binlog event could be set wrongly to ER_SERVER_SHUTDOWN or
ER_QUERY_INTERRUPTED.
This patch fixed the problem by ignoring the kill status when
constructing the event for DDL statements.
This patch also included the following changes in order to
provide the test case.
1) modified mysqltest to support variable for connection command
2) modified mysql-test-run.pl, add new variable MYSQL_SLAVE to
run mysql client against the slave mysqld.
mysqld w/ --valgrind
- Fixed by implementing parsing of error log messages generated outside of
test case runs (eg. during server shutdown).
Also make mysql-test-run.pl not delete the error log after server restart,
which looses information about which warnings were found.
Finally, make multi_update2 a --big test.
mysql-test/lib/My/Test.pm:
Fix home-brewed (and broken) serialization in My::Test to use the standard
Storable serializer.
mysql-test/mysql-test-run.pl:
- Stop mysqld servers gracefully rather than kill -9 when
warnings are being checked.
- After stopping mysqld servers, do an additional parse of the error
log to check for any warnings generated during shutdown.
- Fix error log parsing to be careful not to skip parsing part of the
file, by keeping track of previous file position rather than
relying on mark_log markers.
- Workers report warnings during shutdown to the master process with
a new packet 'WARNINGS' which includes a list of names of test that
might have caused the problem (could be any test run since last
server start).
- Fail entire test suite if warnings are found.
- When we remove the server data dir before server restart, preserve the
error log (don't delete it between restarts), as it may contain
valuable information even for test cases which don't show direct
failures.
mysql-test/t/multi_update2.test:
Make test --big, as it takes a _long_ time to run and only tests a single bug.
This is a backport of code from MySQL 6.0 with cleanups and extensions
The following new options are supported
configure options:
--with-libevent ; Enable use of libevent, which is needed for pool of threads
mysqld options:
--thread-handling=pool-of-threads ; Use a pool of threads to handle queries
--thread-pool-size=# ; Define how many threads should be created to handle all queries
--extra-port=# ; Extra tcp port that uses the old one-thread-per-connection method
--extra-max-connections=# ; Number of connections to accept to 'extra-port'
--test-ignore-wrong-options ; Ignore setting an enum value to a wrong option (for mysql-test-run)
BUILD/SETUP.sh:
Added libevents (and thus pool-of-threads) to max builds
CMakeLists.txt:
Added libevent
Makefile.am:
Added libevents
config/ac-macros/libevent.m4:
Libevent code for configure
config/ac-macros/libevent_configure.m4:
Libevent code for configure
configure.in:
Added libevents
dbug/dbug.c:
Added _db_is_pushed(); Needed for pool-of-threads code
extra/Makefile.am:
Added libevents
extra/libevent:
Libevent initial code
extra/libevent/CMakeLists.txt:
Libevent initial code
extra/libevent/Makefile.am:
Libevent initial code
extra/libevent/README:
Libevent initial code
extra/libevent/WIN32-Code:
Libevent initial code
extra/libevent/WIN32-Code/config.h:
Libevent initial code
extra/libevent/WIN32-Code/misc.c:
Libevent initial code
extra/libevent/WIN32-Code/misc.h:
Libevent initial code
extra/libevent/WIN32-Code/tree.h:
Libevent initial code
extra/libevent/WIN32-Code/win32.c:
Libevent initial code
extra/libevent/buffer.c:
Libevent initial code
extra/libevent/compat:
Libevent initial code
extra/libevent/compat/sys:
Libevent initial code
extra/libevent/compat/sys/_time.h:
Libevent initial code
extra/libevent/compat/sys/queue.h:
Libevent initial code
extra/libevent/compat/sys/tree.h:
Libevent initial code
extra/libevent/devpoll.c:
Libevent initial code
extra/libevent/epoll.c:
Libevent initial code
extra/libevent/epoll_sub.c:
Libevent initial code
extra/libevent/evbuffer.c:
Libevent initial code
extra/libevent/evdns.c:
Libevent initial code
extra/libevent/evdns.h:
Libevent initial code
extra/libevent/event-config.h:
Libevent initial code
extra/libevent/event-internal.h:
Libevent initial code
extra/libevent/event.c:
Libevent initial code
extra/libevent/event.h:
Libevent initial code
extra/libevent/event_tagging.c:
Libevent initial code
extra/libevent/evhttp.h:
Libevent initial code
extra/libevent/evport.c:
Libevent initial code
extra/libevent/evrpc-internal.h:
Libevent initial code
extra/libevent/evrpc.c:
Libevent initial code
extra/libevent/evrpc.h:
Libevent initial code
extra/libevent/evsignal.h:
Libevent initial code
extra/libevent/evutil.c:
Libevent initial code
extra/libevent/evutil.h:
Libevent initial code
extra/libevent/http-internal.h:
Libevent initial code
extra/libevent/http.c:
Libevent initial code
extra/libevent/kqueue.c:
Libevent initial code
extra/libevent/log.c:
Libevent initial code
extra/libevent/log.h:
Libevent initial code
extra/libevent/min_heap.h:
Libevent initial code
extra/libevent/poll.c:
Libevent initial code
extra/libevent/select.c:
Libevent initial code
extra/libevent/signal.c:
Libevent initial code
extra/libevent/strlcpy-internal.h:
Libevent initial code
extra/libevent/strlcpy.c:
Libevent initial code
include/config-win.h:
Libevent support
include/my_dbug.h:
ADded _db_is_pushed
include/mysql.h.pp:
Update to handle new prototypes
include/typelib.h:
Split find_type_or_exit() into two functions
include/violite.h:
Added vio_is_pending()
libmysqld/Makefile.am:
Added libevent
mysql-test/include/have_pool_of_threads.inc:
Added test for pool-of-threads
mysql-test/mysql-test-run.pl:
Don't abort based on time and don't retry test cases when run under --gdb or --debug
mysql-test/r/crash_commit_before.result:
USE GLOBAL for debug variable
mysql-test/r/have_pool_of_threads.require:
Added test for pool-of-threads
mysql-test/r/pool_of_threads.result:
Added test for pool-of-threads
mysql-test/r/subselect_debug.result:
USE GLOBAL for debug variable
mysql-test/t/crash_commit_before.test:
USE GLOBAL for debug variable
mysql-test/t/merge-big.test:
USE GLOBAL for debug variable
mysql-test/t/pool_of_threads-master.opt:
Added test for pool-of-threads
mysql-test/t/pool_of_threads.test:
Added test for pool-of-threads
mysys/typelib.c:
Split find_type_or_exit() into find_type_with_warning()
sql/Makefile.am:
Added libevent
sql/handler.cc:
Indentation fix.
Fixed memory loss bug
Fixed crash on exit when handler plugin failed
sql/mysql_priv.h:
Added extra_max_connections and mysqld_extra_port
Added extern functions from sql_connect.cc
sql/mysqld.cc:
Added support for new mysqld options
Added code for 'extra-port' and 'extra-max-connections'
Split some functions into smaller pieces to be able to reuse code
Added code for test-ignore-wrong-options
sql/scheduler.cc:
Updated schduler code from MySQL 6.0
sql/scheduler.h:
Updated schduler code from MySQL 6.0
sql/set_var.cc:
Added support for changing "extra_max_connections"
sql/sql_class.cc:
Iniitalize thread schduler options in THD
sql/sql_class.h:
Added to extra_port and scheduler to 'THD'
sql/sql_connect.cc:
Use thd->schduler to check number of connections and terminate connection
Made some local functions global (for scheduler.cc)
vio/viosocket.c:
Added 'vio_pending', needed for scheduler..c
Error log gets truncated when mysqld is restarted by MTR
mysql-test/include/check-warnings.test:
Since server doesn't have log-error, get it from env. var. set by MTR
mysql-test/lib/My/ConfigFactory.pm:
"Hide" log-error in a comment
mysql-test/mysql-test-run.pl:
"Hide" log-error in my.cnf by comment
add --console to arguments on Windows
Move .err file to var/log
mysql-test/t/variables.test:
Reset delay_key_write, otherwise maria.maria test may fail
sql/set_var.cc:
Reset ha_open_options if one resets the delay_key_write variable.
Before there was no way to reset it without restarting mysqld, which caused some tests to fail
testcase checks are made.
MTR spawns mysqltest to run check-testcase test before and after each testcase
it runs. It can also run check-warnings using mysqltest. Since it happened on PB
that these checks hanged, this patch provides additional feedback to help
investigating such failures:
- mysqltest is modified to give feedback about main steps in execution of a
testcase if run in verbose mode (including connection to the server),
- MTR is modified to run mysqltest in verbose mode when doing check-testcase or
check-warnings. The diagnostic output from mysqltest is preserved so that it is
saved upon test failure.
client/mysqltest.cc:
Add verbose messages informing about main steps in execution of a testcase.
mysql-test/mysql-test-run.pl:
- When doing check-testcase or check-warnings run mysqltest in verbose mode.
- Do not delete the mysqltest's error log if errors are detected during these
- Since we are only using the auto cleanup in one place of mtr.pl today, disable the
autocleanup and write our own END handler that clean up the tmpdir only when the process
that created it exits.
- adding back Serg's "mtr --list-options"
- safe_mutex deadlock detector started raising wrong deadlock warnings, fixed
here by a backport from 6.0-main.
include/my_pthread.h:
Porting changes done to 6.0-main which satisfy the safe_mutex deadlock detector (those
in 5.1-main don't), see chad@mysql.com-20090126155607-n0j3zbmgbfepnmmo for explanations
mysql-test/mysql-test-run.pl:
adding back Serg's --list-options
mysys/my_init.c:
Porting changes done to 6.0-main which satisfy the safe_mutex deadlock detector (those
in 5.1-main don't), see chad@mysql.com-20090126155607-n0j3zbmgbfepnmmo for explanations
mysys/my_thr_init.c:
Porting changes done to 6.0-main which satisfy the safe_mutex deadlock detector (those
in 5.1-main don't), see chad@mysql.com-20090126155607-n0j3zbmgbfepnmmo for explanations
There were so many changes into mtr (this is the new mtr coming) that I rather
copied mtr from 6.0-main here (at least this one knows how to run Maria tests).
I also fixed suite/maria tests to be accepted by the new mtr.
mysys/thr_mutex.c:
adding DBUG_PRINT here, so that we can locate where the warning is issued.
- Make a rough filtering of the servers error log and write
all suspicious warnings to $error_log.warnings
The .warnings file is then examined more carefully by check_warnings.test
- This will speed things up, doing all of this in a server running
under valgrind takes far too long time.
- Add a "skip-ssl=1" to [mysqltest] section so that
mysqltest will not run with ssl turned on by default
but stil be able to turn it on when requested
- This avoids that check_warnings and check_testcase
connects to the server woth SSL turned on
