database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-07-30 16:24:05 +03:00
Code Activity

Added HandlerSocket plugin

Browse Source 
- Fixed compiler errors
- Modified Makefiles to be part of plugin directory
- Some minor changes in database.cpp to use the new MariaDB handler interface
This commit is contained in:
Michael Widenius
2011-02-20 15:22:10 +02:00
parent 26aa83bfc0
commit 2c7d6f12ee
109 changed files with 21060 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
plugin/handler_socket/docs-en/about-handlersocket.en.txt Normal file
View File

@ -0,0 +1,72 @@
-----------------------------------------------------------------------------
HandlerSocket plugin for MySQL
Copyright (c) 2010 DeNA Co.,Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of DeNA Co.,Ltd. nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY DeNA Co.,Ltd. "AS IS" AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL DeNA Co.,Ltd. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-----------------------------------------------------------------------------
About HandlerSocket
HandlerSocket is a NoSQL plugin for MySQL. It works as a daemon inside the
mysqld process, accept tcp connections, and execute requests from clients.
HandlerSocket does not support SQL queries. Instead, it supports simple CRUD
operations on tables.
Because of the following reasons, HandlerSocket is much faster than the
mysqld/libmysql pair in some circumstances:
- HandlerSocket manipulates data without parsing SQL, which causes less
CPU usage.
- HandlerSocket reads many requests from clients and executes their
requests in bulk, which causes less CPU and disk usage.
- HandlerSocket client/server protocol is more compact than the
mysql/libmysql pair, which causes less network usage.
The current version of HandlerSocket only works with GNU/Linux. The source
archive of HandlerSocket includes a C++ and a Perl client libraries.
Here is a list of other language bindings:
- PHP
http://openpear.org/package/Net_HandlerSocket
http://github.com/tz-lom/HSPHP
http://code.google.com/p/php-handlersocket/
- Java
http://code.google.com/p/handlersocketforjava/
- Python
https://code.launchpad.net/~songofacandy/+junk/pyhandlersocket
- Ruby
https://github.com/winebarrel/ruby-handlersocket
https://github.com/miyucy/handlersocket
- JavaScript(Node.js)
https://github.com/koichik/node-handlersocket
The home of HandlerSocket is here:
https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL
More documents are available in docs-en/ and docs-ja/ directories.

87
plugin/handler_socket/docs-en/configuration-options.en.txt Normal file
View File

@ -0,0 +1,87 @@
-----------------------------------------------------------------
handlersocket_verbose (default = 10, min = 0, max = 10000)
Specify the logging verboseness.
-----------------------------------------------------------------
handlersocket_address (default = '')
Specify the address to bind. If empty, it binds to 0.0.0.0.
-----------------------------------------------------------------
handlersocket_port (default = '9998')
Specify the port to bind. This option is for the listener for
read requests. If empty, the listener is disabled.
-----------------------------------------------------------------
handlersocket_port_wr (default = '9999')
Specify the port to bind. This option is for the listener for
write requests. If empty, the listener is disabled.
-----------------------------------------------------------------
handlersocket_epoll (default = 1, min = 0, max = 1)
Specify whether handlersocket uses epoll for I/O multiplexing.
-----------------------------------------------------------------
handlersocket_threads (default = 16, min = 1, max = 3000)
Specify the number of handlersocket worker threads. This option
is for the listener for read requests. Recommended value is
(the number of CPU cores * 2).
-----------------------------------------------------------------
handlersocket_threads_wr (default = 1, min = 1, max = 3000)
Specify the number of handlersocket worker threads. This option
is for the listener for write requests. Recommended value is 1.
-----------------------------------------------------------------
handlersocket_timeout (default = 300, min = 30, max = 3600)
Specify the socket timeout in seconds.
-----------------------------------------------------------------
handlersocket_backlog (default = 32768, min = 5, max = 1000000)
Specify the length of the listen backlog.
-----------------------------------------------------------------
handlersocket_sndbuf (default = 0, min = 0, max = 1677216)
Specify the maximum socket send buffer in bytes. If 0, the
system-wide default value is set.
-----------------------------------------------------------------
handlersocket_rcvbuf (default = 0, min = 0, max = 1677216)
Specify the maximum socket receive buffer in bytes. If 0, the
system-wide default value is set.
-----------------------------------------------------------------
handlersocket_readsize (default = 0, min = 0, max = 1677216)
Specify the minimum length of the handlersocket request buffer.
Larger value can make handlersocket faster for large requests,
but can consume memory. The default value is possibly 4096.
-----------------------------------------------------------------
handlersocket_accept_balance (default = 0, min = 0, max = 10000)
When this option is set to non-zero, handlersocket tries to
balance accepted connections among threads. Non-zero is
recommended if you use persistent connections (i.e., connection
pooling on the client side).
-----------------------------------------------------------------
handlersocket_wrlock_timeout (default = 12, min = 0, max = 3600)
Specify the lock timeout in seconds. When a write request is
performed, handlersocket locks an advisory lock named
'handlersocket_wr'. This option sets the timeout for the
locking.

91
plugin/handler_socket/docs-en/installation.en.txt Normal file
View File

@ -0,0 +1,91 @@
1. Building Handlersocket
Handlersocket mainly consists of libhsclient, handlersocket, and C++/Perl clients. libhsclient is a common library shared from both client and server(plugin). handlersocket is a MySQL daemon plugin.
To build Handlersocket, you need both MySQL source code and MySQL binary. It is not required to pre-build MySQL source code, but source itself is needed because Handlersocket depends on MySQL header files that only MySQL source distribution contains. MySQL binary is just a normal MySQL binary distribution. You can use official MySQL binaries provided by Oracle.
Since Handlersocket uses daemon plugin interface supported from MySQL 5.1,
MySQL 5.1 or higher version is required.
Please make sure that you use identical MySQL version between MySQL source
and MySQL binary. Otherwise you might encounter serious problems (i.e. server
crash, etc).
Here are steps to build Handlersocket.
* Get MySQL source code
* Get MySQL binary
* Build Handlersocket
$ ./autogen.sh
$ ./configure --with-mysql-source=/work/mysql-5.1.50 --with-mysql-bindir=/work/mysql-5.1.50-linux-x86_64-glibc23/bin --with-mysql-plugindir=/work/mysql-5.1.50-linux-x86_64-glibc23/lib/plugin
--with-mysql-source refers to the top of MySQL source directory,
--with-mysql-bindir refers to where MySQL binary executables (i.e.
mysql_config) are located, and --with-mysql-plugindir refers to a plugin
directory where plugin libraries (*.so) are installed.
$ make
$ sudo make install
Both libhsclient and the handlersocket plugin will be installed.
2. Using Handlersocket
Append configuration options for handlersocket to my.cnf.
[mysqld]
loose_handlersocket_port = 9998
# the port number to bind to (for read requests)
loose_handlersocket_port_wr = 9999
# the port number to bind to (for write requests)
loose_handlersocket_threads = 16
# the number of worker threads (for read requests)
loose_handlersocket_threads_wr = 1
# the number of worker threads (for write requests)
open_files_limit = 65535
# to allow handlersocket accept many concurrent
# connections, make open_files_limit as large as
# possible.
Log in to mysql as root, and execute the following query.
mysql> install plugin handlersocket soname 'handlersocket.so';
If handlersocket.so is successfully installed, it starts
accepting connections on port 9998 and 9999. Running
'show processlist' should show handlersocket worker threads.
-----------------------------------------------------------------
On the client side, you need to install libhsclient for c++ apps
and perl-Net-HandlerSocket for perl apps. They do not require
MySQL to compile.
$ ./autogen.sh
$ ./configure --disable-handlersocket-server
$ make
$ sudo make install
$ cd perl-Net-HandlerSocket
$ perl Makefile.PL
$ make
$ sudo make install
-----------------------------------------------------------------
Alternatively, you can use the rpm installation. If your OS
supports rpms, you can use the following commands to build and
install handlersocket rpm packages.
(Server side, installs HandlerSocket plugin)
$ ./autogen.sh
$ ./configure --with-mysql-source=/work/mysql-5.1.50 --with-mysql-bindir=/work/mysql-5.1.50-linux-x86_64-glibc23/bin --with-mysql-plugindir=/work/mysql-5.1.50-linux-x86_64-glibc23/lib/plugin
$ make rpm_cli
$ sudo rpm -U dist/RPMS/*/libhsclient*.rpm
$ make rpm_c
$ sudo rpm -U dist/RPMS/*/handlersocket*.rpm
(Client side, installs client libraries)
$ ./autogen.sh
$ ./configure --disable-handlersocket-server
$ make rpm_cli
$ sudo rpm -U dist/RPMS/*/libhsclient*.rpm
$ make rpm_perl
$ sudo rpm -U dist/RPMS/*/perl-Net-HandlerSocket*.rpm

126
plugin/handler_socket/docs-en/perl-client.en.txt Normal file
View File

@ -0,0 +1,126 @@
-----------------------------------------------------------------
To open a connection to the handlersocket plugin, you need to
create a Net::HandlerSocket object.
use Net::HandlerSocket;
my $args = { host => 'localhost', port => 9998 };
my $hs = new Net::HandlerSocket($args);
-----------------------------------------------------------------
Before executing table operations, you need to open an index to
work with.
my $err = $hs->open_index(3, 'database1', 'table1', 'PRIMARY',
'f1,f2');
die $hs->get_error() if $res->[0] != 0;
The first argument for open_index is an integer value which is
used to identify an open table, which is only valid within the
same Net::HandlerSocket object. The 4th argument is the name of
index to open. If 'PRIMARY' is specified, the primary index is
open. The 5th argument is a comma-separated list of column names.
-----------------------------------------------------------------
To read a record from a table using an index, call the
execute_single method.
my $res = $hs->execute_single(3, '=', [ 'foo' ], 1, 0);
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
The first argument must be an integer which has specified as the
first argument for open_index on the same Net::HandlerSocket
object. The second argument specifies the search operation. The
current version of handlersocket supports '=', '>=', '<=', '>',
and '<'. The 3rd argument specifies the key to find, which must
an arrayref whose length is equal to or smaller than the number
of key columns of the index. The 4th and the 5th arguments
specify the maximum number of records to be retrieved, and the
number of records skipped before retrieving records. The columns
to be retrieved are specified by the 5th argument for the
corresponding open_index call.
The execute_single method always returns an arrayref. The first
element is the error code, which is 0 when no error is occured.
The remaining are the field values. If more than one record is
returned, it is flatten to an 1-dimensional array. For example,
when 5 records that have 3 columns are returned, you can retrieve
values using the following code.
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
for (my $row = 0; $row < 5; ++$row) {
for (my $col = 0; $col < 3; ++$col) {
my $value = $res->[$row * 5 + $col];
# ...
}
}
-----------------------------------------------------------------
To update or delete records, you need to specify more arguments
for the execute_single method. Note that the Net::HandlerSocket
object must be connected to a handlersocket worker for write
operations, which is port 9999 by default.
(For safety, the port 9998 only allows read operations, and the
port 9999 allows write operations also. The port 9999 allows
read operations too, but slower than 9998 because of record
locking etc.. Port numbers can be changed using the
'handlersocket_port' and the 'handlersocket_port_wr'
configuration options of mysqld.)
my $args = { host => 'localhost', port => 9999 };
my $hs = new Net::HandlerSocket($args);
my $res = $hs->execute_single(3, '=', [ 'bar' ], 1, 0, 'U',
[ 'fubar', 'hoge' ]);
die $hs->get_error() if $res->[0] != 0;
my $num_updated_rows = $res->[1];
my $res = $hs->execute_single(3, '=', [ 'baz' ], 1, 0, 'D');
die $hs->get_error() if $res->[0] != 0;
my $num_deleted_rows = $res->[1];
The 6th argument for execute_single specifies the modification
operation. The current version supports 'U' and 'D'. For the 'U'
operation, the 7th argument specifies the new value for the row.
The columns to be modified are specified by the 5th argument for
the corresponding open_index call. For the 'D' operation, the
7th argument can be omitted.
-----------------------------------------------------------------
The execute_single method can be used for inserting records also.
my $res = $hs->execute_single(3, '+', [ 'foo', 'bar', 'baz' ]);
die $hs->get_error() if $res->[0] != 0;
my $num_inserted_rows = $res->[1];
The 3rd argument must be an arrayref whose elements correspond to
the 5th argument for the corresponding open_index call. If there
is a column which is not appeared in the 5th argument for the
open_index, the default value for the column is set.
-----------------------------------------------------------------
Multiple operations can be executed in a single call. Executing
multiple operations in a single call is much faster than
executing them separatedly.
my $rarr = $hs->execute_multi([
[ 0, '>=', [ 'foo' ], 5, 0 ],
[ 2, '=', [ 'bar' ], 1, 0 ],
[ 4, '<', [ 'baz' ], 10, 5 ],
]);
for my $res (@$rarr) {
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
# ...
}
-----------------------------------------------------------------
When an error is occured, the first element of the returned
arrayref becomes a non-zero value. A negative value indicates
that an I/O error is occured and the Net::HandlerSocket object
should be disposed. A positive value means that the connection is
still active and the Net::HandlerSocket object can be reused
later.

148
plugin/handler_socket/docs-en/protocol.en.txt Normal file
View File

@ -0,0 +1,148 @@
----------------------------------------------------------------------------
The HandlerSocket protocol
----------------------------------------------------------------------------
Basic syntax
- The HandlerSocket protocol is line-based. Each line ends with LF(0x0a).
- Each line consists a concatenation of tokens separated by HT(0x09).
- A token is either NULL or an encoded string. Note that you need to
distinguish NULL from an empty string, as most DBMs does so.
- NULL is expressed as a single NUL(0x00).
- An encoded string is a string with the following encoding rules.
- Characters in the range [0x10 - 0xff] are not encoded.
- A character in the range [0x00 - 0x0f] is prefixed by 0x01 and
shifted by 0x40. For example, 0x03 is encoded as 0x01 0x43.
- Note that a string can be empty. A continuation of 0x09 0x09 means that
there is an empty string between them. A continuation of 0x09 0x0a means
that there is an empty string at the end of the line.
----------------------------------------------------------------------------
Request and Response
- The HandlerSocket protocol is a simple request/response protocol. After a
connection is established, the client side sends a request, and then the
server side sends a response.
- A request/response consists of a single line.
- Requests can be pipelined; That is, you can send multiple requests (ie.
lines) at one time, and receive responses for them at one time.
----------------------------------------------------------------------------
'open_index' request
The 'open_index' request has the following syntax.
P <indexid> <dbname> <tablename> <indexname> <columns>
- <indexid> is a number in decimal.
- <dbname>, <tablename>, and <indexname> are strings. To open the primary
key, use PRIMARY as <indexname>.
- <columns> is a comma-separated list of column names.
Once an 'open_index' request is issued, the HandlerSocket plugin opens the
specified index and keep it open until the client connection is closed. Each
open index is identified by <indexid>. If <indexid> is already open, the old
open index is closed. You can open the same combination of <dbname>
<tablename> <indexname> multple times, possibly with different <columns>.
For efficiency, keep <indexid> small as far as possible.
----------------------------------------------------------------------------
Getting data
The 'find' request has the following syntax.
<indexid> <op> <vlen> <v1> ... <vn> <limit> <offset>
- <indexid> is a number. This number must be an <indexid> specified by a
'open_index' request executed previously on the same connection.
- <op> specifies the comparison operation to use. The current version of
HandlerSocket supports '=', '>', '>=', '<', and '<='.
- <vlen> indicates the length of the trailing parameters <v1> ... <vn>. This
must be smaller than or equal to the number of index columns specified by
specified by the corresponding 'open_index' request.
- <v1> ... <vn> specify the index column values to fetch.
- <limit> and <offset> are numbers. These parameters can be omitted. When
omitted, it works as if 1 and 0 are specified.
----------------------------------------------------------------------------
Updating/Deleting data
The 'find_modify' request has the following syntax.
<indexid> <op> <vlen> <v1> ... <vn> <limit> <offset> <mop> <m1> ... <mk>
- <mop> is either 'U' (update) or 'D' (delete).
- <m1> ... <mk> specifies the column values to set. The length of <m1> ...
<mk> must be smaller than or equal to the length of <columns> specified by
the corresponding 'open_index' request. If <mop> is 'D', these parameters
are ignored.
----------------------------------------------------------------------------
Inserting data
The 'insert' request has the following syntax.
<indexid> '+' <vlen> <v1> ... <vn>
- <vlen> indicates the length of the trailing parameters <v1> ... <vn>. This
must be smaller than or equal to the length of <columns> specified by the
corresponding 'open_index' request.
- <v1> ... <vn> specify the column values to set. For columns not in
<columns>, the default values for each column are set.
----------------------------------------------------------------------------
Response syntax
HandlerSocket returns a response of the following syntax for each request.
<errorcode> <numcolumns> <r1> ... <rn>
- <errorcode> indicates whether the request has successfully executed or not.
'0' means success. Non-zero means an error.
- <numcolumns> indicates the number of columns of the result set.
- <r1> ... <rn> is the result set. The length of <r1> ... <rn> is always a
multiple of <numcolumns>. It is possible that <r1> ... <rn> is empty.
If <errorcode> is non-zero, <numcolumns> is always 1 and <r1> indicates a
human-readable error message, though sometimes <r1> is not provided.
----------------------------------------------------------------------------
Response for 'open_index'
If 'open_index' is succeeded, HandlerSocket returns a line of the following
syntax.
0 1
----------------------------------------------------------------------------
Response for 'find'
If 'find' is succeeded, HandlerSocket returns a line of the following
syntax.
0 <numcolumns> <r1> ... <rn>
- <numcolumns> always equals to the length of <columns> of the corresponding
'open_index' request.
- <r1> ... <rn> is the result set. If N rows are found, the length of <r1>
... <rn> becomes ( <numcolumns> * N ).
----------------------------------------------------------------------------
Response for 'find_modify'
If 'find_modify' is succeeded, HandlerSocket returns a line of the following
syntax.
0 1 <nummod>
- <nummod> is the number of modified rows.
----------------------------------------------------------------------------
Response for 'insert'
If 'insert' is succeeded, HanderSocket returns a line of the following
syntax.
0 1