I'm pleased to announce the release of Spider storage engine version 2.11(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add table link background monitoring feature.
- Add table parameter "monitoring_bg_interval" and "monitoring_bg_kind".
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
12/28/2009
12/15/2009
[MySQL][Spider]Spider-2.10 released
I'm pleased to announce the release of Spider storage engine version 2.10(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "ssl_ca", "ssl_capath", "ssl_cert", "ssl_cipher", "ssl_key", "ssl_verify_server_cert", "default_file" and "default_group".
- Add UDF parameter "ssl_ca", "ssl_capath", "ssl_cert", "ssl_cipher", "ssl_key", "ssl_verify_server_cert", "default_file" and "default_group".
You can use ssl for Spider's communication to remote servers.
Spider can read config file(my.cnf) at connecting to remote servers.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Shinya for bug report.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "ssl_ca", "ssl_capath", "ssl_cert", "ssl_cipher", "ssl_key", "ssl_verify_server_cert", "default_file" and "default_group".
- Add UDF parameter "ssl_ca", "ssl_capath", "ssl_cert", "ssl_cipher", "ssl_key", "ssl_verify_server_cert", "default_file" and "default_group".
You can use ssl for Spider's communication to remote servers.
Spider can read config file(my.cnf) at connecting to remote servers.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Shinya for bug report.
Enjoy!
11/29/2009
[MySQL][Spider]Spider-2.9 released
I'm pleased to announce the release of Spider storage engine version 2.9(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add UDFs "spider_ping_table".
- Add table parameter "monitoring_kind", "monitoring_limit" and "monitoring_server_id".
- Add server parameter "spider_udf_table_mon_mutex_count".
Add Spider's link fault monitor at this release.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to shinichiro and merikonjatta for bug report.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add UDFs "spider_ping_table".
- Add table parameter "monitoring_kind", "monitoring_limit" and "monitoring_server_id".
- Add server parameter "spider_udf_table_mon_mutex_count".
Add Spider's link fault monitor at this release.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to shinichiro and merikonjatta for bug report.
Enjoy!
11/04/2009
[MySQL][Spider]Spider-2.8 released
I'm pleased to announce the release of Spider storage engine version 2.8(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "link_status".
You can change link_status using "alter table" statement.
Spider's link fault management is table level.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "link_status".
You can change link_status using "alter table" statement.
Spider's link fault management is table level.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
10/30/2009
[MySQL][Spider]Spider-2.7 released
I'm pleased to announce the release of Spider storage engine version 2.7(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Spider table can load balance using multiple links at searching without locking. Load balance rule is "((server_id + thread_id) % number_of_links)".
Please write multiple connection informations splitting space characters at "host", "user", "table", "server", "socket", "wrapper", "database" and "password" table parameters.
Example: "host 'h1 h2 h3'"
I will continue to develop Spider's functions for Active-Active type clustering.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Spider table can load balance using multiple links at searching without locking. Load balance rule is "((server_id + thread_id) % number_of_links)".
Please write multiple connection informations splitting space characters at "host", "user", "table", "server", "socket", "wrapper", "database" and "password" table parameters.
Example: "host 'h1 h2 h3'"
I will continue to develop Spider's functions for Active-Active type clustering.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
10/19/2009
[MySQL][Spider]Spider-2.6 released
I'm pleased to announce the release of Spider storage engine version 2.6(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add server parameter "spider_remote_access_charset", "spider_remote_autocommit", "spider_remote_sql_log_off" and "spider_remote_trx_isolation".
These parameters are setted by Spider at connecting remote server. But it does not need to be setted if you know it previously and no need to change it. In this case, you can set these parameters for improvement connecting performance.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Giuseppe for bug report.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add server parameter "spider_remote_access_charset", "spider_remote_autocommit", "spider_remote_sql_log_off" and "spider_remote_trx_isolation".
These parameters are setted by Spider at connecting remote server. But it does not need to be setted if you know it previously and no need to change it. In this case, you can set these parameters for improvement connecting performance.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Giuseppe for bug report.
Enjoy!
10/14/2009
[MySQL][VP]Vartical Partitioning-0.6 released
I'm pleased to announce the release of Vertical Partitioning storage engine version 0.6(alpha).
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
The main changes in this version are following.
- Add UDF "vp_copy_tables".
This UDF is used for syncronizing Vertical Partitioning's child table data from old child tables to new added child tables.
- Add table parameter "choose_ignore_table_list", "choose_ignore_table_list_for_lock" and "zero_record_update_mode".
This parameters are used for avoiding to use new added child tables at searching operation and separating child tables for searching with lock and without lock.
- Add server parameter "vp_udf_ct_bulk_insert_interval" and "vp_udf_ct_bulk_insert_rows".
This parameters are used for changing the load of updating immediately from vp_copy_tables.
Example
-------------------------------------------------------------------------------
Initial tables:
create table tbl_a(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a),
key idx1(col_c, col_a)
)engine=VP comment='tnl "tbl_b tbl_c"';
Add a table:
alter table tbl_a comment='tnl "tbl_b tbl_c tbl_d", cit "3", cil "3", zru "1"';
Copy table data from tbl_c to tbl_d:
select vp_copy_tables("tbl_a", "tbl_c", "tbl_d");
Finalize:
alter table tbl_a comment='tnl "tbl_b tbl_c tbl_d"';
-------------------------------------------------------------------------------
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
The main changes in this version are following.
- Add UDF "vp_copy_tables".
This UDF is used for syncronizing Vertical Partitioning's child table data from old child tables to new added child tables.
- Add table parameter "choose_ignore_table_list", "choose_ignore_table_list_for_lock" and "zero_record_update_mode".
This parameters are used for avoiding to use new added child tables at searching operation and separating child tables for searching with lock and without lock.
- Add server parameter "vp_udf_ct_bulk_insert_interval" and "vp_udf_ct_bulk_insert_rows".
This parameters are used for changing the load of updating immediately from vp_copy_tables.
Example
-------------------------------------------------------------------------------
Initial tables:
create table tbl_a(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a),
key idx1(col_c, col_a)
)engine=VP comment='tnl "tbl_b tbl_c"';
Add a table:
alter table tbl_a comment='tnl "tbl_b tbl_c tbl_d", cit "3", cil "3", zru "1"';
Copy table data from tbl_c to tbl_d:
select vp_copy_tables("tbl_a", "tbl_c", "tbl_d");
Finalize:
alter table tbl_a comment='tnl "tbl_b tbl_c tbl_d"';
-------------------------------------------------------------------------------
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
10/05/2009
[MySQL][Spider][VP]Spider-2.5 Vartical Partitioning-0.5 released
I'm pleased to announce the release of Spider storage engine version 2.5(beta) and Vertical Partitioning storage engine version 0.5(alpha).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
The main changes in this version are following.
- The standalone compilation with Vertical Partitioning storage engine is now available.
- Vertical Partitioning can be used for multi-step partitioning.
Example:
create table a(a int primary key)engine=myisam;
create table b(a int primary key)engine=vp comment 'tnl "a"';
create table c(a int primary key)engine=vp comment 'tnl "b"';
- Add table parameter "bgs_mode", "bgi_mode" and "bgu_mode".
- Add server parameter "vp_bgs_mode", "vp_bgi_mode" and "vp_bgu_mode".
Parallel search and update from Vertical Partitioning to Spider are now available by using this parameters.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
The main changes in this version are following.
- The standalone compilation with Vertical Partitioning storage engine is now available.
- Vertical Partitioning can be used for multi-step partitioning.
Example:
create table a(a int primary key)engine=myisam;
create table b(a int primary key)engine=vp comment 'tnl "a"';
create table c(a int primary key)engine=vp comment 'tnl "b"';
- Add table parameter "bgs_mode", "bgi_mode" and "bgu_mode".
- Add server parameter "vp_bgs_mode", "vp_bgi_mode" and "vp_bgu_mode".
Parallel search and update from Vertical Partitioning to Spider are now available by using this parameters.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
9/27/2009
[MySQL][Spider][VP]Spider-2.4 Vartical Partitioning-0.4 released
I'm pleased to announce the release of Spider storage engine version 2.4(beta) and Vertical Partitioning storage engine version 0.4(alpha).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
The main changes in this version are following.
- "Engine-condition-pushdown" is available on Vertical Partitioning with Spider.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Gennady for bug report.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
The main changes in this version are following.
- "Engine-condition-pushdown" is available on Vertical Partitioning with Spider.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Gennady for bug report.
Enjoy!
9/21/2009
[MySQL][VP]Vertical Partitioning storage engine 0.3 released
I'm pleased to announce the release of Vertical Partitioning storage engine version 0.3.
http://launchpad.net/vpformysql
The main changes in this version are following.
- Add table parameter "choose_table_mode_for_lock".
- Add server parameter "vp_choose_table_mode_for_lock".
These parameters are used for avoiding deadlock by fixing lock order to table list order.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://launchpad.net/vpformysql
The main changes in this version are following.
- Add table parameter "choose_table_mode_for_lock".
- Add server parameter "vp_choose_table_mode_for_lock".
These parameters are used for avoiding deadlock by fixing lock order to table list order.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
9/12/2009
[MySQL][Spider]Spider storage engine 2.3 released
I'm pleased to announce the release of Spider storage engine version 2.3(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add UDF parameter "connection_channel".
This parameter is used for increasing background parallelism to same remote server by using multiple connections.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add UDF parameter "connection_channel".
This parameter is used for increasing background parallelism to same remote server by using multiple connections.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
9/06/2009
[MySQL][Spider]Spider storage engine 2.2 released
I'm pleased to announce the release of Spider storage engine version 2.2(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add UDF parameter "iop000".
You can use "ignore" and "replace" at inserting records to temporary table list's tables using this parameter.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
The main changes in this version are following.
- Add UDF parameter "iop000".
You can use "ignore" and "replace" at inserting records to temporary table list's tables using this parameter.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
8/29/2009
[MySQL][Spider]Spider storage engine 2.1 released
I'm pleased to announce the release of Spider storage engine version 2.1(beta).
http://spiderformysql.com/
The main changes in this version are following.
- Add UDFs "spider_direct_sql" and "spider_bg_direct_sql".
These UDFs execute hand writing SQLs on remote server by using Spider's table link and store result sets in temporary tables.
Main usages are following.
1.Maintenance remote servers.
(Create table on remote servers for Spider's link tables, etc.)
2.Use every resources (excluding table) on remote servers.
3.Mass update of the bulk data. (Update all rows in tables, etc.
There are next advantages. It is possible to update it in parallel
with multiple remote servers. The amount of the data transfer of
a local server and remote servers can be reduced)
4.Aggregate by 2-steps. (For DWH etc)
First step: Aggregate on remote servers and collect result
to local server.
Second step: Aggregate on local server.
(There are next advantages. It is possible to aggregate it in
parallel with multiple remote servers. The amount of the data
transfer of a local server and remote servers can be reduced)
Example of 2-steps aggregating
(Count the number of rows in 7 shards tables from local server)
-------------------------------------------------------------------------------
Prepare for aggregating:
mysql> create table target_hosts (host char(17))engine=myisam;
Query OK, 0 rows affected (0.01 sec)
mysql> insert into target_hosts (host) values ('192.168.22.13'),('192.168.22.14'),('192.168.22.15'),('192.168.22.16'),('192.168.22.51'),('192.168.22.52'),('192.168.22.53');
Query OK, 7 rows affected (0.00 sec)
Records: 7 Duplicates: 0 Warnings: 0
mysql> create temporary table a (a int)engine=myisam;
Query OK, 0 rows affected (0.00 sec)
Execute aggregating:
mysql> select sum(a.a) from a, (select sum(spider_direct_sql('select count(*) from data01', 'a', concat('srv "s", host "', host, '"'))) cnt from target_hosts) c;
+-----------+
| sum(a.a) |
+-----------+
| 117641811 |
+-----------+
1 row in set (40.86 sec)
mysql> truncate table a;
Query OK, 0 rows affected (0.00 sec)
mysql> select sum(a.a) from a, (select spider_bg_direct_sql('select count(*) from data01', 'a', concat('srv "s", host "', host, '"')) cnt from target_hosts) c;
+-----------+
| sum(a.a) |
+-----------+
| 117641811 |
+-----------+
1 row in set (5.91 sec)
-------------------------------------------------------------------------------
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://spiderformysql.com/
The main changes in this version are following.
- Add UDFs "spider_direct_sql" and "spider_bg_direct_sql".
These UDFs execute hand writing SQLs on remote server by using Spider's table link and store result sets in temporary tables.
Main usages are following.
1.Maintenance remote servers.
(Create table on remote servers for Spider's link tables, etc.)
2.Use every resources (excluding table) on remote servers.
3.Mass update of the bulk data. (Update all rows in tables, etc.
There are next advantages. It is possible to update it in parallel
with multiple remote servers. The amount of the data transfer of
a local server and remote servers can be reduced)
4.Aggregate by 2-steps. (For DWH etc)
First step: Aggregate on remote servers and collect result
to local server.
Second step: Aggregate on local server.
(There are next advantages. It is possible to aggregate it in
parallel with multiple remote servers. The amount of the data
transfer of a local server and remote servers can be reduced)
Example of 2-steps aggregating
(Count the number of rows in 7 shards tables from local server)
-------------------------------------------------------------------------------
Prepare for aggregating:
mysql> create table target_hosts (host char(17))engine=myisam;
Query OK, 0 rows affected (0.01 sec)
mysql> insert into target_hosts (host) values ('192.168.22.13'),('192.168.22.14'),('192.168.22.15'),('192.168.22.16'),('192.168.22.51'),('192.168.22.52'),('192.168.22.53');
Query OK, 7 rows affected (0.00 sec)
Records: 7 Duplicates: 0 Warnings: 0
mysql> create temporary table a (a int)engine=myisam;
Query OK, 0 rows affected (0.00 sec)
Execute aggregating:
mysql> select sum(a.a) from a, (select sum(spider_direct_sql('select count(*) from data01', 'a', concat('srv "s", host "', host, '"'))) cnt from target_hosts) c;
+-----------+
| sum(a.a) |
+-----------+
| 117641811 |
+-----------+
1 row in set (40.86 sec)
mysql> truncate table a;
Query OK, 0 rows affected (0.00 sec)
mysql> select sum(a.a) from a, (select spider_bg_direct_sql('select count(*) from data01', 'a', concat('srv "s", host "', host, '"')) cnt from target_hosts) c;
+-----------+
| sum(a.a) |
+-----------+
| 117641811 |
+-----------+
1 row in set (5.91 sec)
-------------------------------------------------------------------------------
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
8/22/2009
[MySQL][Spider]Spider storage engine 2.0 released
I'm pleased to announce the release of Spider storage engine version 2.0(beta).
http://spiderformysql.com/
From this time, version 1.x is stable releases, version 2.x is development releases for new features.
The main changes in this version are following.
- Add table parameter "semi_table_lock_connection".
- Add server parameter "spider_semi_table_lock_connection".
Spider has "spider_semi_trx", "spider_semi_trx_isolation", (for transactional tables like InnoDB) "spider_semi_table_lock" and "semi_table_lock" (for non transactional tables like MyISAM) options for consistent reading at remote servers during executing 1 SQL at local server. But you must use different connections for transactional tables and non transactional tables because "semi_table_lock" causes implicit transaction commit.
"semi_table_lock_connection" and "spider_semi_table_lock_connection" can use different connection for transactional tables and non transactional tables by checking that "semi_table_lock" is enabled.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://spiderformysql.com/
From this time, version 1.x is stable releases, version 2.x is development releases for new features.
The main changes in this version are following.
- Add table parameter "semi_table_lock_connection".
- Add server parameter "spider_semi_table_lock_connection".
Spider has "spider_semi_trx", "spider_semi_trx_isolation", (for transactional tables like InnoDB) "spider_semi_table_lock" and "semi_table_lock" (for non transactional tables like MyISAM) options for consistent reading at remote servers during executing 1 SQL at local server. But you must use different connections for transactional tables and non transactional tables because "semi_table_lock" causes implicit transaction commit.
"semi_table_lock_connection" and "spider_semi_table_lock_connection" can use different connection for transactional tables and non transactional tables by checking that "semi_table_lock" is enabled.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
8/18/2009
[MySQL][Spider]Spider storage engine 1.0 released
I'm pleased to announce the release of Spider storage engine version 1.0.
Development status is GA from this version.
http://spiderformysql.com/
From this version, you can download shared library for Linux x86_64.
Thanks to Kazuho for information.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
Development status is GA from this version.
http://spiderformysql.com/
From this version, you can download shared library for Linux x86_64.
Thanks to Kazuho for information.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
8/15/2009
[MySQL][Spider]Spider storage engine 0.18 released
I'm pleased to announce the release of Spider storage engine version 0.18.(RC)
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "use_pushdown_udf".
- Add server parameter "spider_use_pushdown_udf".
"use_pushdown_udf" and "spider_use_pushdown_udf" are used for sending UDF string to remote server if engine_condition_pushdown is used.
From this version, you can use '\r' and '\n' like a space between table parameter name and table parameter value.
Thanks to Akihiro for report.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "use_pushdown_udf".
- Add server parameter "spider_use_pushdown_udf".
"use_pushdown_udf" and "spider_use_pushdown_udf" are used for sending UDF string to remote server if engine_condition_pushdown is used.
From this version, you can use '\r' and '\n' like a space between table parameter name and table parameter value.
Thanks to Akihiro for report.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
8/09/2009
[MySQL][Spider]Spider storage engine 0.17 released
I'm pleased to announce the release of Spider storage engine version 0.17.(RC)
Development status is RC from this version.
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "use_table_charset".
- Add server parameter "spider_use_table_charset" and "spider_local_lock_table".
"use_table_charset" and "spider_use_table_charset" are used for choosing the communication charset (UTF8 or table charset) at connections from Spider to remote server. Basically, you don't need changing this option but you can change this option for performance improvement if the SQL has very longer table and column name string than column data string.
"spider_local_lock_table" is used for suppressing sending "lock table" statement to remote server for locking table at local server only. This option is used with non-blocking resharding.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
Development status is RC from this version.
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "use_table_charset".
- Add server parameter "spider_use_table_charset" and "spider_local_lock_table".
"use_table_charset" and "spider_use_table_charset" are used for choosing the communication charset (UTF8 or table charset) at connections from Spider to remote server. Basically, you don't need changing this option but you can change this option for performance improvement if the SQL has very longer table and column name string than column data string.
"spider_local_lock_table" is used for suppressing sending "lock table" statement to remote server for locking table at local server only. This option is used with non-blocking resharding.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
8/01/2009
[MySQL][Spider]Spider storage engine 0.16 released
I'm pleased to announce the release of Spider storage engine version 0.16.(beta)
http://spiderformysql.com/
The main changes in this version are following.
- Add server parameter "spider_same_server_link" and "spider_table_init_error_interval".
"spider_same_server_link" is used for creating table link to same MySQL server's table.
"spider_table_init_error_interval" is used for protection from connection consumption of eternal loop of table link. Please increase this parameter using "set global" command if eternal loop of table link causes connection consumption.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://spiderformysql.com/
The main changes in this version are following.
- Add server parameter "spider_same_server_link" and "spider_table_init_error_interval".
"spider_same_server_link" is used for creating table link to same MySQL server's table.
"spider_table_init_error_interval" is used for protection from connection consumption of eternal loop of table link. Please increase this parameter using "set global" command if eternal loop of table link causes connection consumption.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
7/25/2009
[MySQL][Spider]Spider storage engine 0.15 released
I'm pleased to announce the release of Spider storage engine version 0.15.(beta)
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "auto_increment_mode".
- Add server parameter "spider_auto_increment_mode".
Spider supports 3 modes.
0:Use a count that Spider get from the remote server with exclusive lock for auto increment value.
1:Use a count that Spider has internal for auto increment value.
2:Auto increment value is given by the remote server.
- Add server parameter "spider_ping_interval_at_trx_start".
Performance improvement by frequency decrease of checking broken connections.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "auto_increment_mode".
- Add server parameter "spider_auto_increment_mode".
Spider supports 3 modes.
0:Use a count that Spider get from the remote server with exclusive lock for auto increment value.
1:Use a count that Spider has internal for auto increment value.
2:Auto increment value is given by the remote server.
- Add server parameter "spider_ping_interval_at_trx_start".
Performance improvement by frequency decrease of checking broken connections.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
7/21/2009
[MySQL][Spider]Spider storage engine 0.14 released
I'm pleased to announce the release of Spider storage engine version 0.14.(beta)
http://spiderformysql.com/
The main changes in this version are following.
- The standalone compilation with Spider storage engine is now available.
Please see "03_install_spider_storage_engine.txt" for more detail.
Tatsuhiko and Tetsuro,
Thank you for your advice!
Please see "99_change_logs.txt" in the download documents for checking other changes.
Spider storage engine was introduced at following with the usage.
The Data Charmer(English)
http://datacharmer.blogspot.com/2009/04/test-driving-spider-storage-engine.html
http://datacharmer.blogspot.com/2009/07/sharding-for-masses-spider-storage.html
Giuseppe,
Thank you for introducing Spider.
Unoh Labs(Japanese)
http://labs.unoh.net/2009/07/mysqlspider.html
Satoru,
Thank you for introducing Spider.
Enjoy!
http://spiderformysql.com/
The main changes in this version are following.
- The standalone compilation with Spider storage engine is now available.
Please see "03_install_spider_storage_engine.txt" for more detail.
Tatsuhiko and Tetsuro,
Thank you for your advice!
Please see "99_change_logs.txt" in the download documents for checking other changes.
Spider storage engine was introduced at following with the usage.
The Data Charmer(English)
http://datacharmer.blogspot.com/2009/04/test-driving-spider-storage-engine.html
http://datacharmer.blogspot.com/2009/07/sharding-for-masses-spider-storage.html
Giuseppe,
Thank you for introducing Spider.
Unoh Labs(Japanese)
http://labs.unoh.net/2009/07/mysqlspider.html
Satoru,
Thank you for introducing Spider.
Enjoy!
7/08/2009
[MySQL][Spider]Spider storage engine 0.13 released
I'm pleased to announce the release of Spider storage engine version 0.13.(beta)
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "crd_bg_mode" and "sts_bg_mode".
- Add server parameter "spider_crd_bg_mode" and "spider_sts_bg_mode".
MySQL needs table's analysis informations when select statement is executed. Spider storage engine gets this informations from remote table but this action often generates over-heads.
This options remove generated over-heads by getting analysis informations at background but you must set this option to 0 if the table's analysis informations change frequently.
* There are other options for the over-heads reduction of the analysis informations acquisition. I will introduce this at another time.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://spiderformysql.com/
The main changes in this version are following.
- Add table parameter "crd_bg_mode" and "sts_bg_mode".
- Add server parameter "spider_crd_bg_mode" and "spider_sts_bg_mode".
MySQL needs table's analysis informations when select statement is executed. Spider storage engine gets this informations from remote table but this action often generates over-heads.
This options remove generated over-heads by getting analysis informations at background but you must set this option to 0 if the table's analysis informations change frequently.
* There are other options for the over-heads reduction of the analysis informations acquisition. I will introduce this at another time.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
7/03/2009
[MySQL][Other]Condition Pushdown for ha_partition 0.1 released
I'm pleased to announce the release of Condition Pushdown for ha_partition version 0.1.
http://launchpad.net/partitionconditionpushdownformysql
This release is a patch file for MySQL's table partitioning feature. (ndb is excluded)
This patch file works to add methods("cond_push" and "cond_pop") to ha_partition. As a result, every storage engine's table that use table partitioning can get condition-pushdown through ha_partition.
There was no problem because only ndb supported engine-condition-pushdown up to now.
This patch was needed because the Spider storage engine supported engine-condition-pushdown recently.
Enjoy!
http://launchpad.net/partitionconditionpushdownformysql
This release is a patch file for MySQL's table partitioning feature. (ndb is excluded)
This patch file works to add methods("cond_push" and "cond_pop") to ha_partition. As a result, every storage engine's table that use table partitioning can get condition-pushdown through ha_partition.
There was no problem because only ndb supported engine-condition-pushdown up to now.
This patch was needed because the Spider storage engine supported engine-condition-pushdown recently.
Enjoy!
7/02/2009
[MySQL][VP]Vertical Partitioning storage engine 0.2 released
I'm pleased to announce the release of Vertical Partitioning storage engine version 0.2.
http://launchpad.net/vpformysql
The main changes in this version are following.
- Support MySQL's table partitioning.
MySQL's table partitioning that is only introduced last release is now supported.
note: After this version, you need to apply a patch for installing Vertical Partitioning storage engine.
- After this version, the primary key part of InnoDB's secondary index is effectively used.
You can use the primary key columns with InnoDB's secondary index by using following statements.
create table tbl_b(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a),
key idx1(col_c)
)engine=InnoDB;
create table tbl_c(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a),
key idx1(col_c)
)engine=InnoDB;
create table tbl_a(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a),
key idx1(col_c, col_a)
)engine=VP comment='table_name_list "tbl_b tbl_c"';
note: Vertical Partitioning storage engine can use index with primary key together with index without primary key, because Vertical Partitioning storage engine can use different storage engines for child tables at same table. In this case, the purpose of writing the column of primary key to tbl_a's idx1 is to absorb the difference by this coexistence.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://launchpad.net/vpformysql
The main changes in this version are following.
- Support MySQL's table partitioning.
MySQL's table partitioning that is only introduced last release is now supported.
note: After this version, you need to apply a patch for installing Vertical Partitioning storage engine.
- After this version, the primary key part of InnoDB's secondary index is effectively used.
You can use the primary key columns with InnoDB's secondary index by using following statements.
create table tbl_b(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a),
key idx1(col_c)
)engine=InnoDB;
create table tbl_c(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a),
key idx1(col_c)
)engine=InnoDB;
create table tbl_a(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a),
key idx1(col_c, col_a)
)engine=VP comment='table_name_list "tbl_b tbl_c"';
note: Vertical Partitioning storage engine can use index with primary key together with index without primary key, because Vertical Partitioning storage engine can use different storage engines for child tables at same table. In this case, the purpose of writing the column of primary key to tbl_a's idx1 is to absorb the difference by this coexistence.
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
6/30/2009
[MySQL][Spider]Spider storage engine 0.12 released
I released Spider storage engine version 0.12.(beta)
http://spiderformysql.com/
Spider storage engine can be used for database sharding by using table partitioning and table link.
Spider storage engine can synchronize an update for remote MySQL servers by XA transaction at Spider-internal.
Introduction document:
http://www.mysqlconf.com/mysql2009/public/schedule/detail/6837
The main changes in this version are following.
("Table parameter" that is used following is parameters for each table. Please see "04_table_create.txt" and "06_table_parameters.txt" in the download documents for more details.)
- Add table parameter "table_count_mode".
If storage engine can return correct full record count from table status like MyISAM, you should better to set this option to 1 for getting full record count from table status.
- Add table parameter "select_column_mode", add server parameter "spider_select_column_mode".
After this time, you can choose method of deciding used columns at select clause from "easy deciding(existing method)" and "detailed deciding(new method)".
You can reduce the wire traffic to remote servers by using "detailed deciding".
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
http://spiderformysql.com/
Spider storage engine can be used for database sharding by using table partitioning and table link.
Spider storage engine can synchronize an update for remote MySQL servers by XA transaction at Spider-internal.
Introduction document:
http://www.mysqlconf.com/mysql2009/public/schedule/detail/6837
The main changes in this version are following.
("Table parameter" that is used following is parameters for each table. Please see "04_table_create.txt" and "06_table_parameters.txt" in the download documents for more details.)
- Add table parameter "table_count_mode".
If storage engine can return correct full record count from table status like MyISAM, you should better to set this option to 1 for getting full record count from table status.
- Add table parameter "select_column_mode", add server parameter "spider_select_column_mode".
After this time, you can choose method of deciding used columns at select clause from "easy deciding(existing method)" and "detailed deciding(new method)".
You can reduce the wire traffic to remote servers by using "detailed deciding".
Please see "99_change_logs.txt" in the download documents for checking other changes.
Enjoy!
6/25/2009
[MySQL][VP]Vertical Partitioning storage engine 0.1 released
I'm pleased to announce a release of a new storage engine "Vertical Partitioning storage engine".
http://launchpad.net/vpformysql
You can use relation tables of one to one like one table by using Vertical Partitioning storage engine.
It is near by a view of joined relation tables of one to one but different point is following.
- It can insert.
- It can do partition pruning.(Not use partitions that are no necessity)
Vertical Partitioning storage engine can use following purposes.
(Vertical Partitioning table is called "parent table", integrated tables are called "child table")
1.A MySQL's InnoDB table that has the big record size is very slow if a select statement needs table scanning with a lot of records.
Vertical Partitioning is good for resolving this performance problem.
For example, you can solve a performance issue for "select col_d from tbl_a" by using following partitions.
Before partitioning
create table tbl_a(
col_a int primary key,
col_b int,
col_c int,
col_d int,
col_e varchar(255),
col_f varchar(255),
col_g varchar(255),
col_h varchar(255),
col_i varchar(255),
col_j text,
key idx_a(col_b, col_c),
key idx_b(col_c, col_j(100))
)engine=InnoDB;
After partitioning
create table tbl_b(
col_a int primary key,
col_b int,
col_c int,
col_d int,
key idx_a(col_b, col_c)
)engine=InnoDB;
create table tbl_c(
col_a int primary key,
col_c int,
col_e varchar(255),
col_f varchar(255),
col_g varchar(255),
col_h varchar(255),
col_i varchar(255),
col_j text,
key idx_b(col_c, col_j(100))
)engine=InnoDB;
create table tbl_a(
col_a int primary key,
col_b int,
col_c int,
col_d int,
col_e varchar(255),
col_f varchar(255),
col_g varchar(255),
col_h varchar(255),
col_i varchar(255),
col_j text,
key idx_a(col_b, col_c),
key idx_b(col_c, col_j(100))
)engine=VP
comment 'table_name_list "tbl_b tbl_c"';
Note: Vertical Partitioning storage engine can use same column in different child tables. You can use flexible partition design by this feature but you should note that a lot of same columns causes updating performance issues.
2.MySQL Cluster (ndb) is not good for using text columns.
Vertical Partitioning can use different storage engines for one table.
As a result, MySQL Cluster can use only for strong point columns.
Example of mixed storage engines.
create table tbl_b(
col_a int primary key,
col_b int,
col_c int,
key idx_a(col_b, col_c)
)engine=ndb ......;
create table tbl_c(
col_a int primary key,
col_c int,
col_d varchar(255),
col_e text,
key idx_b(col_c, col_e(100))
)engine=Spider ......;
create table tbl_a(
col_a int primary key,
col_b int,
col_c int,
col_d varchar(255),
col_e text,
key idx_a(col_b, col_c),
key idx_b(col_c, col_e(100))
)engine=VP
comment 'table_name_list "tbl_b tbl_c"';
3.MySQL does not support multi shaped partitioning.
Vertical Partitioning storage engine supports multi shaped partitioning.
Multi shaped partitioning is dividing a table by multiple partitioning rules.
You can use tables that dividing into detail information table and index tables like one table by this feature. And you can shard each table by using Spider storage engine.
Before partitioning
create table employees (
id int primary key,
fname varchar(30),
lname varchar(30),
hired date not null default '1970-01-01',
separated date not null default '9999-12-31',
job_code int,
store_id int
)engine=InnoDB;
After partitioning
create table emp_pk (
id int not null,
separated date not null default '9999-12-31',
unique idx_a(id),
key idx_b(id, separated)
)engine=InnoDB
partition by hash(id)
partitions 4;
create table emp_detail (
id int not null,
fname varchar(30),
lname varchar(30),
hired date not null default '1970-01-01',
separated date not null default '9999-12-31',
job_code int,
store_id int,
key idx_a(id, separated)
)engine=InnoDB
partition by range ( year(separated) ) (
partition p0 values less than (1991),
partition p1 values less than (1996),
partition p2 values less than (2001),
partition p3 values less than maxvalue
);
create table employees (
id int not null,
fname varchar(30),
lname varchar(30),
hired date not null default '1970-01-01',
separated date not null default '9999-12-31',
job_code int,
store_id int,
primary key(id, separated)
)engine=VP
comment 'table_name_list "emp_pk emp_detail", pk_correspond_mode "1"';
Note: In this case, parent table's primary key has columns that using both child table partitioning rules for using partition pruning.
(In the future, I will improve this to become "parent table's primary key has only real primary key (id) and child tables can use partition pruning")
4.Vertical Partitioning can add "parallel columns searching" to other storage engines with supporting "parallel searching" like spider storage engine.(Now developping)
5.Vertical Partitioning storage engine can use for partitioning wrapper because Vertical Partitioning storage engine supports MySQL's table partitioning.
You can use each partition like tables by this feature.(Don't use this feature because now developping)
Example
create table tbl_b(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a)
)engine=InnoDB;
create table tbl_c(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a)
)engine=InnoDB;
create table tbl_d(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a)
)engine=InnoDB;
create table tbl_a(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a)
)engine=VP
partition by hash(col_a)
(
partition pt1 comment='table_name_list "tbl_b"',
partition pt2 comment='table_name_list "tbl_c"',
partition pt3 comment='table_name_list "tbl_d"'
);
In this time, Vertical Partitioning storage engine is alpha version and some features can not use yet but I will develop fast for adding features.
Enjoy!
http://launchpad.net/vpformysql
You can use relation tables of one to one like one table by using Vertical Partitioning storage engine.
It is near by a view of joined relation tables of one to one but different point is following.
- It can insert.
- It can do partition pruning.(Not use partitions that are no necessity)
Vertical Partitioning storage engine can use following purposes.
(Vertical Partitioning table is called "parent table", integrated tables are called "child table")
1.A MySQL's InnoDB table that has the big record size is very slow if a select statement needs table scanning with a lot of records.
Vertical Partitioning is good for resolving this performance problem.
For example, you can solve a performance issue for "select col_d from tbl_a" by using following partitions.
Before partitioning
create table tbl_a(
col_a int primary key,
col_b int,
col_c int,
col_d int,
col_e varchar(255),
col_f varchar(255),
col_g varchar(255),
col_h varchar(255),
col_i varchar(255),
col_j text,
key idx_a(col_b, col_c),
key idx_b(col_c, col_j(100))
)engine=InnoDB;
After partitioning
create table tbl_b(
col_a int primary key,
col_b int,
col_c int,
col_d int,
key idx_a(col_b, col_c)
)engine=InnoDB;
create table tbl_c(
col_a int primary key,
col_c int,
col_e varchar(255),
col_f varchar(255),
col_g varchar(255),
col_h varchar(255),
col_i varchar(255),
col_j text,
key idx_b(col_c, col_j(100))
)engine=InnoDB;
create table tbl_a(
col_a int primary key,
col_b int,
col_c int,
col_d int,
col_e varchar(255),
col_f varchar(255),
col_g varchar(255),
col_h varchar(255),
col_i varchar(255),
col_j text,
key idx_a(col_b, col_c),
key idx_b(col_c, col_j(100))
)engine=VP
comment 'table_name_list "tbl_b tbl_c"';
Note: Vertical Partitioning storage engine can use same column in different child tables. You can use flexible partition design by this feature but you should note that a lot of same columns causes updating performance issues.
2.MySQL Cluster (ndb) is not good for using text columns.
Vertical Partitioning can use different storage engines for one table.
As a result, MySQL Cluster can use only for strong point columns.
Example of mixed storage engines.
create table tbl_b(
col_a int primary key,
col_b int,
col_c int,
key idx_a(col_b, col_c)
)engine=ndb ......;
create table tbl_c(
col_a int primary key,
col_c int,
col_d varchar(255),
col_e text,
key idx_b(col_c, col_e(100))
)engine=Spider ......;
create table tbl_a(
col_a int primary key,
col_b int,
col_c int,
col_d varchar(255),
col_e text,
key idx_a(col_b, col_c),
key idx_b(col_c, col_e(100))
)engine=VP
comment 'table_name_list "tbl_b tbl_c"';
3.MySQL does not support multi shaped partitioning.
Vertical Partitioning storage engine supports multi shaped partitioning.
Multi shaped partitioning is dividing a table by multiple partitioning rules.
You can use tables that dividing into detail information table and index tables like one table by this feature. And you can shard each table by using Spider storage engine.
Before partitioning
create table employees (
id int primary key,
fname varchar(30),
lname varchar(30),
hired date not null default '1970-01-01',
separated date not null default '9999-12-31',
job_code int,
store_id int
)engine=InnoDB;
After partitioning
create table emp_pk (
id int not null,
separated date not null default '9999-12-31',
unique idx_a(id),
key idx_b(id, separated)
)engine=InnoDB
partition by hash(id)
partitions 4;
create table emp_detail (
id int not null,
fname varchar(30),
lname varchar(30),
hired date not null default '1970-01-01',
separated date not null default '9999-12-31',
job_code int,
store_id int,
key idx_a(id, separated)
)engine=InnoDB
partition by range ( year(separated) ) (
partition p0 values less than (1991),
partition p1 values less than (1996),
partition p2 values less than (2001),
partition p3 values less than maxvalue
);
create table employees (
id int not null,
fname varchar(30),
lname varchar(30),
hired date not null default '1970-01-01',
separated date not null default '9999-12-31',
job_code int,
store_id int,
primary key(id, separated)
)engine=VP
comment 'table_name_list "emp_pk emp_detail", pk_correspond_mode "1"';
Note: In this case, parent table's primary key has columns that using both child table partitioning rules for using partition pruning.
(In the future, I will improve this to become "parent table's primary key has only real primary key (id) and child tables can use partition pruning")
4.Vertical Partitioning can add "parallel columns searching" to other storage engines with supporting "parallel searching" like spider storage engine.(Now developping)
5.Vertical Partitioning storage engine can use for partitioning wrapper because Vertical Partitioning storage engine supports MySQL's table partitioning.
You can use each partition like tables by this feature.(Don't use this feature because now developping)
Example
create table tbl_b(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a)
)engine=InnoDB;
create table tbl_c(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a)
)engine=InnoDB;
create table tbl_d(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a)
)engine=InnoDB;
create table tbl_a(
col_a int not null,
col_b varchar(20),
col_c int not null,
primary key(col_a)
)engine=VP
partition by hash(col_a)
(
partition pt1 comment='table_name_list "tbl_b"',
partition pt2 comment='table_name_list "tbl_c"',
partition pt3 comment='table_name_list "tbl_d"'
);
In this time, Vertical Partitioning storage engine is alpha version and some features can not use yet but I will develop fast for adding features.
Enjoy!
Subscribe to:
Posts (Atom)
