Wild Growth

[MySQL][Spider][VP]Spider-2.27 VP-0.16 released

2011-10-19T22:35:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.27(beta) and Vertical Partitioning storage engine version 0.16(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

Please use the following for downloading binary file.
http://spiderformysql.com/download_spider.html

The main changes in this version are following.
Q4M is bundled for Linux 64bit.
Bundled Q4M supports replication, but please use it carefully.

Spider
- Support R-Tree index.
- Support direct updating for SQL access.
- Support handlersocket increment and decrement.
- Change table parameter from "net_timeout" to "connect_timeout", "net_read_timeout" and "net_write_timeout".
- Change server parameter from "spider_net_timeout" to "spider_connect_timeout", "spider_net_read_timeout" and "spider_net_write_timeout".
- Add UDF paramter "access_mode". "spider_direct_sql" support handlersocket access.
- Add server parameter "spider_hs_ping_interval", "spider_error_read_mode" and "spider_error_write_mode".
- Add table parameter "hs_write_to_read", "error_read_mode" and "error_write_mode".
- Performance improvement for "COUNT(*)", "MAX" and "MIN" without clause.
Note
- Change parameter from "net_timeout" and "spider_net_timeout" to "connect_timeout", "net_read_timeout", "net_write_timeout", "spider_connect_timeout", "spider_net_read_timeout" and "spider_net_write_timeout" from this version.
- From this version, you can't compile with static link option for Spider.

Vertical Partitioning
- Support R-Tree index.
- Support direct updating.
- Support handlersocket increment and decrement.
Note
- From this version, you can't compile with static link option for VP.

handlersocket
- Add server parameter "handlersocket_general_log".
Logging handlersocket request into general log with "general_log=on". This parameter is global parameter and you can change it from SQL interface.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider][VP]Spider-2.26 VP-0.15 released

2011-07-22T01:46:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.26(beta) and Vertical Partitioning storage engine version 0.15(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

Recently, I can't upload the binary file to Launchpad, so I put the binary file at the following. Please use the following for downloading binary file.
http://spiderformysql.com/download_spider.html

The main changes in this version are following.
Spider
- Add table parameter "active_link_count", "read_only_mode", "first_read" and "second_read".
- Add server parameter "spider_read_only_mode", "spider_first_read", "spider_second_read", "spider_remote_time_zone" and "spider_sync_time_zone".
  "active_link_count" is a parameter for using Active-Standby database for data node. Normally, set "active_link_count" to 1 for using Active-Standby data node. Currently, this feature supports only Active-Standby that can switch immediately. For this changing, Spider's management table was changed from previous version, please execute "install_spider.sql" for upgrading.
- Reducing access to data nodes.
  By this change, even if a shard is crushed, you can continue accessing to other shard.

Vertical Partitioning
  This release is bug fix release.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider][VP]Spider-2.25 VP-0.14 released

2011-05-12T02:07:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.25(beta) and Vertical Partitioning storage engine version 0.14(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
Spider
- Add table parameter "skip_default_condition" and "direct_order_limit".
- Add server parameter "spider_skip_default_condition" and "spider_direct_order_limit".
  "direct_order_limit" improves some SQL with "order by" and "limit" performance.
- Add UDF "spider_flush_table_mon_cache".
  "spider_flush_table_mon_cache" is used for reflecting changing of table monitoring server information.
- Wild card for monitoring server information is now available.
  Wild card like used for user information can be used for monitoring server information(db_name、table_name、link_id), and you can set a monitoring server information for multiple tables. For this changing, Spider's management table was changed from previous version, please execute "install_spider.sql" for upgrading.

Vertical Partitioning
  This release is bug fix release.

- Table partitioning supports fulltext searching.

- Added following parameters to bundled handlersocket.
  - handlersocket_bulk_insert
    Handlersocket supports bulk inserting.
    This parameter is bulk size of inserting.
    There is a feature that insert from multiple users becomes bulk insert, and there is the speed-up of insert of the application to which insert of a single line is done with the MultiThreading.
     0:Disable bulk inserting.
    1-:Execute bulk inserting when the stocked records increase more than the values of the parameter.
  - handlersocket_bulk_insert_timeout
    The maximum standby time of the state that it doesn't reach the number of records of handlersocket_bulk_insert to execute bulk insert. (second)

- Table partitioning supports "handlersocket_direct_update_mode" with Spider. (It only works with unlimited updating and deleting with handlersocket_unlimited_boundary.)

Please see "99_change_logs.txt" in the download documents for checking other changes.

Thanks to Kazuhide, Takahiro, Yuki, Hiroshi and Sangjin for bug reporting.
Thanks to Mikiya for your advice.

Enjoy!

[MySQL][Spider][VP][Other]Spider-2.24 VP-0.13 released

2011-02-13T03:34:00.004+09:00

I'm pleased to announce the release of Spider storage engine version 2.24(beta) and Vertical Partitioning storage engine version 0.13(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
Spider
- MySQL 5.5 support.
- handlersocket support.
  You can use handlersocket through Spider table that called by handlersocket. It means handlersocket can access to sharded databases. Currently, handlersocket support only Linux.
  Set "spider_use_hs_read=1" and "spider_use_hs_write=1" for activating this feature.
  Added following parameters to bundled handlersocket.
  #1 handlersocket_support_merge_table
    Please set this parameter to 1, if you want to use merge_myisam and vp tables through handlersocket.
  #2 handlersocket_direct_update_mode
    Improvement performance for a lot of rows updating. Currentry, this parameter works only for Spider table(except partitioned Spider table). I will add a direct update feature for partitioned Spider table at next release.
    There are following modes.
    0:Normal
    1:direct update mode 1
      This mode can write binary logs on local server at direct updating.
    2:direct update mode 2
      This mode cannot write binary logs on local server at direct updating, but this mode is the fastest. (This mode can write binary logs on remote server)
  #3 handlersocket_unlimited_boundary
    If limit value is more than this parameter, limit value means unlimited.
- Support fulltext search.
  Except partitioned Spider table. I will add a fulltext search feature for partitioned Spider table at next release.
- It's available to set monitoring_node for per table, per link_id and per pertition to mysql.spider_link_mon_servers.
- Add table parameter "hs_read_port", "hs_write_port", "use_handler", "use_hs_read", "use_hs_write", "hs_read_socket" and "hs_write_socket".
- Add server parameter "spider_use_handler", "spider_use_hs_read", "spider_use_hs_write", "spider_hs_r_conn_recycle_mode", "spider_hs_r_conn_recycle_strict", "spider_hs_w_conn_recycle_mode" and "spider_hs_w_conn_recycle_strict".
- Add value of 3 to "auto_increment_mode"(table parameter) and "spider_auto_increment_mode"(server parameter).

Vertical Partitioning
- MySQL 5.5 support.
- Support full text search.
  Except partitioned VP table. I will add a fulltext search feature for partitioned VP table at next release.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Thanks to Gen, Yoshihiko, Satoko and Makoto for testing and bug reporting for online schema changing.
Thanks to Takahiro and Kazuhide for testing and bug reporting for Spider's HA feature and VP.
Thanks to Kamipo and Laurent for bug report.

Enjoy!

[MySQL][Spider][VP]Spider-2.23 VP-0.12 released

2010-10-17T02:02:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.23(beta) and Vertical Partitioning storage engine version 0.12(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
Spider
- Add server parameter "spider_udf_ct_bulk_insert_interval" and "spider_udf_ct_bulk_insert_rows".

Vertical Partitioning
This release is bug fix release.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Thanks to Gen, Yoshihiko, Satoko and Makoto for testing for clustering and online schema changing.
Thanks to Frederic for bug report.

Enjoy!

[MySQL][Spider][VP][Other]Spider-2.22 VP-0.11 BKA-for-ha_partition-0.3 released

2010-08-31T01:35:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.22(beta), Vertical Partitioning storage engine version 0.11(beta) and BKA-for-ha_partition version 0.3.
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
BKA-for-ha_partition is a patch file of supporting "Batched Key Access" for table partitioning feature.
http://launchpad.net/partitionmrrformysql

The main changes in this version are following.
Spider
- MS Windows support.
- Add UDF "spider_copy_tables".
Spider HA feature is now available!

Vertical Partitioning
- MS Windows support.

Pathced MySQL source code and compiled MySQL binary are available from this release.

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Toru for working for supporting Windows.

Enjoy!

[MySQL][Spider]Spider-2.21 released

2010-06-27T02:19:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.21(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "bka_mode" and "bka_engine".
- Add server parameter "spider_bka_mode" and "spider_bka_engine".
This parameters are for improvement performance for using BKA and using a lot of conditions with IN clause and so on.

"mysql-5.1.44.slave-trx-retry.diff" is included in the source tar ball from this release.
mysql-5.1.44.slave-trx-retry.diff adds a option "slave_transaction_retry_errors". This parameter is used like "slave_transaction_retry_errors=1158,1159,2013,12701". Slave sql thread retries the transaction if slave sql thread encounters the errors at this parameter. Number of retries are at "slave_transaction_retries".

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Mikiya for your advice.

Enjoy!

[MySQL][Spider][VP][Other]Spider-2.20 VP-0.10 BKA-for-ha_partition-0.2 released

2010-05-18T00:38:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.20(beta), Vertical Partitioning storage engine version 0.10(beta) and BKA-for-ha_partition version 0.2.
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
BKA-for-ha_partition is a patch file of supporting "Batched Key Access" for table partitioning feature.
http://launchpad.net/partitionmrrformysql

The main changes in this version are following.
Spider
- "spider_multi_split_read" supports dividing condition count.
From this release, there is "install_spider.sql" file for installing and updating Spider. Please use this file.

Vertical Partitioning
- Support "Batched Key Access".
This feature is now available on MariaDB 5.3 with "join_cache_level=6 and more".

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider-2.19 released

2010-05-04T01:36:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.19(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Support "Batched Key Access".
This feature is now available on MariaDB 5.3 with "join_cache_level=6 and more".

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider-2.18 released

2010-04-29T16:28:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.18(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
This release is bug fix release.

Please see "99_change_logs.txt" in the download documents for more detail.

Enjoy!

[MySQL][Spider]Spider-2.17 released

2010-03-27T05:54:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.17(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "semi_split_read_limit".
- Add server parameter "spider_semi_split_read_limit".
This parameters are for searching performance improvement.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider][VP]Spider-2.16 Vartical Partitioning-0.9 released

2010-03-15T04:04:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.16(beta) and Vertical Partitioning storage engine version 0.9(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
(This release for Vertical Partitioning is a bug fix release)
- Add table parameter "semi_split_read".
- Add server parameter "spider_semi_split_read".
This parameters are for searching performance improvement.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider-2.15 released

2010-03-08T03:30:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.15(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
This release is bug fix release.

Please see "99_change_logs.txt" in the download documents for more detail.

Enjoy!

[MySQL][Spider]Spider-2.14 released

2010-03-02T04:00:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.14(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "bulk_update_mode" and "bulk_update_size".
- Add server parameter "spider_bulk_update_mode" and "spider_bulk_update_size".
This parameters are for performance improvement.
Please examine to use "spider_direct_dup_insert", if "REPLACE" and "INSERT IGNORE" are slow.
Please examine to use "spider_bulk_update_mode", if "UPDATE" and "DELETE" are slow.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider-2.13 released

2010-02-21T15:41:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.13(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
This release is bug fix release.

Please see "99_change_logs.txt" in the download documents for more detail.
Thanks to Mitsuhiro for bug report.

Enjoy!

[MySQL][VP]VP-0.8 released

2010-02-18T04:56:00.001+09:00

I'm pleased to announce the release of Vertical Partitioning storage engine version 0.8(alpha).
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
This release is bug fix release.

Please see "99_change_logs.txt" in the download documents for more detail.

Enjoy!

[MySQL][VP][Other]Start creating Engine Independent Test & VP-0.7 released

2010-01-31T18:28:00.002+09:00

I started creating "Engine Independent Test".
http://launchpad.net/engineindependenttestformysql

This test aims to be testable with all storage engines.
There is following wiki but test was not created yet.
http://forge.mysql.com/wiki/EngineIndependentTestSuite
I think this test is need for Storage Engine Developers(including me).
Additionally, I already released 2 Storage Engines so I felt needing this test strongly.
So I started creating "Engine Independent Test".

Please tell me without hesitate if you have a demand.

I'm pleased to announce the release of Vertical Partitioning storage engine version 0.7(alpha).
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
MariaDB support.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Please enjoy and go forward!

[MySQL][Spider]Spider-2.12 released

2010-01-22T03:22:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.12(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
MariaDB support.
- Add server parameter "spider_connect_mutex", "spider_connect_retry_count" and "spider_connect_retry_interval".
This parameters are used for avoiding failed connected processing caused by intense connected processing. (Especially for using "spider_conn_recycle_mode=0")

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Mohammad and Jay for bug report.

Enjoy!

[MySQL][Spider]Spider-2.11 released

2009-12-28T01:48:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.11(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add table link background monitoring feature.
- Add table parameter "monitoring_bg_interval" and "monitoring_bg_kind".

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider-2.10 released

2009-12-15T02:14:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.10(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "ssl_ca", "ssl_capath", "ssl_cert", "ssl_cipher", "ssl_key", "ssl_verify_server_cert", "default_file" and "default_group".
- Add UDF parameter "ssl_ca", "ssl_capath", "ssl_cert", "ssl_cipher", "ssl_key", "ssl_verify_server_cert", "default_file" and "default_group".
You can use ssl for Spider's communication to remote servers.
Spider can read config file(my.cnf) at connecting to remote servers.

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Shinya for bug report.

Enjoy!

[MySQL][Spider]Spider-2.9 released

2009-11-29T02:02:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.9(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add UDFs "spider_ping_table".
- Add table parameter "monitoring_kind", "monitoring_limit" and "monitoring_server_id".
- Add server parameter "spider_udf_table_mon_mutex_count".
Add Spider's link fault monitor at this release.

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to shinichiro and merikonjatta for bug report.

Enjoy!

[MySQL][Spider]Spider-2.8 released

2009-11-04T07:13:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.8(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "link_status".
You can change link_status using "alter table" statement.
Spider's link fault management is table level.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider-2.7 released

2009-10-30T03:49:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 2.7(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Spider table can load balance using multiple links at searching without locking. Load balance rule is "((server_id + thread_id) % number_of_links)".
Please write multiple connection informations splitting space characters at "host", "user", "table", "server", "socket", "wrapper", "database" and "password" table parameters.
Example: "host 'h1 h2 h3'"

I will continue to develop Spider's functions for Active-Active type clustering.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider-2.6 released

2009-10-19T01:11:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.6(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add server parameter "spider_remote_access_charset", "spider_remote_autocommit", "spider_remote_sql_log_off" and "spider_remote_trx_isolation".
These parameters are setted by Spider at connecting remote server. But it does not need to be setted if you know it previously and no need to change it. In this case, you can set these parameters for improvement connecting performance.

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Giuseppe for bug report.

Enjoy!

[MySQL][VP]Vartical Partitioning-0.6 released

2009-10-14T01:53:00.000+09:00

I'm pleased to announce the release of Vertical Partitioning storage engine version 0.6(alpha).
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
- Add UDF "vp_copy_tables".
  This UDF is used for syncronizing Vertical Partitioning's child table data from old child tables to new added child tables.

- Add table parameter "choose_ignore_table_list", "choose_ignore_table_list_for_lock" and "zero_record_update_mode".
  This parameters are used for avoiding to use new added child tables at searching operation and separating child tables for searching with lock and without lock.

- Add server parameter "vp_udf_ct_bulk_insert_interval" and "vp_udf_ct_bulk_insert_rows".
  This parameters are used for changing the load of updating immediately from vp_copy_tables.

Example
-------------------------------------------------------------------------------
Initial tables:
    create table tbl_a(
      col_a int not null,
      col_b varchar(20),
      col_c int not null,
      primary key(col_a),
      key idx1(col_c, col_a)
    )engine=VP comment='tnl "tbl_b tbl_c"';

Add a table:
    alter table tbl_a comment='tnl "tbl_b tbl_c tbl_d", cit "3", cil "3", zru "1"';

Copy table data from tbl_c to tbl_d:
    select vp_copy_tables("tbl_a", "tbl_c", "tbl_d");

Finalize:
    alter table tbl_a comment='tnl "tbl_b tbl_c tbl_d"';
-------------------------------------------------------------------------------

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider][VP]Spider-2.5 Vartical Partitioning-0.5 released

2009-10-05T06:32:00.002+09:00

I'm pleased to announce the release of Spider storage engine version 2.5(beta) and Vertical Partitioning storage engine version 0.5(alpha).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
- The standalone compilation with Vertical Partitioning storage engine is now available.

- Vertical Partitioning can be used for multi-step partitioning.
  Example:
    create table a(a int primary key)engine=myisam;
    create table b(a int primary key)engine=vp comment 'tnl "a"';
    create table c(a int primary key)engine=vp comment 'tnl "b"';

- Add table parameter "bgs_mode", "bgi_mode" and "bgu_mode".
- Add server parameter "vp_bgs_mode", "vp_bgi_mode" and "vp_bgu_mode".
  Parallel search and update from Vertical Partitioning to Spider are now available by using this parameters.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider][VP]Spider-2.4 Vartical Partitioning-0.4 released

2009-09-27T18:09:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.4(beta) and Vertical Partitioning storage engine version 0.4(alpha).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
- "Engine-condition-pushdown" is available on Vertical Partitioning with Spider.

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Gennady for bug report.

Enjoy!

[MySQL][VP]Vertical Partitioning storage engine 0.3 released

2009-09-21T12:45:00.001+09:00

I'm pleased to announce the release of Vertical Partitioning storage engine version 0.3.
http://launchpad.net/vpformysql

The main changes in this version are following.
- Add table parameter "choose_table_mode_for_lock".
- Add server parameter "vp_choose_table_mode_for_lock".
These parameters are used for avoiding deadlock by fixing lock order to table list order.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 2.3 released

2009-09-12T19:00:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.3(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add UDF parameter "connection_channel".
This parameter is used for increasing background parallelism to same remote server by using multiple connections.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 2.2 released

2009-09-06T08:30:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.2(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add UDF parameter "iop000".
You can use "ignore" and "replace" at inserting records to temporary table list's tables using this parameter.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 2.1 released

2009-08-29T13:06:00.005+09:00

I'm pleased to announce the release of Spider storage engine version 2.1(beta).
http://spiderformysql.com/

The main changes in this version are following.
- Add UDFs "spider_direct_sql" and "spider_bg_direct_sql".
  These UDFs execute hand writing SQLs on remote server by using Spider's table link and store result sets in temporary tables.
  Main usages are following.
  1.Maintenance remote servers.
    (Create table on remote servers for Spider's link tables, etc.)
  2.Use every resources (excluding table) on remote servers.
  3.Mass update of the bulk data. (Update all rows in tables, etc.
    There are next advantages. It is possible to update it in parallel
    with multiple remote servers. The amount of the data transfer of
    a local server and remote servers can be reduced)
  4.Aggregate by 2-steps. (For DWH etc)
    First step: Aggregate on remote servers and collect result
                    to local server.
    Second step: Aggregate on local server.
    (There are next advantages. It is possible to aggregate it in
    parallel with multiple remote servers. The amount of the data
    transfer of a local server and remote servers can be reduced)

Example of 2-steps aggregating
(Count the number of rows in 7 shards tables from local server)
-------------------------------------------------------------------------------
Prepare for aggregating:
mysql> create table target_hosts (host char(17))engine=myisam;
Query OK, 0 rows affected (0.01 sec)

mysql> insert into target_hosts (host) values ('192.168.22.13'),('192.168.22.14'),('192.168.22.15'),('192.168.22.16'),('192.168.22.51'),('192.168.22.52'),('192.168.22.53');
Query OK, 7 rows affected (0.00 sec)
Records: 7  Duplicates: 0  Warnings: 0

mysql> create temporary table a (a int)engine=myisam;
Query OK, 0 rows affected (0.00 sec)

Execute aggregating:
mysql> select sum(a.a) from a, (select sum(spider_direct_sql('select count(*) from data01', 'a', concat('srv "s", host "', host, '"'))) cnt from target_hosts) c;
+-----------+
| sum(a.a)  |
+-----------+
| 117641811 |
+-----------+
1 row in set (40.86 sec)

mysql> truncate table a;
Query OK, 0 rows affected (0.00 sec)

mysql> select sum(a.a) from a, (select spider_bg_direct_sql('select count(*) from data01', 'a', concat('srv "s", host "', host, '"')) cnt from target_hosts) c;
+-----------+
| sum(a.a)  |
+-----------+
| 117641811 |
+-----------+
1 row in set (5.91 sec)
-------------------------------------------------------------------------------

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 2.0 released

2009-08-22T18:38:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 2.0(beta).
http://spiderformysql.com/
From this time, version 1.x is stable releases, version 2.x is development releases for new features.

The main changes in this version are following.
- Add table parameter "semi_table_lock_connection".
- Add server parameter "spider_semi_table_lock_connection".
Spider has "spider_semi_trx", "spider_semi_trx_isolation", (for transactional tables like InnoDB) "spider_semi_table_lock" and "semi_table_lock" (for non transactional tables like MyISAM) options for consistent reading at remote servers during executing 1 SQL at local server. But you must use different connections for transactional tables and non transactional tables because "semi_table_lock" causes implicit transaction commit.
"semi_table_lock_connection" and "spider_semi_table_lock_connection" can use different connection for transactional tables and non transactional tables by checking that "semi_table_lock" is enabled.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 1.0 released

2009-08-18T19:23:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 1.0.
Development status is GA from this version.
http://spiderformysql.com/

From this version, you can download shared library for Linux x86_64.
Thanks to Kazuho for information.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 0.18 released

2009-08-15T11:43:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 0.18.(RC)
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "use_pushdown_udf".
- Add server parameter "spider_use_pushdown_udf".
"use_pushdown_udf" and "spider_use_pushdown_udf" are used for sending UDF string to remote server if engine_condition_pushdown is used.

From this version, you can use '\r' and '\n' like a space between table parameter name and table parameter value.
Thanks to Akihiro for report.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 0.17 released

2009-08-09T15:50:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 0.17.(RC)
Development status is RC from this version.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "use_table_charset".
- Add server parameter "spider_use_table_charset" and "spider_local_lock_table".
"use_table_charset" and "spider_use_table_charset" are used for choosing the communication charset (UTF8 or table charset) at connections from Spider to remote server. Basically, you don't need changing this option but you can change this option for performance improvement if the SQL has very longer table and column name string than column data string.
"spider_local_lock_table" is used for suppressing sending "lock table" statement to remote server for locking table at local server only. This option is used with non-blocking resharding.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 0.16 released

2009-08-01T14:18:00.000+09:00

I'm pleased to announce the release of Spider storage engine version 0.16.(beta)
http://spiderformysql.com/

The main changes in this version are following.
- Add server parameter "spider_same_server_link" and "spider_table_init_error_interval".
"spider_same_server_link" is used for creating table link to same MySQL server's table.
"spider_table_init_error_interval" is used for protection from connection consumption of eternal loop of table link. Please increase this parameter using "set global" command if eternal loop of table link causes connection consumption.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 0.15 released

2009-07-25T20:02:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 0.15.(beta)
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "auto_increment_mode".
- Add server parameter "spider_auto_increment_mode".
  Spider supports 3 modes.
  0:Use a count that Spider get from the remote server with exclusive lock for auto increment value.
  1:Use a count that Spider has internal for auto increment value.
  2:Auto increment value is given by the remote server.

- Add server parameter "spider_ping_interval_at_trx_start".
  Performance improvement by frequency decrease of checking broken connections.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 0.14 released

2009-07-21T01:41:00.001+09:00

I'm pleased to announce the release of Spider storage engine version 0.14.(beta)
http://spiderformysql.com/

The main changes in this version are following.
- The standalone compilation with Spider storage engine is now available.
Please see "03_install_spider_storage_engine.txt" for more detail.
Tatsuhiko and Tetsuro,
Thank you for your advice!

Please see "99_change_logs.txt" in the download documents for checking other changes.

Spider storage engine was introduced at following with the usage.

The Data Charmer(English)
http://datacharmer.blogspot.com/2009/04/test-driving-spider-storage-engine.html
http://datacharmer.blogspot.com/2009/07/sharding-for-masses-spider-storage.html
　Giuseppe,
　　Thank you for introducing Spider.

Unoh Labs(Japanese)
http://labs.unoh.net/2009/07/mysqlspider.html
Satoru,
　　Thank you for introducing Spider.

Enjoy!

[MySQL][Spider]Spider storage engine 0.13 released

2009-07-08T04:36:00.004+09:00

I'm pleased to announce the release of Spider storage engine version 0.13.(beta)
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "crd_bg_mode" and "sts_bg_mode".
- Add server parameter "spider_crd_bg_mode" and "spider_sts_bg_mode".
  MySQL needs table's analysis informations when select statement is executed. Spider storage engine gets this informations from remote table but this action often generates over-heads.
  This options remove generated over-heads by getting analysis informations at background but you must set this option to 0 if the table's analysis informations change frequently.
  * There are other options for the over-heads reduction of the analysis informations acquisition. I will introduce this at another time.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Other]Condition Pushdown for ha_partition 0.1 released

2009-07-03T19:47:00.002+09:00

I'm pleased to announce the release of Condition Pushdown for ha_partition version 0.1.
http://launchpad.net/partitionconditionpushdownformysql

This release is a patch file for MySQL's table partitioning feature. (ndb is excluded)
This patch file works to add methods("cond_push" and "cond_pop") to ha_partition. As a result, every storage engine's table that use table partitioning can get condition-pushdown through ha_partition.

There was no problem because only ndb supported engine-condition-pushdown up to now.
This patch was needed because the Spider storage engine supported engine-condition-pushdown recently.

Enjoy!

[MySQL][VP]Vertical Partitioning storage engine 0.2 released

2009-07-02T02:17:00.002+09:00

I'm pleased to announce the release of Vertical Partitioning storage engine version 0.2.
http://launchpad.net/vpformysql

The main changes in this version are following.

- Support MySQL's table partitioning.
  MySQL's table partitioning that is only introduced last release is now supported.
  note: After this version, you need to apply a patch for installing Vertical Partitioning storage engine.

- After this version, the primary key part of InnoDB's secondary index is effectively used.
  You can use the primary key columns with InnoDB's secondary index by using following statements.
    create table tbl_b(
      col_a int not null,
      col_b varchar(20),
      col_c int not null,
      primary key(col_a),
      key idx1(col_c)
    )engine=InnoDB;
    create table tbl_c(
      col_a int not null,
      col_b varchar(20),
      col_c int not null,
      primary key(col_a),
      key idx1(col_c)
    )engine=InnoDB;
    create table tbl_a(
      col_a int not null,
      col_b varchar(20),
      col_c int not null,
      primary key(col_a),
      key idx1(col_c, col_a)
    )engine=VP comment='table_name_list "tbl_b tbl_c"';
  note: Vertical Partitioning storage engine can use index with primary key together with index without primary key, because Vertical Partitioning storage engine can use different storage engines for child tables at same table. In this case, the purpose of writing the column of primary key to tbl_a's idx1 is to absorb the difference by this coexistence.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][Spider]Spider storage engine 0.12 released

2009-06-30T00:23:00.001+09:00

I released Spider storage engine version 0.12.(beta)
http://spiderformysql.com/

Spider storage engine can be used for database sharding by using table partitioning and table link.
Spider storage engine can synchronize an update for remote MySQL servers by XA transaction at Spider-internal.
Introduction document:
http://www.mysqlconf.com/mysql2009/public/schedule/detail/6837

The main changes in this version are following.
("Table parameter" that is used following is parameters for each table. Please see "04_table_create.txt" and "06_table_parameters.txt" in the download documents for more details.)
- Add table parameter "table_count_mode".
  If storage engine can return correct full record count from table status like MyISAM, you should better to set this option to 1 for getting full record count from table status.

- Add table parameter "select_column_mode", add server parameter "spider_select_column_mode".
  After this time, you can choose method of deciding used columns at select clause from "easy deciding(existing method)" and "detailed deciding(new method)".
  You can reduce the wire traffic to remote servers by using "detailed deciding".

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

[MySQL][VP]Vertical Partitioning storage engine 0.1 released

2009-06-25T23:14:00.003+09:00

I'm pleased to announce a release of a new storage engine "Vertical Partitioning storage engine".
http://launchpad.net/vpformysql
You can use relation tables of one to one like one table by using Vertical Partitioning storage engine.
It is near by a view of joined relation tables of one to one but different point is following.
- It can insert.
- It can do partition pruning.(Not use partitions that are no necessity)

Vertical Partitioning storage engine can use following purposes.
(Vertical Partitioning table is called "parent table", integrated tables are called "child table")

1.A MySQL's InnoDB table that has the big record size is very slow if a select statement needs table scanning with a lot of records.
  Vertical Partitioning is good for resolving this performance problem.

  For example, you can solve a performance issue for "select col_d from tbl_a" by using following partitions.
  Before partitioning
    create table tbl_a(
      col_a int primary key,
      col_b int,
      col_c int,
      col_d int,
      col_e varchar(255),
      col_f varchar(255),
      col_g varchar(255),
      col_h varchar(255),
      col_i varchar(255),
      col_j text,
      key idx_a(col_b, col_c),
      key idx_b(col_c, col_j(100))
    )engine=InnoDB;

  After partitioning
    create table tbl_b(
      col_a int primary key,
      col_b int,
      col_c int,
      col_d int,
      key idx_a(col_b, col_c)
    )engine=InnoDB;
    create table tbl_c(
      col_a int primary key,
      col_c int,
      col_e varchar(255),
      col_f varchar(255),
      col_g varchar(255),
      col_h varchar(255),
      col_i varchar(255),
      col_j text,
      key idx_b(col_c, col_j(100))
    )engine=InnoDB;
    create table tbl_a(
      col_a int primary key,
      col_b int,
      col_c int,
      col_d int,
      col_e varchar(255),
      col_f varchar(255),
      col_g varchar(255),
      col_h varchar(255),
      col_i varchar(255),
      col_j text,
      key idx_a(col_b, col_c),
      key idx_b(col_c, col_j(100))
    )engine=VP
    comment 'table_name_list "tbl_b tbl_c"';
  Note: Vertical Partitioning storage engine can use same column in different child tables. You can use flexible partition design by this feature but you should note that a lot of same columns causes updating performance issues.

2.MySQL Cluster (ndb) is not good for using text columns.
  Vertical Partitioning can use different storage engines for one table.
  As a result, MySQL Cluster can use only for strong point columns.
  Example of mixed storage engines.
    create table tbl_b(
      col_a int primary key,
      col_b int,
      col_c int,
      key idx_a(col_b, col_c)
    )engine=ndb ......;
    create table tbl_c(
      col_a int primary key,
      col_c int,
      col_d varchar(255),
      col_e text,
      key idx_b(col_c, col_e(100))
    )engine=Spider ......;
    create table tbl_a(
      col_a int primary key,
      col_b int,
      col_c int,
      col_d varchar(255),
      col_e text,
      key idx_a(col_b, col_c),
      key idx_b(col_c, col_e(100))
    )engine=VP
    comment 'table_name_list "tbl_b tbl_c"';

3.MySQL does not support multi shaped partitioning.
  Vertical Partitioning storage engine supports multi shaped partitioning.
  Multi shaped partitioning is dividing a table by multiple partitioning rules.
  You can use tables that dividing into detail information table and index tables like one table by this feature. And you can shard each table by using Spider storage engine.

  Before partitioning
    create table employees (
      id int primary key,
      fname varchar(30),
      lname varchar(30),
      hired date not null default '1970-01-01',
      separated date not null default '9999-12-31',
      job_code int,
      store_id int
    )engine=InnoDB;

  After partitioning
    create table emp_pk (
      id int not null,
      separated date not null default '9999-12-31',
      unique idx_a(id),
      key idx_b(id, separated)
    )engine=InnoDB
    partition by hash(id)
    partitions 4;
    create table emp_detail (
      id int not null,
      fname varchar(30),
      lname varchar(30),
      hired date not null default '1970-01-01',
      separated date not null default '9999-12-31',
      job_code int,
      store_id int,
      key idx_a(id, separated)
    )engine=InnoDB
    partition by range ( year(separated) ) (
      partition p0 values less than (1991),
      partition p1 values less than (1996),
      partition p2 values less than (2001),
      partition p3 values less than maxvalue
    );
    create table employees (
      id int not null,
      fname varchar(30),
      lname varchar(30),
      hired date not null default '1970-01-01',
      separated date not null default '9999-12-31',
      job_code int,
      store_id int,
      primary key(id, separated)
    )engine=VP
    comment 'table_name_list "emp_pk emp_detail", pk_correspond_mode "1"';

  Note: In this case, parent table's primary key has columns that using both child table partitioning rules for using partition pruning.
  (In the future, I will improve this to become "parent table's primary key has only real primary key (id) and child tables can use partition pruning")

4.Vertical Partitioning can add "parallel columns searching" to other storage engines with supporting "parallel searching" like spider storage engine.(Now developping)

5.Vertical Partitioning storage engine can use for partitioning wrapper because Vertical Partitioning storage engine supports MySQL's table partitioning.
  You can use each partition like tables by this feature.(Don't use this feature because now developping)

  Example
    create table tbl_b(
      col_a int not null,
      col_b varchar(20),
      col_c int not null,
      primary key(col_a)
    )engine=InnoDB;
    create table tbl_c(
      col_a int not null,
      col_b varchar(20),
      col_c int not null,
      primary key(col_a)
    )engine=InnoDB;
    create table tbl_d(
      col_a int not null,
      col_b varchar(20),
      col_c int not null,
      primary key(col_a)
    )engine=InnoDB;
    create table tbl_a(
      col_a int not null,
      col_b varchar(20),
      col_c int not null,
      primary key(col_a)
    )engine=VP
    partition by hash(col_a)
    (
      partition pt1 comment='table_name_list "tbl_b"',
      partition pt2 comment='table_name_list "tbl_c"',
      partition pt3 comment='table_name_list "tbl_d"'
    );

In this time, Vertical Partitioning storage engine is alpha version and some features can not use yet but I will develop fast for adding features.
Enjoy!