MyCat垂直分库

一.什么是垂直分库

将一类功能的表从一个实例切分到另一个实例，横向扩展实例，增加写负载

目标：将1个实例的4类表拆分多4个实例中

二.垂直切分步骤

2.1收集分析业务模块间的关系，能分几个库

2.2全量复制需要的数据到其他实例

root@localhost 21:51:  [imooc_db]> show tables;
+-----------------------+
| Tables_in_imooc_db    |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
| order_cart            |
| order_customer_addr   |
| order_detail          |
| order_master          |
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
| region_info           |
| serial                |
| shipping_info         |
| warehouse_info        |
| warehouse_proudct     |
+-----------------------+
21 rows in set (0.01 sec)

root@localhost 21:51:  [imooc_db]> system ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:a3:fe:0a brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.141/24 brd 172.16.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.10.139/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea3:fe0a/64 scope link 
       valid_lft forever preferred_lft forever

将上面一个实例中的一个库拆成4个库，分别是imooc_db/order_db/product_db/customer_db

• 1.备份原数据库并记录事务点

# master导出单个库的备份
SET GLOBAL log_timestamps = SYSTEM;（
/usr/local/mysql/bin/mysqldump -uroot -p123456 -hlocalhost -P3306 --socket=/usr/local/mysql/mysql.sock --single-transaction --master-data=2  imooc_db --triggers --routines --events > /tmp/all.sql

• 2.原数据库中建立复制用户

grant replication slave on *.* to 'repl'@'%' identified by '123456';

• 3.新实例上恢复备份数据库，并更改数据库名

mysql -uroot -p123456 -e"create database order_db";
mysql -uroot -p123456 order_db < /tmp/all.sql
mysql -uroot -p123456 -e"create database product_db";
mysql -uroot -p123456 product_db < /tmp/all.sql
mysql -uroot -p123456 -e"create database customer_db";
mysql -uroot -p123456 customer_db < /tmp/all.sql
######################################
ERROR 1840 (HY000) at line 24: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty
Query OK, 0 rows affected (0.01 sec)

root@localhost 23:55:  [(none)]> show master status;
+---------------+----------+--------------+------------------+-------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+---------------+----------+--------------+------------------+-------------------+
| binlog.000001 |      154 |              |                  |                   |
+---------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

root@localhost 23:55:  [(none)]> show master status;
+---------------+----------+--------------+------------------+-------------------------------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                         |
+---------------+----------+--------------+------------------+-------------------------------------------+
| binlog.000001 |      154 |              |                  | ad083f85-9264-11e8-9e91-005056a3fe0a:1-85 |
+---------------+----------+--------------+------------------+-------------------------------------------+
1 row in set (0.00 sec)

• 4.新实例上配置复制链路

help change master to
CHANGE MASTER TO
  MASTER_HOST='172.16.10.141',
  MASTER_USER='repl',
  MASTER_PASSWORD='123456',
  MASTER_PORT=3306,
  master_auto_position=1;

help change replication filter
CHANGE REPLICATION FILTER
  REPLICATE_REWRITE_DB = ((imooc_db, order_db));
CHANGE REPLICATION FILTER
  REPLICATE_REWRITE_DB = ((imooc_db, product_db));
CHANGE REPLICATION FILTER
  REPLICATE_REWRITE_DB = ((imooc_db, customer_db));
start slave;
# 主库插入数据验证
root@localhost 00:12:  [imooc_db]> insert into region_info(parent_id,region_name,region_level) values(1,'guangdong',2);
# 从库查看数据
root@localhost 00:13:  [order_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
+-----------+-----------+-------------+--------------+
1 row in set (0.00 sec)
# 其他2个也这么同步

• 5.新实例上启动复制追上主库，结果如下：

IP	dbname
172.16.10.141	imooc_db
172.16.10.142	order_db
172.16.10.143	product_db
172.16.10.144	customer_db

2.3配置mycat垂直分库

• 1.配置schema.xml，需要授权grant select,insert,update,delete on *.* to 'bm_mycat'@'%' identified by '123456';

<schema name="imooc_db" checkSQLschema="false" sqlMaxLimit="100">
        <table name="company" primaryKey="ID" dataNode="dn3,dn2,dn1" rule="mod-long"/>
        <table name="order_cart" primaryKey="cart_id" dataNode="dn_orderdb" />
        <table name="order_customer_addr" primaryKey="customer_addr_id" dataNode="dn_orderdb" />
        <table name="order_detail" primaryKey="order_detail_id" dataNode="dn_orderdb" />
        <table name="order_master" primaryKey="order_id" dataNode="dn_orderdb" />
        
        <table name="region_info" primaryKey="region_id" dataNode="dn_orderdb" />        
        <table name="serial" primaryKey="id" dataNode="dn_orderdb" />          
        <table name="shipping_info" primaryKey="ship_id" dataNode="dn_orderdb" />     
        <table name="warehouse_info" primaryKey="w_id" dataNode="dn_orderdb" />    
        <table name="warehouse_proudct" primaryKey="wp_id" dataNode="dn_orderdb" />
        
        <table name="product_brand_info" primaryKey="brand_id" dataNode="dn_productdb" />
        <table name="product_category" primaryKey="category_id" dataNode="dn_productdb" />
        <table name="product_comment" primaryKey="comment_id" dataNode="dn_productdb" />
        <table name="product_info" primaryKey="product_id" dataNode="dn_productdb" />
        <table name="product_pic_info" primaryKey="product_pic_id" dataNode="dn_productdb" />
        <table name="product_supplier_info" primaryKey="supplier_id" dataNode="dn_productdb" />
        
        <table name="customer_balance_log" primaryKey="balance_id" dataNode="dn_customerdb" />
        <table name="customer_inf" primaryKey="customer_inf_id" dataNode="dn_customerdb" />
        <table name="customer_level_inf" primaryKey="customer_level" dataNode="dn_customerdb" />
        <table name="customer_login" primaryKey="customer_id" dataNode="dn_customerdb" />
        <table name="customer_login_log" primaryKey="login_id" dataNode="dn_customerdb" />
        <table name="customer_point_log" primaryKey="point_id" dataNode="dn_customerdb" />
        
</schema>

<dataNode name="dn_orderdb" dataHost="mysql10142" database="order_db" />
<dataNode name="dn_productdb" dataHost="mysql10143" database="product_db" />
<dataNode name="dn_customerdb" dataHost="mysql10144" database="customer_db" />

<dataHost name="mysql10142" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>show slave status</heartbeat>
        <writeHost host="172.16.10.142" url="172.16.10.142:3306" user="bm_mycat" password="123456">
        </writeHost>
</dataHost>
<dataHost name="mysql10143" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>show slave status</heartbeat>
        <writeHost host="172.16.10.143" url="172.16.10.143:3306" user="bm_mycat" password="123456">
        </writeHost>
</dataHost>
<dataHost name="mysql10144" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>show slave status</heartbeat>
        <writeHost host="172.16.10.144" url="172.16.10.144:3306" user="bm_mycat" password="123456">
        </writeHost>
</dataHost>

• 2.配置server.xml，

<user name="app_imooc">
        <property name="password">123456</property>
        <property name="schemas">imooc_db</property>
</user>

2.4通过mycat访问db，应用端做访问更改

[root@mycat01 logs]# mysql -uapp_imooc -p123456 -P8066
Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'app_imooc'@'localhost' (using password: YES)
[root@mycat01 logs]# mysql -uapp_imooc -p123456 -P8066 -h172.16.10.142
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 1
Server version: 5.5.8-mycat-1.5.1-RELEASE-20160525110043 MyCat Server (OpenCloundDB)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

app_imooc@172.16.10.142 01:59:  [(none)]> show databases;
+----------+
| DATABASE |
+----------+
| imooc_db |
+----------+
1 row in set (0.00 sec)

app_imooc@172.16.10.142 01:59:  [(none)]> use imooc_db;
Database changed
app_imooc@172.16.10.142 01:59:  [imooc_db]> show tables;
+-----------------------+
| Tables in imooc_db    |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
| order_cart            |
| order_customer_addr   |
| order_detail          |
| order_master          |
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
| region_info           |
| serial                |
| shipping_info         |
| warehouse_info        |
| warehouse_proudct     |
+-----------------------+
21 rows in set (0.01 sec)

app_imooc@172.16.10.142 01:59:  [imooc_db]> 

2.5删除原库中已迁移的表

• 172.16.10.141 | imooc_db
• 172.16.10.142 | order_db

drop table product_brand_info   ;
drop table product_category     ;
drop table product_comment      ;
drop table product_info         ;
drop table product_pic_info     ;
drop table product_supplier_info;

drop table customer_balance_log ;
drop table customer_inf         ;
drop table customer_level_inf   ;
drop table customer_login       ;
drop table customer_login_log   ;
drop table customer_point_log   ;

root@localhost 02:07:  [order_db]> show tables;
+---------------------+
| Tables_in_order_db  |
+---------------------+
| order_cart          |
| order_customer_addr |
| order_detail        |
| order_master        |
| region_info         |
| serial              |
| shipping_info       |
| warehouse_info      |
| warehouse_proudct   |
+---------------------+
9 rows in set (0.00 sec)

• 172.16.10.143 | product_db

drop table order_cart           ;
drop table order_customer_addr  ;
drop table order_detail         ;
drop table order_master         ;
drop table region_info          ;
drop table serial               ;
drop table shipping_info        ;
drop table warehouse_info       ;
drop table warehouse_proudct    ;

drop table customer_balance_log ;
drop table customer_inf         ;
drop table customer_level_inf   ;
drop table customer_login       ;
drop table customer_login_log   ;
drop table customer_point_log   ;

root@localhost 02:10:  [product_db]> show tables;
+-----------------------+
| Tables_in_product_db  |
+-----------------------+
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
+-----------------------+
6 rows in set (0.00 sec)

• 172.16.10.144 | customer_db

drop table product_brand_info   ;
drop table product_category     ;
drop table product_comment      ;
drop table product_info         ;
drop table product_pic_info     ;
drop table product_supplier_info;

drop table order_cart           ;
drop table order_customer_addr  ;
drop table order_detail         ;
drop table order_master         ;
drop table region_info          ;
drop table serial               ;
drop table shipping_info        ;
drop table warehouse_info       ;
drop table warehouse_proudct    ;
root@localhost 02:08:  [customer_db]> show t
ables;
+-----------------------+
| Tables_in_customer_db |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
+-----------------------+
6 rows in set (0.00 sec)

2.6在mycat逻辑库中查看，和拆分之前的物理库是一样的

app_imooc@172.16.10.142 02:11:  [imooc_db]> show tables;
+-----------------------+
| Tables in imooc_db    |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
| order_cart            |
| order_customer_addr   |
| order_detail          |
| order_master          |
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
| region_info           |
| serial                |
| shipping_info         |
| warehouse_info        |
| warehouse_proudct     |
+-----------------------+
21 rows in set (0.00 sec)

2.7访问测试

app_imooc@172.16.10.142 23:14:  [imooc_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
+-----------+-----------+-------------+--------------+
2 rows in set (0.01 sec)

# 跨分片查询是不允许的
app_imooc@172.16.10.142 23:14:  [imooc_db]> select * from product_supplier_info a join region_info b on a.province=b.region_id;
ERROR 1064 (HY000): invalid route in sql, multi tables found but datanode has no intersection  sql:select * from product_supplier_info a join region_info on a.province=b.region_id

三.全局表

• 适用字典表/元数据表
• 每个物理库中都存在的表
• 由mycat来进行多写进行数据一致性保证

3.1配置全局表过程

• 1.在原物理库中将数据备份导出

mysqldump -uroot -p123456 --set-gtid-purged=OFF order_db region_info > /tmp/region_info.sql
scp /tmp/region_info.sql 172.16.10.143:/tmp
scp /tmp/region_info.sql 172.16.10.144:/tmp

• 2.将备份导入另外两个物理库

mysql -uroot -p123456 product_db < /tmp/region_info.sql
mysql -uroot -p123456 customer_db < /tmp/region_info.sql

• 3.此时3个物理库中都存在全局表region_info

root@localhost 23:28:  [order_db]> show tables;
+---------------------+
| Tables_in_order_db  |
+---------------------+
| order_cart          |
| order_customer_addr |
| order_detail        |
| order_master        |
| region_info         |
| serial              |
| shipping_info       |
| warehouse_info      |
| warehouse_proudct   |
+---------------------+
9 rows in set (0.00 sec)

root@localhost 23:28:  [customer_db]> show tables;
+-----------------------+
| Tables_in_customer_db |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
| region_info           |
+-----------------------+
7 rows in set (0.00 sec)

root@localhost 23:28:  [product_db]> show tables;
+-----------------------+
| Tables_in_product_db  |
+-----------------------+
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
| region_info           |
+-----------------------+
7 rows in set (0.00 sec)

• 4.在schema.xml配置全局表属性，只需要在表属性中增加type="global"，并增加对应的物理节点信息即可

<table name="region_info" primaryKey="region_id" dataNode="dn_orderdb,dn_productdb,dn_customerdb" type="global"/>

• 5.重启生效验证，此时可以在一个物理库内关联查询

app_imooc@172.16.10.142 23:35:  [imooc_db]> select * from product_supplier_info a join region_info b on a.province=b.region_id;
+-------------+---------------+---------------+---------------+----------+--------------+--------------+----------------+----------+------+----------+---------+-----------------+---------------------+-----------+-----------+-------------+--------------+
| supplier_id | supplier_code | supplier_name | supplier_type | link_man | phone_number | bank_name    | bank_account   | province | city | district | address | supplier_status | modified_time       | region_id | parent_id | region_name | region_level |
+-------------+---------------+---------------+---------------+----------+--------------+--------------+----------------+----------+------+----------+---------+-----------------+---------------------+-----------+-----------+-------------+--------------+
|           3 | 20001         | 供应商-3      |             1 | 王五     | 13800138003  | 中国银行     | 12345656785443 |        2 |   52 |      503 | 北京    |               1 | 2018-07-28 21:50:57 |         2 |         1 | shanghai    |            3 |
+-------------+---------------+---------------+---------------+----------+--------------+--------------+----------------+----------+------+----------+---------+-----------------+---------------------+-----------+-----------+-------------+--------------+
1 row in set (0.02 sec)

• 6.在mycat上维护全局表

app_imooc@172.16.10.142 23:38:  [imooc_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
+-----------+-----------+-------------+--------------+
2 rows in set (0.00 sec)

app_imooc@172.16.10.142 23:38:  [imooc_db]> insert into region_info values(3,3,'beijing',4);
Query OK, 1 row affected (0.05 sec)

app_imooc@172.16.10.142 23:39:  [imooc_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
|         3 |         3 | beijing     |            4 |
+-----------+-----------+-------------+--------------+
3 rows in set (0.00 sec)
# 在其他库上也可以同时更新该表
root@localhost 23:28:  [product_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
|         3 |         3 | beijing     |            4 |
+-----------+-----------+-------------+--------------+
3 rows in set (0.00 sec)

root@localhost 23:28:  [customer_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
|         3 |         3 | beijing     |            4 |
+-----------+-----------+-------------+--------------+
3 rows in set (0.00 sec)

四.垂直切分的优缺点

4.1优点

• 1.拆分简单明了，拆分规则明确
• 2.应用程序模块清晰明确，整合容易
• 3.数据维护方便易行，容易定位

4.2缺点

• 1.部分表关联无法在数据库级别完成，需要在程序中完成
• 2.对于访问机器频繁且数据量超大的表任然存在性能瓶颈
• 3.切分达到一定程度之后，扩展性会遇到限制，单表性能依然存在瓶颈

4.3解决跨分片关联的方式

• 1.适用mycat全局表，在每个库中冗余数据
• 2.在各个表中冗余字段，增加关键数据
• 3.适用API方式获取数据，再进行拼接