mysqlmmm+amoeba+keepalived实现mysql高可用和读写分离（二）

一、利用amoeba实现高可用下的读写分离

1、  关于amoeba的介绍和安装我这里不做介绍了，请查看：http://blog.chinaunix.net/u3/93755/showart.php?id=2363202，有什么问题我们再一起讨论。

2、  Amoeba在mysql-mmm架构下的配置

配置的除了amoeba的监听IP外其他的均采用上面mmm分配的浮动IP，因为当出现故障后，浮动IP会自动分配到正常的server上。amoeba.xml的配置如下：

<?xml version="1.0" encoding="gbk"?>

 

<!DOCTYPE amoeba:configuration SYSTEM "amoeba.dtd">

<amoeba:configuration xmlns:amoeba="http://amoeba.meidusa.com/">

 

       <server>

              <!-- proxy server绑定的端口 -->

                <property name="port">8066</property>

               

                <!-- proxy server绑定的IP -->

                <property name="ipAddress">192.168.1.163</property>

                <!-- proxy server net IO Read thread size -->

                <property name="readThreadPoolSize">20</property>

               

                <!-- proxy server client process thread size -->

                <property name="clientSideThreadPoolSize">30</property>

               

                <!-- mysql server data packet process thread size -->

                <property name="serverSideThreadPoolSize">30</property>

               

                <!-- socket Send and receive BufferSize(unit:K)  -->

                <property name="netBufferSize">128</property>

               

                <!-- Enable/disable TCP_NODELAY (disable/enable Nagle's algorithm). -->

                <property name="tcpNoDelay">true</property>

               

                <property name="user">root</property>

                <property name="password">password</property>

               

                <!-- query timeout( default: 60 second , TimeUnit:second) -->

                <property name="queryTimeout">60</property>

        </server>

       

        <!--

                每个ConnectionManager都将作为一个线程启动。

                manager负责Connection IO读写/死亡检测

        -->

        <connectionManagerList>

                <connectionManager name="defaultManager" class="com.meidusa.amoeba.net.MultiConnectionManagerWrapper">

                         <property name="subManagerClassName">com.meidusa.amoeba.net.AuthingableConnectionManager</property>

                        

                         <!--

                           default value is avaliable Processors

                         <property name="processors">5</property>

                          -->

                </connectionManager>

        </connectionManagerList>

       

        <dbServerList>

                <!--

                         一台mysqlServer 需要配置一个pool，

                         如果多台 平等的mysql需要进行loadBalance，

                         平台已经提供一个具有负载均衡能力的objectPool：com.meidusa.amoeba.mysql.server.MultipleServerPool

                         简单的配置是属性加上 virtual="true",该Pool 不允许配置factoryConfig

                         或者自己写一个ObjectPool。

                -->

                <dbServer name="server1">

                        

                         <!-- PoolableObjectFactory实现类 -->

                         <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory">

                                 <property name="manager">defaultManager</property>

                                

                                 <!-- 真实mysql数据库端口 -->

                                 <property name="port">3306</property>

                                

                                 <!-- 真实mysql数据库IP -->

                                 <property name="ipAddress">192.168.1.113</property>

                                 <property name="schema">test</property>

                                

                                 <!-- 用于登陆mysql的用户名 -->

                                 <property name="user">zhang</property>

                                

                                 <!-- 用于登陆mysql的密码 -->

                                 <property name="password">zhang123</property>

                                

                         </factoryConfig>

                        

                         <!-- ObjectPool实现类 -->

                         <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool">

                                 <property name="maxActive">200</property>

                                 <property name="maxIdle">200</property>

                                 <property name="minIdle">10</property>

                                 <property name="minEvictableIdleTimeMillis">600000</property>

                                 <property name="timeBetweenEvictionRunsMillis">600000</property>

                                 <property name="testOnBorrow">true</property>

                                 <property name="testWhileIdle">true</property>

                         </poolConfig>

                </dbServer>

               

 

                <dbServer name="server2">

                        

                         <!-- PoolableObjectFactory实现类 -->

                         <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory">

                                 <property name="manager">defaultManager</property>

                                

                                 <!-- 真实mysql数据库端口 -->

                                 <property name="port">3306</property>

                                

                                 <!-- 真实mysql数据库IP -->

                                 <property name="ipAddress">192.168.1.111</property>

                                 <property name="schema">test</property>

                                

                                 <!-- 用于登陆mysql的用户名 -->

                                 <property name="user">zhang</property>

                                

                                 <!-- 用于登陆mysql的密码 -->

                                 <property name="password">zhang123</property>

                                

                         </factoryConfig>

                        

                         <!-- ObjectPool实现类 -->

                         <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool">

                                 <property name="maxActive">200</property>

                                 <property name="maxIdle">200</property>

                                 <property name="minIdle">10</property>

                                 <property name="minEvictableIdleTimeMillis">600000</property>

                                 <property name="timeBetweenEvictionRunsMillis">600000</property>

                                 <property name="testOnBorrow">true</property>

                                 <property name="testWhileIdle">true</property>

                         </poolConfig>

                </dbServer>

 

 

                <dbServer name="server3">

                        

                         <!-- PoolableObjectFactory实现类 -->

                         <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory">

                                <property name="manager">defaultManager</property>

                                

                                 <!-- 真实mysql数据库端口 -->

                                 <property name="port">3306</property>

                                

                                 <!-- 真实mysql数据库IP -->

                                 <property name="ipAddress">192.168.1.112</property>

                                 <property name="schema">test</property>

                                

                                 <!-- 用于登陆mysql的用户名 -->

                                 <property name="user">zhang</property>

                                

                                 <!-- 用于登陆mysql的密码 -->

                                 <property name="password">zhang123</property>

                                

                         </factoryConfig>

                        

                         <!-- ObjectPool实现类 -->

                         <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool">

                                 <property name="maxActive">200</property>

                                 <property name="maxIdle">200</property>

                                 <property name="minIdle">10</property>

                                 <property name="minEvictableIdleTimeMillis">600000</property>

                                 <property name="timeBetweenEvictionRunsMillis">600000</property>

                                 <property name="testOnBorrow">true</property>

                                 <property name="testWhileIdle">true</property>

                         </poolConfig>

                </dbServer>

 

 

                <dbServer name="server4">

                        

                         <!-- PoolableObjectFactory实现类 -->

                         <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory">

                                 <property name="manager">defaultManager</property>

                                

                                 <!-- 真实mysql数据库端口 -->

                                 <property name="port">3306</property>

                                

                                 <!-- 真实mysql数据库IP -->

                                 <property name="ipAddress">192.168.1.114</property>

                                 <property name="schema">test</property>

                                

                                 <!-- 用于登陆mysql的用户名 -->

                                 <property name="user">zhang</property>

                                

                                 <!-- 用于登陆mysql的密码 -->

                                 <property name="password">zhang123</property>

                                

                         </factoryConfig>

                        

                         <!-- ObjectPool实现类 -->

                         <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool">

                                 <property name="maxActive">200</property>

                                 <property name="maxIdle">200</property>

                                 <property name="minIdle">10</property>

                                 <property name="minEvictableIdleTimeMillis">600000</property>

                                 <property name="timeBetweenEvictionRunsMillis">600000</property>

                                 <property name="testOnBorrow">true</property>

                                 <property name="testWhileIdle">true</property>

                         </poolConfig>

                </dbServer>

 

 

                <dbServer name="server5">

                        

                         <!-- PoolableObjectFactory实现类 -->

                         <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory">

                                 <property name="manager">defaultManager</property>

                                

                                 <!-- 真实mysql数据库端口 -->

                                 <property name="port">3306</property>

                                

                                 <!-- 真实mysql数据库IP -->

                                 <property name="ipAddress">192.168.1.115</property>

                                 <property name="schema">test</property>

                                

                                 <!-- 用于登陆mysql的用户名 -->

                                 <property name="user">zhang</property>

                                

                                 <!-- 用于登陆mysql的密码 -->

                                 <property name="password">zhang123</property>

                                

                         </factoryConfig>

                        

                         <!-- ObjectPool实现类 -->

                         <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool">

                                 <property name="maxActive">200</property>

                                 <property name="maxIdle">200</property>

                                 <property name="minIdle">10</property>

                                 <property name="minEvictableIdleTimeMillis">600000</property>

                                 <property name="timeBetweenEvictionRunsMillis">600000</property>

                                 <property name="testOnBorrow">true</property>

                                 <property name="testWhileIdle">true</property>

                         </poolConfig>

                </dbServer>

 

 

 

               

                <dbServer name="master" virtual="true">

                         <poolConfig class="com.meidusa.amoeba.server.MultipleServerPool">

                                 <!-- 负载均衡参数 1=ROUNDROBIN , 2=WEIGHTBASED , 3=HA-->

                                 <property name="loadbalance">1</property>

                                

                                 <!-- 参与该pool负载均衡的poolName列表以逗号分割 -->

                                 <property name="poolNames">server1</property>

                         </poolConfig>

                </dbServer>

 

                <dbServer name="slave" virtual="true">

                         <poolConfig class="com.meidusa.amoeba.server.MultipleServerPool">

                                 <!-- 负载均衡参数 1=ROUNDROBIN , 2=WEIGHTBASED , 3=HA-->

                                 <property name="loadbalance">1</property>

                                

                                 <!-- 参与该pool负载均衡的poolName列表以逗号分割 -->

                                 <property name="poolNames">server2,server3,server4,server5</property>

                         </poolConfig>

                </dbServer>

               

        </dbServerList>

       

        <queryRouter class="com.meidusa.amoeba.mysql.parser.MysqlQueryRouter">

                <property name="ruleConfig">${amoeba.home}/conf/rule.xml</property>

                <property name="functionConfig">${amoeba.home}/conf/functionMap.xml</property>

                <property name="ruleFunctionConfig">${amoeba.home}/conf/ruleFunctionMap.xml</property>

                <property name="LRUMapSize">1500</property>

                <property name="defaultPool">master</property>

                <property name="writePool">master</property>

                <property name="readPool">slave</property>

                <property name="needParse">true</property>

        </queryRouter>

</amoeba:configuration>

 

3、  启动amoeba

       /usr/local/amoeba/bin/amoeba &  

检验启动是否成功（使用的是默认的8066端口）：

[root@server3 ~]# ps aux | grep amoeba

root     15390  0.2 21.7 411800 55684 pts/0    Sl   Nov09   0:13 /usr/java/jdk1.6/bin/java -server -Xms256m -Xmx256m -Xss128k -Damoeba.home=/usr/local/amoeba -Dclassworlds.conf=/usr/local/amoeba/bin/amoeba.classworlds -classpath /usr/local/amoeba/lib/classworlds-1.0.jar org.codehaus.classworlds.Launcher

 

4、  测试

为了有权限，请在所有mysql-server上添加上面配置的用户名和密码，比如：

grant all on test.* to zhang@'192.168.1.%' identified by 'zhang123';

flush privileges;

测试的时候比较麻烦，因为如果把主从停掉来看效果的话，mmm会将该同步状态有问题的服务器设置成REPLICATION_FAIL并将该主机上的浮动IP移动到正常服务器，比如：我将server4停掉后，就出出现如下现象：

[root@server3 ~]#  mmm_control show

  db1(192.168.1.161) master/ONLINE. Roles: reader(192.168.1.114), reader(192.168.1.115)

  db2(192.168.1.162) master/ONLINE. Roles: reader(192.168.1.112), writer(192.168.1.113)

  db3(192.168.1.164) slave/REPLICATION_FAIL. Roles:

  db4(192.168.1.165) slave/ONLINE. Roles: reader(192.168.1.111)

暂时没想到很好的测试办法。读写分离测试没有问题。

最近手头没有空闲的机器来做压力测试。等有了机器了再对这个架构做下压力测试！

 

 

二、利用keepalived实现amoeba高可用

Keepalived简介：Keepalived是Linux下面实现VRRP 备份路由的高可靠性运行件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间无缝交接。从而提高系统的可用性

1、  keepalived的安装（server6、server7上都要安装）：

wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz
tar zxvf keepalived-1.1.19.tar.gz
cd keepalived-1.1.19
./configure --prefix=/usr/local/keepalived
make
make install
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
mkdir /etc/keepalived
cd /etc/keepalived/

 

2、  keepalived的配置

配置server6的keepalived配置文件/etc/keepalived/keepalived.conf，内容如下：

bal_defs {

        notification_email {

               jimo291@gmail.com

        }

        notification_email_from jimo291@gmail.com

        smtp_server 127.0.0.1

        smtp_connect_timeout 30

        router_id test1

}

 

 

vrrp_instance VI_1 {

        state MASTER

        interface eth0

        virtual_router_id 51

        priority 100

        advert_int 1

        smtp_alert

        authentication {

                auth_type PASS

                auth_pass 123

        }

 

        virtual_ipaddress {

                192.168.1.170

        }

}

 

配置server6的keepalived配置文件/etc/keepalived/keepalived.conf，内容如下：

bal_defs {

        notification_email {

                jimo291@gmail.com

        }

        notification_email_from jimo291@gmail.com

        smtp_server 127.0.0.1

        smtp_connect_timeout 30

        router_id test2

}

 

vrrp_instance VI_1 {

        state BACKUP

        interface eth0

        virtual_router_id 51

        priority 80

        advert_int 1

        smtp_alert

        authentication {

                auth_type PASS

                auth_pass 123

        }

        virtual_ipaddress {

                192.168.1.170

        }

}

 

3、  keepalived的启动:

在server6、server7上执行如下命令启动：

/etc/rc.d/init.d/keepalived start

看看server6上是否绑定了虚拟IP192.168.1.170

[root@server6 ~]# ip add | grep 192.168.1.170

inet 192.168.1.170/32 scope global eth0

       可以看到已经成功绑定了192.168.1.170。

4、  测试

停掉server6，看看虚拟IP192.168.1.170会不会自动切换到server7上。以及测试mysql -uroot -ppassword -h192.168.1.170登录，看看能否实现读写操作是否正常等等！

 

备注：最后记得将所有的启动命令都写到/etc/rc.local文件中去，实现开机启动！