• 慢查询阻塞了xtrabackup进而阻塞以后的sql导致的系统瘫痪问题


    收到开发反应一库的sql频繁超时,系统几乎瘫痪,无法执行任何操作,我登上库先查看到当前的线程,发现有大量的线程状态是

     Waiting for table flush

    查看当前的事务

    从昨天开始执行,到今天早晨还没执行完,具体原因还没深究,先将此线程释放,然后备份才可以flush table成功继而备份完成后后面一系列被阻塞的sql都得以正常运行

    mysql> select * from information_schema.innodb_trxG
    *************************** 1. row ***************************
                        trx_id: 192611452
                     trx_state: RUNNING
                   trx_started: 2017-11-30 18:33:58
         trx_requested_lock_id: NULL
              trx_wait_started: NULL
                    trx_weight: 3688
           trx_mysql_thread_id: 352932171
                     trx_query: DELETE FROM xx WHERE xx IN(SELECT xx
                                                    FROM xx WHERE Remarks LIKE xx)
           trx_operation_state: unlock_row
             trx_tables_in_use: 2
             trx_tables_locked: 2
              trx_lock_structs: 3688
         trx_lock_memory_bytes: 368848
               trx_rows_locked: 4
             trx_rows_modified: 0
       trx_concurrency_tickets: 0
           trx_isolation_level: READ COMMITTED
             trx_unique_checks: 1
        trx_foreign_key_checks: 1
    trx_last_foreign_key_error: NULL
     trx_adaptive_hash_latched: 0
     trx_adaptive_hash_timeout: 0
              trx_is_read_only: 0
    trx_autocommit_non_locking: 0

    后来想了一下每天的凌晨两点有物理备份,于是查看备份日志,发现果然是上面的事务阻塞了物理备份;

    物理备份的整个流程

    先记录当前redo log的序列号
    171201 02:00:02 >> log scanned up to (54138135415)
    xtrabackup: Generating a list of tablespaces
    xtrabackup: using the full scan for incremental backup
    xtrabackup: Starting 4 threads for parallel data files transfer
    然后备份innodb库表
    171201 02:00:12 [01] Copying .
    备份完之后flush table;因为被阻塞,所以知道释放完事务后才成功
    171201 02:00:17 Executing FLUSH NO_WRITE_TO_BINLOG TABLES...
    接着开始备份非事务库表
    171201 09:36:13 Executing FLUSH TABLES WITH READ LOCK... 171201 09:36:13 >> log scanned up to (54147795188) 171201 09:36:14 Starting to backup non-InnoDB tables and files 171201 09:36:14 [01] Copying .... xtrabackup: The latest check point (for incremental): '54138858140' xtrabackup: Stopping log copying thread. .171201 09:36:14 >> log scanned up to (54147795198) 171201 09:36:14 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
    备份完之后释放表锁
    171201 09:36:14 Executing UNLOCK TABLES 171201 09:36:14 All tables unlocked 171201 09:36:14 [00] Copying ib_buffer_pool to xxx 171201 09:36:14 [00] ...done 171201 09:36:14 Backup created in directory xxxx MySQL binlog position: xxx 171201 09:36:14 [00] Writing backup-my.cnf 171201 09:36:14 [00] ...done 171201 09:36:14 [00] Writing xtrabackup_info 171201 09:36:14 [00] ...done xtrabackup: Transaction log of lsn (54138129801) to (54147795198) was copied. 171201 09:36:15 completed OK!

    被阻塞的语句是FLUSH NO_WRITE_TO_BINLOG TABLES...

    官方解释flush tables

    Closes all open tables, forces all tables in use to be closed, and flushes the query cache and prepared statement cache.

    没有涉及到锁相关的字眼;但是测试表明在执行查询或者变更还未完成时,如果另起一个会话执行flush tables 则会被阻塞,

    如果此后如果有操作慢查询中的表的任何sql都会被阻塞;

              

  • 相关阅读:
    常见压缩/解压缩及打包命令
    黑盒测试和白盒测试
    基础命令的操作

    转]DLL-多个进程间共享动态链接库的原理
    Ansi UNICODE,GBK,UTF-8区别
    Cppunit 源码 SynchronizedObject
    二分查找实现
    Androdi 开发学习四 Activity和Intent
    Android开发学习三:adb启动失败
  • 原文地址:https://www.cnblogs.com/Bccd/p/7940809.html
Copyright © 2020-2023  润新知