[20191127]探究等待事件的本源4.txt
--//昨天使用ash_wait_chains.sql脚本把各个生产库执行1遍,才发现我对一套系统性能理解错误.
--//我一直以为这套系统存储有点问题,性能不是很好.
1.环境:
xxxxxx> @ ver1
PORT_STRING VERSION BANNER
------------------------------ -------------- --------------------------------------------------------------------------------
x86_64/Linux 2.4.xx 11.2.0.4.0 Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
2.分析:
xxxxxx> @ tpt/ash/ash_wait_chains event2 1=1 trunc(sysdate)+8/24 sysdate
-- Display ASH Wait Chain Signatures script v0.2 BETA by Tanel Poder ( http://blog.tanelpoder.com )
%This SECONDS AAS WAIT_CHAIN
------ ---------- ---------- -------------------------------------------------------------
39% 2667 .5 -> ON CPU
18% 1274 .2 -> LNS wait on SENDREQ
18% 1249 .2 -> log file sync -> LGWR-LNS wait on channel
18% 1215 .2 -> LGWR-LNS wait on channel
1% 62 0 -> log file sync
1% 60 0 -> log file parallel write
1% 46 0 -> control file sequential read
1% 39 0 -> db file sequential read
1% 36 0 -> log file sync -> log file parallel write
0% 29 0 -> gc cr block 2-way
0% 24 0 -> gc current block 2-way
0% 23 0 -> null event
0% 21 0 -> direct path read
0% 20 0 -> log file sync -> ON CPU
0% 18 0 -> gcs log flush sync -> LGWR-LNS wait on channel
0% 18 0 -> gc cr block busy
0% 16 0 -> db file parallel write
0% 12 0 -> SQL*Net more data to client
0% 10 0 -> gc cr multi block request
0% 9 0 -> gc current grant busy
0% 7 0 -> Disk file operations I/O
0% 6 0 -> ASM file metadata operation
0% 5 0 -> log file sequential read
0% 4 0 -> LGWR wait on LNS
0% 4 0 -> log file switch completion -> LGWR-LNS wait on channel
0% 4 0 -> IPC send completion sync
0% 4 0 -> control file parallel write
0% 3 0 -> reliable message
0% 3 0 -> log file sync -> LGWR wait on LNS
0% 2 0 -> CGS wait for IPC msg
30 rows selected.
xxxxxx> select sysdate from dual ;
SYSDATE
-------------------
2019-11-27 09:27:51
xxxxxx> @ tpt/ash/ash_wait_chains event2 "event='log file sync'" trunc(sysdate)+8/24 sysdate
-- Display ASH Wait Chain Signatures script v0.2 BETA by Tanel Poder ( http://blog.tanelpoder.com )
%This SECONDS AAS WAIT_CHAIN
------ ---------- ---------- ----------------------------------------------
91% 1249 .2 -> log file sync -> LGWR-LNS wait on channel
5% 62 0 -> log file sync
3% 36 0 -> log file sync -> log file parallel write
1% 20 0 -> log file sync -> ON CPU
0% 3 0 -> log file sync -> LGWR wait on LNS
--//实际上主要问题在于log_archive_dest_2参数设置不合理,采用sync参数.
xxxxxx> show parameter log_archive_dest_2
NAME TYPE VALUE
------------------ ------ ----------------------------------------------------------------------------------------------------
log_archive_dest_2 string service=xxxxxx lgwr sync reopen=15 max_failure=10 net_timeout=30 optional noaffirm db_unique_name=xxxxx
xxxxxx> show parameter log_archive_config
NAME TYPE VALUE
------------------ ------ -----------
log_archive_config string nodg_config
--//实际上我第一次看应该是去年的春节前后,刚上线,当时这个问题没有严重,现在显得越来越明显.
--//"可怕"的是我不能修改这个参数,这个所谓的dg是一个第三方安装的东西,根本不是什么dg,我一修改参数,对方的软件视乎就检测我的改动,自动重置回来.
--//alter system set log_archive_dest_2="service=xxxxxx lgwr async reopen=15 max_failure=10 net_timeout=30 optional noaffirm db_unique_name=xxxxxx";
--//btw:好像这个参数可以修改,也许我记错了,不能修改log_archive_config参数,等一段时间观察看看.
--//看awr报表:
Top 10 Foreground Events by Total Wait Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tota Wait % DB
Event Waits Time Avg(ms) time Wait Class
------------------------------ ------------ ---- ------- ------ ----------
DB CPU 1063 78.5
log file sync 46,848 311. 7 23.0 Commit
control file sequential read 68,350 21.1 0 1.6 System I/O
gc cr block 2-way 17,827 8.2 0 .6 Cluster
gc current block 2-way 17,190 7.4 0 .5 Cluster
gc cr block busy 218 6.8 31 .5 Cluster
SQL*Net more data to client 541,828 6.4 0 .5 Network
db file sequential read 936 6 6 .4 User I/O
direct path read 1,591 5.2 3 .4 User I/O
Disk file operations I/O 23,740 4.6 0 .3 User I/O
--//log file sync 平均等待7ms,到业务高峰可以到达13-14ms.我开始以为磁盘io不行,或者使用asm的原因.实际是就是上面的参数设置不合理.
--//如果不小心选择优化磁盘IO,那就选择错误的优化方向.实际上这套系统cpu充足,磁盘性能也不差,典型的大马拉小车.
--//主要问题在于应用软件垃圾(不仅仅指这个dg设置)!!也就是我以前提得良好的硬件掩盖拙劣的应用设计.
--//再来看看control file sequential read等待事件:
IOStat by Filetype summary DB/Inst: zzzzz/zzzzz1 Snaps: 25477-25478
-> 'Data' columns suffixed with M,G,T,P are in multiples of 1024
other columns suffixed with K,M,G,T,P are in multiples of 1000
-> Small Read and Large Read are average service times, in milliseconds
-> Ordered by (Data Read + Write) desc
Reads: Reqs Data Writes: Reqs Data Small Large
Filetype Name Data per sec per sec Data per sec per sec Read Read
--------------- ------- ------- ------- ------- ------- ------- ------- -------
Control File 8.3G 27.2 2.382M 228M 3.6 .064M 0.1 0.8
Log File 206M 0.1 .058M 228M 20.4 .064M 1.5 6.0
Data File 64M 2.2 .018M 235M 4.6 .066M 2.3 N/A
Archive Log 1M 0.2 0M 205M 0.1 .057M 0.0 N/A
Temp File 23M 0.0 .006M 23M 0.0 .006M 0.0 1.7
TOTAL: 8.6G 29.7 2.464M 919M 28.6 .257M 0.3 0.9
------------------------------------------------------
--//实际上对方的软件简直是变态,关于这个问题的描述在链接如下,不再分析展开贴出:
--//http://blog.itpub.net/267265/viewspace-2222146/=>[20181129]大量的control file sequential read.txt.
3.修改参数后的观察:
--//检查alert发现,是Wed Nov 27 09:33:31 2019修改参数
Wed Nov 27 09:33:31 2019
ALTER SYSTEM SET log_archive_dest_2='service=xxxxx lgwr async reopen=15 max_failure=10 net_timeout=30 optional noaffirm db_unique_name=xxxxx' SCOPE=BOTH;
xxxxxx> select sysdate from dual ;
SYSDATE
-------------------
2019-11-27 10:08:20
xxxxxx> @ tpt/ash/ash_wait_chains event2 "event='log file sync'" trunc(sysdate)+9/24+33/1440 sysdate
-- Display ASH Wait Chain Signatures script v0.2 BETA by Tanel Poder ( http://blog.tanelpoder.com )
%This SECONDS AAS WAIT_CHAIN
------ ---------- ---------- -----------------------------------------------------------------------------------------------
57% 26 0 -> log file sync -> LGWR-LNS wait on channel
33% 15 0 -> log file sync
9% 4 0 -> log file sync -> log file parallel write
2% 1 0 -> log file sync -> ON CPU
xxxxxx> @ tpt/ash/ash_wait_chains event2 "event='log file sync'" trunc(sysdate)+9/24+34/1440 sysdate
-- Display ASH Wait Chain Signatures script v0.2 BETA by Tanel Poder ( http://blog.tanelpoder.com )
%This SECONDS AAS WAIT_CHAIN
------ ---------- ---------- -----------------------------------------------------------------------------------------------
74% 14 0 -> log file sync
21% 4 0 -> log file sync -> log file parallel write
5% 1 0 -> log file sync -> ON CPU
--//你可以对比看出取值范围9:33换成9:34,多了1分钟由LGWR-LNS wait on channel引起的log file sync占26秒,而9:34后的查询完全看不到这个情况.
xxxxxx> @ tpt/ash/ash_wait_chains program2||event2 1=1 trunc(sysdate)+9/24+34/1440 sysdate
-- Display ASH Wait Chain Signatures script v0.2 BETA by Tanel Poder ( http://blog.tanelpoder.com )
%This SECONDS AAS WAIT_CHAIN
------ ---------- ---------- -----------------------------------------------------------------------------------------------
36% 547 .2 -> (zzzzzz.EXE) ON CPU
13% 204 .1 -> (wnwp.exe) ON CPU
9% 145 .1 -> (zzzzzz.EXE) ON CPU
7% 107 0 -> (NSAn) ON CPU
5% 80 0 -> (CAPAA-PIPE) ON CPU
3% 40 0 -> (httpd.exe) ON CPU
2% 31 0 -> (sqlplus) ON CPU
2% 31 0 -> (DIAn) ON CPU
2% 24 0 -> (oracle) ON CPU
1% 22 0 -> (LGWR) ON CPU
1% 22 0 -> (PSPn) ON CPU
1% 21 0 -> (LGWR) log file parallel write
1% 16 0 -> (NSAn) LNS wait on SENDREQ
1% 15 0 -> (sqlplus) control file sequential read
1% 15 0 -> (LMSn) ON CPU
1% 13 0 -> (zzzzzz.EXE) db file sequential read
1% 12 0 -> (zzzzzz.EXE) gc cr block 2-way
1% 12 0 -> (routine.exe) ON CPU
1% 12 0 -> (PlSqlDev.exe) ON CPU
1% 10 0 -> (Toad.exe) ON CPU
1% 10 0 -> (DBWn) db file parallel write
0% 7 0 -> (DBWn) ON CPU
0% 7 0 -> (wnwp.exe) log file sync
0% 6 0 -> (zzzzzz.EXE) gc current block 2-way
0% 6 0 -> (zzzzzz.EXE) log file sync
0% 6 0 -> (zzzzzz.EXE) direct path read
0% 5 0 -> (zzzzzz.EXE) log file sync
0% 5 0 -> (LMON) ON CPU
0% 4 0 -> (wnwp.exe) log file sync -> (LGWR) log file parallel write
0% 4 0 -> (MMON) ON CPU
30 rows selected.
