During the performance test, observe the following condition in the database server from EM,
And the AWR report shows the top event is "Latch:Row Cache Objects",
And the ASH report also confirms this,
Did some query, and find the most of the row cache are in "dc_users",
select * from v$latch_children where latch#=280 and to_number(addr, 'xxxxxxxxxxxxxxxx') = 14688581512; -- child# 8, level# 4 select distinct s.kqrstcln latch#, r.cache#, r.parameter name, r.type, r.subordinate#, r.gets from v$rowcache r, x$kqrst s where r.cache# = s.kqrstcid and s.kqrstcln = 8 ;
Try some search in MOS, and found the following article (Doc ID 1417373.1 - Row Cache Latch Contention for DC_USERS While Using
VPD) mentioning one Oracle bug (12772404) could cause this problem.
===============================================
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.0 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.
SYMPTOMS
Significant Latch: row cache objects contention.
The Dictionary Cache Stats section of the awr report shows high Get Requests for the dc_users row cache. Soft parsing contributes most to the total DB Time. The top SQLs doing huge parse calls (Sqls Ordered by Parse Calls section of the awr report) uses VPD.
The queries in the top list are executed by the users who were granted the "exempt access policy" privilege.
CAUSE
The problem is caused by << Bug 12772404 >>. The SQL statements accessing VPD-protected objects are constantly soft reparsed and this in turns generates significant contention on access to row cache objects, in particular dc_users. The heavy hits of row cache during soft parse of VPD cursors is due to repeated system privilege check for VPD bypass against the subheap of user row cache. The cause of the latch contention is the row cache look-up while checking for EXEMPT ACCESS POLICY system privilege.
When VPD is used, intense row cache objects latch contention (on dc_users) may be caused (because of Exempt Access Policy privilege check).
SOLUTION
Apply the patch 12772404 SIGNIFICANT ROW CACHE OBJECTS LATCH CONTENTION WHEN USING VPD
=====================
So, just download the patch and using Opatch to apply it to the database.
Now the database Top 5 events has changed to....
Seems the "latch:row cache objects" issue is gone. Will wait and see.