客户的某套系统上有一个表空间近日使用率异常增长，该表空间用以存储索引段，经过定位发现一个原本只有200M左右的索引使用将近30+G的空间，而且表现为绝大多数是未格式化的数据块。以下为通过 show_space脚本收集的段信息：
Unformatted Blocks = 1772568
Blocks with 00-25% free space = 0
Blocks with 26-50% free space = 2173
Blocks with 51-75% free space = 0
Blocks with 76-100% free space = 0
Full Blocks = 60762
Unformatted Blocks总数1772568,共使用1772568*16k=27G. 一个索引占用如此多的未格式化块似乎不可思议，首先想到的可能是一个威力惊人的Bug。
探索MOS，可以发现以下这个有趣的note:
Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.3
Information in this document applies to any platform.
Symptoms
The problem as observed is that the space allocated for a table/index(partition) is far too high and keeps growing.
Using the AVG_ROW_SIZE * NUM_ROWS shows that the space as needed would be far less that the allocated space (in some cases allocated xxGb while xxMb would be expected).
Cause
Bug 5987262 TABLESPACE IS ABNORMALLY INCREASED BY UNFORMATTED BLOCKS. This bug was closed as duplicate of unpublished Bug 5890312 HANG OBSERVED WHILE CREATING CTXCAT INDEX
The following PL/SQL block will show the space usage in the (partitioned) table
set serveroutput on
exec dbms_output.enable(1000000);
declare
unf number;
unfb number;
fs1 number;
fs1b number;
fs2 number;
fs2b number;
fs3 number;
fs3b number;
fs4 number;
fs4b number;
full number;
fullb number;
own dba_tables.owner%type;
tab dba_tables.table_name%type;
yesno varchar2(3);
type parts is table of dba_tab_partitions%rowtype;
partlist parts;
type cursor_ref is ref cursor;
c_cur cursor_ref;
begin
own:=upper('&owner');
tab:=upper('&table_name');
dbms_output.put_line('--------------------------------------------------------------------------------');
open c_cur for select partitioned from dba_tables
where owner=own and table_name=tab;
fetch c_cur into yesno;
close c_cur;
dbms_output.put_line('Owner: '||own);
dbms_output.put_line('Table: '||tab);
dbms_output.put_line('------------------------------------------------');
if yesno='NO'
then
dbms_space.space_usage(own,tab,'TABLE',unf,unfb,fs1,fs1b,fs2,fs2b,fs3,fs3b,fs4,fs4b,full,fullb);
dbms_output.put_line('unf: '||unf||' fs1: '||fs1||' fs2: '||fs2||' fs3: '||fs3||' fs4: '||fs4||' full: '||full);
else
open c_cur for select * from dba_tab_partitions
where table_owner=own and table_name=tab;
fetch c_cur bulk collect into partlist;
close c_cur;
for i in partlist.first .. partlist.last
loop
dbms_space.space_usage(partlist(i).table_owner,partlist(i).table_name,'TABLE PARTITION',unf,unfb,fs1,fs1b,fs2,fs2b,fs3,fs3b,fs4,fs4b,full,fullb,partlist(i).partition_name);
dbms_output.put_line('Partition: '||partlist(i).partition_name);
dbms_output.put_line('unf: '||unf||' fs1: '||fs1||' fs2: '||fs2||' fs3: '||fs3||' fs4: '||fs4||' full: '||full);
end loop;
end if;
dbms_output.put_line('--------------------------------------------------------------------------------');
end;
/
In case there is a need to check a single table partition:
set serveroutput on
exec dbms_output.enable(1000000);
declare
unf number;
unfb number;
fs1 number;
fs1b number;
fs2 number;
fs2b number;
fs3 number;
fs3b number;
fs4 number;
fs4b number;
full number;
fullb number;
own dba_tables.owner%type;
tab dba_tables.table_name%type;
par dba_tab_partitions.partition_name%type;
begin
own:=upper('&owner');
tab:=upper('&table_name');
par:=upper('&partition_name');
dbms_output.put_line('--------------------------------------------------------------------------------');
dbms_output.put_line('Owner: '||own);
dbms_output.put_line('Table: '||tab);
dbms_output.put_line('Partition: '||par);
dbms_output.put_line('------------------------------------------------');
dbms_space.space_usage(own,tab,'TABLE PARTITION',unf,unfb,fs1,fs1b,fs2,fs2b,fs3,fs3b,fs4,fs4b,full,fullb,par);
dbms_output.put_line('unf: '||unf||' fs1: '||fs1||' fs2: '||fs2||' fs3: '||fs3||' fs4: '||fs4||' full: '||full);
dbms_output.put_line('--------------------------------------------------------------------------------');
end;
/
The following PL/SQL block will show the space usage in the (partitioned) index:
set serveroutput on
exec dbms_output.enable(1000000);
declare
unf number;
unfb number;
fs1 number;
fs1b number;
fs2 number;
fs2b number;
fs3 number;
fs3b number;
fs4 number;
fs4b number;
full number;
fullb number;
own dba_indexes.owner%type;
ind dba_indexes.index_name%type;
yesno varchar2(3);
type parts is table of dba_ind_partitions%rowtype;
partlist parts;
type cursor_ref is ref cursor;
c_cur cursor_ref;
begin
own:=upper('&owner');
ind:=upper('&index_name');
dbms_output.put_line('--------------------------------------------------------------------------------');
open c_cur for select partitioned from dba_indexes
where owner=own and index_name=ind;
fetch c_cur into yesno;
close c_cur;
dbms_output.put_line('Owner: '||own);
dbms_output.put_line('Index: '||ind);
dbms_output.put_line('------------------------------------------------');
if yesno='NO'
then
dbms_space.space_usage(own,ind,'INDEX',unf,unfb,fs1,fs1b,fs2,fs2b,fs3,fs3b,fs4,fs4b,full,fullb);
dbms_output.put_line('unf: '||unf||' fs1: '||fs1||' fs2: '||fs2||' fs3: '||fs3||' fs4: '||fs4||' full: '||full);
else
open c_cur for select * from dba_ind_partitions
where index_owner=own and index_name=ind;
fetch c_cur bulk collect into partlist;
close c_cur;
for i in partlist.first .. partlist.last
loop
dbms_space.space_usage(partlist(i).index_owner,partlist(i).index_name,'INDEX PARTITION',unf,unfb,fs1,fs1b,fs2,fs2b,fs3,fs3b,fs4,fs4b,full,fullb,partlist(i).partition_name);
dbms_output.put_line('Partition: '||partlist(i).partition_name);
dbms_output.put_line('unf: '||unf||' fs1: '||fs1||' fs2: '||fs2||' fs3: '||fs3||' fs4: '||fs4||' full: '||full);
end loop;
end if;
dbms_output.put_line('--------------------------------------------------------------------------------');
end;
/
In case there is a need to check a single index partition:
set serveroutput on
exec dbms_output.enable(1000000);
declare
unf number;
unfb number;
fs1 number;
fs1b number;
fs2 number;
fs2b number;
fs3 number;
fs3b number;
fs4 number;
fs4b number;
full number;
fullb number;
own dba_indexes.owner%type;
ind dba_indexes.index_name%type;
par dba_ind_partitions.partition_name%type;
begin
own:=upper('&owner');
ind:=upper('&index_name');
par:=upper('&partition_name');
dbms_output.put_line('--------------------------------------------------------------------------------');
dbms_output.put_line('Owner: '||own);
dbms_output.put_line('Index: '||ind);
dbms_output.put_line('Partition: '||par);
dbms_output.put_line('------------------------------------------------');
dbms_space.space_usage(own,ind,'INDEX PARTITION',unf,unfb,fs1,fs1b,fs2,fs2b,fs3,fs3b,fs4,fs4b,full,fullb,par);
dbms_output.put_line('unf: '||unf||' fs1: '||fs1||' fs2: '||fs2||' fs3: '||fs3||' fs4: '||fs4||' full: '||full);
dbms_output.put_line('--------------------------------------------------------------------------------');
end;
/
If the value of UNF makes up the major part of the space as allocated, unpublished bug 5890312 is hit.
In a specific case a table had a value of 4GB unformatted blocks out of 4.3GB allocated blocks.
Solution
The actions as needed to resolve this issue are:
1) Install the patch for unpublished Bug 5890312 in order to prevent future occurrences
2) Resolve the current space allocation, options:
- shrink the table (partition)
- recreate the table (partition)
- recreate the index (partition)
以上关于表空间异常增长且表(或表分区)和索引(或索引分区)中分配了大量未格式化的块的描述与我们的问题十分相似，但该文档的指出可能发生该问题的版本是10.2.0.3,而我们的版本已经是10.2.0.4了，是否又一次是Oracle的"伪修复"呢?怀着疑虑我们提交了SR，希望MOS帮助我们确认是否"hit Bug 729149.1"。
来自MOS的回复:
"The bug is fixed in 10.2.0.4.3 as well as 11.1.0.6.
You might as well apply the lastest PSU available for your platform: 10.2.0.4.4 Patch:9352164
I'm afraid there is no one off (interim) patch available for 10.2.0.4 on Solaris SPARC.
It was only provided for 9.2.0.8 and 10.2.0.3.
The one off
patch 5890312 has been made available in Recommended Patch Sets (PSU): 10.2.0.4.3 and 10.2.0.4.4.
More information about Oracle Recommended Patches and their availability can be found in:
NOTE.756671.1 Oracle Recommended Patches -- Oracle Database
If you have trouble applying any of the recommended PSUs, please let me know and i will log a one off backport request on your behalf to ask development is it would be possible to provide you with the one off patch on 10.2.0.4."
看起来我们并不走运，在10.2.0.4的基础上甚至没有one-off补丁，如果要彻底解决只能打PSU。
这个Bug最早出现是在9.2.0.8上，有一个相关的说明:
Hdr: 5987262 9.2.0.8.0 RDBMS 9.2.0.8.0 SPACE PRODID-5 PORTID-197 5890312
Abstract: TABLESPACE IS ABNORMALLY INCREASED BY UNFORMATTED BLOCKS
*** 04/12/07 12:32 am ***
TAR:
----
6265975.992
PROBLEM:
--------
SQL> select count(*) from pb.PBTAJUMT;
COUNT(*)
----------
897807
SQL> select sum(bytes)/1024/1024 from dba_extents where
segment_name='PBTAJUMT';
SUM(BYTES)/1024/1024
--------------------
22008
SQL> select count(*) from dba_extents where segment_name='PBTAJUMT';
COUNT(*)
----------
5502
SQL>
UNF
----------
1404407
SQL>
FS4
----------
87
SQL>
FS3
----------
8
SQL>
FS2
----------
1
SQL>
FS1
----------
99
SQL>
FULL
----------
2291
AFTER 18 rows inserted >>
SQL> select count(*) from pb.PBTAJUMT;
COUNT(*)
----------
897825
SQL> select sum(bytes)/1024/1024 from dba_extents where
segment_name='PBTAJUMT';
SUM(BYTES)/1024/1024
--------------------
23004
SQL> select count(*) from dba_extents where segment_name='PBTAJUMT';
COUNT(*)
----------
5751
SQL>
UNF
----------
1468072
SQL>
FS4
----------
103
SQL>
FS3
----------
7
SQL>
FS2
----------
2
SQL>
FS1
----------
56
SQL>
FULL
----------
2334
After 18 rows inserted, extents increased about 1G btyes.
DIAGNOSTIC ANALYSIS:
--------------------
ALERT LOG : node2 >
Thu Apr 12 09:40:15 2007
ORA-1653: unable to extend table PB.PBTAJUMT by 256 in
tablespace PB
Thu Apr 12 09:42:48 2007
ORA-1653: unable to extend table PB.PBTAJUMT by 256 in
tablespace PB
..
Thu Apr 12 09:44:37 2007
ORA-1653: unable to extend table PB.PBTAJUMT by 256 in
tablespace PB
Thu Apr 12 09:48:06 2007
ORA-1653: unable to extend table PB.PBTAJUMT by 256 in
tablespace PB
Thu Apr 12 09:49:49 2007
alter database datafile '/dev/oravg05/roravg05_04' resize 1000m
Thu Apr 12 09:50:21 2007
Completed: alter database datafile '/dev/oravg05/roravg05_04'
Thu Apr 12 09:50:39 2007
ORA-1653: unable to extend table PB.PBTAJUMT by 256 in
tablespace PB
...
Thu Apr 12 11:43:07 2007
alter database datafile '/dev/oravg05/roravg05_04' resize 3300m
Completed: alter database datafile '/dev/oravg05/roravg05_04
Thu Apr 12 11:43:51 2007
ORA-1653: unable to extend table PB.PBTAJUMT by 256 in
tablespace PB
Thu Apr 12 11:44:35 2007
alter database datafile '/dev/oravg05/roravg05_04' resize 3500
Thu Apr 12 11:44:37 2007
ORA-3214 signalled during: alter database datafile
'/dev/oravg05/roravg05_04'...
Thu Apr 12 11:44:54 2007
alter database datafile '/dev/oravg05/roravg05_04' resize 3500m
Thu Apr 12 11:45:05 2007
Completed: alter database datafile '/dev/oravg05/roravg05_04'
kjbdb1_ora_23026.trc :: segment dump >>
After a specific extents , we can find owner is only inst 2 .
...
0x05073607 Free: 2 Inst: 2
0x05073608 Free: 1 Inst: 2
0x05482905 Free: 2 Inst: 2
0x05482906 Free: 1 Inst: 1
0x05482907 Free: 5 Inst: 1
0x05482908 Free: 5 Inst: 1
0x11400405 Free: 5 Inst: 2 <======
0x11400406 Free: 5 Inst: 2
0x11400407 Free: 5 Inst: 2
0x11400408 Free: 5 Inst: 2
0x11400505 Free: 5 Inst: 2
0x11400506 Free: 5 Inst: 2
0x11400507 Free: 5 Inst: 2
0x11400508 Free: 5 Inst: 2
0x11400605 Free: 5 Inst: 2
0x11400606 Free: 5 Inst: 2
0x11400607 Free: 5 Inst: 2
0x11400608 Free: 5 Inst: 2
0x11400705 Free: 5 Inst: 2
0x11400706 Free: 5 Inst: 2
0x11400707 Free: 5 Inst: 2
0x11400708 Free: 5 Inst: 2
0x11400805 Free: 5 Inst: 2
0x11400806 Free: 5 Inst: 2
0x11400807 Free: 5 Inst: 2
0x11400808 Free: 5 Inst: 2
0x11400905 Free: 5 Inst: 2
0x11400a05 Free: 5 Inst: 2
0x11400b05 Free: 5 Inst: 2
0x11400c05 Free: 5 Inst: 2
0x11400d05 Free: 5 Inst: 2
0x11400e05 Free: 5 Inst: 2
0x11400f05 Free: 5 Inst: 2
0x11401005 Free: 5 Inst: 2
0x11401105 Free: 5 Inst: 2
0x11401205 Free: 5 Inst: 2
0x11401305 Free: 5 Inst: 2
0x11401405 Free: 5 Inst: 2
0x11401505 Free: 5 Inst: 2
0x11401605 Free: 5 Inst: 2
0x11401705 Free: 5 Inst: 2
0x11401805 Free: 5 Inst: 2
0x11401905 Free: 5 Inst: 2
0x11401a05 Free: 5 Inst: 2
0x11401b05 Free: 5 Inst: 2
0x11401c05 Free: 5 Inst: 2
0x11401d05 Free: 5 Inst: 2
0x11401e05 Free: 5 Inst: 2
0x11401f05 Free: 5 Inst: 2
0x11402005 Free: 5 Inst: 2
0x11402105 Free: 5 Inst: 2
0x11402205 Free: 5 Inst: 2
0x11402305 Free: 5 Inst: 2
0x11402405 Free: 5 Inst: 2
0x11402505 Free: 5 Inst: 2
0x11402605 Free: 5 Inst: 2
0x11402705 Free: 5 Inst: 2
0x11402805 Free: 5 Inst: 2
0x11402905 Free: 5 Inst: 2
0x11402a05 Free: 5 Inst: 2
0x11402b05 Free: 5 Inst: 2
0x11402c05 Free: 5 Inst: 2
0x11403605 Free: 5 Inst: 2
0x11403705 Free: 5 Inst: 2
WORKAROUND:
-----------
none
RELATED BUGS:
-------------
bug.5945288, bug.4887955, bug.2447046
REPRODUCIBILITY:
----------------
No, cannot reproduce intentionally. But happened 4 times on product machine.
TEST CASE:
----------
STACK TRACE:
------------
SUPPORTING INFORMATION:
-----------------------
This system migrated on 19th of Mar. It happened 4 times from 20th of Mar.
24 HOUR CONTACT INFORMATION FOR P1 BUGS:
MOS无法重现该Bug,但可以确认该Bug的存在，具体触发的条件也不得而知。
如果您的环境无法在短期内应用补丁的话，可以通过 shrink/recreate 表(或表分区)和索引(或索引分区)来workaround这个问题，事实上我感觉这个Bug的触发条件并不简单，也就是说出现概率并不高；但如果您觉得自己不是那么幸运，那么还尽快将您的产品数据库升级到10.2.0.4.4以上或者10.2.0.5(我们是激进派)吧！