【ORA-16038】Oracle Rman备份故障处理

某日在检查数据库备份时发现 EMC NetWorker 某数据库备份报错,如下图:

network1


备份状态显示Fail 但完成度是100%

随后查看EMC Networker后台的报告日志

network2

日志内容如下:

已取消输出 24478 字节。  

channel t5: finished piece 1 at 13-JUL-15  piece handle=bak_level_0_HISTORYD_blqbvnoq tag=TAG20150713T214524 comment=API Version 2.0,MMS Version 1.5.0.0  

channel t5: backup set complete, elapsed time: 00:04:51  

channel t2: finished piece 1 at 13-JUL-15  piece handle=bak_level_0_HISTORYD_bkqbvnlh tag=TAG20150713T214524 comment=API Version 2.0,MMS Version 1.5.0.0  

channel t2: backup set complete, elapsed time: 00:07:26  

channel t1: finished piece 1 at 13-JUL-15  piece handle=bak_level_0_HISTORYD_bmqbvntv tag=TAG20150713T214524 comment=API Version 2.0,MMS Version 1.5.0.0  

channel t1: backup set complete, elapsed time: 00:04:55  

channel t3: finished piece 1 at 13-JUL-15  piece handle=bak_level_0_HISTORYD_bnqbvnu0 tag=TAG20150713T214524 comment=API Version 2.0,MMS Version 1.5.0.0  

channel t3: backup set complete, elapsed time: 00:04:55  

Finished backup at 13-JUL-15    

Starting Control File and SPFILE Autobackup at 13-JUL-15  piece handle=c-2367244127-20150713-01 comment=API Version 2.0,MMS Version 1.5.0.0  

Finished Control File and SPFILE Autobackup at 13-JUL-15  

RMAN-08591: WARNING: invalid archived log deletion policy    

sql statement: alter system archive log current  

released channel: t1  

released channel: t2  

released channel: t3  

released channel: t4  

released channel: t5

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of sql command on default channel at 07/13/2015 22:36:44

RMAN-11003: failure during parse/execution of SQL statement: alter system archive log current  

ORA-16038: ?? 1 sequence# 85974 ????  

ORA-19504: ??????””  

ORA-00312: ???? 1 ?? 1: ‘+DATA/test/onlinelog/group_1.597.844946543’    Recovery Manager complete.

 The backup process failed:   RMAN exited with return code ‘1’.  Exiting with error:

RMAN exited with return code ‘1’.  test:

RMAN exited with return code ‘1’.  test:/nsr/script/ora_level_0: retried 1 times. 

看来问题出在alter system archive log current  ORA-16038 日志不能正常归档导致的。

查看数据库alert日志

Tue Jul 14 14:51:07 2015

ARCH: Archival stopped, error occurred. Will continue retrying

ORACLE Instance test – Archival Error

ORA-16014:  9 sequence# 19754

ORA-00312:  9  2: ‘+DATA/test/onlinelog/group_9.17337.839522043′

日志不能正常归档?查看存储空间是否正常。

进入ASM 存储查看,果然存储空间用尽导致的日志没有空间可以归档。

ASMCMD>  lsdg

State           Type      Rebal  Sector  Block       AU    Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN    N         5        4096    1048     2097          2                       0                           2             0                     N           DATA/

为什么会造成ASM存储空间突然爆满呢?原来是配置了Oracle CDC 此数据库做为目标库在接收源库的归档日志,而接受过来的归档日志没有做日志清理,造成的空间被挤满。

还好这是一个临时测试环境,如果是一个生产环境请设置定时删除传输过来的归档日志文件,此文的重点在于排查问题的思路。

补充说明:

[oracle@test01 ~]$ oerr ora 16038
16038, 00000, “log %s sequence# %s cannot be archived”
// *Cause:  An attempt was made to archive the named file, but the
//          file could not be archived. Examine the secondary error
//          messages to determine the cause of the error.
// *Action: No action is required.

发表评论