FAILED TO READ MORROR SIDE 1 问题描述： 环境： RAC p740 p550 -bash-3.2$ srvctl status database -d kmis Instance kmis1 is running on node p550 Instance kmis2 is not running on node p740 AIX -bash-3.2$ oslevel -r 6100-08 数据库 oracle 11.2.2

FAILED TO READ MORROR SIDE 1

问题描述：

环境：
RAC p740 p550
-bash-3.2$ srvctl status database -d kmis

Instance kmis1 is running on node p550
Instance kmis2 is not running on node p740

AIX
-bash-3.2$ oslevel -r
6100-08

数据库 oracle 11.2.2

$ srvctl start database -d kmi本文来源gaodai#ma#com搞@@代~&码网s

PRCR-1079 : Failed to start resource ora.kmis.db

CRS-5017: The resource action “ora.ARCHDG.dg start” encountered the following error:

ORA-15032: not all alterations performed

ORA-15017: diskgroup “ARCHDG” cannot be mounted

ORA-15063: ASM discovered an insufficient number of disks for diskgroup “ARCHDG”

ORA-15080: synchronous I/O operation to a disk failed

CRS-2674: Start of 'ora.ARCHDG.dg' on 'p740' failed

CRS-2632: There are no more servers to try to place resource 'ora.kmis.db' on that would satisfy its placement policy

$

$ srvctl start instance -d kmis -i kmis2 -o open;

PRCR-1013 : Failed to start resource ora.kmis.db

PRCR-1064 : Failed to start resource ora.kmis.db on node p740

CRS-5017: The resource action “ora.ARCHDG.dg start” encountered the following error:

ORA-15032: not all alterations performed

ORA-15017: diskgroup “ARCHDG” cannot be mounted

ORA-15063: ASM discovered an insufficient number of disks for diskgroup “ARCHDG”

ORA-15080: synchronous I/O operation to a disk failed

1

p550

ASMCMD> ls -lState Type Rebal NameMOUNTED EXTERN N ARCHDG/MOUNTED NORMAL N CRS/MOUNTED EXTERN N DATADG/ASMCMD> host

p740上

MOUNTED NORMAL N CRS/MOUNTED EXTERN N DATADG/挂载不上 ARCDG

一挂载就报错

odm：

Hdr: 10422126 11.2.0.1 RDBMS 11.2.0.1 ASM PRODID-5 PORTID-212Abstract: FAILED TO READ MORROR SIDE 1 <strong>*** 12/22/10 03:46 am ***</strong>---- PROBLEM:--------Alert log spooling this error continously and filling up available space.Here is the message.-----------------------WARNING: failed to read mirror side 1 of virtual extent 665 logical extent 0 of file 280 in group [1.445758315] from disk ASM_ESBQA001_DATA1_0005 allocation unit 75811 reason error; if possible,will try another mirror side----------------------- When the problem starts, errors spin in the alert log until disk space is exausted. We have found that the process that is refrenced in the trace is usually an orphaned ebiz form. Recent Changes - Upgrade to 11.2.0.1 Over the weekend we have upgraded Production Database to 11.2.0.1.  We've had 2 occurrence of the issue on 2 of the RAC nodes. The issue had happened in QA and we have tried to drop the disk and add new disk again, but it did not solve the problem. One more thing to note is, when this happens, there was a "run-away" process that also generates the same error message in a trace file. As long as that session/process is killed, the error messages stops.  Please investigate. DIAGNOSTIC ANALYSIS:--------------------This looks closely related to bug.10051315 and bug.10055063But customer has applied these patches already, even then we still see the same warning messages. We are still trying to get information from customer to understand how that session is being killed(not sure could be from database end --or application end). Customer confirmed the following :---------------1.) Are they in-house scripts scheduled in database/os which are going to kill the inactive sessions at database/os level ?-- There are not any scripts scheduled to kill sessions.  When this error starts happening, we kill the process manually. 2.) Is this happening only during specific intervals.-- I cannot see a pattern as to when it is happening.  It seems random. 3.) Please confirm - the ASM instance is functional (one can connect to it, run queries on the various v$asm% views, like: v$asm_diskgroup, v$asm_disk)-- Yes, ASM is functional during the error.  All other sessions continue to run without errors.--------------- WORKAROUND:-----------As long as that session/process is killed, the error messages stops. RELATED BUGS:-------------bug.10051315 and bug.10055063 REPRODUCIBILITY:----------------Its random. TEST CASE:----------NA STACK TRACE:------------Function List   (to Full stack)   (to Summary stack)      ksedsts <- ksdxfstk <- ksdxcb <- sspuser <- f0        <- sdbgrfucf_close_file <- sdbgrfcf_close_file <- dbgrlWriteTextOnly <- dbgrlWriteAlertDetail_int <- dbgrlWriteAlertDetail         <- dbgrlWriteAlertText <- dbkrlWriteAlertText <- dbkrlPril <- ksdfmw <- ksdwrt          <- kfioCompleteIO <- kfioWaitIO <- kfioRequestPriv <- kfioRequest <- ksfd_kfioRequest           <- ksfd_osmio <- ksfd_io <- ksfdread <- kcfrbd1 <- kcbzib            <- kcbgcur <- kturRollbackToSavepoint <- ktcrsp1 <- ksures <- ksuxds             <- kssdel <- kssdch <- ksudlp <- opidcl <- opidrv              <- sou2o <- opimai_real <- ssthrdmain <- main SUPPORTING INFORMATION:-----------------------Customer’s Management name:      Al FischerCustomer's Management 24x7 contact number:      +1-616-648-6244Customer’s Management email:      [email protected] Customer Email address:      eric.laiSecondary Contact Name:      George ZhangSecondary Contact Email:      (EMAIL ADDRESS)  [email protected] 24 HOUR CONTACT INFORMATION FOR P1 BUGS:---------------------------------------- DIAL-IN INFORMATION:-------------------- IMPACT DATE:------------" system at risk of crashing as alert logging is filling up; causing forms to hang at Call Center" <strong>*** 12/22/10 03:57 am ***</strong><strong>*** 12/22/10 04:20 am ***</strong><strong>*** 12/22/10 04:21 am ***</strong><strong>*** 12/22/10 04:21 am *** (CHG: Sta->16)</strong><strong>*** 12/22/10 04:33 am ***</strong><strong>*** 12/22/10 05:35 am ***</strong><strong>*** 12/22/10 06:08 am ***</strong><strong>*** 12/22/10 06:09 am ***</strong><strong>*** 12/22/10 06:10 am ***</strong><strong>*** 12/22/10 08:06 am ***</strong><strong>*** 12/22/10 11:25 am *** </strong><strong>*** 12/22/10 11:25 am ***</strong><strong>*** 12/22/10 01:47 pm *** (CHG: Sta->10)</strong><strong>*** 12/22/10 01:47 pm ***</strong><strong>*** 12/22/10 07:46 pm ***</strong><strong>*** 12/22/10 07:58 pm ***</strong>File versions form GI & RDBMS homes. Below is the info from GI_HOME. [usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfd.orw-rw-rw- 10000/1026  393236 Nov 15 13:29 2010 kfd.o[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfdp.orw-rw-rw- 10000/1026  271741 Nov 15 13:29 2010 kfdp.o[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfkio.orw-rw-rw- 10000/1026  103575 Nov 15 13:29 2010 kfkio.o[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfksubs.orw-rw-rw- 10000/1026  166215 Nov 15 13:29 2010 kfksubs.o   Below is the info from RDBMS_HOME [usax02w] $ echo $ORACLE_HOME/u01/apps/oracle/product/11.2.0/db_1[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfd.orw-rw-rw- 10000/1026  393236 Nov 15 13:29 2010 kfd.o[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfdp.orw-rw-rw- 10000/1026  271741 Nov 15 13:29 2010 kfdp.o[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfkio.orw-rw-rw- 10000/1026  103575 Nov 15 13:29 2010 kfkio.o[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfksubs.orw-rw-rw- 10000/1026  166215 Nov 15 13:29 2010 kfksubs.o  Regards,Prathap<strong>*** 12/22/10 08:01 pm *** (CHG: Sta->16)</strong><strong>*** 12/22/10 08:25 pm ***</strong><strong>*** 12/23/10 01:48 am ***</strong><strong>*** 12/23/10 02:46 am ***</strong><strong>*** 12/23/10 06:25 am *** (CHG: Sta->11)</strong><strong>*** 12/23/10 06:25 am ***</strong><strong>*** 12/23/10 06:26 am *** </strong><strong>*** 12/23/10 06:26 am ***</strong><strong>*** 12/24/10 01:43 am *** </strong><strong>*** 12/24/10 01:43 am ***</strong><strong>*** 12/27/10 11:33 am ***</strong><strong>*** 12/27/10 11:45 am ***</strong><strong>*** 12/27/10 11:51 am ***</strong><strong>*** 12/27/10 12:22 pm *** ESCALATED</strong><strong>*** 12/27/10 12:22 pm ***</strong><strong>*** 12/27/10 12:29 pm ***</strong><strong>*** 12/28/10 12:42 am *** (CHG: Sta->30)</strong><strong>*** 12/28/10 12:42 am ***</strong><strong>*** 12/28/10 05:00 am *** (CHG: Sta->16)</strong><strong>*** 12/28/10 05:00 am ***</strong><strong>*** 12/28/10 05:08 am *** (CHG: Sta->11)</strong><strong>*** 12/28/10 09:37 am ***</strong><strong>*** 12/29/10 02:51 am ***</strong><strong>*** 12/31/10 03:05 am ***</strong><strong>*** 01/02/11 08:28 pm ***</strong><strong>*** 01/04/11 01:43 am *** (CHG: Sta->30)</strong><strong>*** 01/04/11 01:43 am ***</strong><strong>*** 01/04/11 01:54 am ***</strong><strong>*** 01/04/11 04:50 am *** (CHG: Sta->11)</strong><strong>*** 01/04/11 04:50 am ***</strong><strong>*** 01/05/11 01:26 am *** (CHG: Sta->30)</strong><strong>*** 01/05/11 01:26 am ***</strong><strong>*** 01/10/11 10:46 am ***</strong><strong>*** 01/10/11 10:46 am *** (CHG: Sta->11)</strong><strong>*** 01/11/11 03:14 am ***</strong><strong>*** 01/13/11 11:32 pm ***</strong>RELEASE NOTES:]]  kfk SO is deleted, while there are still allocated sessions.  REDISCOVERY INFORMATION: The following error, even with fix 10051315:  WARNING: IO Failed. group:1 disk(number.incarnation):5.0xe0d14bb7  disk_path:/dev/ASM_1457_4           AU:75811 disk_offset(bytes):79494299648 io_size:8192 operation:Read  type:synchronous           result:I/O error process_id:2338940           subsys:System iop:0x110939310 bufp:0x70000016f2ec000  osderr:0x434c5344 osderr1:0x0WORKAROUND:None<strong>*** 01/13/11 11:44 pm *** </strong><strong>*** 01/13/11 11:44 pm *** </strong><strong>*** 01/13/11 11:44 pm *** (CHG: Sta->80)</strong><strong>*** 01/16/11 11:05 pm ***</strong><strong>*** 01/16/11 11:17 pm ***</strong><strong>*** 01/16/11 11:36 pm ***</strong><strong>*** 01/17/11 12:01 am ***</strong><strong>*** 01/17/11 07:09 am ***</strong><strong>*** 01/17/11 07:11 am ***</strong><strong>*** 01/17/11 09:25 am ***</strong><strong>*** 01/17/11 10:16 am ***</strong><strong>*** 01/17/11 11:14 pm ***</strong><strong>*** 01/18/11 05:06 am ***</strong><strong>*** 01/18/11 11:47 am ***</strong><strong>*** 01/18/11 11:52 pm ***</strong><strong>*** 01/19/11 02:05 am ***</strong><strong>*** 01/19/11 05:39 am ***</strong><strong>*** 01/19/11 05:45 am ***</strong><strong>*** 01/19/11 07:43 am ***</strong><strong>*** 01/19/11 08:44 am ***</strong><strong>*** 01/19/11 06:12 pm ***</strong><strong>*** 01/19/11 06:14 pm ***</strong><strong>*** 01/19/11 10:53 pm ***</strong><strong>*** 01/20/11 12:11 am ***</strong><strong>*** 01/20/11 04:04 am ***</strong><strong>*** 01/20/11 06:11 am ***</strong><strong>*** 01/20/11 06:59 am ***</strong><strong>*** 01/20/11 09:13 am ***</strong><strong>*** 01/20/11 10:44 am ***</strong><strong>*** 01/20/11 10:44 am ***</strong><strong>*** 01/20/11 11:00 am ***</strong><strong>*** 01/20/11 11:38 am ***</strong><strong>*** 01/20/11 04:58 pm ***</strong><strong>*** 01/21/11 04:05 am ***</strong><strong>*** 01/24/11 11:33 am ***</strong><strong>*** 01/25/11 02:17 am ***</strong><strong>*** 01/27/11 02:58 am ***</strong><strong>*** 01/27/11 03:16 am ***</strong><strong>*** 01/27/11 03:37 am ***</strong><strong>*** 01/27/11 03:49 am ***</strong><strong>*** 02/02/11 04:37 am ***</strong><strong>*** 02/04/11 08:25 pm ***</strong><strong>*** 02/05/11 02:46 am ***</strong><strong>*** 02/05/11 02:58 am ***</strong><strong>*** 02/05/11 02:59 am ***</strong><strong>*** 02/05/11 02:29 pm ***</strong><strong>*** 02/05/11 07:19 pm ***</strong><strong>*** 02/05/11 09:37 pm ***</strong><strong>*** 02/07/11 01:37 am ***</strong><strong>*** 02/07/11 06:14 am ***</strong><strong>*** 02/07/11 12:35 pm ***</strong><strong>*** 02/07/11 06:46 pm ***</strong><strong>*** 02/07/11 08:33 pm ***</strong><strong>*** 02/08/11 01:52 pm ***</strong><strong>*** 02/08/11 02:42 pm ***</strong><strong>*** 02/08/11 10:32 pm ***</strong><strong>*** 02/09/11 11:51 pm ***</strong><strong>*** 02/10/11 08:56 am ***</strong><strong>*** 02/10/11 08:59 am ***</strong><strong>*** 02/10/11 09:30 am ***</strong><strong>*** 02/11/11 11:52 pm ***</strong><strong>*** 02/12/11 05:10 pm ***</strong><strong>*** 02/15/11 02:55 am ***</strong><strong>*** 02/15/11 10:13 am ***</strong><strong>*** 02/16/11 01:07 am ***</strong><strong>*** 02/16/11 06:34 am ***</strong><strong>*** 02/18/11 01:39 am ***</strong><strong>*** 02/18/11 01:40 am ***</strong><strong>*** 02/24/11 07:20 am ***</strong><strong>*** 02/24/11 07:35 am ***</strong><strong>*** 02/25/11 12:34 am ***</strong><strong>*** 03/02/11 07:59 am ***</strong><strong>*** 03/02/11 08:25 am ***</strong><strong>*** 03/03/11 01:22 am ***</strong><strong>*** 03/03/11 03:34 pm ***</strong><strong>*** 03/07/11 11:54 am ***</strong><strong>*** 03/08/11 08:53 pm ***</strong><strong>*** 03/09/11 10:44 pm ***</strong><strong>*** 04/20/11 01:17 pm ***</strong><strong>*** 04/21/11 09:34 am ***</strong><strong>*** 04/21/11 01:46 pm ***</strong><strong>*** 04/22/11 03:47 pm ***</strong><strong>*** 04/29/11 12:33 pm ***</strong><strong>*** 04/29/11 01:21 pm ***</strong><strong>*** 05/02/11 07:53 am ***</strong><strong>*** 05/03/11 01:12 am ***</strong><strong>*** 05/03/11 01:37 am ***</strong><strong>*** 05/03/11 02:11 am ***</strong><strong>*** 05/11/11 03:00 am ***</strong><strong>*** 05/11/11 03:44 am ***</strong><strong>*** 05/16/11 08:07 am ***</strong><strong>*** 05/17/11 11:31 am ***</strong><strong>*** 05/17/11 11:34 am ***</strong><strong>*** 05/17/11 03:14 pm ***</strong><strong>*** 05/17/11 03:16 pm ***</strong><strong>*** 05/18/11 06:00 am ***</strong><strong>*** 05/18/11 06:11 am ***</strong><strong>*** 05/18/11 08:46 am ***</strong><strong>*** 06/09/11 12:44 pm ***</strong><strong>*** 06/10/11 10:21 am ***</strong><strong>*** 06/10/11 10:21 am ***</strong><strong>*** 06/11/11 04:44 am ***</strong><strong>*** 06/14/11 10:14 pm ***</strong><strong>*** 06/17/11 12:33 pm ***</strong><strong>*** 07/04/11 01:09 pm ***</strong><strong>*** 07/06/11 04:03 pm ***</strong><strong>*** 07/06/11 04:56 pm ***</strong><strong>*** 07/08/11 11:47 am ***</strong><strong>*** 07/08/11 12:16 pm ***</strong><strong>*** 07/12/11 07:28 am ***</strong><strong>*** 07/14/11 07:58 pm *** </strong><strong>*** 07/17/11 05:29 am ***</strong><strong>*** 07/17/11 05:30 am ***</strong><strong>*** 07/18/11 03:23 pm ***</strong><strong>*** 07/26/11 03:20 pm ***</strong><strong>*** 07/26/11 03:23 pm ***</strong><strong>*** 07/27/11 03:13 am ***</strong><strong>*** 07/27/11 03:15 am ***</strong><strong>*** 07/27/11 07:33 am ***</strong><strong>*** 08/02/11 01:34 pm ***</strong><strong>*** 08/03/11 01:03 am *** </strong><strong>*** 08/03/11 01:03 am ***</strong><strong>*** 08/03/11 01:15 am *** </strong><strong>*** 08/03/11 01:15 am ***</strong><strong>*** 08/05/11 08:00 am ***</strong><strong>*** 08/05/11 08:11 am ***</strong><strong>*** 08/05/11 10:42 am ***</strong><strong>*** 08/11/11 01:22 am ***</strong><strong>*** 08/12/11 04:00 am ***</strong><strong>*** 08/12/11 10:03 am ***</strong><strong>*** 08/18/11 12:46 pm ***</strong><strong>*** 08/22/11 12:14 pm ***</strong><strong>*** 09/07/11 04:49 am ***</strong><strong>*** 09/09/11 02:55 pm ***</strong><strong>*** 09/14/11 02:40 pm ***</strong><strong>*** 10/14/11 02:15 am ***</strong><strong>*** 10/14/11 02:15 am *** ESCALATION -> CLOSED</strong><strong>*** 10/14/11 02:15 am *** (CHG: Sta->90)</strong><strong>*** 10/14/11 02:21 am *** </strong><strong>*** 10/14/11 02:21 am ***</strong><strong>*** 10/14/11 09:56 am ***</strong><strong>*** 11/01/11 01:55 am ***</strong><strong>*** 11/01/11 10:49 am ***</strong><strong>*** 11/01/11 10:49 am ***</strong><strong>*** 11/27/11 01:15 am ***</strong><strong>*** 12/01/11 09:11 am ***</strong><strong>*** 01/17/12 03:29 pm ***</strong><strong>*** 01/23/12 08:26 pm ***</strong><strong>*** 01/23/12 08:26 pm ***</strong><strong>*** 03/14/12 01:07 am ***</strong><strong>*** 03/18/12 11:22 pm ***</strong><strong>*** 03/22/12 12:44 am ***</strong><strong>*** 04/02/12 06:10 am ***</strong><strong>*** 04/24/12 10:45 am ***</strong><strong>*** 01/26/13 08:03 am ***</strong><strong>*** 01/31/13 07:35 am ***</strong><strong>*** 02/24/13 11:27 pm ***</strong><strong>*** 04/16/13 02:14 am ***</strong><strong>*** 04/19/13 07:23 am ***</strong><strong>*** 05/12/13 09:12 pm ***</strong><strong>*** 05/20/13 06:28 am ***</strong><strong>*** 06/25/13 01:16 am ***</strong><strong>*** 07/08/13 09:41 pm ***</strong><strong>*** 07/09/13 09:44 pm ***</strong><strong>*** 07/09/13 10:24 pm ***</strong><strong>*** 07/17/13 03:02 am ***</strong><strong>*** 07/18/13 02:35 am ***</strong><strong>*** 08/12/13 03:54 am ***</strong><strong>*** 08/20/13 12:21 am ***</strong><strong>*** 09/30/13 12:26 pm ***</strong><strong>*** 10/23/13 05:08 pm ***</strong><strong>*** 10/29/13 08:22 am ***</strong><strong>*** 10/29/13 09:56 am ***</strong><strong>*** 10/29/13 09:56 am ***</strong><strong>*** 12/18/13 03:38 am ***</strong><strong>*** 01/23/14 09:24 pm ***</strong><strong>*** 02/11/14 09:58 am ***</strong><strong>*** 02/24/14 03:11 am ***</strong><strong>*** 02/24/14 06:02 am ***</strong><strong>*** 02/24/14 12:34 pm ***</strong><strong>*** 02/24/14 03:58 pm ***</strong><strong>*** 02/25/14 02:28 am ***</strong><strong>*** 03/18/14 04:34 am ***</strong><strong>*** 03/18/14 04:55 am ***</strong><strong>*** 03/18/14 05:09 am ***</strong><strong>*** 03/18/14 10:37 pm ***</strong><strong>*** 03/19/14 12:39 am ***</strong>

喜欢 (0)赏

[搞代码]

分享 (0)