Reading in configuration from /home/sanmgmt/database_refresh/conf/prod-oracle-1a-rman.conf Running - ssh root@prod-oracle-1a.sea2.jobster.us "/usr/local/sbin/restore-from-prod-oracle.sh stop-oracle-rman-refresh -f" =============================[ Sending System Announcement ]============================ =========================[ Finished Sending System Announcement ]======================= =========================[ Stopping Oracle Listener ]=================================== LSNRCTL for Linux: Version 10.2.0.3.0 - Production on 19-MAR-2008 23:09:02 Copyright (c) 1991, 2006, Oracle. All rights reserved. Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521)) The command completed successfully ======================[ Finished Stopping Oracle Listener ]============================= =========================[ Stopping Oracle Jobster instance ]=========================== SQL*Plus: Release 10.2.0.3.0 - Production on Wed Mar 19 23:09:05 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to: Oracle Database 10g Release 10.2.0.3.0 - 64bit Production SQL> ORACLE instance shut down. SQL> Disconnected from Oracle Database 10g Release 10.2.0.3.0 - 64bit Production ====================[ Finished Stopping Oracle Jobster instance ]======================= =========================[ Stopping Oracle ASM instance ]=============================== SQL*Plus: Release 10.2.0.3.0 - Production on Wed Mar 19 23:09:06 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to: Oracle Database 10g Release 10.2.0.3.0 - 64bit Production SQL> ASM instance shutdown SQL> Disconnected from Oracle Database 10g Release 10.2.0.3.0 - 64bit Production ===================[ Finished Stopping Oracle ASM instance ]============================ =====================[ Stopping Oracle Automatic Storage Management ]=================== Unmounting ASMlib driver filesystem: [ OK ] Unloading module "oracleasm": [ OK ] ==============[ Finished Stopping Oracle Automatic Storage Management ]================= =================[ Stopping Oracle Cluster Synchronization Services ]=================== Stopping CSSD. Unable to communicate with the CSS daemon. Shutdown has begun. The daemons should exit soon. Sleeping 10 seconds for oracle stuff to exit..done sleeping =============[ Finished Stopping Oracle Cluster Synchronization Services ]============== ==============[ Attempting to Hard kill any remaining Oracle processes ]================ ===========[ Finished Attempting to Hard kill any remaining Oracle processes ]========== =====================[ Unmounting and exporting SAN filesystems ]======================= Scanning and un-mounting multipathed filesystems.....[/san/prod-e200-1/oracle/app/primary]....[/san/prod-e200-1/oracle/log/primary]....[/san/prod-e200-1/oracle/rman/backups]..done! Scanning and exporting iSCSI based volume groups.. =================[ Finished Unmounting and exporting SAN filesystems ]================== ==============================[ Stopping Multipath Services ]=========================== Stopping multipathd daemon: [ OK ] ========================[ Finished Stopping Multipath Services ]======================== ==============================[ Stopping iSCSI Subsystem ]============================== device-mapper: remove ioctl failed: Device or resource busy Command failed device-mapper: remove ioctl failed: Device or resource busy Command failed device-mapper: remove ioctl failed: Device or resource busy Command failed device-mapper: remove ioctl failed: Device or resource busy Command failed device-mapper: remove ioctl failed: Device or resource busy Command failed device-mapper: remove ioctl failed: Device or resource busy Command failed device-mapper: remove ioctl failed: Device or resource busy Command failed Searching for iscsi-based multipath maps Found 4 maps Flushing iscsi-based multipath map, 350002ac07a9903e7 Flushing iscsi-based multipath map, 350002ac07a9b03e7 Flushing iscsi-based multipath map, 350002ac07a9a03e7 Flushing iscsi-based multipath map, 350002ac02bad03e7 Stopping iscsid: [ OK ] Removing iscsi driver: [ OK ] ==========================[ Finished Stopping iSCSI Subsystem ]========================= Running - ssh prod-e200-1.sea2.jobster.us removevlun -f p-orcl-app-1a.rw 80 prod-oracle-1a Issuing removevlun p-orcl-app-1a.rw 80 prod-oracle-1a Running - ssh prod-e200-1.sea2.jobster.us removevlun -f p-orcl-db-1a.rw 81 prod-oracle-1a Issuing removevlun p-orcl-db-1a.rw 81 prod-oracle-1a Running - ssh prod-e200-1.sea2.jobster.us removevlun -f p-orcl-log-1a.rw 82 prod-oracle-1a Issuing removevlun p-orcl-log-1a.rw 82 prod-oracle-1a Removing vv p-orcl-app-1a.rw Removing vv p-orcl-log-1a.rw Removing vv p-orcl-db-1a.rw Removing vv p-orcl-app-1a.ro Removing vv p-orcl-log-1a.ro Removing vv p-orcl-db-1a.ro Running - ssh root@prod-oracle-1d.sea2.jobster.us "/usr/local/sbin/production-oracle-snapshot-hotbackup.sh remove-old-logs -f" =============================[ Removing old Archive logs ]============================== removed `/san/prod-e200-1/oracle/log/primary/arch/1_18195_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18196_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18197_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18198_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18199_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18200_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18201_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18202_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18203_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18204_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18205_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18206_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18207_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18208_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18209_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18210_634215597.dbf' removed `/san/prod-e200-1/oracle/log/primary/arch/1_18211_634215597.dbf' =======================[ Finished Removing old Archive logs ]=========================== Running - ssh root@prod-oracle-1d.sea2.jobster.us "/usr/local/sbin/production-oracle-snapshot-hotbackup.sh enable-hotbackup -f" =============================[ Sending System Announcement ]============================ =========================[ Finished Sending System Announcement ]======================= ====================[ Putting all Tablespaces into HOTBACKUP mode ]===================== System altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 1 ACTIVE 5103747264 19-MAR-08 2 ACTIVE 5103747275 19-MAR-08 3 ACTIVE 5103747269 19-MAR-08 4 ACTIVE 5103747239 19-MAR-08 5 ACTIVE 5103747227 19-MAR-08 6 ACTIVE 5103747203 19-MAR-08 7 ACTIVE 5103747233 19-MAR-08 8 ACTIVE 5103747209 19-MAR-08 9 ACTIVE 5103747253 19-MAR-08 10 ACTIVE 5103747215 19-MAR-08 11 ACTIVE 5103747221 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 12 ACTIVE 5103747259 19-MAR-08 13 ACTIVE 5103747227 19-MAR-08 14 ACTIVE 5103747227 19-MAR-08 15 ACTIVE 5103747227 19-MAR-08 16 ACTIVE 5103747227 19-MAR-08 17 ACTIVE 5103747275 19-MAR-08 18 ACTIVE 5103747275 19-MAR-08 19 ACTIVE 5103747275 19-MAR-08 20 ACTIVE 5103747275 19-MAR-08 21 ACTIVE 5103747275 19-MAR-08 22 ACTIVE 5103747275 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 23 ACTIVE 5103747275 19-MAR-08 24 ACTIVE 5103747275 19-MAR-08 25 ACTIVE 5103747275 19-MAR-08 26 ACTIVE 5103747275 19-MAR-08 27 ACTIVE 5103747275 19-MAR-08 28 ACTIVE 5103747275 19-MAR-08 29 ACTIVE 5103747275 19-MAR-08 30 ACTIVE 5103747275 19-MAR-08 31 ACTIVE 5103747233 19-MAR-08 32 ACTIVE 5103747233 19-MAR-08 33 ACTIVE 5103747233 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 34 ACTIVE 5103747233 19-MAR-08 35 ACTIVE 5103747233 19-MAR-08 36 ACTIVE 5103747233 19-MAR-08 37 ACTIVE 5103747233 19-MAR-08 38 ACTIVE 5103747233 19-MAR-08 39 ACTIVE 5103747233 19-MAR-08 40 ACTIVE 5103747233 19-MAR-08 41 ACTIVE 5103747233 19-MAR-08 42 ACTIVE 5103747233 19-MAR-08 43 ACTIVE 5103747233 19-MAR-08 44 ACTIVE 5103747233 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 45 ACTIVE 5103747233 19-MAR-08 46 ACTIVE 5103747233 19-MAR-08 47 ACTIVE 5103747233 19-MAR-08 48 ACTIVE 5103747233 19-MAR-08 49 ACTIVE 5103747233 19-MAR-08 50 ACTIVE 5103747233 19-MAR-08 51 ACTIVE 5103747233 19-MAR-08 52 ACTIVE 5103747233 19-MAR-08 53 ACTIVE 5103747233 19-MAR-08 54 ACTIVE 5103747233 19-MAR-08 55 ACTIVE 5103747233 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 56 ACTIVE 5103747233 19-MAR-08 57 ACTIVE 5103747233 19-MAR-08 58 ACTIVE 5103747233 19-MAR-08 59 ACTIVE 5103747233 19-MAR-08 60 ACTIVE 5103747233 19-MAR-08 61 ACTIVE 5103747233 19-MAR-08 62 ACTIVE 5103747233 19-MAR-08 63 ACTIVE 5103747233 19-MAR-08 64 ACTIVE 5103747233 19-MAR-08 65 ACTIVE 5103747233 19-MAR-08 66 ACTIVE 5103747233 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 67 ACTIVE 5103747233 19-MAR-08 68 ACTIVE 5103747233 19-MAR-08 69 ACTIVE 5103747233 19-MAR-08 70 ACTIVE 5103747233 19-MAR-08 71 ACTIVE 5103747233 19-MAR-08 72 ACTIVE 5103747233 19-MAR-08 73 ACTIVE 5103747233 19-MAR-08 74 ACTIVE 5103747233 19-MAR-08 75 ACTIVE 5103747233 19-MAR-08 76 ACTIVE 5103747233 19-MAR-08 77 ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 78 ACTIVE 5103747239 19-MAR-08 79 ACTIVE 5103747239 19-MAR-08 80 ACTIVE 5103747239 19-MAR-08 81 ACTIVE 5103747239 19-MAR-08 82 ACTIVE 5103747239 19-MAR-08 83 ACTIVE 5103747239 19-MAR-08 84 ACTIVE 5103747239 19-MAR-08 85 ACTIVE 5103747239 19-MAR-08 86 ACTIVE 5103747239 19-MAR-08 87 ACTIVE 5103747239 19-MAR-08 88 ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 89 ACTIVE 5103747239 19-MAR-08 90 ACTIVE 5103747239 19-MAR-08 91 ACTIVE 5103747239 19-MAR-08 92 ACTIVE 5103747239 19-MAR-08 93 ACTIVE 5103747239 19-MAR-08 94 ACTIVE 5103747239 19-MAR-08 95 ACTIVE 5103747239 19-MAR-08 96 ACTIVE 5103747239 19-MAR-08 97 ACTIVE 5103747239 19-MAR-08 98 ACTIVE 5103747239 19-MAR-08 99 ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 100 ACTIVE 5103747239 19-MAR-08 101 ACTIVE 5103747239 19-MAR-08 102 ACTIVE 5103747239 19-MAR-08 103 ACTIVE 5103747239 19-MAR-08 104 ACTIVE 5103747239 19-MAR-08 105 ACTIVE 5103747239 19-MAR-08 106 ACTIVE 5103747239 19-MAR-08 107 ACTIVE 5103747239 19-MAR-08 108 ACTIVE 5103747239 19-MAR-08 109 ACTIVE 5103747239 19-MAR-08 110 ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 111 ACTIVE 5103747239 19-MAR-08 112 ACTIVE 5103747239 19-MAR-08 113 ACTIVE 5103747239 19-MAR-08 114 ACTIVE 5103747239 19-MAR-08 115 ACTIVE 5103747239 19-MAR-08 116 ACTIVE 5103747239 19-MAR-08 117 ACTIVE 5103747239 19-MAR-08 118 ACTIVE 5103747239 19-MAR-08 119 ACTIVE 5103747239 19-MAR-08 120 ACTIVE 5103747239 19-MAR-08 121 ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 122 ACTIVE 5103747239 19-MAR-08 123 ACTIVE 5103747239 19-MAR-08 124 ACTIVE 5103747239 19-MAR-08 125 ACTIVE 5103747239 19-MAR-08 126 ACTIVE 5103747239 19-MAR-08 127 ACTIVE 5103747239 19-MAR-08 128 ACTIVE 5103747239 19-MAR-08 129 ACTIVE 5103747239 19-MAR-08 130 ACTIVE 5103747239 19-MAR-08 131 ACTIVE 5103747239 19-MAR-08 132 ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 133 ACTIVE 5103747239 19-MAR-08 134 ACTIVE 5103747239 19-MAR-08 135 ACTIVE 5103747239 19-MAR-08 136 ACTIVE 5103747239 19-MAR-08 137 ACTIVE 5103747239 19-MAR-08 138 ACTIVE 5103747239 19-MAR-08 139 ACTIVE 5103747239 19-MAR-08 140 ACTIVE 5103747239 19-MAR-08 141 ACTIVE 5103747239 19-MAR-08 142 ACTIVE 5103747239 19-MAR-08 143 ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 144 ACTIVE 5103747239 19-MAR-08 145 ACTIVE 5103747239 19-MAR-08 146 ACTIVE 5103747239 19-MAR-08 147 ACTIVE 5103747239 19-MAR-08 148 ACTIVE 5103747239 19-MAR-08 149 ACTIVE 5103747239 19-MAR-08 150 ACTIVE 5103747239 19-MAR-08 151 ACTIVE 5103747239 19-MAR-08 152 ACTIVE 5103747239 19-MAR-08 153 ACTIVE 5103747239 19-MAR-08 154 ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 155 ACTIVE 5103747239 19-MAR-08 156 ACTIVE 5103747239 19-MAR-08 157 ACTIVE 5103747239 19-MAR-08 158 ACTIVE 5103747239 19-MAR-08 159 ACTIVE 5103747239 19-MAR-08 160 ACTIVE 5103747239 19-MAR-08 160 rows selected. STATUS ------------------ ACTIVE ===============[ Finished Putting all Tablespaces into HOTBACKUP mode ]================= Running - ssh prod-e200-1.sea2.jobster.us creategroupsv -ro p-orcl-pri-app:p-orcl-app-1a.ro p-orcl-pri-log:p-orcl-log-1a.ro p-orcl-pri-db:p-orcl-db-1a.ro CopyOfVV SnapshotVV p-orcl-pri-app p-orcl-app-1a.ro p-orcl-pri-log p-orcl-log-1a.ro p-orcl-pri-db p-orcl-db-1a.ro Running - ssh prod-e200-1.sea2.jobster.us creategroupsv p-orcl-app-1a.ro:p-orcl-app-1a.rw p-orcl-log-1a.ro:p-orcl-log-1a.rw p-orcl-db-1a.ro:p-orcl-db-1a.rw CopyOfVV SnapshotVV p-orcl-app-1a.ro p-orcl-app-1a.rw p-orcl-log-1a.ro p-orcl-log-1a.rw p-orcl-db-1a.ro p-orcl-db-1a.rw Running - ssh root@prod-oracle-1d.sea2.jobster.us "/usr/local/sbin/production-oracle-snapshot-hotbackup.sh disable-hotbackup -f" ===================[ Removing all Tablespaces from HOTBACKUP mode ]===================== Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. Tablespace altered. System altered. FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 1 NOT ACTIVE 5103747264 19-MAR-08 2 NOT ACTIVE 5103747275 19-MAR-08 3 NOT ACTIVE 5103747269 19-MAR-08 4 NOT ACTIVE 5103747239 19-MAR-08 5 NOT ACTIVE 5103747227 19-MAR-08 6 NOT ACTIVE 5103747203 19-MAR-08 7 NOT ACTIVE 5103747233 19-MAR-08 8 NOT ACTIVE 5103747209 19-MAR-08 9 NOT ACTIVE 5103747253 19-MAR-08 10 NOT ACTIVE 5103747215 19-MAR-08 11 NOT ACTIVE 5103747221 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 12 NOT ACTIVE 5103747259 19-MAR-08 13 NOT ACTIVE 5103747227 19-MAR-08 14 NOT ACTIVE 5103747227 19-MAR-08 15 NOT ACTIVE 5103747227 19-MAR-08 16 NOT ACTIVE 5103747227 19-MAR-08 17 NOT ACTIVE 5103747275 19-MAR-08 18 NOT ACTIVE 5103747275 19-MAR-08 19 NOT ACTIVE 5103747275 19-MAR-08 20 NOT ACTIVE 5103747275 19-MAR-08 21 NOT ACTIVE 5103747275 19-MAR-08 22 NOT ACTIVE 5103747275 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 23 NOT ACTIVE 5103747275 19-MAR-08 24 NOT ACTIVE 5103747275 19-MAR-08 25 NOT ACTIVE 5103747275 19-MAR-08 26 NOT ACTIVE 5103747275 19-MAR-08 27 NOT ACTIVE 5103747275 19-MAR-08 28 NOT ACTIVE 5103747275 19-MAR-08 29 NOT ACTIVE 5103747275 19-MAR-08 30 NOT ACTIVE 5103747275 19-MAR-08 31 NOT ACTIVE 5103747233 19-MAR-08 32 NOT ACTIVE 5103747233 19-MAR-08 33 NOT ACTIVE 5103747233 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 34 NOT ACTIVE 5103747233 19-MAR-08 35 NOT ACTIVE 5103747233 19-MAR-08 36 NOT ACTIVE 5103747233 19-MAR-08 37 NOT ACTIVE 5103747233 19-MAR-08 38 NOT ACTIVE 5103747233 19-MAR-08 39 NOT ACTIVE 5103747233 19-MAR-08 40 NOT ACTIVE 5103747233 19-MAR-08 41 NOT ACTIVE 5103747233 19-MAR-08 42 NOT ACTIVE 5103747233 19-MAR-08 43 NOT ACTIVE 5103747233 19-MAR-08 44 NOT ACTIVE 5103747233 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 45 NOT ACTIVE 5103747233 19-MAR-08 46 NOT ACTIVE 5103747233 19-MAR-08 47 NOT ACTIVE 5103747233 19-MAR-08 48 NOT ACTIVE 5103747233 19-MAR-08 49 NOT ACTIVE 5103747233 19-MAR-08 50 NOT ACTIVE 5103747233 19-MAR-08 51 NOT ACTIVE 5103747233 19-MAR-08 52 NOT ACTIVE 5103747233 19-MAR-08 53 NOT ACTIVE 5103747233 19-MAR-08 54 NOT ACTIVE 5103747233 19-MAR-08 55 NOT ACTIVE 5103747233 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 56 NOT ACTIVE 5103747233 19-MAR-08 57 NOT ACTIVE 5103747233 19-MAR-08 58 NOT ACTIVE 5103747233 19-MAR-08 59 NOT ACTIVE 5103747233 19-MAR-08 60 NOT ACTIVE 5103747233 19-MAR-08 61 NOT ACTIVE 5103747233 19-MAR-08 62 NOT ACTIVE 5103747233 19-MAR-08 63 NOT ACTIVE 5103747233 19-MAR-08 64 NOT ACTIVE 5103747233 19-MAR-08 65 NOT ACTIVE 5103747233 19-MAR-08 66 NOT ACTIVE 5103747233 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 67 NOT ACTIVE 5103747233 19-MAR-08 68 NOT ACTIVE 5103747233 19-MAR-08 69 NOT ACTIVE 5103747233 19-MAR-08 70 NOT ACTIVE 5103747233 19-MAR-08 71 NOT ACTIVE 5103747233 19-MAR-08 72 NOT ACTIVE 5103747233 19-MAR-08 73 NOT ACTIVE 5103747233 19-MAR-08 74 NOT ACTIVE 5103747233 19-MAR-08 75 NOT ACTIVE 5103747233 19-MAR-08 76 NOT ACTIVE 5103747233 19-MAR-08 77 NOT ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 78 NOT ACTIVE 5103747239 19-MAR-08 79 NOT ACTIVE 5103747239 19-MAR-08 80 NOT ACTIVE 5103747239 19-MAR-08 81 NOT ACTIVE 5103747239 19-MAR-08 82 NOT ACTIVE 5103747239 19-MAR-08 83 NOT ACTIVE 5103747239 19-MAR-08 84 NOT ACTIVE 5103747239 19-MAR-08 85 NOT ACTIVE 5103747239 19-MAR-08 86 NOT ACTIVE 5103747239 19-MAR-08 87 NOT ACTIVE 5103747239 19-MAR-08 88 NOT ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 89 NOT ACTIVE 5103747239 19-MAR-08 90 NOT ACTIVE 5103747239 19-MAR-08 91 NOT ACTIVE 5103747239 19-MAR-08 92 NOT ACTIVE 5103747239 19-MAR-08 93 NOT ACTIVE 5103747239 19-MAR-08 94 NOT ACTIVE 5103747239 19-MAR-08 95 NOT ACTIVE 5103747239 19-MAR-08 96 NOT ACTIVE 5103747239 19-MAR-08 97 NOT ACTIVE 5103747239 19-MAR-08 98 NOT ACTIVE 5103747239 19-MAR-08 99 NOT ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 100 NOT ACTIVE 5103747239 19-MAR-08 101 NOT ACTIVE 5103747239 19-MAR-08 102 NOT ACTIVE 5103747239 19-MAR-08 103 NOT ACTIVE 5103747239 19-MAR-08 104 NOT ACTIVE 5103747239 19-MAR-08 105 NOT ACTIVE 5103747239 19-MAR-08 106 NOT ACTIVE 5103747239 19-MAR-08 107 NOT ACTIVE 5103747239 19-MAR-08 108 NOT ACTIVE 5103747239 19-MAR-08 109 NOT ACTIVE 5103747239 19-MAR-08 110 NOT ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 111 NOT ACTIVE 5103747239 19-MAR-08 112 NOT ACTIVE 5103747239 19-MAR-08 113 NOT ACTIVE 5103747239 19-MAR-08 114 NOT ACTIVE 5103747239 19-MAR-08 115 NOT ACTIVE 5103747239 19-MAR-08 116 NOT ACTIVE 5103747239 19-MAR-08 117 NOT ACTIVE 5103747239 19-MAR-08 118 NOT ACTIVE 5103747239 19-MAR-08 119 NOT ACTIVE 5103747239 19-MAR-08 120 NOT ACTIVE 5103747239 19-MAR-08 121 NOT ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 122 NOT ACTIVE 5103747239 19-MAR-08 123 NOT ACTIVE 5103747239 19-MAR-08 124 NOT ACTIVE 5103747239 19-MAR-08 125 NOT ACTIVE 5103747239 19-MAR-08 126 NOT ACTIVE 5103747239 19-MAR-08 127 NOT ACTIVE 5103747239 19-MAR-08 128 NOT ACTIVE 5103747239 19-MAR-08 129 NOT ACTIVE 5103747239 19-MAR-08 130 NOT ACTIVE 5103747239 19-MAR-08 131 NOT ACTIVE 5103747239 19-MAR-08 132 NOT ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 133 NOT ACTIVE 5103747239 19-MAR-08 134 NOT ACTIVE 5103747239 19-MAR-08 135 NOT ACTIVE 5103747239 19-MAR-08 136 NOT ACTIVE 5103747239 19-MAR-08 137 NOT ACTIVE 5103747239 19-MAR-08 138 NOT ACTIVE 5103747239 19-MAR-08 139 NOT ACTIVE 5103747239 19-MAR-08 140 NOT ACTIVE 5103747239 19-MAR-08 141 NOT ACTIVE 5103747239 19-MAR-08 142 NOT ACTIVE 5103747239 19-MAR-08 143 NOT ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 144 NOT ACTIVE 5103747239 19-MAR-08 145 NOT ACTIVE 5103747239 19-MAR-08 146 NOT ACTIVE 5103747239 19-MAR-08 147 NOT ACTIVE 5103747239 19-MAR-08 148 NOT ACTIVE 5103747239 19-MAR-08 149 NOT ACTIVE 5103747239 19-MAR-08 150 NOT ACTIVE 5103747239 19-MAR-08 151 NOT ACTIVE 5103747239 19-MAR-08 152 NOT ACTIVE 5103747239 19-MAR-08 153 NOT ACTIVE 5103747239 19-MAR-08 154 NOT ACTIVE 5103747239 19-MAR-08 FILE# STATUS CHANGE# TIME ---------- ------------------ ---------- --------- 155 NOT ACTIVE 5103747239 19-MAR-08 156 NOT ACTIVE 5103747239 19-MAR-08 157 NOT ACTIVE 5103747239 19-MAR-08 158 NOT ACTIVE 5103747239 19-MAR-08 159 NOT ACTIVE 5103747239 19-MAR-08 160 NOT ACTIVE 5103747239 19-MAR-08 160 rows selected. STATUS ------------------ NOT ACTIVE ==============[ Finished Removing all Tablespaces from HOTBACKUP mode ]================= =============================[ Sending System Announcement ]============================ =========================[ Finished Sending System Announcement ]======================= Running - ssh prod-e200-1.sea2.jobster.us createvlun p-orcl-app-1a.rw 80 prod-oracle-1a Warning: Host prod-oracle-1a has no active paths. Template may be created but no active VLUNs will be created. Running - ssh prod-e200-1.sea2.jobster.us createvlun p-orcl-db-1a.rw 81 prod-oracle-1a Warning: Host prod-oracle-1a has no active paths. Template may be created but no active VLUNs will be created. Running - ssh prod-e200-1.sea2.jobster.us createvlun p-orcl-log-1a.rw 82 prod-oracle-1a Warning: Host prod-oracle-1a has no active paths. Template may be created but no active VLUNs will be created. Running - ssh root@prod-oracle-1a.sea2.jobster.us "/usr/local/sbin/restore-from-prod-oracle.sh start-oracle-rman-refresh -f" ==============================[ Starting iSCSI Subsystem ]============================== Checking iscsi config: [ OK ] Loading iscsi driver: [ OK ] Starting iscsid: [ OK ] ==========================[ Finished Starting iSCSI Subsystem ]========================= ==============================[ Starting Multipath Services ]=========================== Stopping multipathd daemon: [FAILED] Starting multipathd daemon: 0:0:0:0: sg_io failed status 0x8 0x0 0x0 0x2 0:0:0:0: sense key 0x5 ASC 0x24 ASCQ 0x0 0:0:0:0: Unable to get INQUIRY vpd 1 page 0x0. error calling out /sbin/scsi_id -g -u -s /block/sda [ OK ] ========================[ Finished Starting Multipath Services ]======================== ==================[ Scanning, Checking and mounting SAN filesystems ]=================== Scanning and activating SAN-based volume groups PV /dev/dm-7 VG san-p-orcl-pri-log lvm2 [200.00 GB / 1020.00 MB free] PV /dev/dm-4 VG san-p-orcl-pri-app lvm2 [74.99 GB / 0 free] PV /dev/dm-6 VG san-p-orcl-rman-bkup lvm2 [1023.99 GB / 603.99 GB free] Total: 3 [1.27 TB] / in use: 3 [1.27 TB] / in no VG: 0 [0 ] Volume group "san-p-orcl-pri-log" is not exported Volume group "san-p-orcl-pri-app" is not exported Volume group "san-p-orcl-rman-bkup" is not exported Checking LVM SAN filesystems.. e2fsck 1.35 (28-Feb-2004) /dev/san-p-orcl-pri-log/san-p-orcl-pri-log: recovering journal Clearing orphaned inode 5423143 (uid=502, gid=503, mode=0100640, size=930) Clearing orphaned inode 5423144 (uid=502, gid=503, mode=0100640, size=34948) Clearing orphaned inode 5423401 (uid=502, gid=503, mode=0100640, size=899) Clearing orphaned inode 5423399 (uid=502, gid=503, mode=0100640, size=757) /dev/san-p-orcl-pri-log/san-p-orcl-pri-log: clean, 63678/26083328 files, 9895438/52166656 blocks e2fsck 1.35 (28-Feb-2004) /dev/san-p-orcl-pri-app/san-p-orcl-pri-app: recovering journal Clearing orphaned inode 3902267 (uid=502, gid=503, mode=0100660, size=0) Clearing orphaned inode 4017608 (uid=502, gid=503, mode=0100660, size=0) Clearing orphaned inode 3901214 (uid=502, gid=503, mode=0100770, size=20363086) /dev/san-p-orcl-pri-app/san-p-orcl-pri-app: clean, 43436/9830400 files, 2318847/19658752 blocks e2fsck 1.35 (28-Feb-2004) /dev/san-p-orcl-rman-bkup/san-p-orcl-rman-bkup: clean, 35/107520 files, 32424708/110100480 blocks Finished checking LVM SAN filesystems.. Scanning and mounting multipathed filesystems.....[/san/prod-e200-1/oracle/app/primary]....[/san/prod-e200-1/oracle/log/primary]....[/san/prod-e200-1/oracle/rman/backups]..done! ========================[ Finished mounting SAN filesystems ]========================== =====================[ Starting Oracle Automatic Storage Management ]=================== Loading module "oracleasm": [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ] ==============[ Finished Starting Oracle Automatic Storage Management ]================= ===============[ Configuring Oracle Cluster Synchronization Services ]================== Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Configuration for local CSS has been initialized Adding to inittab Startup will be queued to init within 30 seconds. Checking the status of new Oracle init process... Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. prod-oracle-1a CSS is active on all nodes. Oracle CSS service is installed and running under init(1M) Waiting 120 seconds for CSS to start up just incase..done waiting. Re-working /etc/inittab to work around bug with Oracle start up on boot removed `/root/.init_sed.script' done fixing /etc/inittab, hope it's good, if it's not the previous file is /etc/inittab.bak ===========[ Finished Configuring Oracle Cluster Synchronization Services ]============= =======================[ Starting up Oracle ASM instance ]============================== SQL*Plus: Release 10.2.0.3.0 - Production on Wed Mar 19 23:13:47 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to an idle instance. SQL> ASM instance started Total System Global Area 130023424 bytes Fixed Size 2071000 bytes Variable Size 102786600 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> Disconnected from Oracle Database 10g Release 10.2.0.3.0 - 64bit Production ===================[ Finished Starting up Oracle ASM instance ]========================= =====================[ Reconfiguring Oracle Jobster instance ]========================== `/san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/dbs/initjobster.ora' -> `/san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/dbs/old_initjobster.ora' create symbolic link `/san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/dbs/initjobster.ora' to `/home/oracle/conf/oracle-snapshot-configs/prod-oracle-1a/initjobster.ora_standby' `/san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/network/admin/tnsnames.ora' -> `/san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/network/admin/old_tnsnames.ora' create symbolic link `/san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/network/admin/tnsnames.ora' to `/home/oracle/conf/oracle-snapshot-configs/prod-oracle-1a/tnsnames.ora_standby' ==================[ Finished Reconfiguring Oracle Jobster instance ]==================== ==================[ Synchronizing Archive Logs with Primary ]=========================== receiving file list ... done 1_18501_634215597.dbf sent 36 bytes received 1144969 bytes 2290010.00 bytes/sec total size is 36758780416 speedup is 32103.60 receiving file list ... done sent 16 bytes received 7637 bytes 15306.00 bytes/sec total size is 36758780416 speedup is 4803185.73 =============[ Finished Synchronizing Archive Logs with Primary ]======================== =============[ Starting Oracle Jobster instance, ARCHIVELOG, END BACKUP ]=============== SQL*Plus: Release 10.2.0.3.0 - Production on Wed Mar 19 23:14:01 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to an idle instance. 23:14:04 SQL> startup mount; ORACLE instance started. Total System Global Area 805306368 bytes Fixed Size 2076136 bytes Variable Size 218104344 bytes Database Buffers 578813952 bytes Redo Buffers 6311936 bytes Database mounted. 23:14:18 SQL> recover database until cancel; ORA-00279: change 5103747203 generated at 03/19/2008 23:09:57 needed for thread 1 ORA-00289: suggestion : /san/prod-e200-1/oracle/log/primary/arch/1_18501_634215597.dbf ORA-00280: change 5103747203 for thread 1 is in sequence #18501 ORA-00279: change 5103747435 generated at 03/19/2008 23:10:03 needed for thread 1 ORA-00289: suggestion : /san/prod-e200-1/oracle/log/primary/arch/1_18502_634215597.dbf ORA-00280: change 5103747435 for thread 1 is in sequence #18502 ORA-00278: log file '/san/prod-e200-1/oracle/log/primary/arch/1_18501_634215597.dbf' no longer needed for this recovery ORA-00308: cannot open archived log '/san/prod-e200-1/oracle/log/primary/arch/1_18502_634215597.dbf' ORA-27037: unable to obtain file status Linux-x86_64 Error: 2: No such file or directory Additional information: 3 23:14:23 SQL> alter database open RESETLOGS; Database altered. 23:16:25 SQL> quit Disconnected from Oracle Database 10g Release 10.2.0.3.0 - 64bit Production ========[ Finished Starting Oracle Jobster instance, ARCHIVELOG, END BACKUP ]=========== =========================[ Starting Oracle Listener ]=================================== LSNRCTL for Linux: Version 10.2.0.3.0 - Production on 19-MAR-2008 23:16:25 Copyright (c) 1991, 2006, Oracle. All rights reserved. Starting /san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/bin/tnslsnr: please wait... TNSLSNR for Linux: Version 10.2.0.3.0 - Production Log messages written to /san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/network/log/listener.log Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=prod-oracle-1a.sea2.jobster.us)(PORT=1521))) Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521)) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 10.2.0.3.0 - Production Start Date 19-MAR-2008 23:16:25 Uptime 0 days 0 hr. 0 min. 0 sec Trace Level off Security ON: Local OS Authentication SNMP ON Listener Log File /san/prod-e200-1/oracle/app/primary/product/10.2.0/db_1/network/log/listener.log Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=prod-oracle-1a.sea2.jobster.us)(PORT=1521))) The listener supports no services The command completed successfully ======================[ Finished Starting Oracle Listener ]============================= =============================[ Sending System Announcement ]============================ =========================[ Finished Sending System Announcement ]======================= ===============================[ Deleting Old RMAN Backups ]============================ =========================[ Finished Deleting Old RMAN Backups ]========================= ================================[ Moving Old RMAN Backups ]============================= mv: cannot stat `/oracle_exports/rman-backups/*': No such file or directory ==========================[ Finished Moving Old RMAN Backups ]========================== Running - rsync -ae ssh /home/sanmgmt/logs/oracle-restore-prod-oracle-1a_20080319*.log root@prod-oracle-1a.sea2.jobster.us:/tmp/ removed `/home/sanmgmt/var/run/prod-oracle-1a.sea2.jobster.us.pid'