Configure the Observer

Documentation is here

Search for:Fast-Start failover.

 

I have decided to put the observer on the grid control server…

 

So first step, is to configure the network to be able to see the primary and the standby from the observer.

PRD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = primary.mydomain.fr)(PORT = 1521))
)
(CONNECT_DATA =
(SID = PRD)
(SERVICE_NAME = PRD_DGMGRL.mydomain.fr)
(SERVER = DEDICATED)
)
)

STB =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = standby.mydomain.fr)(PORT = 1521))
)
(CONNECT_DATA =
(SID = STB)
(SERVICE_NAME = STB_DGMGRL.mydomain.fr)
(SERVER = DEDICATED)
)
)

STY =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = single.mydomain.fr)(PORT = 1521))
)
(CONNECT_DATA =
(SID = STY)
(SERVICE_NAME = STY_DGMGRL.mydomain.fr)
(SERVER = DEDICATED)
)
)

 

Start the observer

 

dgmgrl sys/xxxx@prd

DGMGRL> stop observer;
Done.
DGMGRL> start observer;
Observer started

A script to run with nohup is ok

Then we go to the primary

DGMGRL> show configuration verbose;

Configuration – MYPRRD

Protection Mode: MaxAvailability
Databases:
PRD – Primary database
STB – Physical standby database
STY – Physical standby database

Properties:
FastStartFailoverThreshold = ’30’
OperationTimeout = ’30’
FastStartFailoverLagLimit = ’30’
CommunicationTimeout = ‘180’
FastStartFailoverAutoReinstate = ‘TRUE’
FastStartFailoverPmyShutdown = ‘TRUE’
BystandersFollowRoleChange = ‘ALL’

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> enable fast_start failover;
Error: ORA-16651: requirements not met for enabling fast-start failover

Failed.

As we have two standbys databases we have to specify the relationship.

Let’s adjust some parameters

 

DGMGRL> edit configuration set property faststartfailoverlaglimit=20;
Property “faststartfailoverlaglimit” updated

DGMGRL> edit configuration set property faststartfailoverthreshold=10;

Property “faststartfailoverthreshold” updated
DGMGRL> edit database ‘PRD’ set property faststartfailovertarget=’STB’;
Property “faststartfailovertarget” updated
DGMGRL> edit database ‘STB’ set property faststartfailovertarget=’PRD’;
Property “faststartfailovertarget” updated
DGMGRL> enable fast_start failover;
Enabled.
DGMGRL> show configuration verbose;

Configuration – MYPRRD

Protection Mode: MaxAvailability
Databases:
PRD – Primary database
STB – (*) Physical standby database
STY – Physical standby database

(*) Fast-Start Failover target

Properties:
FastStartFailoverThreshold = ’10’
OperationTimeout = ’30’
FastStartFailoverLagLimit = ’20’
CommunicationTimeout = ‘180’
FastStartFailoverAutoReinstate = ‘TRUE’
FastStartFailoverPmyShutdown = ‘TRUE’
BystandersFollowRoleChange = ‘ALL’

Fast-Start Failover: ENABLED

Threshold: 25 seconds
Target: STB
Observer: grid
Lag Limit: 20 seconds (not in use)
Shutdown Primary: TRUE
Auto-reinstate: TRUE

Configuration Status:
SUCCESS

Test is done by disable NIC adaptor from primary server.

Here is the log of the dataguard trace file on the standby server.

Protection Mode: Maximum Availability
Fast-Start Failover (FSFO): Disabled, flags=0x0, version=0
Primary Database: PRD (0x01010000)
Standby Database: STB, Enabled Physical Standby (0x02010000)
Standby Database: STY, Enabled Physical Standby (0x03010000)
08/21/2013 08:52:30
STB version check successfully completed
STB has current configuration metadata,
completing bootstrap
Creating process RSM0

[oracle@standby trace]$
[oracle@standby trace]$ pwd
/u01/app/oracle/diag/rdbms/stb/STB/trace
[oracle@standby trace]$ tail -f drcSTB.log
Protection Mode: Maximum Availability
Fast-Start Failover (FSFO): Disabled, flags=0x0, version=0
Primary Database: PRD (0x01010000)
Standby Database: STB, Enabled Physical Standby (0x02010000)
Standby Database: STY, Enabled Physical Standby (0x03010000)
08/21/2013 08:52:30
STB version check successfully completed
STB has current configuration metadata,
completing bootstrap
Creating process RSM0

[oracle@standby trace]$ clear

[oracle@standby trace]$ tail -f drcSTB.log
Protection Mode: Maximum Availability
Fast-Start Failover (FSFO): Disabled, flags=0x0, version=0
Primary Database: PRD (0x01010000)
Standby Database: STB, Enabled Physical Standby (0x02010000)
Standby Database: STY, Enabled Physical Standby (0x03010000)
08/21/2013 08:52:30
STB version check successfully completed
STB has current configuration metadata,
completing bootstrap
Creating process RSM0
08/21/2013 20:35:13
FAILOVER TO STB
Beginning failover to database STB
Notifying Oracle Clusterware to teardown database for FAILOVER
08/21/2013 20:35:27
DMON: Old primary “PRD” needs reinstatement
08/21/2013 20:35:29
Protection mode set to MAXIMUM AVAILABILITY
08/21/2013 20:35:37
Deferring associated archivelog destinations of sites permanently disabled due to Failover
Notifying Oracle Clusterware to buildup primary database after FAILOVER
Data Guard notifying Oracle Clusterware to start services and other instances change
Posting DB_DOWN alert …
… with reason Data Guard Fast-Start Failover – Primary Disconnected
Command FAILOVER TO STB completed
08/21/2013 20:35:43
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration MYPRRD Warning ORA-16608
Primary Database STB Warning ORA-16817
Physical Standby Database PRD Error ORA-16661
Physical Standby Database STY Success ORA-00000
08/21/2013 20:36:43
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration MYPRRD Warning ORA-16608
Primary Database STB Warning ORA-16817
Physical Standby Database PRD Error ORA-16661
Physical Standby Database STY Success ORA-00000
08/21/2013 20:37:38
REINSTATE DATABASE PRD
Database PRD can be reinstated
08/21/2013 20:37:45
Command REINSTATE DATABASE PRD completed with warning ORA-16570
08/21/2013 20:37:46
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration MYPRRD Warning ORA-16608
Primary Database STB Warning ORA-16817
Physical Standby Database PRD Error ORA-16661
Physical Standby Database STY Success ORA-00000
08/21/2013 20:38:16
REINSTATE DATABASE PRD
Database PRD can be reinstated
08/21/2013 20:38:31
Successfully completed reinstatement of database 0x01001000; removing ReinstateContextArray property
08/21/2013 20:38:34
Command REINSTATE DATABASE PRD completed
EDIT DATABASE PRD SET PROPERTY ActualApplyInstance = PRD
Apply Instance for database PRD is PRD
Command EDIT DATABASE PRD SET PROPERTY ActualApplyInstance = PRD completed

 

Here is the log of the alertSTB.log file on the standby server.

Wed Aug 21 16:00:08 2013
Standby controlfile consistent with primary
RFS[2]: Selected log 7 for thread 1 sequence 260 dbid 1883658296 branch 819994683
Wed Aug 21 16:00:08 2013
Media Recovery Waiting for thread 1 sequence 260 (in transit)
Recovery of Online Redo Log: Thread 1 Group 7 Seq 260 Reading mem 0
Mem# 0: +DATA/stb/onlinelog/group_7.267.820966063
Mem# 1: +FRA/stb/onlinelog/group_7.261.820966063
Wed Aug 21 16:00:09 2013
Archived Log entry 286 added for thread 1 sequence 259 ID 0x7056ffcb dest 1:
Wed Aug 21 20:35:13 2013
Attempting Fast-Start Failover because the threshold of 25 seconds has elapsed.
Wed Aug 21 20:35:13 2013
Data Guard Broker: Beginning failover
Wed Aug 21 20:35:14 2013
NSV2 started with pid=47, OS id=18453
Wed Aug 21 20:35:21 2013
ALTER SYSTEM SET log_archive_dest_3=’service=”STY”‘,’LGWR ASYNC NOAFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name=”STY” net_timeout=30′,’valid_for=(all_logfiles,primary_role)’ SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
Wed Aug 21 20:35:21 2013
MRP0: Background Media Recovery cancelled with status 16037
Errors in file /u01/app/oracle/diag/rdbms/stb/STB/trace/STB_pr00_5626.trc:
ORA-16037: user requested cancel of managed recovery operation
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 3680259
Wed Aug 21 20:35:22 2013
MRP0: Background Media Recovery process shutdown (STB)
Managed Standby Recovery Canceled (STB)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH FORCE
Attempt to do a Terminal Recovery (STB)
Media Recovery Start: Managed Standby Recovery (STB)
started logmerger process
Wed Aug 21 20:35:23 2013
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 2 slaves
Media Recovery Waiting for thread 1 sequence 260 (in transit)
krsv_proc_kill: Killing 3 processes (all RFS, wait for I/O)
Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
Terminal Recovery timestamp is ’08/21/2013 20:35:26′
Terminal Recovery: applying standby redo logs.
Terminal Recovery: thread 1 seq# 260 redo required
Terminal Recovery:
Recovery of Online Redo Log: Thread 1 Group 7 Seq 260 Reading mem 0
Mem# 0: +DATA/stb/onlinelog/group_7.267.820966063
Mem# 1: +FRA/stb/onlinelog/group_7.261.820966063
Identified End-Of-Redo (failover) for thread 1 sequence 260 at SCN 0xffff.ffffffff
Incomplete Recovery applied until change 3680260 time 08/21/2013 20:34:46
Media Recovery Complete (STB)
Terminal Recovery: successful completion
Wed Aug 21 20:35:27 2013
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance STB – Archival Error
ORA-16014: log 7 sequence# 260 not archived, no available destinations
ORA-00312: online log 7 thread 1: ‘+DATA/stb/onlinelog/group_7.267.820966063’
ORA-00312: online log 7 thread 1: ‘+FRA/stb/onlinelog/group_7.261.820966063’
Forcing ARSCN to IRSCN for TR 0:3680260
Attempt to set limbo arscn 0:3680260 irscn 0:3680260
Resetting standby activation ID 1884749771 (0x7056ffcb)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH FORCE
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER DATABASE SWITCHOVER TO PRIMARY (STB)
Maximum wait for role transition is 15 minutes.
Backup controlfile written to trace file /u01/app/oracle/diag/rdbms/stb/STB/trace/STB_rsm0_5598.trc
Standby terminal recovery start SCN: 3680259
RESETLOGS after incomplete recovery UNTIL CHANGE 3680260
Online logfile pre-clearing operation disabled by switchover
Online log +DATA/stb/onlinelog/group_1.263.820966063: Thread 1 Group 1 was previously cleared
Online log +FRA/stb/onlinelog/group_1.257.820966063: Thread 1 Group 1 was previously cleared
Online log +DATA/stb/onlinelog/group_2.264.820966063: Thread 1 Group 2 was previously cleared
Online log +FRA/stb/onlinelog/group_2.258.820966063: Thread 1 Group 2 was previously cleared
Online log +DATA/stb/onlinelog/group_3.265.820966063: Thread 1 Group 3 was previously cleared
Online log +FRA/stb/onlinelog/group_3.259.820966063: Thread 1 Group 3 was previously cleared
Standby became primary SCN: 3680258
Wed Aug 21 20:35:27 2013
Setting recovery target incarnation to 6
Switchover: Complete – Database mounted as primary
Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
Completed: ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
ALTER SYSTEM SET log_archive_dest_state_3=’ENABLE’ SCOPE=BOTH;
ALTER DATABASE OPEN
Data Guard Broker initializing…
Wed Aug 21 20:35:27 2013
Assigning activation ID 1887675199 (0x7083a33f)
LGWR: Primary database is in MAXIMUM AVAILABILITY mode
LGWR: Destination LOG_ARCHIVE_DEST_3 is using asynchronous network I/O
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
LGWR: Destination LOG_ARCHIVE_DEST_1 is not serviced by LGWR
Thread 1 advanced to log sequence 2 (thread open)
Wed Aug 21 20:35:27 2013
ARC3: Becoming the ‘no SRL’ ARCH
ARC0: Becoming the ‘no SRL’ ARCH
ARC3: LGWR is scheduled to archive destination LOG_ARCHIVE_DEST_3 after log switch
Wed Aug 21 20:35:27 2013
NSA3 started with pid=38, OS id=18616
Thread 1 opened at log sequence 2
Current log# 2 seq# 2 mem# 0: +DATA/stb/onlinelog/group_2.264.820966063
Current log# 2 seq# 2 mem# 1: +FRA/stb/onlinelog/group_2.258.820966063
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 21 20:35:27 2013
SMON: enabling cache recovery
ARC3: Standby redo logfile selected for thread 1 sequence 1 for destination LOG_ARCHIVE_DEST_3
Archived Log entry 287 added for thread 1 sequence 1 ID 0x7083a33f dest 1:
Archiver process freed from errors. No longer stopped
[5598] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:41383024 end:41383194 diff:170 (1 seconds)
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Wed Aug 21 20:35:28 2013
ARC1: Attempting destination LOG_ARCHIVE_DEST_3 network reconnect (3113)
Starting background process QMNC
ARC1: Error 16058 attaching to RFS for reconnect
PING[ARC1]: Error 3113 when pinging standby STY.
Wed Aug 21 20:35:29 2013
QMNC started with pid=39, OS id=18618
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: ALTER DATABASE OPEN
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID=’STB’;
ALTER SYSTEM SET log_archive_format=’%t_%s_%r.dbf’ SCOPE=SPFILE SID=’STB’;
ALTER SYSTEM SET standby_file_management=’AUTO’ SCOPE=BOTH SID=’*’;
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID=’*’;
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID=’*’;
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID=’*’;
Wed Aug 21 20:35:30 2013
Starting background process CJQ0
Thread 1 advanced to log sequence 3 (LGWR switch)
Current log# 3 seq# 3 mem# 0: +DATA/stb/onlinelog/group_3.265.820966063
Current log# 3 seq# 3 mem# 1: +FRA/stb/onlinelog/group_3.259.820966063
ARC3: STARTING ARCH PROCESSES
Wed Aug 21 20:35:30 2013
CJQ0 started with pid=41, OS id=18660
Wed Aug 21 20:35:30 2013
ARC4 started with pid=43, OS id=18662
******************************************************************
LGWR: Setting ‘active’ archival for destination LOG_ARCHIVE_DEST_3
******************************************************************
LNS: Standby redo logfile selected for thread 1 sequence 3 for destination LOG_ARCHIVE_DEST_3
ARC4: Archival started
ARC3: STARTING ARCH PROCESSES COMPLETE
ARC3: Becoming the ‘no SRL’ ARCH
krsk_srl_archive_int: Enabling archival of deferred physical standby SRLs
Archived Log entry 289 added for thread 1 sequence 2 ID 0x7083a33f dest 1:
Archived Log entry 290 added for thread 1 sequence 260 ID 0x7056ffcb dest 1:
Shutting down archive processes
ARCH shutting down
ARC4: Archival stopped
ARC3: Becoming the ‘no SRL’ ARCH
ARC0: Becoming the ‘no SRL’ ARCH
ARC0: Becoming the ‘no SRL’ ARCH
ARC0: Standby redo logfile selected for thread 1 sequence 2 for destination LOG_ARCHIVE_DEST_3
ARC0: Becoming the ‘no SRL’ ARCH
Wed Aug 21 20:35:37 2013
ALTER SYSTEM SET log_archive_dest_state_3=’ENABLE’ SCOPE=BOTH;
Wed Aug 21 20:35:37 2013
ALTER SYSTEM SET log_archive_dest_state_2=’RESET’ SCOPE=BOTH;
Failover succeeded. Primary database is now STB.
Wed Aug 21 20:35:37 2013
ALTER SYSTEM SET service_names=’APP’ SCOPE=MEMORY SID=’STB’;
Wed Aug 21 20:35:39 2013
FSFP started with pid=45, OS id=18797
Wed Aug 21 20:35:46 2013

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:35:46
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

Wed Aug 21 20:35:46 2013

***********************************************************************
TNS-00513: Destination host unreachable

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))
nt secondary err code: 113
nt OS err code: 0

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:35:46
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 113
nt OS err code: 0

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:35:49
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 113
nt OS err code: 0

***********************************************************************

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:35:55
Time: 21-AUG-2013 20:35:55
Tracing not turned on.
Tracing not turned on.
Tns error struct:
Tns error struct:
ns main err code: 12543
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
ns secondary err code: 12560
nt main err code: 513
nt main err code: 513

TNS-00513: Destination host unreachable
TNS-00513: Destination host unreachable
nt secondary err code: 113
nt secondary err code: 113
nt OS err code: 0
nt OS err code: 0
Error 12543 received logging on to the standby
FAL[server, ARC0]: Error 12543 creating remote archivelog file ‘PRD’
FAL[server, ARC0]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance STB – Archival Error. Archiver continuing.
ARC0: Becoming the ‘no SRL’ ARCH
Wed Aug 21 20:35:58 2013

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:35:58
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 113
nt OS err code: 0
Error 12543 received logging on to the standby
FAL[server, ARC3]: Error 12543 creating remote archivelog file ‘PRD’
FAL[server, ARC3]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance STB – Archival Error. Archiver continuing.
Wed Aug 21 20:36:17 2013

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:36:17
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 113
nt OS err code: 0

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:36:20
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 113
nt OS err code: 0

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=primary.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SID=PRD)(SERVICE_NAME=PRD_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=standby)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:36:23
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 113
nt OS err code: 0
Error 12543 received logging on to the standby
PING[ARC1]: Heartbeat failed to connect to standby ‘PRD’. Error is 12543.
Wed Aug 21 20:37:43 2013
ALTER SYSTEM SET log_archive_dest_2=’service=”PRD”‘,’LGWR SYNC AFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=1 db_unique_name=”PRD” net_timeout=20′,’valid_for=(all_logfiles,primary_role)’ SCOPE=BOTH;
Wed Aug 21 20:37:43 2013
ARC3: Becoming the ‘no SRL’ ARCH
ALTER SYSTEM SET log_archive_dest_state_2=’ENABLE’ SCOPE=BOTH;
ALTER SYSTEM SET log_archive_dest_state_2=’RESET’ SCOPE=BOTH;
Wed Aug 21 20:37:46 2013
Destination LOG_ARCHIVE_DEST_2 no longer supports SYNCHRONIZATION
Thread 1 advanced to log sequence 4 (LGWR switch)
Current log# 1 seq# 4 mem# 0: +DATA/stb/onlinelog/group_1.263.820966063
Current log# 1 seq# 4 mem# 1: +FRA/stb/onlinelog/group_1.257.820966063
Wed Aug 21 20:37:46 2013
ARC0: Becoming the ‘no SRL’ ARCH
Wed Aug 21 20:37:46 2013
******************************************************************
LGWR: Setting ‘active’ archival for destination LOG_ARCHIVE_DEST_3
******************************************************************
Archived Log entry 294 added for thread 1 sequence 3 ID 0x7083a33f dest 1:
LNS: Standby redo logfile selected for thread 1 sequence 4 for destination LOG_ARCHIVE_DEST_3
Wed Aug 21 20:38:15 2013
ARC0: Becoming the ‘no SRL’ ARCH
Wed Aug 21 20:38:16 2013
ARC2: Becoming the ‘no SRL’ ARCH
Wed Aug 21 20:38:21 2013
ALTER SYSTEM SET log_archive_dest_state_2=’ENABLE’ SCOPE=BOTH;
Wed Aug 21 20:38:22 2013
Thread 1 cannot allocate new log, sequence 5
Checkpoint not complete
Current log# 1 seq# 4 mem# 0: +DATA/stb/onlinelog/group_1.263.820966063
Current log# 1 seq# 4 mem# 1: +FRA/stb/onlinelog/group_1.257.820966063
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
******************************************************************
LGWR: Setting ‘active’ archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
LGWR: Standby redo logfile selected to archive thread 1 sequence 5
LGWR: Standby redo logfile selected for thread 1 sequence 5 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 5 (LGWR switch)
Current log# 2 seq# 5 mem# 0: +DATA/stb/onlinelog/group_2.264.820966063
Current log# 2 seq# 5 mem# 1: +FRA/stb/onlinelog/group_2.258.820966063
Wed Aug 21 20:38:26 2013
Archived Log entry 299 added for thread 1 sequence 4 ID 0x7083a33f dest 1:
Wed Aug 21 20:38:26 2013
ARC0: Standby redo logfile selected for thread 1 sequence 4 for destination LOG_ARCHIVE_DEST_2
Wed Aug 21 20:38:26 2013
LNS: Standby redo logfile selected for thread 1 sequence 5 for destination LOG_ARCHIVE_DEST_3
ARC3: Archive log rejected (thread 1 sequence 4) at host ‘STY’
FAL[server, ARC3]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance STB – Archival Error. Archiver continuing.
Thread 1 cannot allocate new log, sequence 6
Checkpoint not complete
Current log# 2 seq# 5 mem# 0: +DATA/stb/onlinelog/group_2.264.820966063
Current log# 2 seq# 5 mem# 1: +FRA/stb/onlinelog/group_2.258.820966063
Wed Aug 21 20:38:31 2013
ALTER SYSTEM SET log_archive_dest_2=’service=”PRD”‘,’LGWR SYNC AFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name=”PRD” net_timeout=20′,’valid_for=(all_logfiles,primary_role)’ SCOPE=BOTH;
ALTER SYSTEM ARCHIVE LOG
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
******************************************************************
LGWR: Setting ‘active’ archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
LGWR: Standby redo logfile selected to archive thread 1 sequence 6
LGWR: Standby redo logfile selected for thread 1 sequence 6 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 6 (LGWR switch)
Current log# 3 seq# 6 mem# 0: +DATA/stb/onlinelog/group_3.265.820966063
Current log# 3 seq# 6 mem# 1: +FRA/stb/onlinelog/group_3.259.820966063
Archived Log entry 303 added for thread 1 sequence 5 ID 0x7083a33f dest 1:
LNS: Standby redo logfile selected for thread 1 sequence 6 for destination LOG_ARCHIVE_DEST_3
Wed Aug 21 20:40:30 2013
Starting background process SMCO
Wed Aug 21 20:40:30 2013
SMCO started with pid=36, OS id=19393

Here is the log of the dataguard trace file on the primary server.

EDIT DATABASE STB SET PROPERTY faststartfailovertarget = PRD
Command EDIT DATABASE STB SET PROPERTY faststartfailovertarget = PRD completed
08/21/2013 19:45:00
ENABLE FAST_START FAILOVER
Fast-Start Failover (FSFO) has been enabled between:
Primary = “PRD”
Standby = “STB”
08/21/2013 19:45:04
Command ENABLE FAST_START FAILOVER completed
08/21/2013 20:35:12
LGWR: FSFO SetState(“UNSYNC”, 0x2) operation requires an ack
LGWR: Notifying Oracle Clusterware to disable services and monitoring for Primary Shutdown on Failover
LGWR: Primary has heard from neither observer nor target standby
within FastStartFailoverThreshold seconds. It is
likely an automatic failover has already occurred.
The primary is shutting down.
Data Guard notifying Oracle Clusterware of database role change
08/21/2013 20:35:26
>> Starting Data Guard Broker bootstrap <<
Broker Configuration File Locations:
dg_broker_config_file1 = “+DATA/prd/dataguardconfig/dgconffile1.cfg”
dg_broker_config_file2 = “+DATA/prd/dataguardconfig/dgconffile2.cfg”
08/21/2013 20:35:29
DMON Registering service PRD_DGB with listener(s)
Broker Configuration: “MYPRRD”
Protection Mode: Maximum Availability
Fast-Start Failover (FSFO): Enabled, flags=0x40021, version=2
Primary Database: PRD (0x01010000)
Standby Database: STB, Enabled Physical Standby (FSFO target) (0x02010000)
Standby Database: STY, Enabled Physical Standby (0x03010000)
08/21/2013 20:35:33
Failed to connect to remote database STB. Error is ORA-12543
Failed to send message to site STB. Error code is ORA-12543.
08/21/2013 20:35:37
Failed to connect to remote database STY. Error is ORA-12543
Failed to send message to site STY. Error code is ORA-12543.
Version Check Results:
Database STB returned ORA-12543
Database STY returned ORA-12543
Creating process RSM0
Data Guard: detected that a role change has occurred or is likely to have occurred. This is an old primary in need of reinstatement. Database will not be opened.
08/21/2013 20:35:47
Failed to connect to remote database STB. Error is ORA-12543
Failed to connect to remote database STB. Error is ORA-12543
08/21/2013 20:36:02
Failed to connect to remote database STB. Error is ORA-12543
Failed to connect to remote database STB. Error is ORA-12543
08/21/2013 20:36:17
Failed to connect to remote database STB. Error is ORA-12543
Failed to connect to remote database STB. Error is ORA-12543
08/21/2013 20:36:32
Failed to connect to remote database STB. Error is ORA-12543
Failed to connect to remote database STB. Error is ORA-12543
Failed to connect to remote database STB. Error is ORA-12543
Failed to send message to site STB. Error code is ORA-12543.
Failed to connect to remote database STY. Error is ORA-12543
Failed to send message to site STY. Error code is ORA-12543.
08/21/2013 20:36:33
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration MYPRRD Warning ORA-16607
Primary Database PRD Error ORA-16825
Physical Standby Database STB Error ORA-12543
Physical Standby Database STY Error ORA-12543
08/21/2013 20:36:47
Failed to connect to remote database STB. Error is ORA-12543
Failed to connect to remote database STB. Error is ORA-12543
08/21/2013 20:37:33
Site STB returned ORA-16623.
Site STY returned ORA-16623.
Site STY returned ORA-16623.
Site STB returned ORA-16623.
Database STB indicates failover has occurred
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration MYPRRD Warning ORA-16607
Primary Database PRD Error ORA-16782
Physical Standby Database STB Error ORA-16623
Physical Standby Database STY Error ORA-16623
Database failover has occurred to another database
This database will be prepared for reinstatement
DMON: Old primary “PRD” needs reinstatement
08/21/2013 20:37:43
Data Guard notifying Oracle Clusterware of database role change
08/21/2013 20:37:44
Physical RSM: Reinstatement… Converting old primary control file to physical standby control file
Purging diverged redos on resetlogs branch 819994683, starting SCN 3680259
Purged 0 archived logs
08/21/2013 20:37:51
Data Guard Broker shutting down
RSM0 successfully terminated
08/21/2013 20:37:53
>> DMON Process Shutdown <<
08/21/2013 20:38:13
>> Starting Data Guard Broker bootstrap <<
Broker Configuration File Locations:
dg_broker_config_file1 = “+DATA/prd/dataguardconfig/dgconffile1.cfg”
dg_broker_config_file2 = “+DATA/prd/dataguardconfig/dgconffile2.cfg”
08/21/2013 20:38:16
Database needs to be reinstated or re-created, Data Guard broker ready
08/21/2013 20:38:17
DMON Registering service PRD_DGB with listener(s)
08/21/2013 20:38:20
Apply Instance for Database PRD set to PRD
Data Guard notifying Oracle Clusterware of database role change
Creating process RSM0
08/21/2013 20:38:33
Notifying Oracle Clusterware to buildup after REINSTATEMENT
Data Guard notifying Oracle Clusterware to start services and other instances change
Command REINSTATE DATABASE PRD completed
08/21/2013 20:38:35
Command EDIT DATABASE PRD SET PROPERTY ActualApplyInstance = PRD completed

 

Here is the log of the alertPRD.log file on the primary server.

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=standby.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=STB_DGB.mydomain.fr)(SERVICE_NAME=STB_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=primary)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:35:33
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 101
nt OS err code: 0

***********************************************************************

….

….

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=standby.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=STB_DGB.mydomain.fr)(SERVICE_NAME=STB_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=primary)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:36:47
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 101
nt OS err code: 0

***********************************************************************

Fatal NI connect error 12543, connecting to:
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=standby.mydomain.fr)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=STB_DGB.mydomain.fr)(SERVICE_NAME=STB_DGMGRL.mydomain.fr)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=primary)(USER=grid))))

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 – Production
Time: 21-AUG-2013 20:36:47
Tracing not turned on.
Tns error struct:
ns main err code: 12543

TNS-12543: TNS:destination host unreachable
ns secondary err code: 12560
nt main err code: 513

TNS-00513: Destination host unreachable
nt secondary err code: 101
nt OS err code: 0
Wed Aug 21 20:37:40 2013
NSV1 started with pid=31, OS id=6692
Wed Aug 21 20:37:43 2013
FLASHBACK DATABASE TO SCN 3680258
Flashback Restore Start
Flashback Restore Complete
Flashback Media Recovery Start
started logmerger process
Parallel Media Recovery started with 2 slaves
Wed Aug 21 20:37:44 2013
Recovery of Online Redo Log: Thread 1 Group 2 Seq 260 Reading mem 0
Mem# 0: +DATA/prd/onlinelog/group_2.262.819994685
Mem# 1: +FRA/prd/onlinelog/group_2.258.819994685
Incomplete Recovery applied until change 3680259 time 08/21/2013 20:34:47
Flashback Media Recovery Complete
Completed: FLASHBACK DATABASE TO SCN 3680258
alter database convert to physical standby
ALTER DATABASE CONVERT TO PHYSICAL STANDBY (PRD)
Flush standby redo logfile failed:1649
Clearing standby activation ID 1884749771 (0x7056ffcb)
The primary database controlfile was created using the
‘MAXLOGFILES 16’ clause.
There is space for up to 13 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE ‘srl1.f’ SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE ‘srl2.f’ SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE ‘srl3.f’ SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE ‘srl4.f’ SIZE 52428800;
Waiting for all non-current ORLs to be archived…
All non-current ORLs have been archived.
Clearing online redo logfile 1 +DATA/prd/onlinelog/group_1.261.819994683
Clearing online log 1 of thread 1 sequence number 259
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 +DATA/prd/onlinelog/group_2.262.819994685
Clearing online log 2 of thread 1 sequence number 260
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 +DATA/prd/onlinelog/group_3.263.819994685
Clearing online log 3 of thread 1 sequence number 258
Clearing online redo logfile 3 complete
Completed: alter database convert to physical standby
Wed Aug 21 20:37:45 2013
Shutting down instance (immediate)
Shutting down instance: further logons disabled
Stopping background process MMNL
Stopping background process MMON
License high water mark = 6
All dispatchers and shared servers shutdown
alter database CLOSE NORMAL
ORA-1109 signalled during: alter database CLOSE NORMAL…
alter database DISMOUNT
Shutting down archive processes
Archiving is disabled
Wed Aug 21 20:37:49 2013
NOTE: Deferred communication with ASM instance
NOTE: deferred map free for map id 5
Wed Aug 21 20:37:49 2013
NOTE: Deferred communication with ASM instance
NOTE: deferred map free for map id 146
Wed Aug 21 20:37:51 2013
NOTE: Deferred communication with ASM instance
NOTE: deferred map free for map id 2
Wed Aug 21 20:37:51 2013
NOTE: Deferred communication with ASM instance
Completed: alter database DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Shutting down Data Guard Broker processes
Wed Aug 21 20:37:53 2013
Completed: Data Guard Broker shutdown
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Wed Aug 21 20:37:55 2013
Stopping background process VKTM
Wed Aug 21 20:37:55 2013
NOTE: Shutting down MARK background process
NOTE: force a map free for map id 143
NOTE: force a map free for map id 141
NOTE: force a map free for map id 142
NOTE: force a map free for map id 140
Wed Aug 21 20:37:57 2013
Instance shutdown complete
Wed Aug 21 20:37:57 2013
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =27
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1
System name: Linux
Node name: primary
Release: 2.6.18-164.el5
Version: #1 SMP Thu Sep 3 02:16:47 EDT 2009
Machine: i686
VM name: VMWare Version: 6
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/initPRD.ora
System parameters with non-default values:
processes = 150
spfile = “+DATA/prd/spfileprd.ora”
memory_target = 812M
control_files = “+DATA/prd/controlfile/current.260.819994681”
control_files = “+FRA/prd/controlfile/current.256.819994681”
db_block_size = 8192
compatible = “11.2.0.0.0”
log_archive_dest_1 = “location=USE_DB_RECOVERY_FILE_DEST”
log_archive_dest_1 = “valid_for=(ALL_LOGFILES, ALL_ROLES)”
log_archive_dest_2 = “service=”STB””
log_archive_dest_2 = “LGWR SYNC AFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name=”STB” net_timeout=20″
log_archive_dest_2 = “valid_for=(all_logfiles,primary_role)”
log_archive_dest_3 = “service=”STY””
log_archive_dest_3 = “LGWR ASYNC NOAFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name=”STY” net_timeout=30″
log_archive_dest_3 = “valid_for=(all_logfiles,primary_role)”
log_archive_dest_state_2 = “ENABLE”
log_archive_dest_state_3 = “ENABLE”
log_archive_min_succeed_dest= 1
fal_server = “STB”
log_archive_trace = 0
log_archive_config = “dg_config=(PRD,STB,STY)”
log_archive_format = “%t_%s_%r.dbf”
log_archive_max_processes= 4
archive_lag_target = 0
db_create_file_dest = “+DATA”
db_recovery_file_dest = “+FRA”
db_recovery_file_dest_size= 9000M
standby_file_management = “AUTO”
undo_tablespace = “UNDOTBS1”
remote_login_passwordfile= “EXCLUSIVE”
db_domain = “mydomain.fr”
dispatchers = “(PROTOCOL=TCP) (SERVICE=PRDXDB)”
audit_file_dest = “/u01/app/oracle/admin/PRD/adump”
audit_trail = “DB”
db_name = “PRD”
open_cursors = 300
dg_broker_start = TRUE
dg_broker_config_file1 = “+DATA/prd/dataguardconfig/dgconffile1.cfg”
dg_broker_config_file2 = “+DATA/prd/dataguardconfig/dgconffile2.cfg”
diagnostic_dest = “/u01/app/oracle”
Wed Aug 21 20:37:58 2013
PMON started with pid=2, OS id=6841
Wed Aug 21 20:37:58 2013
PSP0 started with pid=3, OS id=6843
Wed Aug 21 20:37:59 2013
VKTM started with pid=4, OS id=6845 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Wed Aug 21 20:37:59 2013
GEN0 started with pid=5, OS id=6849
Wed Aug 21 20:37:59 2013
DIAG started with pid=6, OS id=6851
Wed Aug 21 20:37:59 2013
DBRM started with pid=7, OS id=6853
Wed Aug 21 20:38:00 2013
DIA0 started with pid=8, OS id=6855
Wed Aug 21 20:38:00 2013
MMAN started with pid=9, OS id=6857
Wed Aug 21 20:38:00 2013
DBW0 started with pid=10, OS id=6859
Wed Aug 21 20:38:00 2013
LGWR started with pid=11, OS id=6861
Wed Aug 21 20:38:00 2013
CKPT started with pid=12, OS id=6863
Wed Aug 21 20:38:00 2013
SMON started with pid=13, OS id=6865
Wed Aug 21 20:38:00 2013
RECO started with pid=14, OS id=6867
Wed Aug 21 20:38:00 2013
RBAL started with pid=15, OS id=6869
Wed Aug 21 20:38:00 2013
ASMB started with pid=16, OS id=6871
Wed Aug 21 20:38:00 2013
MMON started with pid=17, OS id=6873
Wed Aug 21 20:38:00 2013
MMNL started with pid=18, OS id=6877
starting up 1 dispatcher(s) for network address ‘(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))’…
NOTE: initiating MARK startup
starting up 1 shared server(s) …
Starting background process MARK
Wed Aug 21 20:38:00 2013
MARK started with pid=20, OS id=6881
NOTE: MARK has subscribed
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /u01/app/oracle
Wed Aug 21 20:38:00 2013
alter database mount
Wed Aug 21 20:38:00 2013
DMON started with pid=24, OS id=6896
Wed Aug 21 20:38:00 2013
ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.102.6)(PORT=1521))))’ SCOPE=MEMORY SID=’PRD’;
NOTE: Loaded library: /opt/oracle/extapi/32/asm/orcl/1/libasm.so
NOTE: Loaded library: System
SUCCESS: diskgroup DATA was mounted
SUCCESS: diskgroup FRA was mounted
NOTE: dependency between database PRD and diskgroup resource ora.DATA.dg is established
NOTE: dependency between database PRD and diskgroup resource ora.FRA.dg is established
Wed Aug 21 20:38:07 2013
NSS2 started with pid=26, OS id=6962
ARCH: STARTING ARCH PROCESSES
Wed Aug 21 20:38:07 2013
ARC0 started with pid=27, OS id=6964
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Wed Aug 21 20:38:08 2013
ARC1 started with pid=28, OS id=6967
Successful mount of redo thread 1, with mount id 1887742632
Allocated 3981204 bytes in shared pool for flashback generation buffer
Starting background process RVWR
Wed Aug 21 20:38:08 2013
ARC2 started with pid=29, OS id=6969
Wed Aug 21 20:38:08 2013
RVWR started with pid=30, OS id=6971
Wed Aug 21 20:38:08 2013
ARC3 started with pid=31, OS id=6973
ARC1: Archival started
ARC2: Archival started
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the ‘no FAL’ ARCH
ARC0: Becoming the ‘no SRL’ ARCH
ARC1: Becoming the heartbeat ARCH
Physical Standby Database mounted.
Lost write protection disabled
Completed: alter database mount
Wed Aug 21 20:38:13 2013
Starting Data Guard Broker (DMON)
Wed Aug 21 20:38:13 2013
INSV started with pid=34, OS id=6998
Wed Aug 21 20:38:17 2013
NSV1 started with pid=35, OS id=7007
Wed Aug 21 20:38:21 2013
RSM0 started with pid=36, OS id=7013
Wed Aug 21 20:38:23 2013
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
Wed Aug 21 20:38:23 2013
RFS[1]: Assigned to RFS process 7028
RFS[1]: Opened log for thread 1 sequence 260 dbid 1883658296 branch 819994683
Archived Log entry 561 added for thread 1 sequence 260 rlc 819994683 ID 0x7056ffcb dest 2:
RFS[1]: Opened log for thread 1 sequence 3 dbid 1883658296 branch 824070927
RFS[2]: Assigned to RFS process 7022
RFS[2]: Opened log for thread 1 sequence 1 dbid 1883658296 branch 824070927
Wed Aug 21 20:38:24 2013
RFS[3]: Assigned to RFS process 7042
RFS[3]: Opened log for thread 1 sequence 2 dbid 1883658296 branch 824070927
RFS[1]: New Archival REDO Branch(resetlogs_id): 824070927 Prior: 819994683
RFS[1]: Archival Activation ID: 0x7083a33f Current: 0x0
RFS[1]: Effect of primary database OPEN RESETLOGS
RFS[1]: Incarnation entry added for Branch(resetlogs_id): 824070927 (PRD)
Wed Aug 21 20:38:24 2013
Setting recovery target incarnation to 3
Archived Log entry 562 added for thread 1 sequence 3 rlc 824070927 ID 0x7083a33f dest 2:
Archived Log entry 563 added for thread 1 sequence 1 rlc 824070927 ID 0x7083a33f dest 2:
Archived Log entry 564 added for thread 1 sequence 2 rlc 824070927 ID 0x7083a33f dest 2:
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID=’PRD’;
ALTER SYSTEM SET log_archive_format=’%t_%s_%r.dbf’ SCOPE=SPFILE SID=’PRD’;
ALTER SYSTEM SET standby_file_management=’AUTO’ SCOPE=BOTH SID=’*’;
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID=’*’;
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID=’*’;
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID=’*’;
ALTER SYSTEM SET fal_server=’STB’,’STY’ SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (PRD)
Wed Aug 21 20:38:25 2013
MRP0 started with pid=41, OS id=7048
MRP0: Background Managed Standby Recovery process started (PRD)
Wed Aug 21 20:38:25 2013
Primary database is in MAXIMUM AVAILABILITY mode
Changing standby controlfile to RESYNCHRONIZATION level
Standby controlfile consistent with primary
RFS[4]: Assigned to RFS process 7050
RFS[4]: Selected log 6 for thread 1 sequence 5 dbid 1883658296 branch 824070927
Wed Aug 21 20:38:26 2013
RFS[5]: Assigned to RFS process 7052
RFS[5]: Selected log 7 for thread 1 sequence 4 dbid 1883658296 branch 824070927
Wed Aug 21 20:38:26 2013
Archived Log entry 565 added for thread 1 sequence 4 ID 0x7083a33f dest 1:
started logmerger process
Wed Aug 21 20:38:30 2013
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Media Recovery start incarnation depth : 1, target inc# : 3, irscn : 3680260
Waiting for all non-current ORLs to be archived…
All non-current ORLs have been archived.
Wed Aug 21 20:38:30 2013
ARC1: Becoming the active heartbeat ARCH
Media Recovery Log +FRA/prd/archivelog/2013_08_21/thread_1_seq_260.438.824071103
Identified End-Of-Redo (failover) for thread 1 sequence 260 at SCN 0x0.382804
Resetting standby activation ID 1884749771 (0x7056ffcb)
Media Recovery End-Of-Redo indicator encountered
Media Recovery Continuing
Media Recovery Log +FRA/prd/archivelog/2013_08_21/thread_1_seq_1.436.824071105
Media Recovery Log +FRA/prd/archivelog/2013_08_21/thread_1_seq_2.435.824071105
Wed Aug 21 20:38:31 2013
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Wed Aug 21 20:38:31 2013
Archived Log entry 566 added for thread 1 sequence 5 ID 0x7083a33f dest 1:
Media Recovery Log +FRA/prd/archivelog/2013_08_21/thread_1_seq_3.437.824071105
Media Recovery Log +FRA/prd/archivelog/2013_08_21/thread_1_seq_4.434.824071107
Media Recovery Log +FRA/prd/archivelog/2013_08_21/thread_1_seq_5.433.824071111
Media Recovery Waiting for thread 1 sequence 6
Wed Aug 21 20:38:32 2013
Primary database is in MAXIMUM AVAILABILITY mode
Changing standby controlfile to MAXIMUM AVAILABILITY level
Standby controlfile consistent with primary
RFS[6]: Assigned to RFS process 7077
RFS[6]: Selected log 6 for thread 1 sequence 6 dbid 1883658296 branch 824070927
Recovery of Online Redo Log: Thread 1 Group 6 Seq 6 Reading mem 0
Mem# 0: +DATA/prd/onlinelog/group_6.267.820964705
Mem# 1: +FRA/prd/onlinelog/group_6.273.820964705

 

And the status from the single server

DGMGRL> show configuration;

Configuration – MYPRRD

Protection Mode: MaxAvailability
Databases:
STB – Primary database
PRD – (*) Physical standby database
STY – Physical standby database

Fast-Start Failover: ENABLED

Configuration Status:
SUCCESS

 

So now we can do the switchover to put back the production to the right place.

DGMGRL> switchover to ‘PRD’;
Performing switchover NOW, please wait…
New primary database “PRD” is opening…
Operation requires shutdown of instance “STB” on database “STB”
Shutting down instance “STB”…
ORACLE instance shut down.
Operation requires startup of instance “STB” on database “STB”
Starting instance “STB”…
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is “PRD”

DGMGRL> show configuration verbose;

Configuration – MYPRRD

Protection Mode: MaxAvailability
Databases:
PRD – Primary database
STB – (*) Physical standby database
STY – Physical standby database

(*) Fast-Start Failover target

Properties:
FastStartFailoverThreshold = ’25’
OperationTimeout = ’30’
FastStartFailoverLagLimit = ’20’
CommunicationTimeout = ‘180’
FastStartFailoverAutoReinstate = ‘TRUE’
FastStartFailoverPmyShutdown = ‘TRUE’
BystandersFollowRoleChange = ‘ALL’

Fast-Start Failover: ENABLED

Threshold: 25 seconds
Target: STB
Observer: grid
Lag Limit: 20 seconds (not in use)
Shutdown Primary: TRUE
Auto-reinstate: TRUE

Configuration Status:
SUCCESS

The following two tabs change content below.

Jacques

I am Oracle Certified Master 11g & 12c database architect with significant experience in heterogeneous environments, and strong ability to lead complex and critical projects requiring multiple technical implementations. at Trivadis SA

3 thoughts on “Configure the Observer

  1. Latika

    Can I enable Fast Start Fail over without enabling flashback on the primary or standby databases.
    or any other way I can do it.

    Reply
  2. BestJett

    I see you don’t monetize jko-licorne.com, don’t waste your traffic, you can earn extra bucks every month with new monetization method.
    This is the best adsense alternative for any type of website
    (they approve all websites), for more info simply search
    in gooogle: murgrabia’s tools

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *