Oracle 11gR2: Stretched Cluster

      14,688 Comments on Oracle 11gR2: Stretched Cluster

Warning!: the following article will present Stretched cluster functionality using 2 VMWAre Esxi servers.
This is NOT the Oracle supported configuration. Do NOT implement this solution for a productive environment.
 
For openfiler or dns, this link might be helpful: here
 
(Article in progress…)
 

Architecture design

2 Esxi servers
ESXi 1

  • One dns server
  • One iSCSI target server (openfiler 2.9)
  • 4 nodes (node1,node2,node3,node4)

ESXi 2

  • One dns server
  • One iSCSI target server (openfiler 2.9)
  • 4 nodes (node5,node6,node7,node8)

1 Windows 7 box with iSCSI target capabilities (fox third voting file)
 

Here is the graphical representation of the cluster


 
 

Hosts configuration

Hardware and OS

DNS servers

  • Oracle enterprise linux 5.6 x64
  • 1 VCPU
  • 640 Mb RAM
  • 20 GB HD
  • 1 NIC (192.168.1.x) public

Storage servers

  • OpenFiler 2.9 x64
  • 2 VCPU
  • 2 Gb RAM
  • 20 GB HD
  • Shared storage (see later on)
  • 2 NIC (192.168.1.x) public (192.168.102.x) storage with jumbo frame

Nodes

  • Oracle enterprise Linux 6.4 x64
  • 4 VCPU
  • 5 GB RAM
  • 50 HD
  • Shared storage configuration (see later)
  • 3 NIC (192.168.1.x) public, (192.168.101.x) private with jumbo frame, (192.168.102.x) storage with jumbo frame

 
Network configuration
DNS1: 192.168.1.5 (public network)
DNS2: 192.168.1.6 (public network)
Storage1

  • 192.168.1.10 (public network)
  • 192.168.102.5 (storage network) enable jumbo frame MTU 9000

Storage2

  • 192.168.1.11 (public network)
  • 192.168.102.6 (storage network) enable jumbo frame MTU 9000

Node1

  • 192.168.1.201 (public network)
  • 192.168.101.201 (private network) enable jumbo frame MTU 9000
  • 192.168.102.201 (storage network) enable jumbo frame MTU 9000

Node2

  • 192.168.1.202 (public network)
  • 192.168.101.202 (private network) enable jumbo frame MTU 9000
  • 192.168.102.202 (storage network) enable jumbo frame MTU 9000

Node3

  • 192.168.1.203 (public network)
  • 192.168.101.203 (private network) enable jumbo frame MTU 9000
  • 192.168.102.203 (storage network) enable jumbo frame MTU 9000

Node4

  • 192.168.1.204 (public network)
  • 192.168.101.204 (private network) enable jumbo frame MTU 9000
  • 192.168.102.204 (storage network) enable jumbo frame MTU 9000

Node5

  • 192.168.1.205 (public network)
  • 192.168.101.205 (private network) enable jumbo frame MTU 9000
  • 192.168.102.205 (storage network) enable jumbo frame MTU 9000

Node6

  • 192.168.1.206 (public network)
  • 192.168.101.206 (private network) enable jumbo frame MTU 9000
  • 192.168.102.206 (storage network) enable jumbo frame MTU 9000

Node7

  • 192.168.1.207 (public network)
  • 192.168.101.207 (private network) enable jumbo frame MTU 9000
  • 192.168.102.207 (storage network) enable jumbo frame MTU 9000

Node8

  • 192.168.1.208 (public network)
  • 192.168.101.208 (private network) enable jumbo frame MTU 9000
  • 192.168.102.208 (storage network) enable jumbo frame MTU 9000

Libraries to install on nodes

  • gcc-c++-4.4.7-3.el6.x86_64.rpm (yum install gcc)
  • compat-libcap1-1.10-1.x86_64.rpm
  • compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm
  • ksh-20100621-19.el6.x86_64.rpm
  • libaio-devel-0.3.107-10.el6.x86_64.rpm
  • libstdc++-devel-4.4.7-3.el6.x86_64.rpm

 
SSH configuration (will be done during grid infrastructure installation)

DNS Configuration

Named configuration:/var/named/chroot/etc/named.conf
// Enterprise Linux BIND Configuration Tool
// Default initial “Caching Only” name server configuration
//
options {
directory “/var/named”;
dump-file “/var/named/data/cache_dump.db”;
statistics-file “/var/named/data/named_stats.txt”;
};
// Zone for this RAC configuration is mydomain.fr
zone “mydomain.fr” in {
type master;
file “mydomain.fr.zone”;
allow-update { none; };
};
// For reverse lookups
zone “1.168.192.in-addr.fr” in {
type master;
file “1.168.192.in-addr.fr.zone”;
allow-update { none; };
};
DNS1: cat /var/named/chroot/var/named/1.168.192.in-addr.fr.zone
$TTL 1d
@ IN SOA dns1.mydomain.fr. root.mydomain.fr. (
100 ; se = serial number
8h ; ref = refresh
5m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS dns1.mydomain.fr.
201 IN PTR node1.mydomain.fr.
211 IN PTR node1-vip.mydomain.fr.
202 IN PTR node2.mydomain.fr.
212 IN PTR node2-vip.mydomain.fr.
203 IN PTR node3.mydomain.fr.
213 IN PTR node3-vip.mydomain.fr.
204 IN PTR node4.mydomain.fr.
214 IN PTR node4-vip.mydomain.fr.
205 IN PTR node5.mydomain.fr.
215 IN PTR node5-vip.mydomain.fr.
206 IN PTR node6.mydomain.fr.
216 IN PTR node6-vip.mydomain.fr.
207 IN PTR node7.mydomain.fr.
217 IN PTR node7-vip.mydomain.fr.
208 IN PTR node8.mydomain.fr.
218 IN PTR node8-vip.mydomain.fr.
201 IN PTR node1.
211 IN PTR node1-vip.
202 IN PTR node2.
212 IN PTR node2-vip.
203 IN PTR node3.
213 IN PTR node3-vip.
204 IN PTR node4.
214 IN PTR node4-vip.
205 IN PTR node5.
215 IN PTR node5-vip.
206 IN PTR node6.
216 IN PTR node6-vip.
207 IN PTR node7.
217 IN PTR node7-vip.
208 IN PTR node8.
218 IN PTR node8-vip.
; RAC Nodes SCAN VIPs in Reverse
73 IN PTR clujko-scan.mydomain.fr.
74 IN PTR clujko-scan.mydomain.fr.
75 IN PTR clujko-scan.mydomain.fr.
73 IN PTR clujko-scan.
74 IN PTR clujko-scan.
75 IN PTR clujko-scan.
DNS1: cat /var/named/chroot/var/named/mydomain.fr.zone
$TTL 1d
mydomain.fr. IN SOA dns1.mydomain.fr. root.mydomain.fr. (
100 ; se = serial number
8h ; ref = refresh
5m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS dns1.mydomain.fr.
node1 IN A 192.168.1.201
node1-vip IN A 192.168.1.211
node2 IN A 192.168.1.202
node2-vip IN A 192.168.1.212
node3 IN A 192.168.1.203
node3-vip IN A 192.168.1.213
node4 IN A 192.168.1.204
node4-vip IN A 192.168.1.214
node5 IN A 192.168.1.205
node5-vip IN A 192.168.1.215
node6 IN A 192.168.1.206
node6-vip IN A 192.168.1.216
node7 IN A 192.168.1.207
node7-vip IN A 192.168.1.217
node8 IN A 192.168.1.208
node8-vip IN A 192.168.1.218
; 3 SCAN VIPs
clujko-scan IN A 192.168.1.73
clujko-scan IN A 192.168.1.74
clujko-scan IN A 192.168.1.75
DNS2: cat /var/named/chroot/var/named/1.168.192.in-addr.fr.zone
$TTL 1d
@ IN SOA dns2.mydomain.fr. root.mydomain.fr. (
100 ; se = serial number
8h ; ref = refresh
5m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS dns2.mydomain.fr.
201 IN PTR node1.mydomain.fr.
211 IN PTR node1-vip.mydomain.fr.
202 IN PTR node2.mydomain.fr.
212 IN PTR node2-vip.mydomain.fr.
203 IN PTR node3.mydomain.fr.
213 IN PTR node3-vip.mydomain.fr.
204 IN PTR node4.mydomain.fr.
214 IN PTR node4-vip.mydomain.fr.
205 IN PTR node5.mydomain.fr.
215 IN PTR node5-vip.mydomain.fr.
206 IN PTR node6.mydomain.fr.
216 IN PTR node6-vip.mydomain.fr.
207 IN PTR node7.mydomain.fr.
217 IN PTR node7-vip.mydomain.fr.
208 IN PTR node8.mydomain.fr.
218 IN PTR node8-vip.mydomain.fr.
201 IN PTR node1.
211 IN PTR node1-vip.
202 IN PTR node2.
212 IN PTR node2-vip.
203 IN PTR node3.
213 IN PTR node3-vip.
204 IN PTR node4.
214 IN PTR node4-vip.
205 IN PTR node5.
215 IN PTR node5-vip.
206 IN PTR node6.
216 IN PTR node6-vip.
207 IN PTR node7.
217 IN PTR node7-vip.
208 IN PTR node8.
218 IN PTR node8-vip.
; 3 SCAN VIPs
73 IN PTR clujko-scan.mydomain.fr.
74 IN PTR clujko-scan.mydomain.fr.
75 IN PTR clujko-scan.mydomain.fr.
73 IN PTR clujko-scan.
74 IN PTR clujko-scan.
75 IN PTR clujko-scan.
 
DNS2: cat /var/named/chroot/var/named/mydomain.fr.zone
$TTL 1d
mydomain.fr. IN SOA dns2.mydomain.fr. root.mydomain.fr. (
100 ; se = serial number
8h ; ref = refresh
5m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS dns2.mydomain.fr.
; RAC Nodes Public name
node1 IN A 192.168.1.201
node1-vip IN A 192.168.1.211
node2 IN A 192.168.1.202
node2-vip IN A 192.168.1.212
node3 IN A 192.168.1.203
node3-vip IN A 192.168.1.213
node4 IN A 192.168.1.204
node4-vip IN A 192.168.1.214
node5 IN A 192.168.1.205
node5-vip IN A 192.168.1.215
node6 IN A 192.168.1.206
node6-vip IN A 192.168.1.216
node7 IN A 192.168.1.207
node7-vip IN A 192.168.1.217
node8 IN A 192.168.1.208
node8-vip IN A 192.168.1.218
; 3 SCAN VIPs
clujko-scan IN A 192.168.1.73
clujko-scan IN A 192.168.1.74
clujko-scan IN A 192.168.1.75
 
DNS1:service named start
DNS2:service named start
 
Users creation on Node hosts
groupadd -g 1001 oinstall
groupadd -g 1011 asmadmin
groupadd -g 1021 asmdba
groupadd -g 1031 dba
useradd -u 1100 -g oinstall -G asmadmin,asmdba,dba grid
useradd -u 1101 -g oinstall -G dba,asmdba oracle
umask 022
Folders creation on Node hosts
mkdir -p /oracle/gridbase/
mkdir -p /oracle/grid/
mkdir -p /oracle/grid/11.2.0.3
mkdir -p /oracle/db/
mkdir -p /oracle/db/11.2.0.3/
mkdir -p /oracle/oraInventory
chown -R grid:oinstall /oracle/gridbase/
chown -R grid:oinstall /oracle/grid
chown -R grid:oinstall /oracle/oraInventory
chown -R oracle:oinstall /oracle/db/

Shared storage configuration

On the 2 openfiler server define 4 targets:

  • GRID (2GB)
  • DATA (360 GB)
  • RECO (60 GB)
  • FRA (640 GB)

On the windows 7 box define 1 target:

  • GRID (2GB)

Enable iSCSI service on nodes

chkconfig –level 35 iscsi on
service iscsi restart
Update or create /etc/scsi_id.config on each nodes

# cat > /etc/scsi_id.config
vendor=”ATA”,options=-p 0x80
options=-g
Connect storage to each nodes
[root@]# iscsiadm -m discovery -t sendtargets -p 192.168.102.5
192.168.102.5:3260,1 iqn.2013-04.mydomain.fr:GRID
192.168.102.5:3260,1 iqn.2013-04.mydomain.fr:FRA
192.168.102.5:3260,1 iqn.2013-04.mydomain.fr:DATA
192.168.102.5:3260,1 iqn.2013-04.mydomain.fr:RECO
[root@]# iscsiadm -m discovery -t sendtargets -p 192.168.102.6
192.168.102.6:3260,1 iqn2.2013-04.mydomain.fr:RECO
192.168.102.6:3260,1 iqn2.2013-04.mydomain.fr:GRID
192.168.102.6:3260,1 iqn2.2013-04.mydomain.fr:FRA
192.168.102.6:3260,1 iqn2.2013-04.mydomain.fr:DATA
iscsiadm -m discovery -t sendtargets -p 192.168.102.7
192.168.102.7:3260,1 iqn3.2013-04.mydomain.fr:GRID
Connect storage 1
[root@]#iscsiadm -m node -T iqn.2013-04.mydomain.fr:GRID -p 192.168.102.5 -l
[root@]#iscsiadm -m node -T iqn.2013-04.mydomain.fr:DATA -p 192.168.102.5 -l
[root@]#iscsiadm -m node -T iqn.2013-04.mydomain.fr:FRA -p 192.168.102.5 -l
[root@]#iscsiadm -m node -T iqn.2013-04.mydomain.fr:RECO -p 192.168.102.5 -l
[root@]#iscsiadm -m node -T iqn.2013-04.mydomain.fr:GRID -p 192.168.102.5 –op update -n node.startup -v automatic
[root@]#iscsiadm -m node -T iqn.2013-04.mydomain.fr:DATA -p 192.168.102.5 –op update -n node.startup -v automatic
[root@]#iscsiadm -m node -T iqn.2013-04.mydomain.fr:FRA -p 192.168.102.5 –op update -n node.startup -v automatic
[root@]#iscsiadm -m node -T iqn.2013-04.mydomain.fr:RECO -p 192.168.102.5 –op update -n node.startup -v automatic
Connect storage 2
[root@]#iscsiadm -m node -T iqn2.2013-04.mydomain.fr:GRID -p 192.168.102.6 -l
[root@]#iscsiadm -m node -T iqn2.2013-04.mydomain.fr:DATA -p 192.168.102.6 -l
[root@]#iscsiadm -m node -T iqn2.2013-04.mydomain.fr:FRA -p 192.168.102.6 -l
[root@]#iscsiadm -m node -T iqn2.2013-04.mydomain.fr:RECO -p 192.168.102.6 -l
[root@]#iscsiadm -m node -T iqn2.2013-04.mydomain.fr:GRID -p 192.168.102.6 –op update -n node.startup -v automatic
[root@]#iscsiadm -m node -T iqn2.2013-04.mydomain.fr:DATA -p 192.168.102.6 –op update -n node.startup -v automatic
[root@]#iscsiadm -m node -T iqn2.2013-04.mydomain.fr:FRA -p 192.168.102.6 –op update -n node.startup -v automatic
[root@]#iscsiadm -m node -T iqn2.2013-04.mydomain.fr:RECO -p 192.168.102.6 –op update -n node.startup -v automatic
Connect storage 3 (for third voting file)
[root@]#iscsiadm -m node -T iqn3.2013-04.mydomain.fr:GRID -p 192.168.102.7 -l
[root@]#iscsiadm -m node -T iqn3.2013-04.mydomain.fr:GRID -p 192.168.102.7 –op update -n node.startup -v automatic
From first node create partitions using fdisk on each imported disk.
For each disk, query the iScsi ID: scsi_id -g -u -d /dev/sdx
14f504e46494c455238746c686c682d30466b512d70393770

 
Create /etc/udev/rules.d/99-oracle-asmdevices.rules file
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”14f504e46494c455238746c686c682d30466b512d70393770″, NAME+=”asmGRID1″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”14f504e46494c45524f4e64386a472d497035772d655a557a”, NAME+=”asmGRID2″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”2d116c4434a7bf37a”, NAME+=”asmGRID3″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”14f504e46494c4552774e617456732d6a6470652d32766138″, NAME+=”asmRECO1″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”14f504e46494c45526661454e44792d59646c372d666f7261″, NAME+=”asmRECO2″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”14f504e46494c455259573870414d2d4d4747642d58685254″, NAME+=”asmDATA1″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”14f504e46494c45526478385a77702d346472792d50536154″, NAME+=”asmDATA2″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”14f504e46494c455246424c7365712d5a346a632d61396653″, NAME+=”asmFRA1″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, SUBSYSTEM==”block”, ENV{DEVTYPE}==”disk”, ENV{ID_SERIAL}==”14f504e46494c4552496a46536f392d366c75332d58365364″, NAME+=”asmFRA2″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
Enable changes
/sbin/start_udev
Check changes
[root@]# ll /dev/asm*
brw-rw—- 1 grid asmadmin 65, 0 Jun 19 13:37 /dev/asmDATA1
brw-rw—- 1 grid asmadmin 8, 96 Jun 19 13:37 /dev/asmDATA2
brw-rw—- 1 grid asmadmin 8, 160 Jun 19 13:37 /dev/asmFRA1
brw-rw—- 1 grid asmadmin 65, 16 Jun 19 13:37 /dev/asmFRA2
brw-rw—- 1 grid asmadmin 8, 144 Jun 19 13:37 /dev/asmGRID1
brw-rw—- 1 grid asmadmin 8, 48 Jun 19 13:37 /dev/asmGRID2
brw-rw—- 1 grid asmadmin 8, 224 Jun 19 12:38 /dev/asmGRID3
brw-rw—- 1 grid asmadmin 8, 80 Jun 19 13:37 /dev/asmRECO1
brw-rw—- 1 grid asmadmin 8, 208 Jun 19 13:37 /dev/asmRECO2
Copy /etc/udev/rules.d/99-oracle-asmdevices.rules file to other nodes.
Run /sbin/start_udev from each nodes and check changes.
At this stage you can start the installation of the Grid infrastructure.

This process will not be described as this topic is already covered y many articles. Remember just to setup ssh connectivity for grid and oracle users.
During the installation steps, when defining ASM diskgroup for the grid infrastructure, select the 3 GRID disks and specify normal redundancy.
Targets nodes are:

  • node1
  • node2
  • node3
  • node4
  • node5
  • node6
  • node7
  • node8

At the end, create the DATA,RECO and FRA diskgroups. Here is the cluster states at the end of the installation phase:
[root@node1]# crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ONLINE ONLINE node7
ONLINE ONLINE node8
ora.FRA.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ONLINE ONLINE node7
ONLINE ONLINE node8
ora.GRID.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ONLINE ONLINE node7
ONLINE ONLINE node8
ora.LISTENER.lsnr
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ONLINE ONLINE node7
ONLINE ONLINE node8
ora.RECO.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ONLINE ONLINE node7
ONLINE ONLINE node8
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ONLINE ONLINE node3 Started
ONLINE ONLINE node4 Started
ONLINE ONLINE node5 Started
ONLINE ONLINE node6 Started
ONLINE ONLINE node7 Started
ONLINE ONLINE node8 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
OFFLINE OFFLINE node3
OFFLINE OFFLINE node4
OFFLINE OFFLINE node5
OFFLINE OFFLINE node6
OFFLINE OFFLINE node7
OFFLINE OFFLINE node8
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ONLINE ONLINE node7
ONLINE ONLINE node8
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ONLINE ONLINE node7
ONLINE ONLINE node8
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node7
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE node6
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE node8
ora.cvu
1 ONLINE ONLINE node8
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.node3.vip
1 ONLINE ONLINE node3
ora.node4.vip
1 ONLINE ONLINE node4
ora.node5.vip
1 ONLINE ONLINE node5
ora.node6.vip
1 ONLINE ONLINE node6
ora.node7.vip
1 ONLINE ONLINE node7
ora.node8.vip
1 ONLINE ONLINE node8
ora.oc4j
1 ONLINE ONLINE node5
ora.scan1.vip
1 ONLINE ONLINE node7
ora.scan2.vip
1 ONLINE ONLINE node6
ora.scan3.vip
1 ONLINE ONLINE node8
[root@node1]#
 

Features

Purpose

This demo infrastructure will host an Apex 4.2.2 demo application: http://jko-licorne.com:7001/apex/f?p=106
During normal operation, 6 nodes will be used to satisfy application needs 🙂
2 Server pools will be used (PMU and TST)
-bash-4.1$ srvctl status srvpool -g PMU
Server pool name: PMU
Active servers count: 6
-bash-4.1$ srvctl status srvpool -g TST
Server pool name: TST
Active servers count: 2
-bash-4.1$
After RDBMS installation and PMU,TST databases creation, here is the databases status:
-bash-4.1$ srvctl status srvpool -g PMU
Server pool name: PMU
Active servers count: 6
-bash-4.1$ srvctl status database -d PMU
Instance PMU_2 is running on node node1
Instance PMU_5 is running on node node2
Instance PMU_4 is running on node node3
Instance PMU_1 is running on node node4
Instance PMU_6 is running on node node5
Instance PMU_3 is running on node node6
-bash-4.1$ srvctl status database -d TST
Instance TST_1 is running on node node7
Instance TST_2 is running on node node8
 

What happen if one storage is lost?

Suppose now we lose the storage that handle GRID3 disk! This disk handle the third storage where the quorum disk is located that stored the third voting file!
When the disk is back, a query into v$asm_disk shows that the name attribute is null and the mount state is CLOSED.
In this case, as the whole LUN is down, the disk_repair time is not active and the disk is removed from the disk group immediately.
Recovery the situation was done recreating the failure group:
alter diskgroup grid add failgroup grid_0002 disk ‘/dev/asmGrid3’ force;
After the operation, crsctl query css votedisk shows again the 3 voting files.
This behavior is normal since when creating a diskgroup with asmca, the default compatible.rdbms is set to 10.1.0.0.0
In this case, the disk_repair_time parameters is not honored and therefore the whole lun is removed immediately.
To avoid this, the compatible.rdbms must be set to at least 11.2.0.0.0
Here is the chronology on what is happening in this case:
SQL> select name,MOUNT_STATUS,STATE from v$asm_disk;
NAME MOUNT_S STATE
—————————— ——- ——–
GRID_0002 CACHED NORMAL
FRA_0000 CACHED NORMAL
DATA_0000 CACHED NORMAL
FRA_0001 CACHED NORMAL
RECO_0001 CACHED NORMAL
GRID_0001 CACHED NORMAL
GRID_0000 CACHED NORMAL
DATA_0001 CACHED NORMAL
RECO_0000 CACHED NORMAL
9 rows selected.
 
root@node1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———

  1. 1. ONLINE 9c14b3478c0b4fc9bf4eb1473a6381ad (/dev/asmGRID1) [GRID]
  2. 2. ONLINE 023b136537d14fbbbf23af712bc2d71f (/dev/asmGRID2) [GRID]
  3. 3. ONLINE 470f81bd12044fffbf957fc21cc2ad56 (/dev/asmGRID3) [GRID]

Located 3 voting disk(s).


kill asmGrid3!
in alertnode1.log
2013-06-03 18:55:13.630
[cssd(3048)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/asmGRID3 will be considered not functional in 99340 milliseconds
2013-06-03 18:55:33.534
[cssd(3048)]CRS-1649:An I/O error occured for voting file: /dev/asmGRID3; details at (:CSSNM00059:) in /oracle/grid/11.2.0.3/log/node1/cssd/ocssd.log.
2013-06-03 18:55:33.534
[cssd(3048)]CRS-1649:An I/O error occured for voting file: /dev/asmGRID3; details at (:CSSNM00060:) in /oracle/grid/11.2.0.3/log/node1/cssd/ocssd.log.
2013-06-03 18:55:33.842
[cssd(3048)]CRS-1626:A Configuration change request completed successfully
2013-06-03 18:55:33.867
[cssd(3048)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 node2 node3 node4 node5 node6 node7 node8 .


SQL> select name,MOUNT_STATUS,STATE from v$asm_disk;
NAME MOUNT_S STATE
—————————— ——- ——–
CLOSED NORMAL
GRID_0002 MISSING NORMAL
FRA_0000 CACHED NORMAL
DATA_0000 CACHED NORMAL
FRA_0001 CACHED NORMAL
RECO_0001 CACHED NORMAL
GRID_0001 CACHED NORMAL
GRID_0000 CACHED NORMAL
DATA_0001 CACHED NORMAL
RECO_0000 CACHED NORMAL
10 rows selected.
 
[root@node1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———

  1. 1. ONLINE 9c14b3478c0b4fc9bf4eb1473a6381ad (/dev/asmGRID1) [GRID]
  2. 2. ONLINE 023b136537d14fbbbf23af712bc2d71f (/dev/asmGRID2) [GRID]

Located 2 voting disk(s).


Bring asmGrid3 back
SQL> select name,MOUNT_STATUS,STATE from v$asm_disk;
 
NAME MOUNT_S STATE
—————————— ——- ——–
CLOSED NORMAL
GRID_0002 MISSING NORMAL
FRA_0000 CACHED NORMAL
DATA_0000 CACHED NORMAL
FRA_0001 CACHED NORMAL
RECO_0001 CACHED NORMAL
GRID_0001 CACHED NORMAL
GRID_0000 CACHED NORMAL
DATA_0001 CACHED NORMAL
RECO_0000 CACHED NORMAL
10 rows selected.


SQL> alter diskgroup grid online all;
SQL> select name,MOUNT_STATUS,STATE from v$asm_disk;
NAME MOUNT_S STATE
—————————— ——- ——–
GRID_0002 CACHED NORMAL
FRA_0000 CACHED NORMAL
DATA_0000 CACHED NORMAL
FRA_0001 CACHED NORMAL
RECO_0001 CACHED NORMAL
GRID_0001 CACHED NORMAL
GRID_0000 CACHED NORMAL
DATA_0001 CACHED NORMAL
RECO_0000 CACHED NORMAL
9 rows selected.
 
In alertnode1.log
2013-06-03 19:03:09.731
[cssd(3048)]CRS-1605:CSSD voting file is online: /dev/asmGRID3; details in /oracle/grid/11.2.0.3/log/node1/cssd/ocssd.log.
2013-06-03 19:03:09.732
[cssd(3048)]CRS-1626:A Configuration change request completed successfully
2013-06-03 19:03:09.758
[cssd(3048)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 node2 node3 node4 node5 node6 node7 node8 .
 
[root@node1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———

  1. 1. ONLINE 9c14b3478c0b4fc9bf4eb1473a6381ad (/dev/asmGRID1) [GRID]
  2. 2. ONLINE 023b136537d14fbbbf23af712bc2d71f (/dev/asmGRID2) [GRID]
  3. 3. ONLINE 7c18879ab5c04fa5bf9b818147ecd23d (/dev/asmGRID3) [GRID]

Located 3 voting disk(s).
[root@node1 ~]#
 
Now the LUN is not dropped…
 
 

How Server pool concept works

Two Server POOLS
PMU importance=3 min=5 max=6 node list=node1,node2,node3,node4,node5,node6,node7,node8
TST importance=2 min=1 max=2 node list=node1,node2,node3,node4,node5,node6,node7,node8
 
THe following two commands set the required server pools properties:
-bash-4.1$ srvctl modify srvpool -g PMU -i 3 -l 5 -u 6
-bash-4.1$ srvctl modify srvpool -g TST -i 2 -l 1 -u 2
Let’s query db resources to check our two databases:
-bash-4.1$ crsctl status res -t -w “NAME co ‘db'”
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.pmu.db
1 ONLINE ONLINE node4 Open
2 ONLINE ONLINE node1 Open
3 ONLINE ONLINE node6 Open
4 ONLINE ONLINE node3 Open
5 ONLINE ONLINE node2 Open
6 ONLINE ONLINE node5 Open
ora.tst.db
1 ONLINE ONLINE node7 Open
2 ONLINE ONLINE node8 Open
 
PMU database details:
-bash-4.1$ crsctl status res ora.pmu.db -v
NAME=ora.pmu.db
TYPE=ora.database.type
LAST_SERVER=node4
STATE=ONLINE on node4
TARGET=ONLINE
CARDINALITY_ID=1
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 1 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:52:28
LAST_STATE_CHANGE=06/22/2013 21:52:28
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node1
STATE=ONLINE on node1
TARGET=ONLINE
CARDINALITY_ID=2
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 2 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:05
LAST_STATE_CHANGE=06/22/2013 21:53:05
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node6
STATE=ONLINE on node6
TARGET=ONLINE
CARDINALITY_ID=3
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 3 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:04
LAST_STATE_CHANGE=06/22/2013 21:53:04
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node3
STATE=ONLINE on node3
TARGET=ONLINE
CARDINALITY_ID=4
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 4 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:05
LAST_STATE_CHANGE=06/22/2013 21:53:05
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node2
STATE=ONLINE on node2
TARGET=ONLINE
CARDINALITY_ID=5
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 5 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:52:27
LAST_STATE_CHANGE=06/22/2013 21:52:27
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node5
STATE=ONLINE on node5
TARGET=ONLINE
CARDINALITY_ID=6
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 6 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:26
LAST_STATE_CHANGE=06/22/2013 21:53:26
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
TST database details:
-bash-4.1$ crsctl status res ora.tst.db -v
NAME=ora.tst.db
TYPE=ora.database.type
LAST_SERVER=node7
STATE=ONLINE on node7
TARGET=ONLINE
CARDINALITY_ID=1
CREATION_SEED=554
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.tst.db 1 1
INCARNATION=1
LAST_RESTART=06/24/2013 19:03:19
LAST_STATE_CHANGE=06/24/2013 19:03:19
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node8
STATE=ONLINE on node8
TARGET=ONLINE
CARDINALITY_ID=2
CREATION_SEED=554
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.tst.db 2 1
INCARNATION=1
LAST_RESTART=06/24/2013 19:03:19
LAST_STATE_CHANGE=06/24/2013 19:03:19
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
What happen if we lose for example node5 that belong to PMU server pool?
Here is the result of :crsctl status res -t -w “NAME co ‘db'”
-bash-4.1$ crsctl status res -t -w “NAME co ‘db'”
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.pmu.db
1 ONLINE ONLINE node4 Open
2 ONLINE ONLINE node1 Open
3 ONLINE ONLINE node6 Open
4 ONLINE ONLINE node3 Open
5 ONLINE ONLINE node2 Open
6 ONLINE OFFLINE
ora.tst.db
1 ONLINE ONLINE node7 Open
2 ONLINE ONLINE node8 Open
-bash-4.1$ crsctl status res ora.pmu.db -v
NAME=ora.pmu.db
TYPE=ora.database.type
LAST_SERVER=node4
STATE=ONLINE on node4
TARGET=ONLINE
CARDINALITY_ID=1
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 1 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:52:28
LAST_STATE_CHANGE=06/22/2013 21:52:28
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node1
STATE=ONLINE on node1
TARGET=ONLINE
CARDINALITY_ID=2
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 2 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:05
LAST_STATE_CHANGE=06/22/2013 21:53:05
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node6
STATE=ONLINE on node6
TARGET=ONLINE
CARDINALITY_ID=3
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 3 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:04
LAST_STATE_CHANGE=06/22/2013 21:53:04
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node3
STATE=ONLINE on node3
TARGET=ONLINE
CARDINALITY_ID=4
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 4 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:05
LAST_STATE_CHANGE=06/22/2013 21:53:05
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node2
STATE=ONLINE on node2
TARGET=ONLINE
CARDINALITY_ID=5
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 5 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:52:27
LAST_STATE_CHANGE=06/22/2013 21:52:27
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node5
STATE=OFFLINE
TARGET=ONLINE
CARDINALITY_ID=6
CREATION_SEED=523
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 6 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:26
LAST_STATE_CHANGE=06/24/2013 20:44:35
STATE_DETAILS=
INTERNAL_STATE=STABLE
You can see that the status is now offline on server node5 and the state_details is null.
But the interesting part is that nothing has changed except the server we have killed. Why, because the minim servers required by PMU server pool is 5 and we have still 5 servers allocated which is ok and the TST server pool is also inline with it specification.
Now suppose we kill also node2.
Here is now the result of: crsctl status res -t -w “NAME co ‘db'”
As PMU server pool is now falling bellow it minimum value, and it importance is greater that TST server pool, a server from TST server pool will be given to PMU server pool.
To do this oracle will shutdown one TST instance
-bash-4.1$ crsctl status res -t -w “NAME co ‘db'”
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.pmu.db
1 ONLINE ONLINE node4 Open
2 ONLINE ONLINE node1 Open
3 ONLINE ONLINE node6 Open
4 ONLINE ONLINE node3 Open
5 ONLINE OFFLINE
6 ONLINE OFFLINE
ora.tst.db
1 ONLINE ONLINE node7 Open,STOPPING
2 ONLINE ONLINE node8 Open
Then oracle will start the PMU instance into that server
-bash-4.1$ crsctl status res -t -w “NAME co ‘db'”
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.pmu.db
1 ONLINE ONLINE node4 Open
2 ONLINE ONLINE node1 Open
3 ONLINE ONLINE node6 Open
4 ONLINE ONLINE node3 Open
5 ONLINE OFFLINE STARTING
6 ONLINE OFFLINE
ora.tst.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE node8 Open
At the end of the process, here is the global database picture:
-bash-4.1$ crsctl status res -t -w “NAME co ‘db'”
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.pmu.db
1 ONLINE ONLINE node4 Open
2 ONLINE ONLINE node1 Open
3 ONLINE ONLINE node6 Open
4 ONLINE ONLINE node3 Open
5 ONLINE ONLINE node7 Open
6 ONLINE OFFLINE
ora.tst.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE node8 Open
Here is the detail for TST database:-bash-4.1$ crsctl status res ora.tst.db -v
See state_details for node7
-bash-4.1$ crsctl status res ora.tst.db -v
NAME=ora.tst.db
TYPE=ora.database.type
LAST_SERVER=node7
STATE=OFFLINE
TARGET=ONLINE
CARDINALITY_ID=1
CREATION_SEED=554
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.tst.db 1 1
INCARNATION=1
LAST_RESTART=06/24/2013 19:03:19
LAST_STATE_CHANGE=06/24/2013 20:52:11
STATE_DETAILS=Instance Shutdown
INTERNAL_STATE=STABLE
LAST_SERVER=node8
STATE=ONLINE on node8
TARGET=ONLINE
CARDINALITY_ID=2
CREATION_SEED=554
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.tst.db 2 1
INCARNATION=1
LAST_RESTART=06/24/2013 19:03:19
LAST_STATE_CHANGE=06/24/2013 19:03:19
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
What happen now if we bring back node5 and node 2?
-bash-4.1$ crsctl status res -t -w “NAME co ‘db'”
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.pmu.db
1 ONLINE ONLINE node4 Open
2 ONLINE ONLINE node1 Open
3 ONLINE ONLINE node6 Open
4 ONLINE ONLINE node3 Open
5 ONLINE ONLINE node7 Open
6 ONLINE ONLINE node5 Open
ora.tst.db
1 ONLINE ONLINE node2 Open
2 ONLINE ONLINE node8 Open
The two databases are back to initial states except that the location of instances is not the same. Anyway, this doesn’t matter as the most important here is the quality of the service in term of resource capacity to satisfy business needs!
In a policy managed database:

  • No fixed assignment. Cardinality-based approach (#nodes)
  • Only database and service cluster resources, no instance resources
  • You cannot directly specify where a instance will run

 
Possible wrong interpretation with policy managed database
Looking previous command result, we have our two database up and running with required amount of server. Now lets modify server pool PMU as follow:
-bash-4.1$ srvctl modify srvpool -g PMU -i 3 -l 5 -u 5
PRCS-1011 : Failed to modify server pool PMU
CRS-2736: The operation requires stopping resource ‘ora.pmu.db’ on server ‘node1’
CRS-2738: Unable to modify server pool ‘ora.PMU’ as this will affect running resources, but the force option was not specified
We cannot change the server pool because the consequences is that one instance must be shutdown. To do it, we must use the -f (force) option
-bash-4.1$ srvctl modify srvpool -g PMU -i 3 -l 5 -u 5 -f
Result is:
 
-bash-4.1$ crsctl status res -t -w “NAME co ‘db'”
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.pmu.db
1 ONLINE ONLINE node4 Open
2 ONLINE OFFLINE Instance Shutdown
3 ONLINE ONLINE node6 Open
4 ONLINE ONLINE node3 Open
5 ONLINE ONLINE node7 Open
6 ONLINE ONLINE node5 Open
ora.tst.db
1 ONLINE ONLINE node2 Open
2 ONLINE ONLINE node8 Open
And the PMU database detail, note the STATE_DETAILS value for node1!
This result could be wrongly interpreted by a monitoring tool but in fact it’s just a normal behavior when a server is removed from a server pool.
-bash-4.1$ crsctl status res ora.pmu.db -v
NAME=ora.pmu.db
TYPE=ora.database.type
LAST_SERVER=node4
STATE=ONLINE on node4
TARGET=ONLINE
CARDINALITY_ID=1
CREATION_SEED=555
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 1 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:52:28
LAST_STATE_CHANGE=06/22/2013 21:52:28
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node1
STATE=OFFLINE
TARGET=ONLINE
CARDINALITY_ID=2
CREATION_SEED=555
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 2 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:05
LAST_STATE_CHANGE=06/24/2013 21:34:55
STATE_DETAILS=Instance Shutdown
INTERNAL_STATE=STABLE
LAST_SERVER=node6
STATE=ONLINE on node6
TARGET=ONLINE
CARDINALITY_ID=3
CREATION_SEED=555
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 3 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:04
LAST_STATE_CHANGE=06/22/2013 21:53:04
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node3
STATE=ONLINE on node3
TARGET=ONLINE
CARDINALITY_ID=4
CREATION_SEED=555
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 4 1
INCARNATION=1
LAST_RESTART=06/22/2013 21:53:05
LAST_STATE_CHANGE=06/22/2013 21:53:05
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node7
STATE=ONLINE on node7
TARGET=ONLINE
CARDINALITY_ID=5
CREATION_SEED=555
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 5 1
INCARNATION=2
LAST_RESTART=06/24/2013 20:52:41
LAST_STATE_CHANGE=06/24/2013 20:52:41
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
LAST_SERVER=node5
STATE=ONLINE on node5
TARGET=ONLINE
CARDINALITY_ID=6
CREATION_SEED=555
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.pmu.db 6 1
INCARNATION=2
LAST_RESTART=06/24/2013 21:16:09
LAST_STATE_CHANGE=06/24/2013 21:16:09
STATE_DETAILS=Open
INTERNAL_STATE=STABLE
 

Services with Server pools

There is no instance assignment when creating service for a database in a server pool.
It can be created either SINGLETON (one node) or UNIFORM (all nodes).

  • SINGLETON: Service will run on only one node in the server pool (even if the service pool have more that one node)
  • UNIFORM: Service will run on all nodes that belong to the server pool.

Note:If you use Quality of Service, you must use UNIFORM service definition.
Test of SINGLETON and UNIFORM services
Creation of an SINGLETON OLTP service for our PMU database into the PMU server pool.
-bash-4.1$srvctl add service -d PMU -s OLTP -g PMU


     

14,688 thoughts on “Oracle 11gR2: Stretched Cluster

    1. Emory

      Have you seen any good films recently? ciprofloxacino da sueo The rebound for HCA surprised the market, coming after rivalhospital operator Tenet Healthcare Corp warned in Junethat patient admissions remained soft in the second quarter. InApril, smaller chain Health Management Associates Inc reduced its outlook for full-year earnings and revenue, citingweak patient admissions.

      Reply
    2. Avery

      We need someone with qualifications timeless skin care hyaluronic acid serum 100 pure OGX owns stakes in exploration and production blocks in theoffshore Campos and Santos basins as well as along Brazil’s farnortheastern coast. It also has on-shore fields and blocks whoseprimary production is natural gas. A loss of these blocks,particularly Tubarão Martelo where it hopes to start output inthe Campos Basin next month would seriously jeopardize thecompany’s chance to continue operating.

      Reply
    3. Marcelo

      I’d like to open a personal account rhinocort nevel neusspray For some investors, that new Medicare tax may make municipal bonds – on which the tax would not be due – more appealing. (Of course, you have to consider the risks that other issuers could follow Detroit’s distressing lead into bankruptcy.)

      Reply
    4. Connor

      Have you got a current driving licence? ubat provera Mejia, who is 1-2 with a 2.30 ERA in five starts, said Saturday night that he always expected to have surgery in the off-season, but he is not sure if he will now have it sooner. Collins said it is unlikely he will return this season.

      Reply
    5. Agustin

      This is the job description ivermectina para perros sobredosis Earlier this year, Co-op Bank announced heavy losses and had its credit rating cut to junk by Moody’s Investors Service. The bank, which is owned by the foods-to-funerals business Co-operative Group, recently announced that subordinated bondholders would have to swallow losses as it looks to patch up its balance sheet.

      Reply
      1. Neville

        The manager can i take claritin with prozac Any holidaymakers due to visit the country who do want to cancel or rearrange their trips will not be able to do so free of charge. All those tour operators contacted by Telegraph Travel – except Abercrombie and Kent, which said it would look at requests on a case-by-case basis – confirmed that “normal booking conditions” would apply unless the Foreign Office chose to issue a complete warning against travel to Kenya.

        Reply
      2. Ernest

        International directory enquiries levothyroxine 5mg In the meantime, artists will continue to use whatever leverage they can find to wring a higher rate out of companies like Pandora. Morgan, for his part, points to Spotify – which has only been available to U.S. listeners since 2011 – as the “oncoming good guys” who choose to give artists a bigger chunk; he claims that he makes $1 in royalties for every 200 plays on Spotify. Perhaps there is room in the Internet radio world for market competition after all.

        Reply
    6. Winston

      On another call keflex 500 mg for dogs “I think they were playing with the cat, tossing it back and forth,” said Hassan Zghari, store manager at the Key Food supermarket on 101st Ave. “It was a group of kids, and she threw it in the street.”

      Reply
      1. Eva

        Could you transfer $1000 from my current account to my deposit account? depo provera shot uses While today’s NASCAR stars rave that Larson is the real deal, many believe it’s far too soon to push him into the Sprint Cup Series and Ganassi himself has said repeatedly talk about his future is just speculation. Larson has 12 top-10 finishes in Nationwide and is eighth in the standings.

        Reply
      2. Hobert

        What’s your number? panela tramontina allegra ou solar Demand for high-tech security is particularly strong in Brazil, where violent crime is common. Not only is fingerprinting identification for cash withdrawal machines becoming popular, but banks and consumers are requesting an extra layer of security that allows the machines to analyze blood vessels in the finger to recognize whether the person to whom it belongs is alive or dead.

        Reply
      3. Cornelius

        I’m sorry, he’s manfaat saw palmetto untuk wanita Mefloquine was first developed in the 1970s at the U.S. Department of Defense’s Walter Reed Army Institute of Research as a synthetic analogue of quinine, the first effective treatment for malaria. It was licensed in 1989 by the FDA for use against chloroquine-resistant malaria.

        Reply
      4. Teodoro

        I work for a publishers mirtazapine sandoz odt 15 But the real kicker come in the post-processing. If you choose to crop an image after the fact, Nokia uses a feature that it calls re-framing. Instead of letting you choose a section to zoom in on and deleting the rest of the photo, it will create a locked-in zoom setting for a photo, and leave it that way every time you view it — but it won’t delete the parts of the photo you can’t see. If you decide you want to revisit the full photo later, you can simply tap a button and re-frame the shot.

        Reply
      5. Marlin

        How much were you paid in your last job? propecia prix tunisie “And, yes, we do have to stay on the ground. That’s why we spent 18 months on a shape that will travel in a straight line on the ground. If we don’t leave wheel tracks, we haven’t got a record.”

        Reply
      1. Felton

        Could I order a new chequebook, please? recyclage batterie lithium Global markets were coming under pressure as a shutdown ofthe U.S. government seemed increasingly likely, although theeuro had political troubles of its own as the Italian governmentteetered on the edge of collapse.

        Reply
        1. Lamont

          When do you want me to start? cost of acyclovir Heat a grill or grill-pan to medium-high heat. Brush the halibut fillets with the coconut oil and sprinkle it evenly with salt and pepper. Grill the fillets for 2 minutes on each side, until the fish is opaque, firm to the touch and it flakes easily with a fork. Set aside and tent with foil.

          Reply
        2. Randal

          I’m not working at the moment obat floxigra ciprofloxacin 500 mg untuk apa The Health Dept. will also offer a high-dose flu vaccine to clients age 65 and over. This high-dose vaccine costs $50 per person and is covered by Medicare Part B  The high-dose version of flu vaccine is designed to boost older patients’ immune response to influenza. These patients tend to have a weaker immune response to the standard flu vaccine compared with younger patients, and they are at greater risk for serious complications from influenza.  Clients 65 years of age and older can choose to have the high-dose vaccine or the standard seasonal flu vaccine.  

          Reply
        3. Deangelo

          I’ll put her on allegra stratton Amer is said to have told interrogators that he had hoped to trade the body of the soldier – whom he worked beside in a restaurant in the Israeli city of Bat Yam – to win the release of his brother, who is incarcerated in an Israeli jail for militant offences committed during the second Palestinian intifada.

          Reply
      2. Roman

        There’s a three month trial period clotrimazole betamethasone for dogs Texas Instruments is winding down its wireless chip businessas competition from Qualcomm Inc and other rivals hashurt profitability. TI is focusing on analog and embedded chipsused in products ranging from cars to televisions and whichaccount for the bulk of the company’s revenue.

        Reply
      3. Leroy

        Where do you study? la ciprofloxacina tiene penicilina Drilling around the Lofoten islands is becoming one of thetop issues in coalition talks among the four election-winningparties and a new government is expected to uphold a drillingban, forcing oil companies into more remote and difficultwaters.

        Reply
    1. Connie

      Where’s the nearest cash machine? minoxidil 5 side effects in malayalam I believe agents have a very important role to play. It’s hard for players to negotiate their own deal – it’s like arguing with your future partner. It’s much better for managers and players to work through mediators. What is wrong is when agents are paid a lot for not very much or when agents put themselves first. I’ve had players call me up because their agent has refused to allow a deal to go through because their fee isn’t high enough. That’s shocking but it wouldn’t happen to licensed agents as they would be reported. I now represent clubs instead of players. They were fed up crossing swords with me in the boardroom so they asked me to work for them instead. Clubs give me a shopping list and I try to find suitable players, which involves a lot of meetings and flights out of London City airport. And I still have West Ham season tickets. It’s just not possible to be a fair-weather West Ham fan – we don’t win enough games.

      Reply
    2. Elisha

      I’m doing a phd in chemistry euro pharmacies steroids review “To help manage this unprecedented pent-up demand for BBM, we are implementing a simple line-up system to ensure a smooth rollout,” Andrew Bocking, who heads the BlackBerry’s BBM business, said in a blog post on the company’s website.

      Reply
    3. Irea

      Gloomy tales keppra solucion precio farmacia guadalajara “I’m ready now, and that’s nine days away. The 30th is right around the corner. So I’m fine, man,” Martin said. “I’ll get a few more practices in and if he decides to play me in one of these last few preseason games, he’ll let me know. But if not, the real show starts in nine days and I’m ready. I’m ready to rock. That’s not an issue for me. That’s all on Woody. When Woody wants me out there, I’ll be out there.

      Reply
      1. Simon

        I’ve come to collect a parcel propranolol para que sirve wikipedia Tim Wright, Global Head of Development, Novartis Pharmaceuticals, said, “The positive new data clearly show the potential of omalizumab to treat CSU, a disease where more than 50% of patients don’t respond to approved doses of antihistamines, the only licensed treatment option.”

        Reply
      2. Julia

        I’m doing a phd in chemistry augmentin bambini dosaggio 30 kg Peter Jankovskis, co-chief investment officer at OakBrookInvestments LLC in Lisle, Illinois, said investors will try todecipher what the Fed knows about the U.S. jobs report a coupleof days in advance, which could make Wednesday “even morevolatile than it usually is” on Fed statement days.

        Reply
    1. Noah

      We’ve got a joint account how often do you take pepcid ac The cars were rated for their performance in the insurance institute’s “small overlap” test of crashes that cover only 25 percent of a vehicle’s front end. These tests, added to the IIHS’s evaluations last year, are forcing automakers to bolster the front-end structure of all cars in order to avoid bad publicity from a poor performance.

      Reply
    1. Dro4er

      We need someone with qualifications divalproex vs depakote Priebus gave NBC and CNN until Aug 14, the start of the RNC’s summer meeting, to cancel the programs or else he would seek a “binding vote” that “the RNC will neither partner with these networks in 2016 primary debates nor sanction primary debates they sponsor.”

      Reply
    1. Horace

      Another year ciprofloxacino 500 mg sirve para la garganta “It’s amazing,” Dufner said. “We purchased about 50 acres a couple years ago. We are currently building a home on it. So we’ll have plenty of space to grow the trees and hopefully they will take root. We got a sapling actually the other day from the general manager here at Oak Hill. So at least that one will take root.”

      Reply
      1. Buster

        This is your employment contract diovan vs losartan Glimpses of a more popular Caligula emerge in Beard’s film, and hints of the achievements of his reign. But did he really make his horse a consul? Or sleep with his sisters? “What there is evidence for,” says Beard, “what ancient writers like his biographer Suetonius, writing 80 years after his reign, say, is that people said he planned to make his horse a consul, not that he actually did. They say he was very upset when his sister Drusilla died. Then later others say, ‘Ooh, people say his grandmother caught them in bed together.’ So what’s driving this is not necessarily factual accuracy. What’s doing the job is the myth.”

        Reply
      2. Darren

        Please wait how to use fluticasone propionate and salmeterol inhalation powder The research, by telecommunications firm O2, suggested the lack of female role models at the top of business was partly to blame for why so many women felt the need to look or act like their male colleagues to succeed. One in 10 women said there are no females in senior positions at their company, with 42 per cent said senior women act like alpha females to get ahead – “a fact that is leading many women to assume more traditionally ‘male’ characteristics in order to be successful”, according to the study.

        Reply
      3. Cornell

        Remove card ivermectin za zivotinje She told CNBC at the time that JPMorgan expected clientdemand for commodity finance, risk management and trading togrow “very, very rapidly over the next couple of decades infact. … we are very excited about the prospects for growth.”

        Reply
      4. Logan

        Do you need a work permit? himcolin gel uses side effects “It’s really important to the next mayor that they have the tools. They’ve got to know where everything is and what status it’s in and which buttons to push, and how to keep it going, particularly at the beginning,” Bloomberg said.

        Reply
      5. Romeo

        How much notice do you have to give? terbinafine plus itraconazole There is, however, still a significant gap between theseller’s expectations and what suitors are willing to offer,people familiar with the matter said. The CP Group-Carlyle Groupconsortium has made the highest offer, one person said.

        Reply
    2. Royal

      I’d like to transfer some money to this account pepcid iv side effects
      Born in 1877, she was raised in an impoverished middle-class Nottingham family. An air of genteel poverty never left her – decades later, and by then comfortably off, she was presented to the Prince of Wales and famously insisted on taking the bus home.

      Reply
    3. Teddy

      There’s a three month trial period para que sirve portem paracetamol 500 mg Nursing homes and other long-term care providers for the elderly and disabled generally are required to provide and maintain the trust fund accounts for residents who request that the facility handle their money. The funds are supposed to work like conventional bank accounts, with accrued interest, regular statements and reliable oversight.

      Reply
    4. Buford

      Punk not dead goodrx tizanidine 4mg Nasdaq OMX disclosed a “combined series of technology events” that caused trading on the Nasdaq to be halted for roughly three hours on Aug. 22. And rather than pointing to vague external sources, this time Nasdaq wrote “a number of these issues were clearly within the control of Nasdaq OMX.”

      Reply
    5. Kareem

      Insufficient funds buy cabergoline india “This stock is heavily shorted with a short-to-float ratioof almost 26 percent as of Sept. 13,” said WhatsTrading.comoptions strategist Frederic Ruffy. Shares are now up over 285percent year-to-date.

      Reply
      1. Stevie

        I’m doing an internship flagyl metronidazole 500mg No, his crime is nostalgia. He thinks Twitter is stupid and pernicious. He misses the days when publishers made long-term investments in promising young authors and “every magazine and newspaper had a robust books section.” Amazon, he writes, “wants a world in which books are either self-published or published by Amazon itself, with readers dependent on Amazon reviews in choosing books, and with authors responsible for their own promotion.” Franzen does not want this world. Well, the Internet has an answer to that! #Elitist!!!

        Reply
      2. Britt

        I’d like to send this to girabloc ciprofloxacin 500 mg untuk apa There is no worse combo in sports than a coach with nothing to lose leading a team with everything to lose. Even if Sanchez had not gotten hurt, even if the injury does not turn out to be that serious, the decision making by Rex was so flawed that you have to question the motives behind it.

        Reply
      3. Damian

        I’m interested in this position ciprofloxacino 1 gr According to the patent applications submitted by Dyson: “In a preferred embodiment, the present invention provides a humidifying apparatus for generating a flow of moist air and a flow of air for dispersing the moist air within a domestic environment such as a room, office or the like.”

        Reply
      4. Jackson

        Where do you come from? paracetamol zentiva pret catena “We’re going to continue to support the efforts under way in Detroit and ensure the federal government is an active partner in supporting the revitalization of the city,” said Gene Sperling, director of the White House National Economic Council, who has led federal discussions with Detroit on how best to help.

        Reply
      5. Harvey

        Where’s the nearest cash machine? viagra pillen apotheek The activist investor, the company’s second-largestshareholder, said he would no longer try to head off the deal,and that a war with the billionaire founder and the company’sboard “would be almost impossible to win.”

        Reply
    6. Crazyfrog

      Do you know the number for ? naproxen polfarmex chpl “It’s really pretty empty, and kind of strange walkingaround with so few people here,” said Cauble, who drove intoYellowstone on Monday evening when it became apparent that ashutdown would mean visitors would be barred.

      Reply
    7. Anna

      I love the theatre orlistate 120mg ems 84 cpsulas Uruzgan provincial police chief spokesman Fared Ayel said a father and two of his children were killed Sunday when the motorcycle they were on hit a roadside bomb. He said that three more bombs in the same district on Saturday killed four people and wounded three.

      Reply
    8. Florentino

      Have you got any ? taking pepcid after eating Prince Bandar, who was Saudi ambassador to Washington for 22 years, is seen as a foreign policy hawk, especially on Iran. The Sunni Muslim kingdom’s rivalry with Shi’ite Iran, an ally of Syria, has amplified sectarian tensions across the Middle East.

      Reply
    9. Hubert

      I don’t like pubs actos and weight loss John Manly, the attorney who represented the victims in both cases, said he sees filing for bankruptcy as part of the church’s effort to bury the truth about its pedophile priests. Chapter 11 generally stops or at least temporarily halts pending litigation.

      Reply
    10. Fermin

      Could you tell me the number for ? generic for ursodiol Babycare products producer Goodbaby International surged 9.2 percent after the same newspaper reported a schemeallowing couples to have a second child in the mainland if theyare both only childs could resume soon, with authoritiesconsidering a full liberalisation of the policy in 2015.

      Reply
    11. Faith

      The line’s engaged felodipine er info Tomczyk did not say how much of clients’ stock portfolioswere tied up in Apple stock to have such an outsized impact onvalues. Shares of Apple are down 25.4 percent over the last 12months, including reinvested dividends, but the S&P 500 stockindex is up 27.4 percent.

      Reply
    12. Hilton

      Other amount insomnia artane castle The results of the make-up are good. Most of her time was spent making the foundation really look natural and shading different areas of my face to bring out my features. I normally don’t wear a lot of makeup, so there’s a difference in my usual makeup because the express service doesn’t include eyeliner, but the overall look is very natural and polished.

      Reply
    1. Earnest

      Whereabouts are you from? cardace ramipril obat apa Holmes might not need the practice in training camp that younger players would, but “week-to-week” probably starts to eat into the regular season. After all, the Jets take on the Bucs and Darrelle Revis in under four weeks. Realistically, given the stage Holmes is at now, it seems unlikely that the receiver would be ready for that game. Rex Ryan, however, wouldn’t rule it out.

      Reply
    2. Wilford

      Will I get travelling expenses? ivermectina quanox mifarma “Joe’s just a really good man,” Pettitte said a couple of days ago. “That’s what you see when you’re around him a lot. And it’s why guys like playing for him. He cares a lot about doing what’s best for guys, and maybe people are getting a chance to see a little more of that now.”

      Reply
    3. Adam

      Could you ask her to call me? zamienniki lexapro opinie The group of Ocean County Vehicle Services Department employees, dubbed the “Ocean’s 16,” cheered for each other as they were introduced for the first time today in public as the group that held one of the three lucky tickets from last week’s drawing.

      Reply
    4. Edgardo

      I’m self-employed flonase coupon printable “I’m not going to miss her, because I’m going to stay in touch with her,” he said. ”And that’s one of the great things. I just love her to death. I hope that I can someday be half the person she is.”

      Reply
  1. Jasonprise

    [url=http://ivermectinhtab.com/]price of ivermectin liquid[/url] [url=http://sildenafilltabs.com/]viagra pills online south africa[/url] [url=http://cleocin.today/]clindamycin gel rx[/url] [url=http://sildenafilsuperpills.com/]viagra prescription nz[/url] [url=http://sildenafilxv.com/]where can you buy viagra cheap[/url] [url=http://bestcialisgeneric.com/]cialis 100mg tablets[/url] [url=http://onlinedrugstore.monster/]pill pharmacy[/url] [url=http://yasmin.quest/]yasmin generic cost[/url] [url=http://viagraspr.com/]viagra 100mg tablet price[/url] [url=http://buyprazosin.com/]prazosin 2 mg price[/url]

    Reply
  2. Eyeprise

    [url=http://viagragenericorder.com/]sildenafil 100mg tablets canada[/url] [url=http://gencialistablets.com/]buy cheap cialis online[/url] [url=http://yasmin.quest/]buy yasmin online india[/url] [url=http://buyprazosin.com/]prazosin for insomnia[/url] [url=http://cialisrtabs.com/]generic cialis nz[/url] [url=http://azithromycin.today/]generic zithromax 500mg india[/url] [url=http://sildenafilutab.com/]cheap viagra online without prescription[/url] [url=http://buylevaquin.quest/]levaquin price[/url] [url=http://ordersildenafilcitratepills.com/]how to buy viagra usa[/url] [url=http://onlinecialispills.com/]where can i buy cialis in singapore[/url]

    Reply
    1. Brady

      One moment, please imigran fiale forum Calling ‘All The Single Ladies’! Fed up of missing out on all the action your loved up friends have access to? Fret no more, ‘Zips’ allows users to virtually unzip the front of a man, or woman’s, jeans to reveal their underwear. As an added bonus, you can even choose what type of underwear you want your partner to wear – just in case you can’t handle surprises. Although undeniably moronic, some critics seem more bothered by the limitations of the app. One customer review suggested “it could have been sexier showing part of the men’s belly”! Silly app, only letting us see the front of an unknown, animated man’s jeans, we want more! Just not any time soon please.

      Reply
  3. Denprise

    [url=http://sildenafilxv.com/]where to buy viagra online in australia[/url] [url=http://fildena.today/]fildena 25 mg[/url] [url=http://lipitor.live/]lipitor over the counter[/url] [url=http://viagraptabs.com/]viagra compare prices[/url] [url=http://actadalafil.com/]tadalafil 40 mg price[/url]

    Reply
  4. Eyeprise

    [url=http://ampicillin.live/]where to buy ampicillin 500mg[/url] [url=http://pharmacyonline.quest/]online pharmacy[/url] [url=http://yasmin.quest/]yasmin pill prescription[/url] [url=http://buypaxil.quest/]generic paxil prices[/url] [url=http://viagraptabs.com/]3 viagra[/url] [url=http://tadalafilbtabs.com/]cialis soft tabs online[/url] [url=http://viagra.sbs/]where can i get over the counter viagra[/url] [url=http://viagragtabs.com/]buy sildenafil online uk[/url] [url=http://cialisntabs.com/]cialis generic online canada[/url] [url=http://ordersildenafilcitratepills.com/]sildenafil from india[/url]

    Reply
  5. Denprise

    [url=https://ivermectinrt.online/]ivermectin drug[/url] [url=https://cialischeapmedicinedrugstore.quest/]canadian pharmacy cheap cialis[/url] [url=https://onlineviagra50pill.quest/]best price for viagra in uk[/url] [url=https://viagradrugstore.monster/]viagra professional[/url] [url=https://buygenericcialispills.quest/]canada rx cialis[/url]

    Reply
  6. Denprise

    [url=http://bestviagratabletwithnoprescription.quest/]viagra fast delivery[/url] [url=http://lowcostviagra2022.monster/]sildenafil generic without a prescription[/url] [url=http://cialis40withnorx.monster/]can you buy cialis in canada[/url] [url=http://genericviagratabsbuyonline.quest/]buy real viagra uk[/url] [url=http://kviagra.monster/]viagra gel for sale[/url]

    Reply
  7. healthetarians

    Woah this blog is wonderful i really like reading your posts. Keep up the great paintings! You already know, many individuals are hunting round for this information, you could help them greatly.

    Reply
  8. healthetarians

    I just like the helpful information you provide in your articles. I will bookmark your blog and take a look at again here regularly. I’m reasonably certain I will learn plenty of new stuff proper right here! Best of luck for the following!

    Reply
  9. Denprise

    [url=https://ordercialisdrugwithnorx.monster/]rx pharmacy cialis[/url] [url=https://buygenericviagrapillonline.quest/]viagra online cheap[/url] [url=https://cialisitabs.online/]brand cialis australia[/url] [url=https://bestviagradrugforsaleonline.quest/]sildenafil 20 mg tablet[/url] [url=https://buyingcialistablets.monster/]generic tadalafil without prescription[/url]

    Reply
  10. Denprise

    [url=http://genericcialismedicinewithnorx.quest/]tadalafil 10mg online india[/url] [url=http://bestcialis5mgtablet.monster/]tadalafil usa[/url] [url=http://genericcialistabsrx.quest/]cialis buy europe[/url] [url=http://cialisonlinedrugshop.online/]price tadalafil 20mg[/url] [url=http://buygenericcialis20.online/]cialis daily cost[/url]

    Reply
  11. Denprise

    [url=http://viagragenericonline.quest/]sildenafil brand name in india[/url] [url=http://genericviagrapillsbuying.monster/]buy real viagra from canada[/url] [url=http://genericviagratabletsforsale.monster/]viagra online ordering[/url] [url=http://buyviagratabsonline.quest/]sildenafil gel[/url] [url=http://genericcialismedicinewithoutrx.monster/]generic cialis paypal[/url]

    Reply
  12. Jasonprise

    [url=https://tetracycline.online/]tetracycline pills for sale[/url] [url=https://medrol.quest/]purchase medrol[/url] [url=https://lipitor.quest/]lipitor atorvastatin calcium[/url] [url=https://cafergot.monster/]order cafergot online[/url] [url=https://atomoxetine.quest/]buy strattera 80 mg[/url] [url=https://cialisttabs.com/]tadalafil 5mg daily use[/url] [url=https://cialis.cyou/]buy cialis with paypal[/url] [url=https://tenormin.online/]atenolol price[/url] [url=https://valacyclovir.online/]valtrex no perscrition[/url] [url=https://isotretinoin.quest/]accutane online no rx[/url]

    Reply
  13. Denprise

    [url=https://cafergot.monster/]best price cafergot[/url] [url=https://sertraline.today/]zoloft brand name cost[/url] [url=https://malegra.today/]malegra dxt[/url] [url=https://azithromycina.com/]azithromycin 1mg[/url] [url=https://tizanidine.online/]zanaflex 4 mg coupon[/url]

    Reply
  14. Alanprise

    [url=http://escitalopram.monster/]lexapro 20mg tablet[/url] [url=http://tadalafilutabs.com/]cialis 20 mg tadalafil[/url] [url=http://albuterol.online/]buy ventolin online[/url] [url=http://cialisjtabs.com/]soft cialis online[/url] [url=http://tadacip.monster/]where to buy tadacip online[/url] [url=http://valacyclovir.online/]how to order valtrex online[/url] [url=http://ventolin.live/]buy ventolin pharmacy[/url] [url=http://dexamethasone.today/]dexamethasone 2 tablet[/url] [url=http://seroquel.today/]seroquel 200 mg tablet[/url] [url=http://baclofen.monster/]can i buy baclofen over the counter[/url]

    Reply
  15. Alanprise

    [url=http://paxil.today/]paroxetine tab 40mg[/url] [url=http://lipitor.quest/]buy lipitor[/url] [url=http://piroxicam.quest/]feldene 20 mg[/url] [url=http://viagrartabs.com/]where to get sildenafil[/url] [url=http://prednisone.today/]prednisone 20 mg tablet cost[/url] [url=http://femaleviagra.live/]buy female viagra nz[/url] [url=http://atomoxetine.quest/]strattera 80[/url] [url=http://albendazole.monster/]albendazole 400 mg buy online[/url] [url=http://motilium.quest/]motilium otc canada[/url] [url=http://viagraktabs.com/]viagra pill otc[/url]

    Reply
  16. Eyeprise

    [url=https://cleocin.live/]clindamycin 1 cream[/url] [url=https://cialishtabs.com/]cialis price in mexico[/url] [url=https://femaleviagra.live/]where can you purchase female viagra[/url] [url=https://suhagra.monster/]buy suhagra tablet[/url] [url=https://modafinilgeneric.monster/]modafinil over the counter usa[/url] [url=https://pharmacyonline.monster/]pharmacy mall[/url] [url=https://zanaflex.online/]zanaflex for headaches[/url] [url=https://tadalafilktab.com/]cheap tadalafil tablets[/url] [url=https://finasteride.live/]propecia 5mg sale[/url] [url=https://cialisstabs.com/]tadalafil uk generic[/url]

    Reply
  17. Denprise

    [url=https://cafergot.monster/]buy cafergot[/url] [url=https://silagra.quest/]buy silagra tablets[/url] [url=https://cialishtabs.com/]where can i buy cialis over the counter in uk[/url] [url=https://augmentin.today/]cost of augmentin 875[/url] [url=https://effexor.today/]effexor 375 mg[/url]

    Reply
  18. Pingback: buy ivermectin 3 mg

  19. Pingback: ivermectin 6mg for sale

  20. Pingback: ivermectin cost australia