Issue :-
LISTENER is not running on one of RAC node. After attempting to restart throws below error.
[ora12c@RAC2 ~]$ srvctl start listener -n RAC2
PRCR-1013 : Failed to start resource ora.LISTENER.lsnr
PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node RAC2
CRS-2805: Unable to start 'ora.LISTENER.lsnr' because it has a 'hard' dependency on resource type 'ora.cluster_vip_net1.type' and no resource of that type can satisfy the dependency
CRS-2525: All instances of the resource 'ora.RAC1.vip' are already running; relocate is not allowed because the force option was not specified
Solution :-
Step 1 :- Do pre check list.
[ora12c@RAC2 ~]$ crsctl status res -t
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.FRA.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE RAC1 STABLE
OFFLINE OFFLINE RAC2 STABLE
ora.OCR_VOTING.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.asm
ONLINE ONLINE RAC1 Started,STABLE
ONLINE ONLINE RAC2 Started,STABLE
ora.net1.network
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.net2.network
OFFLINE OFFLINE RAC1 STABLE
OFFLINE OFFLINE RAC2 STABLE
ora.ons
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE RAC2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE RAC1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE RAC1 STABLE
ora.MGMTLSNR
1 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE RAC1 STABLE
ora.RAC1.vip
1 ONLINE ONLINE RAC1 STABLE
ora.RAC2_2.vip
1 OFFLINE OFFLINE STABLE
ora.oc4j
1 ONLINE ONLINE RAC1 STABLE
ora.scan1.vip
1 ONLINE ONLINE RAC2 STABLE
ora.scan2.vip
1 ONLINE ONLINE RAC1 STABLE
ora.scan3.vip
1 ONLINE ONLINE RAC1 STABLE
ora.wacprod.db
1 ONLINE ONLINE RAC1 Open,STABLE
2 ONLINE ONLINE RAC2 Open,STABLE
--------------------------------------------------------------------------------
In one of RAC setup , process ora.LISTENER.lsnr was in OFFLINE state. When I tried to start it throws an subjected error and LISTENER did not come up.
Its good to check for other cluster important process
[ora12c@RAC2 ~]$ crsctl check cluster -all
**************************************************************
RAC1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
RAC2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
CSS,CRS,EVM seems to be fine.
Step 2 :- Check are you able to ping VIP from both nodes
Ping RAC1 virtual IP
[ora12c@RAC1 ~]$ ping RAC1-vip
PING RAC1 (RAC1) 56(84) bytes of data.
64 bytes from RAC1: icmp_seq=1 ttl=63 time=0.378 ms
64 bytes from RAC1: icmp_seq=2 ttl=63 time=0.313 ms
64 bytes from RAC1: icmp_seq=3 ttl=63 time=0.334 ms
64 bytes from RAC1: icmp_seq=4 ttl=63 time=0.438 ms
64 bytes from RAC1: icmp_seq=5 ttl=63 time=0.377 ms
^C
--- RAC1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.313/0.368/0.438/0.043 ms
------------------------------------------------------------------------------------------------
Ping RAC2 virtual IP
[ora12c@RAC1 ~]$ ping RAC2-vip
PING RAC2 (RAC2) 56(84) bytes of data.
Destination host unreachable.
Destination host unreachable.
Destination host unreachable.
Destination host unreachable.
This seems to be an issue.
Step 3 :- After doing some R&D , I found that VIP of RAC2 is not registered with CRS due to which LISTENER on RAC2 is not coming up.
Run srvctl config nodeapps -a to check both VIP's is registered or not
[ora12c@RAC2 ~]$ srvctl config nodeapps -a
Network 1 exists
Subnet IPv4: *.*.*.0/255.255.255.0/eno167832, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC1
VIP Name: RAC1-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
Above out put shows that only RAC1 vip is registered with CRS. It means we need to register RAC2 vip in CRS.
Step 4 :- Run as root
Go to Grid path ----> bin
[root@RAC1 ~]# cd /rdbms/app/12.1.0.2/grid/bin
[root@RAC1 bin]# ./srvctl add vip -n RAC2 -A RAC2-vip/255.255.255.0/eno167832 -k 2
Now both the VIPs are registered.
Run again srvctl config nodeapps -a to check both VIP's is registered or not
[root@rac1 bin]# ./srvctl config nodeapps -a
Network 1 exists
Subnet IPv4: *.*.*.0/255.255.255.0/eno167832, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC1
VIP Name: RAC1-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC2
VIP Name: RAC2-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
Step 5 :- Start RAC2-vip
Attempting to start `ora.RAC2.vip` on member `RAC2`
Start of `ora.RAC2.vip` on member `RAC2` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `RAC2`
Start of `ora.LISTENER.lsnr` on member `RAC2` succeeded.
Now the both nodes VIP is registered and it is reachable.
Step 6:- Check the status again (Step 1 commands)
[ora12c@RAC2 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.FRA.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.OCR_VOTING.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.asm
ONLINE ONLINE RAC1 Started,STABLE
ONLINE ONLINE RAC2 Started,STABLE
ora.net1.network
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.net2.network
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.ons
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE RAC2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE RAC1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE RAC1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE STABLE
ora.cvu
1 ONLINE ONLINE RAC1 STABLE
ora.RAC1.vip
1 ONLINE ONLINE RAC1 STABLE
ora.RAC2_2.vip
1 ONLINE ONLINE STABLE
ora.oc4j
1 ONLINE ONLINE RAC1 STABLE
ora.scan1.vip
1 ONLINE ONLINE RAC2 STABLE
ora.scan2.vip
1 ONLINE ONLINE RAC1 STABLE
ora.scan3.vip
1 ONLINE ONLINE RAC1 STABLE
ora.wacprod.db
1 ONLINE ONLINE RAC1 Open,STABLE
2 ONLINE ONLINE RAC2 Open,STABLE
--------------------------------------------------------------------------------
Issue Resolved :)
Thats It :)
LISTENER is not running on one of RAC node. After attempting to restart throws below error.
[ora12c@RAC2 ~]$ srvctl start listener -n RAC2
PRCR-1013 : Failed to start resource ora.LISTENER.lsnr
PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node RAC2
CRS-2805: Unable to start 'ora.LISTENER.lsnr' because it has a 'hard' dependency on resource type 'ora.cluster_vip_net1.type' and no resource of that type can satisfy the dependency
CRS-2525: All instances of the resource 'ora.RAC1.vip' are already running; relocate is not allowed because the force option was not specified
Solution :-
Step 1 :- Do pre check list.
[ora12c@RAC2 ~]$ crsctl status res -t
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.FRA.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE RAC1 STABLE
OFFLINE OFFLINE RAC2 STABLE
ora.OCR_VOTING.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.asm
ONLINE ONLINE RAC1 Started,STABLE
ONLINE ONLINE RAC2 Started,STABLE
ora.net1.network
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.net2.network
OFFLINE OFFLINE RAC1 STABLE
OFFLINE OFFLINE RAC2 STABLE
ora.ons
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE RAC2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE RAC1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE RAC1 STABLE
ora.MGMTLSNR
1 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE RAC1 STABLE
ora.RAC1.vip
1 ONLINE ONLINE RAC1 STABLE
ora.RAC2_2.vip
1 OFFLINE OFFLINE STABLE
ora.oc4j
1 ONLINE ONLINE RAC1 STABLE
ora.scan1.vip
1 ONLINE ONLINE RAC2 STABLE
ora.scan2.vip
1 ONLINE ONLINE RAC1 STABLE
ora.scan3.vip
1 ONLINE ONLINE RAC1 STABLE
ora.wacprod.db
1 ONLINE ONLINE RAC1 Open,STABLE
2 ONLINE ONLINE RAC2 Open,STABLE
--------------------------------------------------------------------------------
Its good to check for other cluster important process
[ora12c@RAC2 ~]$ crsctl check cluster -all
**************************************************************
RAC1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
RAC2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
CSS,CRS,EVM seems to be fine.
Step 2 :- Check are you able to ping VIP from both nodes
Ping RAC1 virtual IP
[ora12c@RAC1 ~]$ ping RAC1-vip
PING RAC1 (RAC1) 56(84) bytes of data.
64 bytes from RAC1: icmp_seq=1 ttl=63 time=0.378 ms
64 bytes from RAC1: icmp_seq=2 ttl=63 time=0.313 ms
64 bytes from RAC1: icmp_seq=3 ttl=63 time=0.334 ms
64 bytes from RAC1: icmp_seq=4 ttl=63 time=0.438 ms
64 bytes from RAC1: icmp_seq=5 ttl=63 time=0.377 ms
^C
--- RAC1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.313/0.368/0.438/0.043 ms
------------------------------------------------------------------------------------------------
Ping RAC2 virtual IP
[ora12c@RAC1 ~]$ ping RAC2-vip
PING RAC2 (RAC2) 56(84) bytes of data.
Destination host unreachable.
Destination host unreachable.
Destination host unreachable.
Destination host unreachable.
This seems to be an issue.
Step 3 :- After doing some R&D , I found that VIP of RAC2 is not registered with CRS due to which LISTENER on RAC2 is not coming up.
Run srvctl config nodeapps -a to check both VIP's is registered or not
[ora12c@RAC2 ~]$ srvctl config nodeapps -a
Network 1 exists
Subnet IPv4: *.*.*.0/255.255.255.0/eno167832, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC1
VIP Name: RAC1-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
Above out put shows that only RAC1 vip is registered with CRS. It means we need to register RAC2 vip in CRS.
Step 4 :- Run as root
Go to Grid path ----> bin
[root@RAC1 ~]# cd /rdbms/app/12.1.0.2/grid/bin
[root@RAC1 bin]# ./srvctl add vip -n RAC2 -A RAC2-vip/255.255.255.0/eno167832 -k 2
Now both the VIPs are registered.
Run again srvctl config nodeapps -a to check both VIP's is registered or not
[root@rac1 bin]# ./srvctl config nodeapps -a
Network 1 exists
Subnet IPv4: *.*.*.0/255.255.255.0/eno167832, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC1
VIP Name: RAC1-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC2
VIP Name: RAC2-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
Step 5 :- Start RAC2-vip
Attempting to start `ora.RAC2.vip` on member `RAC2`
Start of `ora.RAC2.vip` on member `RAC2` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `RAC2`
Start of `ora.LISTENER.lsnr` on member `RAC2` succeeded.
Now the both nodes VIP is registered and it is reachable.
Step 6:- Check the status again (Step 1 commands)
[ora12c@RAC2 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.FRA.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.OCR_VOTING.dg
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.asm
ONLINE ONLINE RAC1 Started,STABLE
ONLINE ONLINE RAC2 Started,STABLE
ora.net1.network
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.net2.network
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
ora.ons
ONLINE ONLINE RAC1 STABLE
ONLINE ONLINE RAC2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE RAC2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE RAC1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE RAC1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE STABLE
ora.cvu
1 ONLINE ONLINE RAC1 STABLE
ora.RAC1.vip
1 ONLINE ONLINE RAC1 STABLE
ora.RAC2_2.vip
1 ONLINE ONLINE STABLE
ora.oc4j
1 ONLINE ONLINE RAC1 STABLE
ora.scan1.vip
1 ONLINE ONLINE RAC2 STABLE
ora.scan2.vip
1 ONLINE ONLINE RAC1 STABLE
ora.scan3.vip
1 ONLINE ONLINE RAC1 STABLE
ora.wacprod.db
1 ONLINE ONLINE RAC1 Open,STABLE
2 ONLINE ONLINE RAC2 Open,STABLE
--------------------------------------------------------------------------------
Issue Resolved :)
Thats It :)
No comments:
Post a Comment