Oracle RAC Error: CRS-2805: Unable to start 'ora.LISTENER.lsnr' because it has a 'hard' dependency on resource type 'ora.cluster_vip_net2.type' and no resource of that type can satisfy the dependency_part1

Issue :- 
LISTENER is not running on one of RAC node. After attempting to restart throws below error.

[ora12c@RAC2 ~]$ srvctl start listener -n RAC2
PRCR-1013 : Failed to start resource ora.LISTENER.lsnr
PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node RAC2
CRS-2805: Unable to start 'ora.LISTENER.lsnr' because it has a 'hard' dependency on resource type 'ora.cluster_vip_net1.type' and no resource of that type can satisfy the dependency
CRS-2525: All instances of the resource 'ora.RAC1.vip' are already running; relocate is not allowed because the force option was not specified


Solution :-

Step 1 :- Do pre check list.

[ora12c@RAC2 ~]$ crsctl status res -t
 Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
      ONLINE  ONLINE       RAC1            STABLE
      ONLINE  ONLINE       RAC2            STABLE
ora.FRA.dg
      ONLINE  ONLINE       RAC1            STABLE
      ONLINE  ONLINE       RAC2            STABLE
ora.LISTENER.lsnr
      ONLINE  ONLINE       RAC1            STABLE
      OFFLINE OFFLINE      RAC2            STABLE
ora.OCR_VOTING.dg
      ONLINE  ONLINE       RAC1            STABLE
      ONLINE  ONLINE       RAC2            STABLE
ora.asm
      ONLINE  ONLINE       RAC1            Started,STABLE
      ONLINE  ONLINE       RAC2            Started,STABLE
ora.net1.network
      ONLINE  ONLINE       RAC1            STABLE
      ONLINE  ONLINE       RAC2            STABLE
ora.net2.network
      OFFLINE OFFLINE      RAC1            STABLE
      OFFLINE OFFLINE      RAC2            STABLE
ora.ons
      ONLINE  ONLINE       RAC1            STABLE
      ONLINE  ONLINE       RAC2            STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       RAC2            STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       RAC1            STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       RAC1            STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                      STABLE
ora.cvu
      1        ONLINE  ONLINE       RAC1            STABLE
ora.RAC1.vip
      1        ONLINE  ONLINE       RAC1            STABLE
ora.RAC2_2.vip
      1        OFFLINE OFFLINE                      STABLE
ora.oc4j
      1        ONLINE  ONLINE       RAC1            STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       RAC2            STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       RAC1            STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       RAC1            STABLE
ora.wacprod.db
      1        ONLINE  ONLINE       RAC1            Open,STABLE
      2        ONLINE  ONLINE       RAC2            Open,STABLE
--------------------------------------------------------------------------------

In one of RAC setup , process ora.LISTENER.lsnr was in OFFLINE state. When I tried to start it throws an subjected error and LISTENER did not come up.

Its good to check for other cluster important process

[ora12c@RAC2 ~]$ crsctl check cluster -all
**************************************************************
RAC1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
RAC2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

CSS,CRS,EVM seems to be fine.

Step 2 :-  Check are you able to ping VIP from both nodes

                                            Ping RAC1 virtual IP

[ora12c@RAC1 ~]$ ping RAC1-vip

PING RAC1 (RAC1) 56(84) bytes of data.
64 bytes from RAC1: icmp_seq=1 ttl=63 time=0.378 ms
64 bytes from RAC1: icmp_seq=2 ttl=63 time=0.313 ms
64 bytes from RAC1: icmp_seq=3 ttl=63 time=0.334 ms
64 bytes from RAC1: icmp_seq=4 ttl=63 time=0.438 ms
64 bytes from RAC1: icmp_seq=5 ttl=63 time=0.377 ms
^C
--- RAC1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.313/0.368/0.438/0.043 ms

------------------------------------------------------------------------------------------------

                                             Ping RAC2 virtual IP

[ora12c@RAC1 ~]$ ping RAC2-vip

PING RAC2 (RAC2) 56(84) bytes of data.

Destination host unreachable.
Destination host unreachable.
Destination host unreachable.
Destination host unreachable.

This seems to be an issue.

Step 3 :- After doing some R&D , I found that VIP of RAC2 is not registered with CRS due to which LISTENER on RAC2 is not coming up.

Run srvctl config nodeapps -a to check both VIP's is registered or not

[ora12c@RAC2 ~]$ srvctl config nodeapps -a
Network 1 exists
Subnet IPv4: *.*.*.0/255.255.255.0/eno167832, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC1
VIP Name: RAC1-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:

Above out put shows that only RAC1 vip is registered with CRS. It means we need to register RAC2 vip in CRS.

Step 4 :- Run as root
Go to Grid path ----> bin

[root@RAC1 ~]# cd /rdbms/app/12.1.0.2/grid/bin
[root@RAC1 bin]# ./srvctl add vip -n RAC2 -A RAC2-vip/255.255.255.0/eno167832 -k 2

Now both the VIPs are registered.


Run again srvctl config nodeapps -a to check both VIP's is registered or not

[root@rac1 bin]# ./srvctl config nodeapps -a

Network 1 exists
Subnet IPv4: *.*.*.0/255.255.255.0/eno167832, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC1
VIP Name: RAC1-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node RAC2
VIP Name: RAC2-vip.co.in
VIP IPv4 Address: *.*.*.*
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:

Step 5 :- Start RAC2-vip
Attempting to start `ora.RAC2.vip` on member `RAC2`
Start of `ora.RAC2.vip` on member `RAC2` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `RAC2`

Start of `ora.LISTENER.lsnr` on member `RAC2` succeeded.

Now the both nodes VIP is registered and it is reachable.

Step 6:- Check the status again (Step 1 commands)

[ora12c@RAC2 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
       ONLINE  ONLINE       RAC1            STABLE
       ONLINE  ONLINE       RAC2            STABLE
ora.FRA.dg
       ONLINE  ONLINE       RAC1            STABLE
       ONLINE  ONLINE       RAC2            STABLE
ora.LISTENER.lsnr
       ONLINE  ONLINE       RAC1            STABLE
       ONLINE  ONLINE       RAC2            STABLE
ora.OCR_VOTING.dg
       ONLINE  ONLINE       RAC1            STABLE
       ONLINE  ONLINE       RAC2            STABLE
ora.asm
       ONLINE  ONLINE       RAC1            Started,STABLE
       ONLINE  ONLINE       RAC2            Started,STABLE
ora.net1.network
       ONLINE  ONLINE       RAC1            STABLE
       ONLINE  ONLINE       RAC2            STABLE
ora.net2.network
       ONLINE ONLINE        RAC1            STABLE
       ONLINE ONLINE        RAC2            STABLE
ora.ons
       ONLINE  ONLINE       RAC1            STABLE
       ONLINE  ONLINE       RAC2            STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       RAC2            STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       RAC1            STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       RAC1            STABLE
ora.MGMTLSNR
      1        ONLINE ONLINE                        STABLE
ora.cvu
      1        ONLINE  ONLINE       RAC1            STABLE
ora.RAC1.vip
      1        ONLINE  ONLINE       RAC1            STABLE
ora.RAC2_2.vip
      1        ONLINE ONLINE                        STABLE
ora.oc4j
      1        ONLINE  ONLINE       RAC1            STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       RAC2            STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       RAC1            STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       RAC1            STABLE
ora.wacprod.db
      1        ONLINE  ONLINE       RAC1            Open,STABLE
      2        ONLINE  ONLINE       RAC2            Open,STABLE
--------------------------------------------------------------------------------


Issue Resolved :)



Thats It :)

No comments:

Post a Comment