[SRVM(1337)]CRS-10051: CVU found following errors with Clusterware setup : PRVF-5301 : Failed to locate CRS home

Error:-

I was working on issue where cluster is not coming up on node 1 but it is running on node 2.
Following error has been recorded.

Go to    Cd /rdbms/app/oracle/diag/crs/SERVER1/crs/trace
 do      vi alert.log

2017-07-11 14:44:39.105 [CLSECHO(9310)]CRS-10001: 11-Jul-17 14:44 AFD-9204: false
2017-07-11 20:43:52.618 [SRVM(1337)]CRS-10051: CVU found following errors with Clusterware setup : PRVF-5301 : Failed to locate CRS home
PRVF-7573 : Sufficient swap size is not available on node "INP44XPDB1283" [Required = 15.5079GB (1.6261252E7KB) ; Found = 7.8125GB (8191996.0KB)]
PRVG-2033 : Permissions of file "/etc/oracle/olr.loc" did not match the expected octal value on node "INP44XPDB1283". [Expected = "0640" ; Found = "0644"]

PRVG-2031 : Owner of file "/rdbms/app/12.1.0.2/grid/cdata/SERVER1.olr" did not match the expected value on node "INP44XPDB1283". [Expected = "ora12c" ; Found = "root"]

Also we suspect that this may be due DB patching . If you apply patches on RAC1 then in some cases RAC2 cluster may not come up due different patch level.
Error:-
CRS-6706: Oracle Clusterware Release patch level ('1009910258') does not match Software patch level ('93699375'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

Refer below link for Patch level issue

 CRS-6706: Oracle Clusterware Release patch level ('mmmmm') does not match Software patch level ('nnnnn'). Oracle Clusterware cannot be started

Solution:-

Before performing below steps make sure that you have proper DB backup in case of recovery.

Step1 :Stopped cluster on SERVER2        //Running Node

Step2 :  Run on problematic node 

[ora12c@SERVER1 ~]$
[ora12c@SERVER1 install]$ pwd                            // Go to crs path 
/rdbms/app/12.1.0.2/grid/crs/install

[ora12c@SERVER1 install]$ ls -lrt root*
-rwxr-xr-x. 1 root oinstall  9467 Mar 10  2015 roothas.pl
-rwxr-xr-x. 1 root oinstall 17971 Mar 10  2015 rootcrs.pl
-rwxr-x---. 1 root oinstall   822 May  3 21:32 roothas.sh
-rwxr-x---. 1 root oinstall   822 May  3 21:32 rootcrs.sh

[ora12c@SERVER1 install]$ pwd
/rdbms/app/12.1.0.2/grid/crs/install

[ora12c@SERVER1 install]$ sudo /rdbms/app/12.1.0.2/grid/crs/install/rootcrs.sh -deconfig -force                          // To deconfigure grid
Using configuration parameter file: /rdbms/app/12.1.0.2/grid/crs/install/crsconfig_params
Usage: srvctl <command> <object> [<options>]
    commands: enable|disable|start|stop|status|add|remove|modify|update|getenv|setenv|unsetenv|config|upgrade|downgrade
    objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
  srvctl <command> -help [-compatible] or
  srvctl <command> <object> -help [-compatible]
PRKO-2012 : nodeapps object is not supported in Oracle Restart
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.
2017/08/10 12:55:40 CLSRSC-463: The deconfiguration or downgrade script could not stop current Oracle Clusterware stack.

2017/08/10 12:55:41 CLSRSC-557: Oracle Clusterware stack on this node has been successfully deconfigured. There were some errors which can be ignored.

[ora12c@SERVER1 install]$ cd ..
[ora12c@SERVER1 crs]$ cd ..
[ora12c@SERVER1 grid]$ ls -lrt root.sh
-rwxr-x---. 1 ora12c oinstall 495 May  3 21:29 root.sh
[ora12c@SERVER1 grid]$ pwd                                //Go to grid path
/rdbms/app/12.1.0.2/grid                                                      
[ora12c@SERVER1 grid]$ sudo /rdbms/app/12.1.0.2/grid/root.sh
Check /rdbms/app/12.1.0.2/grid/install/root_SERVER1.vsnl.co.in_2017-08-10_12-55-56.log for the output of root script
[ora12c@SERVER1 grid]$

ASM & DB will automatically start on SERVER1

Step 3 :- Start cluster on SERVER2

OUTPUT:- 

Output of /rdbms/app/12.1.0.2/grid/install/root_SERVER1.vsnl.co.in_2017-08-10_12-55-56.log file

[ora12c@SERVER1 ~]$ tail -f /rdbms/app/12.1.0.2/grid/install/root_SERVER1.vsnl.co.in_2017-08-10_13-01-13.log

   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Using configuration parameter file: /rdbms/app/12.1.0.2/grid/crs/install/crsconfig_params

2017/08/10 13:01:14 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2017/08/10 13:03:00 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2017/08/10 13:03:01 CLSRSC-363: User ignored prerequisites during installation

2017/08/10 13:04:06 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'SERVER1'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'SERVER1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'SERVER1'

CRS-2673: Attempting to stop 'ora.gipcd' on 'SERVER1'

CRS-2673: Attempting to stop 'ora.evmd' on 'SERVER1'

CRS-2677: Stop of 'ora.gpnpd' on 'SERVER1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'SERVER1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'SERVER1' succeeded

CRS-2677: Stop of 'ora.gipcd' on 'SERVER1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'SERVER1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-2672: Attempting to start 'ora.evmd' on 'SERVER1'

CRS-2672: Attempting to start 'ora.mdnsd' on 'SERVER1'

CRS-2676: Start of 'ora.evmd' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.mdnsd' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'SERVER1'

CRS-2676: Start of 'ora.gpnpd' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'SERVER1'

CRS-2672: Attempting to start 'ora.gipcd' on 'SERVER1'

CRS-2676: Start of 'ora.cssdmonitor' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.gipcd' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'SERVER1'

CRS-2672: Attempting to start 'ora.diskmon' on 'SERVER1'

CRS-2676: Start of 'ora.diskmon' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.cssd' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'SERVER1'

CRS-2672: Attempting to start 'ora.ctssd' on 'SERVER1'

CRS-2676: Start of 'ora.ctssd' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'SERVER1'

CRS-2676: Start of 'ora.asm' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'SERVER1'

CRS-2676: Start of 'ora.storage' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'SERVER1'

CRS-2676: Start of 'ora.crf' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'SERVER1'

CRS-2676: Start of 'ora.crsd' on 'SERVER1' succeeded

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'SERVER1'

CRS-2673: Attempting to stop 'ora.crsd' on 'SERVER1'

CRS-2677: Stop of 'ora.crsd' on 'SERVER1' succeeded

CRS-2673: Attempting to stop 'ora.mdnsd' on 'SERVER1'

CRS-2673: Attempting to stop 'ora.storage' on 'SERVER1'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'SERVER1'

CRS-2677: Stop of 'ora.storage' on 'SERVER1' succeeded

CRS-2673: Attempting to stop 'ora.crf' on 'SERVER1'

CRS-2673: Attempting to stop 'ora.ctssd' on 'SERVER1'

CRS-2673: Attempting to stop 'ora.evmd' on 'SERVER1'

CRS-2673: Attempting to stop 'ora.asm' on 'SERVER1'

CRS-2677: Stop of 'ora.mdnsd' on 'SERVER1' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'SERVER1' succeeded

CRS-2677: Stop of 'ora.crf' on 'SERVER1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'SERVER1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'SERVER1' succeeded

CRS-2677: Stop of 'ora.asm' on 'SERVER1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'SERVER1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'SERVER1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'SERVER1'

CRS-2677: Stop of 'ora.cssd' on 'SERVER1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'SERVER1'

CRS-2677: Stop of 'ora.gipcd' on 'SERVER1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'SERVER1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.mdnsd' on 'SERVER1'

CRS-2672: Attempting to start 'ora.evmd' on 'SERVER1'

CRS-2676: Start of 'ora.mdnsd' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.evmd' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'SERVER1'

CRS-2676: Start of 'ora.gpnpd' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'SERVER1'

CRS-2676: Start of 'ora.gipcd' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'SERVER1'

CRS-2676: Start of 'ora.cssdmonitor' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'SERVER1'

CRS-2672: Attempting to start 'ora.diskmon' on 'SERVER1'

CRS-2676: Start of 'ora.diskmon' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.cssd' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'SERVER1'

CRS-2672: Attempting to start 'ora.ctssd' on 'SERVER1'

CRS-2676: Start of 'ora.ctssd' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'SERVER1'

CRS-2676: Start of 'ora.asm' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'SERVER1'

CRS-2676: Start of 'ora.storage' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'SERVER1'

CRS-2676: Start of 'ora.crf' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'SERVER1'

CRS-2676: Start of 'ora.crsd' on 'SERVER1' succeeded

CRS-6023: Starting Oracle Cluster Ready Services-managed resources

CRS-6017: Processing resource auto-start for servers: SERVER1

CRS-2672: Attempting to start 'ora.net1.network' on 'SERVER1'

CRS-2672: Attempting to start 'ora.oc4j' on 'SERVER1'

CRS-2676: Start of 'ora.net1.network' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.cvu' on 'SERVER1'

CRS-2672: Attempting to start 'ora.SERVER1.vip' on 'SERVER1'

CRS-2672: Attempting to start 'ora.inp44xpdb1284.vip' on 'SERVER1'

CRS-2672: Attempting to start 'ora.scan1.vip' on 'SERVER1'

CRS-2672: Attempting to start 'ora.scan3.vip' on 'SERVER1'

CRS-2672: Attempting to start 'ora.scan2.vip' on 'SERVER1'

CRS-2676: Start of 'ora.cvu' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.ons' on 'SERVER1'

CRS-2676: Start of 'ora.SERVER1.vip' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.ons' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.inp44xpdb1284.vip' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'SERVER1'

CRS-2676: Start of 'ora.scan3.vip' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'SERVER1'

CRS-2676: Start of 'ora.scan2.vip' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'SERVER1'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'SERVER1'

CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.LISTENER.lsnr' on 'SERVER1' succeeded

CRS-2672: Attempting to start 'ora.oemdbmn.db' on 'SERVER1'

CRS-2676: Start of 'ora.oc4j' on 'SERVER1' succeeded

CRS-2676: Start of 'ora.oemdbmn.db' on 'SERVER1' succeeded

CRS-6016: Resource auto-start has completed for server SERVER1

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2017/08/10 13:07:53 CLSRSC-343: Successfully started Oracle Clusterware stack

PRKO-2190 : VIP exists for node SERVER1, VIP name SERVER1-vip.vsnl.co.in

PRKZ-1072 : SCAN name "XPDBoemdbmn" is already registered on network 1

PRCS-1028 : Single Client Access Name (SCAN) listener resources already exist on network 1

PRCN-3004 : Listener MGMTLSNR already exists

PRCR-1086 : resource ora.cvu is already registered

PRCC-1014 : scan1 was already running

PRCR-1004 : Resource ora.scan1.vip is already running

PRCR-1079 : Failed to start resource ora.scan1.vip

CRS-5702: Resource 'ora.scan1.vip' is already running on 'SERVER1'

PRCC-1014 : scan2 was already running

PRCR-1004 : Resource ora.scan2.vip is already running

PRCR-1079 : Failed to start resource ora.scan2.vip

CRS-5702: Resource 'ora.scan2.vip' is already running on 'SERVER1'

PRCC-1014 : scan3 was already running

PRCR-1004 : Resource ora.scan3.vip is already running

PRCR-1079 : Failed to start resource ora.scan3.vip

CRS-5702: Resource 'ora.scan3.vip' is already running on 'SERVER1'

PRCC-1014 : LISTENER_SCAN1 was already running

PRCR-1004 : Resource ora.LISTENER_SCAN1.lsnr is already running

PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr

CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'SERVER1'

PRCC-1014 : LISTENER_SCAN2 was already running

PRCR-1004 : Resource ora.LISTENER_SCAN2.lsnr is already running

PRCR-1079 : Failed to start resource ora.LISTENER_SCAN2.lsnr

CRS-5702: Resource 'ora.LISTENER_SCAN2.lsnr' is already running on 'SERVER1'

PRCC-1014 : LISTENER_SCAN3 was already running

PRCR-1004 : Resource ora.LISTENER_SCAN3.lsnr is already running

PRCR-1079 : Failed to start resource ora.LISTENER_SCAN3.lsnr

CRS-5702: Resource 'ora.LISTENER_SCAN3.lsnr' is already running on 'SERVER1'

PRCC-1014 : cvu was already running

PRCR-1004 : Resource ora.cvu is already running

PRCR-1079 : Failed to start resource ora.cvu

CRS-5702: Resource 'ora.cvu' is already running on 'SERVER1'

Preparing packages...

cvuqdisk-1.0.9-1.x86_64

PRCN-3004 : Listener LISTENER already exists

PRCC-1015 : LISTENER was already running on SERVER1

PRCR-1004 : Resource ora.LISTENER.lsnr is already running

2017/08/10 13:08:36 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded