Adding Second SCAN listeners on a Second Network to a 12c RAC Cluster

Environment:

OS : Oracle Enterprise Linux 7.4

Oracle : Oracle Database 12 R2 RAC
nodes : rac01.localdomain, rac02.localdomain

Generally if we use single switch and network for public interface of RAC then there might be outage in case of the network failure and network component failure. Lets say that if there is problem in network switch then database will not be available. We can provide network redundancy with the network bonding. With the help of network bonding we can create two set of network acting as one.
We can add second network in RAC and create VIP on second network. I am adding Second network with Switch II and interfaces enp0s9 in each rac nodes. In case there is any issue on first network all the request will go through second network. Adding second network with the subnet of 192.168.3.0/24 for load balancing and failover. Prior to Oracle 12c we can add VIP listeners on second network and scan listener can listen only in the default first network. From Oracle 12c we can add VIP listeners and SCAN listeners on second network too.

Adding DNS record.

Editing my zone file with vi.

Following is my current DNS entry.

Adding new DNS entries.

rac01-bck IN A 192.168.3.101

rac02-bck IN A 192.168.3.102

rac01-bck-vip IN A 192.168.3.201

rac02-bck-vip IN A 192.168.3.202

rac-bck-scan IN A 192.168.3.203

rac-bck-scan IN A 192.168.3.204

rac-bck-scan IN A 192.168.3.205

Restarting DNS service

/etc/init.d/named restart

Verify that newly added DNS entries have been resolved.
nslookup rac01-bck.localdomain
nslookup rac02-bck.localdomain

nslookup rac-bck-scan.localdomain

I am using Virtual Box. Adding new interface on Virtual box. I am adding Adapter 3 that is attached to Host-only Adapter.

Adding adapter for second node 12cR2RAC2

Once network interface has been added on both nodes login to first node as user root. rac01 is the first node in my case and configure network with nmtui GUI tool.

nmtui

Below is the landing page Click on OK to continue.

Click on Add.

Select Ethernet and Click on Create.

Type enp0s9 in the profile name, Device name as enp0s9. Type Address as 192.168.3.101/24 and click on OK.

Once adding is competed click on back

Select and click Quit to quit from the interface.

Once network interface configuration is completed, verify IP address has been set properly with ipconfig command.

ipconfig

Repeat the same task on next node rac02 and verify IP address has been set properly.

Add following host entries on /etc/hosts file.

192.168.3.101 rac01-bck.localdomain
192.168.3.102 rac02-bck.localdomain

192.168.3.201 rac01-bck-vip.localdomain
192.168.3.202 rac02-bck-vip.localdomain

192.168.3.203 rac-bck-scan.localdomain rac-bck-scan
192.168.3.204 rac-bck-scan.localdomain rac-bck-scan
192.168.3.205 rac-bck-scan.localdomain rac-bck-scan

Verify new network addresses have been resolved . Use nslookup command to verify it.

nslookup rac01-bck

nslookup rac02-bck

nslookup rac-bck-scan

Change directory to GRID_HOME/bin and execute following statements to add network 192.168.3.0/24 to net number 2 and add vip associated with net number 2 to respected nodes.
cd /u01/app/12.2.0/grid/bin
./srvctl add network -netnum 2 -subnet 192.168.3.0/255.255.255.0
./srvctl add vip -node rac01 -address 192.168.3.201/255.255.255.0 -netnum 2
./srvctl add vip -node rac02 -address 192.168.3.202/255.255.255.0 -netnum 2

Verify that new network has been added and online on both nodes. Use crsctl stat res -t command to list all the resources and its status.
./crsctl stat res -t

In the same way verify newly added vip is listed on the crsctl stat command output.

Add second listener LISTENER_BCK to second network using TCP with port 1522.
./srvctl add listener -listener LISTENER_bck -netnum 2 -user oracle -endpoints TCP:1522

Try to start new listener with no success. We need to create directory with name oracle under GRID_HOME/network/admin. The new listener configuration files will be stored on that location with names listener.ora.net.rac01 and listener.orac.net.rac02 on nodes rac01 and rac02 respectively.

cd /u01/app/12.2.0/grid/bin
./srvctl start listener -listener LISTENER_BCK

Creating oracle directory under $GRID_HOME/network/admin on rac01.
mkdir -p /u01/app/12.2.0/grid/network/admin/oracle
chown -R oracle:oinstall /u01/app/12.2.0/grid/network/admin/oracle

Creating oracle directory under $GRID_HOME/network/admin on rac02.
mkdir -p /u01/app/12.2.0/grid/network/admin/oracle

Now we can start the newly created listener LISTENER_BCK successfully.
./srvctl start listener -listener LISTENER_BCK

Verify the new listener started successfully with crsctl command
./crsctl stat res -t

Add new scan name
./srvctl add scan -scanname rac-bck-scan -netnum 2

Verify new scan has been created and is currently offline.
./crsctl stat res -t

Add scan listener using TCP and port 1522 to net number 2
./srvctl add scan_listener -listener LISTSCAN -endpoints TCP:1522 -netnum 2

Add following tns configuration on tnsnames files on DB home directory.

on rac01

LISTENER =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.201)(PORT = 1521))

)

LISTENER_BCK =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.201)(PORT = 1522))

)

R_LISTENER=

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.203)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.204)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.205)(PORT = 1521))

)

R_LISTENER_BCK=

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.203)(PORT = 1522))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.204)(PORT = 1522))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.205)(PORT = 1522))

)

on rac02

LISTENER =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.202)(PORT = 1521))

)

LISTENER_BCK =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.202)(PORT = 1522))

)

R_LISTENER=

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.203)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.204)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.205)(PORT = 1521))

)

R_LISTENER_BCK=

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.203)(PORT = 1522))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.204)(PORT = 1522))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.205)(PORT = 1522))

)

Now login to rac database from one of the node and modify listener_networks parameter so that PMON or LREG process can register the services to listeners on both network.

In statement below we are adding two network with following details:

	Network I	Network II	Description
Name	network1	network2	Defines the name it can be anything
Local_Listener	LISTENER	LISTENER_BCK	Defines the local listener. It is defined on tnsnames.ora and contains the VIP of the server
Remote_Listener	rac-scan:1521	rac-bck-scan:1522	Scan name of the first and second network with the port on which listener is active.

alter system set listener_networks='((NAME=network1)(LOCAL_LISTENER=LISTENER)(REMOTE_LISTENER=rac-scan:1521))','((NAME=network2)(LOCAL_LISTENER=LISTENER_BCK)(REMOTE_LISTENER=rac-bck-scan:1522))' sid='*';

Start scan listener on net number 2
./srvctl start scan_listener -netnum 2

Verify the listener has been started using crsctl stat res -t command.
./crsctl stat res -t

Check listener status with lsnrctl command.
lsnrctl status

Check the status of LISTENER_BCK listener.
set current_listener LISTENER_BCK
status

On client site create tns setting to connect to RAC on both the scans. We can use LOAD_BALANCE and FAILOVER for high availability and load balancing.