3-Node RAC Setup

 

Operating System: Oracle Enterprise Linux 5.5 (2.6.18-194.el5PAE):

Server: All the RAC Nodes

 

Oracle Clusterware (11.1.0.7):

Server: All the RAC Nodes

ORACLE_BASE: /u01/app/oracle

ORACLE_HOME: /u01/app/oracle/crs

Owner: oracle (Primary Group: oinstall, Secondary Group:  dba)

Permissions: 775

OCR/Voting Disk Storage Type: Raw Devices

Oracle Inventory Location: /u01/app/oraInventory

               

Oracle Database Software (RAC 11.1.0.7) for ASM_HOME:

 

Server: All the RAC Nodes

ORACLE_BASE: /u01/app/oracle

ORACLE_HOME: /u01/app/oracle/asm

Owner: oracle (Primary Group: oinstall, Secondary Group:  dba)

Permissions: 775

Oracle Inventory Location: /u01/app/oraInventory

Listener: LISTENER (TCP:1521)

 

Oracle Database Software (RAC 11.1.0.7) for DB_HOME:

 

Server: All the RAC Nodes

ORACLE_BASE: /u01/app/oracle

ORACLE_HOME: /u01/app/oracle/EBSDB

Owner: oracle (Primary Group: oinstall, Secondary Group:  dba)

Permissions: 775

Oracle Inventory Location: /u01/app/oraInventory

 

Database Name: LABDB

Listener: LAB_LISTENER (TCP:1530)

 

Openfiler 2.3:

Server: single dedicated server acting as NAS.

OS: Openfiler 2.3 (2.6.26.8-1.0.11.smp.pae.gcc3.4.x86.i686).

 

 

Existing 11Gr1 RAC Network Setup:

 

Machine

Public Name

Private  Name

VIP Name

Resolved By

RAC Node1

node1.hingu.net

node1-prv

node1-vip.hingu.net

DNS and /etc/hosts

RAC Node2

node2.hingu.net

node2-prv

node2-vip.hingu.net

DNS and /etc/hosts

RAC Node3

node3.hingu.net

node3-prv

node3-vip.hingu.net

DNS and /etc/hosts

Storage

nas-server

N/A

N/A

N/A

DNS server

lab-dns.hingu.net

N/A

N/A

N/A

 

 

 

 

 

 

 

 

Public Network: 192.168.2.0/eth2

Private network (cluster Interconnect): 192.168.0.0/eth0

Private network (Storage Network): 192.168.1.0/eth1

 

Machine

Public IP

Private IP

VIP

Storage IP

Resolved By

RAC Node1

192.168.2.1

192.168.0.1

192.168.2.51

192.168.1.1

DNS and /etc/hosts

RAC Node2

192.168.2.2

192.168.0.2

192.168.2.52

192.168.1.2

DNS and /etc/hosts

RAC Node3

192.168.2.3

192.168.0.3

192.168.2.53

192.168.1.3

DNS and /etc/hosts

Storage

N/A

N/A

N/A

192.168.1.101

DNS and /etc/hosts

DNS server

192.168.2.200

N/A

N/A

N/A

N/A

 

 

 

 

 

 

3_node_RAC_diagram.jpg

 

 

Network Configuration for RAC Nodes/NAS Server/DNS Host:

 

Public, VIPs and SCAN VIPs are resolved by DNS. The private IPs for Cluster Interconnects are resolved through /etc/hosts. The hostname along with public/private and NAS network is configured at the time of OEL network installations. The final Network Configurations files are listed here.

 

(a)    hostname:

 

For Node node1:

[root@node1 ~]# hostname node1.hingu.net

 

       node1.hingu.net: /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=yes

HOSTNAME=node1.hingu.net

 

For Node node2:

[root@node2 ~]# hostname node2.hingu.net

 

       node2.hingu.net: /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=yes

HOSTNAME=node2.hingu.net

 

For Node node3:

[root@node3 ~]# hostname node3.hingu.net

 

        node3.hingu.net: /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=yes

HOSTNAME=node3.hingu.net

 

For Node nas-server:

[root@nas-server ~]# hostname nas-server

 

        nas-server: /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=nas-server

 

For Node lab-dns:

[root@lab-dns ~]# hostname lab-dns

 

        lab-dns.hingu.net: /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=yes

HOSTNAME=nas-server

 

(b)   Private Network for Cluster Interconnect:

 

node1.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth0

# Linksys Gigabit Network Adapter

DEVICE=eth0

BOOTPROTO=static

BROADCAST=192.168.0.255

HWADDR=00:22:6B:BF:4E:60

IPADDR=192.168.0.1

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.0.0

ONBOOT=yes

 

node2.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth0

# Linksys Gigabit Network Adapter

DEVICE=eth0

BOOTPROTO=static

BROADCAST=192.168.0.255

HWADDR=00:22:6B:BF:4E:4B

IPADDR=192.168.0.2

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.0.0

ONBOOT=yes

 

node3.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth0

# Linksys Gigabit Network Adapter

DEVICE=eth0

BOOTPROTO=static

BROADCAST=192.168.0.255

HWADDR=00:22:6B:BF:4E:49

IPADDR=192.168.0.3

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.0.0

ONBOOT=yes

 

(c)    Public Network:

 

node1.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth2

# Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express

DEVICE=eth2

BOOTPROTO=static

BROADCAST=192.168.2.255

HWADDR=00:18:8B:04:6A:62

IPADDR=192.168.2.1

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.2.0

ONBOOT=yes

 

node2.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth2

# Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express

DEVICE=eth2

BOOTPROTO=static

BROADCAST=192.168.2.255

HWADDR=00:18:8B:24:F8:58

IPADDR=192.168.2.2

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.2.0

ONBOOT=yes

 

node3.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth2

# Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express

DEVICE=eth2

BOOTPROTO=static

BROADCAST=192.168.2.255

HWADDR=00:19:B9:0C:E6:EF

IPADDR=192.168.2.3

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.2.0

ONBOOT=yes

 

lab-dns.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth0

# Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express

DEVICE=eth0

BOOTPROTO=static

BROADCAST=192.168.2.255

HWADDR=00:13:72:A1:E9:1B

IPADDR=192.168.2.200

NETMASK=255.255.255.0

NETWORK=192.168.2.0

ONBOOT=yes

 

(d)   Private Network for Shared Storage:

 

node1.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth1

# Linksys Gigabit Network Adapter

DEVICE=eth1

BOOTPROTO=static

BROADCAST=192.168.1.255

HWADDR=00:22:6B:BF:4E:60

IPADDR=192.168.1.1

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.1.0

ONBOOT=yes

 

node2.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth1

# Linksys Gigabit Network Adapter

DEVICE=eth1

BOOTPROTO=static

BROADCAST=192.168.1.255

HWADDR=00:22:6B:BF:45:13

IPADDR=192.168.1.2

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.1.0

ONBOOT=yes

 

node3.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth1

# Linksys Gigabit Network Adapter

DEVICE=eth1

BOOTPROTO=static

BROADCAST=192.168.1.255

HWADDR=00:22:6B:BF:4E:48

IPADDR=192.168.1.3

IPV6INIT=yes

IPV6_AUTOCONF=yes

NETMASK=255.255.255.0

NETWORK=192.168.1.0

ONBOOT=yes

 

 

nas-server.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1

BOOTPROTO=static

BROADCAST=192.168.1.255

HWADDR=00:22:6B:BF:43:D6

IPADDR=192.168.1.101

NETMASK=255.255.255.0

NETWORK=192.168.1.0

ONBOOT=yes

TYPE=Ethernet

 

(e)   /etc/hosts files:

 

node1.hingu.net: /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1            localhost.localdomain localhost

::1           localhost6.localdomain6 localhost6

 

##=======================================

# Pulic Network

##=======================================

 

192.168.2.1             node1.hingu.net  node1

192.168.2.2             node2.hingu.net  node2

192.168.2.3             node3.hingu.net  node3

 

 

##=======================================

# VIPs

##=======================================

 

192.168.2.51             node1-vip.hingu.net node1-vip

192.168.2.52             node2-vip.hingu.net node2-vip

192.168.2.53             node3-vip.hingu.net node3-vip

 

##=======================================

# Private Network for Cluster Interconnect

##=======================================

 

192.168.0.1             node1-prv

192.168.0.2             node2-prv

192.168.0.3             node3-prv

 

##=======================================

##=======================================

 

node2.hingu.net: /etc/hosts

# # Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1            localhost.localdomain localhost

::1           localhost6.localdomain6 localhost6

 

##=======================================

# Pulic Network

##=======================================

 

192.168.2.1             node1.hingu.net  node1

192.168.2.2             node2.hingu.net  node2

192.168.2.3             node3.hingu.net  node3

 

 

##=======================================

# VIPs

##=======================================

 

192.168.2.51             node1-vip.hingu.net node1-vip

192.168.2.52             node2-vip.hingu.net node2-vip

192.168.2.53             node3-vip.hingu.net node3-vip

 

##=======================================

# Private Network for Cluster Interconnect

##=======================================

 

192.168.0.1             node1-prv

192.168.0.2             node2-prv

192.168.0.3             node3-prv

 

##=======================================

##=======================================

 

 

node3.hingu.net: /etc/hosts

# # Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1            localhost.localdomain localhost

::1           localhost6.localdomain6 localhost6

 

 

##=======================================

# Pulic Network

##=======================================

 

192.168.2.1             node1.hingu.net  node1

192.168.2.2             node2.hingu.net  node2

192.168.2.3             node3.hingu.net  node3

 

 

##=======================================

# VIPs

##=======================================

 

192.168.2.51             node1-vip.hingu.net node1-vip

192.168.2.52             node2-vip.hingu.net node2-vip

192.168.2.53             node3-vip.hingu.net node3-vip

 

##=======================================

# Private Network for Cluster Interconnect

##=======================================

 

192.168.0.1             node1-prv

192.168.0.2             node2-prv

192.168.0.3             node3-prv

 

##=======================================

##=======================================

 

lab-dns.hingu.net: /etc/hosts

# # Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1            lab-dns.hingu.net lab-dns localhost.localdomain localhost

::1           localhost6.localdomain6 localhost6

 

Configure DNS server Node VIPs and Node IPs:

 

DNS Server: lab-dns.hingu.net

 

RPMs required in setting up DNS server:

 

ypbind-1.19-12.el5

bind-utils-9.3.6-4.P1.el5_4.2

bind-9.3.6-4.P1.el5_4.2

system-config-bind-4.0.3-4.0.1.el5

bind-libs-9.3.6-4.P1.el5_4.2

bind-chroot-9.3.6-4.P1.el5_4.2

 

Configurations files modified/created to set up DNS:

 

lab-dbs.hingu.net

 

/var/named/chroot/etc/named.conf (modified)

/var/named/chroot/var/named/hingu.net.zone (created)

/var/named/chroot/var/named/2.168.192.in-addr.arpa.zone (created)

/var/named/chroot/var/named/1.168.192.in-addr.arpa.zone (created)

 

On node1, node2 and node3

 

/etc/resolv.conf (modified)

 

/var/named/chroot/etc/named.conf

 

// Enterprise Linux BIND Configuration Tool

//

// Default initial "Caching Only" name server configuration

//

 

options {

       directory "/var/named";

       dump-file "/var/named/data/cache_dump.db";

        statistics-file "/var/named/data/named_stats.txt";

};

 

 

// Zone for this RAC configuration is hingu.net

 

zone "hingu.net" in {

  type master;

  file "hingu.net.zone";

  allow-update { none; };

};

 

// For reverse lookups

 

zone "2.168.192.in-addr.arpa" in {

  type master;

  file "2.168.192.in-addr.arpa.zone";

  allow-update { none; };

};

 

zone "1.168.192.in-addr.arpa" in {

  type master;

  file "1.168.192.in-addr.arpa.zone";

  allow-update { none; };

};

 

include "/etc/rndc.key";

 

/var/named/chroot/var/named/hingu.net.zone

 

$TTL    1d

hingu.net.  IN    SOA   lab-dns.hingu.net. root.hingu.net. (

    100        ; se = serial number

    8h         ; ref = refresh

    5m         ; ret = update retry

    3w         ; ex = expiry

    3h         ; min = minimum

    )

 

    IN    NS    lab-dns.hingu.net.

 

; DNS server

 

lab-dns    IN    A    192.168.2.200

 

; Application Nodes Public name

 

appsnode1          IN    A    192.168.2.11

appsnode2          IN    A    192.168.2.22

 

 

; RAC Nodes Public name

 

node1          IN    A    192.168.2.1

node2          IN    A    192.168.2.2

node3          IN    A    192.168.2.3

 

; RAC Nodes Public VIPs

 

node1-vip      IN    A    192.168.2.51

node2-vip      IN    A    192.168.2.52

node3-vip      IN    A    192.168.2.53

 

; Storage Network

 

nas-server IN    A    192.168.1.101

node1-nas  IN    A    192.168.1.1

node2-nas  IN    A    192.168.1.2

node3-nas  IN    A    192.168.1.3

 

 

/var/named/chroot/var/named/2.168.192.in-addr.arpa.zone

 

$TTL    1d

@   IN    SOA   lab-dns.hingu.net. root.hingu.net. (

    100        ; se = serial number

    8h         ; ref = refresh

    5m         ; ret = update retry

    3w         ; ex = expiry

    3h         ; min = minimum

    )

 

    IN    NS    lab-dns.hingu.net.

 

; DNS machine name in reverse

200        IN    PTR lab-dns.hingu.net.

 

; Application Nodes Public Name in Reverse

 

11          IN    PTR      appsnode1.hingu.net.

22          IN    PTR      appsnode2.hingu.net.

 

; RAC Nodes Public Name in Reverse

 

1          IN    PTR node1.hingu.net.

2          IN    PTR node2.hingu.net.

3          IN    PTR node3.hingu.net.

 

; RAC Nodes Public VIPs in Reverse

 

51          IN    PTR      node1-vip.hingu.net.

52          IN    PTR      node2-vip.hingu.net.

53          IN    PTR      node3-vip.hingu.net.

 

 

/var/named/chroot/var/named/1.168.192.in-addr.arpa.zone

 

$TTL    1d

@   IN    SOA   lab-dns.hingu.net. root.hingu.net. (

    100        ; se = serial number

    8h         ; ref = refresh

    5m         ; ret = update retry

    3w         ; ex = expiry

    3h         ; min = minimum

    )

 

    IN    NS    lab-dns.hingu.net.

 

;  Storage Network Reverse Lookup

 

101          IN    PTR     nas-server.hingu.net.

1            IN    PTR     node1-nas.hingu.net.

2            IN    PTR     node2-nas.hingu.net.

3            IN    PTR     node3-nas.hingu.net.

 

/etc/resolv.conf (on RAC nodes):

 

search hingu.net

nameserver 192.168.2.200

 

Start the DNS Service (named):

 

service named start

chkconfig --level 35 named on

 

Enable Name Service Cache Daemon nscd: (On all the RAC Nodes)

 

chkconfig --level 35 nscd on

service ncsd start