Operating System: Oracle Enterprise Linux 5.5 (2.6.18-194.el5PAE):
Server: All the RAC Nodes
Oracle Clusterware (11.1.0.7):
Server: All the RAC Nodes
ORACLE_BASE: /u01/app/oracle
ORACLE_HOME: /u01/app/oracle/crs
Owner: oracle (Primary Group: oinstall, Secondary
Group: dba)
Permissions: 775
OCR/Voting Disk Storage Type: Raw Devices
Oracle Inventory Location: /u01/app/oraInventory
Oracle Database Software (RAC 11.1.0.7) for ASM_HOME:
Server: All the RAC Nodes
ORACLE_BASE: /u01/app/oracle
ORACLE_HOME: /u01/app/oracle/asm
Owner: oracle (Primary Group: oinstall, Secondary
Group: dba)
Permissions: 775
Oracle Inventory Location: /u01/app/oraInventory
Listener: LISTENER (TCP:1521)
Oracle Database Software (RAC 11.1.0.7) for DB_HOME:
Server: All the RAC Nodes
ORACLE_BASE: /u01/app/oracle
ORACLE_HOME: /u01/app/oracle/EBSDB
Owner: oracle (Primary Group: oinstall, Secondary
Group: dba)
Permissions: 775
Oracle Inventory Location: /u01/app/oraInventory
Database Name: LABDB
Listener: LAB_LISTENER (TCP:1530)
Openfiler 2.3:
Server: single dedicated server acting as NAS.
OS: Openfiler 2.3 (2.6.26.8-1.0.11.smp.pae.gcc3.4.x86.i686).
Existing 11Gr1 RAC
Network Setup:
|
Machine |
Public Name |
Private Name |
VIP Name |
Resolved By |
|
RAC Node1 |
node1.hingu.net |
node1-prv |
node1-vip.hingu.net |
DNS
and /etc/hosts |
|
RAC Node2 |
node2.hingu.net |
node2-prv |
node2-vip.hingu.net |
DNS
and /etc/hosts |
|
RAC Node3 |
node3.hingu.net |
node3-prv |
node3-vip.hingu.net |
DNS
and /etc/hosts |
|
Storage |
nas-server |
N/A |
N/A |
N/A |
|
DNS server |
lab-dns.hingu.net |
N/A |
N/A |
N/A |
Public Network: 192.168.2.0/eth2
Private network (cluster Interconnect): 192.168.0.0/eth0
Private network (Storage Network): 192.168.1.0/eth1
|
Machine |
Public IP |
Private IP |
VIP |
Storage IP |
Resolved By |
|
RAC
Node1 |
192.168.2.1 |
192.168.0.1 |
192.168.2.51 |
192.168.1.1 |
DNS and /etc/hosts |
|
RAC
Node2 |
192.168.2.2 |
192.168.0.2 |
192.168.2.52 |
192.168.1.2 |
DNS and /etc/hosts |
|
RAC
Node3 |
192.168.2.3 |
192.168.0.3 |
192.168.2.53 |
192.168.1.3 |
DNS and /etc/hosts |
|
Storage |
N/A |
N/A |
N/A |
192.168.1.101 |
DNS and /etc/hosts |
|
DNS
server |
192.168.2.200 |
N/A |
N/A |
N/A |
N/A |
Network Configuration for RAC Nodes/NAS
Server/DNS Host:
Public, VIPs and SCAN VIPs are resolved by DNS. The
private IPs for Cluster Interconnects are resolved through /etc/hosts. The hostname along with public/private and NAS network
is configured at the time of OEL network installations. The final Network Configurations
files are listed here.
(a) hostname:
For Node
node1:
[root@node1 ~]#
hostname node1.hingu.net
node1.hingu.net: /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=node1.hingu.net
For Node
node2:
[root@node2 ~]# hostname
node2.hingu.net
node2.hingu.net: /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=node2.hingu.net
For Node
node3:
[root@node3 ~]#
hostname node3.hingu.net
node3.hingu.net: /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=node3.hingu.net
For Node
nas-server:
[root@nas-server ~]#
hostname nas-server
nas-server:
/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=nas-server
For Node
lab-dns:
[root@lab-dns ~]#
hostname lab-dns
lab-dns.hingu.net:
/etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=nas-server
(b)
Private Network for Cluster Interconnect:
node1.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth0
# Linksys Gigabit Network Adapter
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.0.255
HWADDR=00:22:6B:BF:4E:60
IPADDR=192.168.0.1
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
node2.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth0
# Linksys Gigabit Network Adapter
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.0.255
HWADDR=00:22:6B:BF:4E:4B
IPADDR=192.168.0.2
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
node3.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth0
# Linksys Gigabit Network Adapter
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.0.255
HWADDR=00:22:6B:BF:4E:49
IPADDR=192.168.0.3
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
(c) Public Network:
node1.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth2
# Broadcom Corporation NetXtreme BCM5751
Gigabit Ethernet PCI Express
DEVICE=eth2
BOOTPROTO=static
BROADCAST=192.168.2.255
HWADDR=00:18:8B:04:6A:62
IPADDR=192.168.2.1
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.2.0
ONBOOT=yes
node2.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth2
# Broadcom Corporation NetXtreme BCM5751
Gigabit Ethernet PCI Express
DEVICE=eth2
BOOTPROTO=static
BROADCAST=192.168.2.255
HWADDR=00:18:8B:24:F8:58
IPADDR=192.168.2.2
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.2.0
ONBOOT=yes
node3.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth2
# Broadcom Corporation NetXtreme BCM5751
Gigabit Ethernet PCI Express
DEVICE=eth2
BOOTPROTO=static
BROADCAST=192.168.2.255
HWADDR=00:19:B9:0C:E6:EF
IPADDR=192.168.2.3
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.2.0
ONBOOT=yes
lab-dns.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth0
# Broadcom Corporation NetXtreme BCM5751
Gigabit Ethernet PCI Express
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.2.255
HWADDR=00:13:72:A1:E9:1B
IPADDR=192.168.2.200
NETMASK=255.255.255.0
NETWORK=192.168.2.0
ONBOOT=yes
(d) Private
Network for Shared Storage:
node1.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth1
# Linksys Gigabit Network Adapter
DEVICE=eth1
BOOTPROTO=static
BROADCAST=192.168.1.255
HWADDR=00:22:6B:BF:4E:60
IPADDR=192.168.1.1
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes
node2.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth1
# Linksys Gigabit Network Adapter
DEVICE=eth1
BOOTPROTO=static
BROADCAST=192.168.1.255
HWADDR=00:22:6B:BF:45:13
IPADDR=192.168.1.2
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes
node3.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth1
# Linksys Gigabit Network Adapter
DEVICE=eth1
BOOTPROTO=static
BROADCAST=192.168.1.255
HWADDR=00:22:6B:BF:4E:48
IPADDR=192.168.1.3
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes
nas-server.hingu.net:
/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
BROADCAST=192.168.1.255
HWADDR=00:22:6B:BF:43:D6
IPADDR=192.168.1.101
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes
TYPE=Ethernet
(e) /etc/hosts files:
node1.hingu.net: /etc/hosts
# Do not remove the following line, or various
programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain
localhost
::1 localhost6.localdomain6
localhost6
##=======================================
# Pulic Network
##=======================================
192.168.2.1 node1.hingu.net node1
192.168.2.2 node2.hingu.net node2
192.168.2.3 node3.hingu.net node3
##=======================================
# VIPs
##=======================================
192.168.2.51 node1-vip.hingu.net node1-vip
192.168.2.52 node2-vip.hingu.net node2-vip
192.168.2.53 node3-vip.hingu.net node3-vip
##=======================================
# Private Network for Cluster Interconnect
##=======================================
192.168.0.1 node1-prv
192.168.0.2 node2-prv
192.168.0.3
node3-prv
##=======================================
##=======================================
node2.hingu.net:
/etc/hosts
# # Do not remove the following line,
or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain
localhost
::1 localhost6.localdomain6
localhost6
##=======================================
# Pulic Network
##=======================================
192.168.2.1 node1.hingu.net node1
192.168.2.2 node2.hingu.net node2
192.168.2.3 node3.hingu.net node3
##=======================================
# VIPs
##=======================================
192.168.2.51 node1-vip.hingu.net node1-vip
192.168.2.52 node2-vip.hingu.net node2-vip
192.168.2.53 node3-vip.hingu.net node3-vip
##=======================================
# Private Network for Cluster Interconnect
##=======================================
192.168.0.1 node1-prv
192.168.0.2 node2-prv
192.168.0.3 node3-prv
##=======================================
##=======================================
node3.hingu.net: /etc/hosts
# # Do not remove the following line,
or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain
localhost
::1 localhost6.localdomain6
localhost6
##=======================================
# Pulic Network
##=======================================
192.168.2.1 node1.hingu.net node1
192.168.2.2 node2.hingu.net node2
192.168.2.3 node3.hingu.net node3
##=======================================
# VIPs
##=======================================
192.168.2.51 node1-vip.hingu.net node1-vip
192.168.2.52 node2-vip.hingu.net node2-vip
192.168.2.53 node3-vip.hingu.net node3-vip
##=======================================
# Private Network for Cluster Interconnect
##=======================================
192.168.0.1 node1-prv
192.168.0.2 node2-prv
192.168.0.3 node3-prv
##=======================================
##=======================================
lab-dns.hingu.net: /etc/hosts
# # Do not remove the following line, or various
programs
# that require network functionality will fail.
127.0.0.1 lab-dns.hingu.net
lab-dns localhost.localdomain localhost
::1 localhost6.localdomain6
localhost6
Configure
DNS server Node VIPs and Node IPs:
DNS Server:
lab-dns.hingu.net
RPMs
required in setting up DNS server:
ypbind-1.19-12.el5
bind-utils-9.3.6-4.P1.el5_4.2
bind-9.3.6-4.P1.el5_4.2
system-config-bind-4.0.3-4.0.1.el5
bind-libs-9.3.6-4.P1.el5_4.2
bind-chroot-9.3.6-4.P1.el5_4.2
Configurations
files modified/created to set up DNS:
lab-dbs.hingu.net
/var/named/chroot/etc/named.conf
(modified)
/var/named/chroot/var/named/hingu.net.zone
(created)
/var/named/chroot/var/named/2.168.192.in-addr.arpa.zone
(created)
/var/named/chroot/var/named/1.168.192.in-addr.arpa.zone
(created)
On node1, node2 and node3
/etc/resolv.conf
(modified)
/var/named/chroot/etc/named.conf
// Enterprise Linux BIND Configuration Tool
//
// Default initial "Caching Only"
name server configuration
//
options {
directory
"/var/named";
dump-file
"/var/named/data/cache_dump.db";
statistics-file
"/var/named/data/named_stats.txt";
};
// Zone for this RAC configuration is hingu.net
zone "hingu.net" in {
type
master;
file
"hingu.net.zone";
allow-update { none; };
};
// For reverse lookups
zone "2.168.192.in-addr.arpa" in {
type
master;
file
"2.168.192.in-addr.arpa.zone";
allow-update { none; };
};
zone "1.168.192.in-addr.arpa" in {
type
master;
file
"1.168.192.in-addr.arpa.zone";
allow-update { none; };
};
include "/etc/rndc.key";
/var/named/chroot/var/named/hingu.net.zone
$TTL
1d
hingu.net.
IN SOA lab-dns.hingu.net. root.hingu.net. (
100 ; se = serial number
8h ; ref = refresh
5m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS lab-dns.hingu.net.
; DNS server
lab-dns
IN A 192.168.2.200
; Application Nodes Public name
appsnode1 IN
A 192.168.2.11
appsnode2 IN
A 192.168.2.22
; RAC Nodes Public name
node1
IN A 192.168.2.1
node2
IN A 192.168.2.2
node3
IN A 192.168.2.3
; RAC Nodes Public VIPs
node1-vip
IN A 192.168.2.51
node2-vip
IN A 192.168.2.52
node3-vip
IN A 192.168.2.53
; Storage Network
nas-server IN
A 192.168.1.101
node1-nas
IN A 192.168.1.1
node2-nas
IN A 192.168.1.2
node3-nas IN
A 192.168.1.3
/var/named/chroot/var/named/2.168.192.in-addr.arpa.zone
$TTL
1d
@
IN SOA lab-dns.hingu.net. root.hingu.net. (
100 ; se = serial number
8h ; ref = refresh
5m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS lab-dns.hingu.net.
; DNS machine name in reverse
200
IN PTR lab-dns.hingu.net.
; Application Nodes Public Name in Reverse
11
IN PTR appsnode1.hingu.net.
22
IN PTR appsnode2.hingu.net.
; RAC Nodes Public Name in Reverse
1
IN PTR node1.hingu.net.
2
IN PTR node2.hingu.net.
3
IN PTR node3.hingu.net.
; RAC Nodes Public VIPs in Reverse
51
IN PTR node1-vip.hingu.net.
52
IN PTR node2-vip.hingu.net.
53
IN PTR node3-vip.hingu.net.
/var/named/chroot/var/named/1.168.192.in-addr.arpa.zone
$TTL
1d
@
IN SOA lab-dns.hingu.net. root.hingu.net. (
100 ; se = serial number
8h ; ref = refresh
5m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS lab-dns.hingu.net.
;
Storage Network Reverse Lookup
101
IN PTR nas-server.hingu.net.
1
IN PTR node1-nas.hingu.net.
2
IN PTR node2-nas.hingu.net.
3
IN PTR node3-nas.hingu.net.
/etc/resolv.conf (on
RAC nodes):
search hingu.net
nameserver 192.168.2.200
Start
the DNS Service (named):
service named start
chkconfig --level 35 named on
Enable Name Service
Cache Daemon nscd: (On all the RAC
Nodes)
chkconfig --level 35 nscd on
service ncsd start