Oracle VM for X86 / RHEL PVM Kickstart Installation without DHCP

Just a quick and short post to share a tip related to installation of RHEL as ParaVirtual Machine using kickstart(or any distribution that can use kickstart) on Oracle VM 3 for X86 without having to rely on a DHCP for the Network Parameters. The only prerequisite that is needed is to have kickstart file and RHEL Installation Path available on the network and reachable from Oracle VM Manager Server.

Once that's done, we create our Virtual Server as usual, except that for Network Path, we're having to add more parameters. Indeed we can use the --args to add any parameters we're willing to pass to the kernel. So, we're having something like :

--args kernel_boot_paramaters http://install_server/rhel6_4

For example, let's say that my kickstart file is located on an http server and accessible via http://install_server/my_kickstart.ks, I also need to set 192.169.0.10 as IP of eth1 interface in order to get access to this kickstart file's URL and the RHEL installation tree is http://install_server/rhel6_4. Then, i'll have the following:

--args "ks=http://install_server/my_kickstart.ks ip=192.169.0.10 ksdevice=eth1 netmask=255.255.255.0 gateway=192.169.0.1" http://install_server/rhel6_4

Note that the --args section are in quote, this is needed to clearly draw the boundary between what is kernel arguments and the installation tree's URL. 

Ansible Hosts / Install alternate upgraded Python version

Following the highlighted error below which happened while trying to configure a RHEL4 node to be manageable by ansible (and that i rightly attributed to the old Python 2.3.4 which is the default version of Python for RHEL4), i've decided to install an alternate Python version on the same RHEL node without impacting the already existing environment. Below i'm sharing the 8 steps I followed to have that working.

Note that although i'm describing this Python Installation in  relation with Ansible Integration, Step 1 to Step 6 can be used for the installation of an alternate Python Environment on any other Linux Systems.

Ansible's Error:

[stivesso@ansible-server ~]$ ansible my_ansible_node -m ping  --ask-pass

SSH password:

my_ansible_node | FAILED >> {

    "failed": true,

    "msg": "\r\nSUDO-SUCCESS-josewbugtgyoijxrgkuxihoejbjsbiuq\r\n  File \"/home/esso_s/.ansible/tmp/ansible-tmp-1406122775.05-117998270117858/ping\", line 1177\r\n    clean_args = \" \".join(pipes.quote(arg) for arg in args)\r\n

     ^\r\nSyntaxError: invalid syntax\r\n", 

    "parsed": false

}

1. Install Prerequisites on the target host

# yum groupinstall "Development tools" # yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel

2. Get an updated python package

# wget http://python.org/ftp/python/2.7.5/Python-2.7.5.tgz --no-check-certificate

3. Untar/unzip the package on the target node

# tar zxvf Python-2.7.5.tgz

4. configure with the alternate path as option (here, i'm planning to install the alternate environment in /opt/python2.7) and compile/install

# ./configure --prefix=/opt/python2.7 --enable-shared # mkdir /opt/python2.7 # make # make altinstall

5. Export Shared Library and Bin Library in PATH and LD_LIBRARY_PATH on User profile 

# tail -3 /etc/profile ## Added for python 2.7 export PATH=$PATH:/opt/python2.7/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/python2.7/lib

6. Create ld.so.conf configuration file and run ldconfig to have library loaded system wide

# cat /etc/ld.so.conf.d/python27.conf /opt/python2.7/lib # ldconfig

7. Add the following to /etc/ansible/hosts on the Ansible Control Machine for the target node

my_ansible_node ansible_python_interpreter=/opt/python2.7/bin/python2.7  

8. Test again the ping module from ansible-server...

[esso_s@ansible-server ~]$ ansible my_ansible_node -m ping --ask-pass SSH password: my_ansible_node | success >> { "changed": false, "ping": "pong" }

Linux KVM / Network Bridge over bonded Interface

This is a short post about KVM Network Bridge Configuration for Guest Domain over a bonded Interface. For this post, we're using a Redhat Enterprise Linux 6 as KVM Physical System, but the procedure can be easily adjusted for any others Linux System which supports KVM. Before delving in the Technical Configuration, let’s have a look on what we’re trying to achieve. We’re having a Physical Server with 02 Physical Interfaces (eth0 and eth1) and are planning to use these two physical Interfaces in a highly available configuration. The same interfaces are also used to support a bridge that is used by KVM DomU (Guest Hosts). Below, we can see a picture describing the details of this configuration.

The first thing to do is to complete the bonding configuration and that's pretty straightforward. We're just creating the bonding.conf file in /etc/modprobe.d (to dynamically load the bonding module, bonding's option  will be modified later on and must reflects ones' needs), then we're editing the ifcfg-bond0 network configuration files.

[root@my_dom0 ]# cat /etc/modprobe.d/bonding.conf alias bond0 bonding

Let us now configure the bond0 Interface by editing its configuration files, note the BRIDGE configuration mentioned below, this is explicitly added to link the bond interface to the bridge interface. We are also setting the bond Interface parameters here.

[root@my_dom0 ]# cat /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE="bond0" ONBOOT="yes" BRIDGE=br0 NM_CONTROLLED=no BONDING_OPTS="mode=active-backup miimon=100 downdelay=200 updelay=200"

Next thing to do is to configure the bridge Interface, as usual we’re creating the ifcfg-br0 file containing our network configuration (IP, DNS...). Note that as described above, this is the Interface that will have the IP settings used to administrate the Physical Server (Note that this is a matter of architecture/design, on a typical production server with four interfaces, we could have for example separated this management network from the Guest Domains Network...)

[root@my_dom0 ]# cat /etc/sysconfig/network-scripts/ifcfg-br0 DEVICE=br0 TYPE=Bridge BOOTPROTO=static ONBOOT=yes # IP Configuration HOSTNAME="my_dom0" IPADDR="192.168.1.10" NETMASK="255.255.255.0" DNS1="192.168.1.2" DNS2="192.168.1.3" GATEWAY="192.168.1.1" IPV6INIT="yes" MTU="1500" NM_CONTROLLED="no"

Finally, we can configure the physical interface and restart the network service (or reboot the node…)

[root@my_dom0 ]# cat ifcfg-eth0 DEVICE="eth0" HWADDR="XX:XX:XX:XX:XX:X8" NM_CONTROLLED="no" ONBOOT="yes" BOOTPROTO=none MASTER=bond0 SLAVE=yes USERCTL=no [root@my_dom0 ]# cat ifcfg-eth2 DEVICE="eth2" HWADDR="XX:XX:XX:XX:XX:X9" NM_CONTROLLED="no" ONBOOT="yes" BOOTPROTO=none MASTER=bond0 SLAVE=yes USERCTL=no [root@my_dom0 ]# shutdown –r now

Check the Bridge Configuration, should have output similar to the one below. The bridged interface is now ready to be used by the Guest Domains.

[root@my_dom0 ~]# brctl show bridge name bridge id STP enabled interfaces br0 8000.b4b52f6d1bc0 no bond0

P2V Migration From Solaris 10 Global Zone to Oracle Solaris 11 Local Zone (Solaris 10 Branded Zone)

The aim of this post is to describe a Physical to Virtual (P2V) migration of a Bare-Metal Solaris 10 (a Physical Node) on a Solaris 11 Global Zone (Solaris 10 Branded Zone on Oracle Solaris 11). The scenario describes below involves a Solaris 10 Physical node named oranode (hosts an Oracle Database 10G and another Oracle application) and a Solaris 11 Global Zone named sol11-gz (already hosting some other Local Zones). 

In order to make this migration easily understandable, It has been technically divided in in 03 parts, the first parts deals with Analysis of the Source System, the second is about System/Data Collection and the third is about Target Zone Configuration and Installation.

I. Analysis of the Source System

The Data Analysis phase aims to collect some initial analysis Data. Oracle has provided a very nice tool for that phase, namely zonep2vchk. zonep2vchk serves two functions. First, it can be used to report issues on the source which might prevent a successful p2v migration. Second, it can output a template zonecfg, which can be used to assist in configuring the non-global zone target (man page).

The /usr/sbin/zonep2vchk script is belonging to pkg:/system/zones on Solaris 11 and can be copied from an Oracle Solaris 11 system to an Oracle Solaris 10 system. It is not required to run this utility from a specific location on the system.

Below, we're copying this utility to our Solaris 10 (oranode) and then we're running it with the option -T S11 which specifies that the target node for our planned migration of this Solaris 10 System is running on Oracle Solaris 11.

root@s11-gz:~# scp /usr/sbin/zonep2vchk stivesso@oranode: Password: zonep2vchk 100% |*******************************| 142 KB 00:00 root@oranode # chmod ug+x /export/home/stivesso/zonep2vchk root@oranode # /export/home/stivesso/zonep2vchk -T S11 --Executing Version: 1.0.5-11-16135 - Source System: oranode Solaris Version: Oracle Solaris 10 8/11 s10s_u10wos_17b SPARC Solaris Kernel: 5.10 Generic_147440-01 Platform: sun4u SUNW,Sun-Fire-15000 - Target System: Solaris_Version: Solaris 11 Zone Brand: solaris10 (Solaris 10 Container) IP type: exclusive --Executing basic checks - The following /etc/system tunables exist. These tunables will not function inside a zone. The /etc/system tunable may be transfered to the target global zone, but it will affect the entire system, including all zones and the global zone. If there is an alternate tunable that can be configured from within the zone, this tunable is described: forceload: drv/emcpsf No alternate tunable exists. forceload: drv/sd No alternate tunable exists. forceload: drv/ssd No alternate tunable exists. forceload: drv/emcp No alternate tunable exists. forceload: misc/emcpgpx No alternate tunable exists. forceload: misc/emcpmpx No alternate tunable exists. forceload: misc/emcpvlumd No alternate tunable exists. forceload: misc/emcpxcrypt No alternate tunable exists. forceload: misc/emcpdm No alternate tunable exists. forceload: misc/emcpioc No alternate tunable exists. set emcp:bPxEnableInit=1 zonep2vchk has no information on tunable set shmsys:shminfo_shmmax=4294967295 zonep2vchk has no information on tunable project.max-shm-memory set shmsys:shminfo_shmmin=1 Tunable is obsolete on target host set shmsys:shminfo_shmmni=100 zonep2vchk has no information on tunable project.max-shm-memory set shmsys:shminfo_shmseg=10 Tunable is obsolete on target host set semsys:seminfo_semmns=1024 Tunable is obsolete on target host set semsys:seminfo_semmsl=256 Alternate tunable exists inside a non-global zone: rctl process.max-sem-nsems set semsys:seminfo_semmni=100 Alternate tunable exists inside a non-global zone: rctl project.max-sem-ids set semsys:seminfo_semvmx=32767 Tunable is obsolete on target host set noexec_user_stack=1 zonep2vchk has no information on tunable - The following SMF services will not work in a zone: svc:/network/iscsi/initiator:default svc:/system/iscsitgt:default - The system is configured with the following non-root ZFS pools. Pools cannot be configured inside a zone, but a zone can be configured to use a pool that was set up in the global zone: swappool zpool_oracle zpool_oracle2 Basic checks compete, 25 issue(s) detected --Total issue(s) detected: 25

Quite verbose! Indeed this initial analysis give us a lot of insights for the whole p2v procedure. For example, we can see that in this study case, we have much more than 01 ZFS Zpool (beside the initial rpool). These are File-system which host the Oracle Database 10G (mentioned above) Data Files.

Thus, we must decide how these  application Data's file-systems will be migrated. The strategy we're applying here is to migrate just the Operating System (excluding the Application Data File-System), and use SAN replication for the Application Data. Note that this decision is based on the source system architecture (we may for example decide to go for a zfs send/receive if these Data file-system are not located on an external storage…).

Once done with that, we complete this analysis phase by creating a template file for the Target Zone Creation (using the same zonep2vchk tool with the option -c). This is just a template , it is used to ease the creation of the target Zone and should be adjusted to suit with the requirement of the target environment.

root@oranode # /export/home/stivesso/zonep2vchk -T S11 -c create -b set zonepath=/zones/oranode set brand=solaris10 add attr set name="zonep2vchk-info" set type=string set value="p2v of host oranode" end set ip-type=exclusive # Uncomment the following to retain original host hostid: # set hostid=94a76031 # Max procs and lwps based on max_uproc/v_proc set max-processes=20000 set max-lwps=40000 add attr set name=num-cpus set type=string set value="original system had 4 cpus" end # Only one of dedicated or capped cpu can be used. # Uncomment the following to use cpu caps: # add capped-cpu # set ncpus=4.0 # end # Uncomment the following to use dedicated cpu: # add dedicated-cpu # set ncpus=4 # end # Uncomment the following to use memory caps. # Values based on physical memory plus swap devices: # add capped-memory # set physical=16384M # set swap=119823M # end # Original ce0 interface configuration: # Statically defined oranode (oranode) # Factory assigned MAC address 0:14:4f:1e:a2:90 add anet set linkname=ce0 set lower-link=change-me # Uncomment the following to retain original link configuration: # set mac-address=0:14:4f:1e:a2:90 end # Original dman0 interface configuration: # Statically defined 192.168.1.3/27 # Factory assigned MAC address 0:0:be:a9:66:71 add anet set linkname=dman0 set lower-link=change-me # Uncomment the following to retain original link configuration: # set mac-address=0:0:be:a9:66:71 end exit

II. System/Data Collection

This phase aims to collect Source System's Data (Solaris 10 System - oranode) and transferred them to the Target System (Solaris 11 Global Zone - s11-gz). As discussed during the analysis phase, I have decided to migrate System Data and Application Data in separate processes.

For the system, I’m making use of the classic flarcreate to generate system archive (due to the presence of Data File-System that i'd like to exclude, i'm using CPIO as method to archive the file). For the Application Data, they're being migrated using External Storage Cloning features (not described here!). 

root@oranode # flarcreate -S -n oranode -x /oracle -x /zpool_oracle -x /zpool_oracle2 -L cpio /zpool_oracle/$(hostname)-$(/usr/bin/date +"%Y-%m-%d").flar Archive format requested is cpio Full Flash Checking integrity... Integrity OK. Running precreation scripts... Precreation scripts done. Creating the archive... cpio: cpio: var/adm/lastlog: too large to archive in current mode 14429359 blocks 1 error(s) Archive creation complete. Running postcreation scripts... Postcreation scripts done. Running pre-exit scripts... Pre-exit scripts done.

The transfer of this archive to the target Global Zone can be done using any conventional method (scp/sftp/ftp, SAN Replication...). Once completed, we can move to the last part.

III. Target Zone Configuration and Installation

Now that we've the Data Collection phase completed, we’re going to create the target zone and start the restoration of the flararchive that was created. We’ll also add the Zpool Dataset which were migrated using the SAN replication features.

The zone will be created using the template that was generated by the zonep2vchk tool (we'll obviously customize it to fit the target environment). Before starting, we have to create the zonepath needed for the target zone and import the zpool that was migrated through the SAN replication.

root@sol11-gz:~# zpool create zpool_oranode <local_device> root@sol11-gz:~# zfs create zpool_oranode/oranode root@sol11-gz:~# chmod 700 /zpool_oranode/oranode root@sol11-gz:~# zpool import -f zpool_oracle zpool_oranode_data

Let’s now configure our new local zone by using the template generated by zonep2vchk tool during the first phase.

root@sol11-gz:~# cat /export/zonecfg/oranode.cfg create -b set zonepath=/zpool_oranode/oranode set brand=solaris10 add attr set name="zonep2vchk-info" set type=string set value="p2v of host oranode" end set ip-type=exclusive # Uncomment the following to retain original host hostid: # set hostid=94a76031 # Max procs and lwps based on max_uproc/v_proc set max-processes=20000 set max-lwps=40000 add attr set name=num-cpus set type=string set value="original system had 4 cpus" end # Only one of dedicated or capped cpu can be used. # Uncomment the following to use cpu caps: # add capped-cpu # set ncpus=8.0 # end # Uncomment the following to use dedicated cpu: add dedicated-cpu set ncpus=8 end # Uncomment the following to use memory caps. # Values based on physical memory plus swap devices: add capped-memory set physical=16384M set swap=119823M end # Added by Steve ESSO # for the new network configuration add anet set linkname=oranode_net0 set vlan-id=10 set lower-link=net0 end add anet set linkname=oranode_net1 set vlan-id=10 set lower-link=net1 end exit root@sol11-gz:~# zonecfg -z oranode -f /export/zonecfg/oranode.cfg root@sol11-gz:~# zonecfg -z oranode zonecfg:oranode> info zonename: oranode zonepath: /zpool_oranode/oranode brand: solaris10 autoboot: false bootargs: file-mac-profile: pool: limitpriv: scheduling-class: ip-type: exclusive hostid: fs-allowed: [max-lwps: 40000] [max-processes: 20000] anet: linkname: oranode_net0 lower-link: net0 allowed-address not specified configure-allowed-address: true defrouter not specified allowed-dhcp-cids not specified link-protection: mac-nospoof mac-address: random mac-prefix not specified mac-slot not specified vlan-id: 10 priority not specified rxrings not specified txrings not specified mtu not specified maxbw not specified rxfanout not specified anet: linkname: oranode_net1 lower-link: net1 allowed-address not specified configure-allowed-address: true defrouter not specified allowed-dhcp-cids not specified link-protection: mac-nospoof mac-address: random mac-prefix not specified mac-slot not specified vlan-id: 10 priority not specified rxrings not specified txrings not specified mtu not specified maxbw not specified rxfanout not specified dedicated-cpu: ncpus: 8 capped-memory: physical: 16G [swap: 117G] attr: name: zonep2vchk-info type: string value: "p2v of host oranode" attr: name: num-cpus type: string value: "original system had 4 cpus" rctl: name: zone.max-processes value: (priv=privileged,limit=20000,action=deny) rctl: name: zone.max-lwps value: (priv=privileged,limit=40000,action=deny) rctl: name: zone.max-swap value: (priv=privileged,limit=125643522048,action=deny)

Having the configuration done, we can start the zone Installation

root@sol11-gz:~# zoneadm -z oranode install -p -a /zpool_oranode_data/oranode-25nov2013.flar root@sol11-gz:~# zoneadm -z oranode install -p -a /zpool_oranode_data/oranode-25nov2013.flar Progress being logged to /var/log/zones/zoneadm.20131126T212904Z.oranode.install Installing: This may take several minutes... Postprocessing: This may take a while... Postprocess: Updating the image to run within a zone Postprocess: Migrating data from: zpool_oranode/oranode/rpool/ROOT/zbe-0 to: zpool_oranode/oranode/rpool/export Postprocess: A backup copy of /export is stored at /export.backup.20131126T213800Z. It can be deleted after verifying it was migrated correctly. Postprocess: Migrating data from: zpool_oranode/oranode/rpool/ROOT/zbe-0 to: zpool_oranode/oranode/rpool Postprocess: A backup copy of /rpool is stored at /rpool.backup.20131126T213856Z. It can be deleted after verifying it was migrated correctly. Result: Installation completed successfully. Log saved in non-global zone as /zpool_oranode/oranode/root/var/log/zones/zoneadm.20131126T212904Z.oranode.install

Finally, We are adding the ZFS dataset migrated through SAN Replication,

root@sol11-gz:~# zonecfg -z oranode zonecfg:oranode> add dataset zonecfg:oranode:dataset> set name=zpool_oranode_data zonecfg:oranode:dataset> set alias=zpool_oranode_data zonecfg:oranode:dataset> end zonecfg:oranode> commit zonecfg:oranode> exit

And the big moment! Boot the new zone...

root@sol11-gz:~# zoneadm -z oranode boot root@sol11-gz:~# zlogin -C oranode

Oracle Solaris 11: EMC PowerPath / powermt display dev ; Device(s) not found.

This is just a small post about Oracle Solaris 11 and EMC PowerPath Configuration. If you've been trying to get EMC PowerPath configured on Solaris 11 and facing error(s) similar to the one below, then follow steps described below (can also follow the same for initial PowerPath Configuration on Oracle Solaris 11)

root@stiv-sol11:~# powermt display Device(s) not found. root@stiv-sol11:~# powermt display dev=all Device(s) not found.


1. Make sure that Zoning/Lun Mapping from SAN/Storage is well configured.

For the zoning , run fcinfo remote-port -p <pwwn_connected_hba> (must list the PWWN and others details of the storage, otherwise must check that the zoning is well complete);

For the Storage Configuration, you can use fcinfo lu -v (list all fibre channel  logical  units, must have one or more "OS Device Name" listed here, otherwise must check that mapping is well done at the SAN Level). 

root@stiv-sol11:~# fcinfo remote-port -p <local_pwwn_port01> Remote Port WWN: <san_pwwn_spa_p1> Active FC4 Types: SCSI SCSI Target: yes Port Symbolic Name: DGC LUNZ 0430 Node WWN: <san_wwn_spa_p1> Remote Port WWN: <san_pwwn_spb_p1> Active FC4 Types: SCSI SCSI Target: yes Port Symbolic Name: DGC LUNZ 0430 Node WWN: <san_wwn_spb_p1> root@stiv-sol11:~# fcinfo remote-port -p <local_pwwn_port02> Remote Port WWN: <san_pwwn_spa_p2> Active FC4 Types: SCSI SCSI Target: yes Port Symbolic Name: DGC LUNZ 0430 Node WWN: <san_wwn_spa_p2> Remote Port WWN: <san_pwwn_spb_p2> Active FC4 Types: SCSI SCSI Target: yes Port Symbolic Name: DGC LUNZ 0430 Node WWN: <san_pwwn_spb_p2> root@stiv-sol11:~# fcinfo lu -v OS Device Name: /dev/rdsk/c0t60060160CD0E260000A8B1D4B789E311d0s2 HBA Port WWN: <local_pwwn_port02> Remote Port WWN: <san_pwwn_spa_p1> LUN: 0 Remote Port WWN: <san_pwwn_spb_p1> LUN: 0 HBA Port WWN: <local_pwwn_port01> Remote Port WWN: <san_pwwn_spa_p2> LUN: 0 Remote Port WWN: <san_pwwn_spb_p2> LUN: 0 Vendor: DGC Product: RAID 5 Device Type: Disk Device


2. Check that Oracle Solaris mpxio is disabled.
That can easilly be achieved by running stmsboot with -L option.  If Solaris  I/O  multipathing  is not enabled, then no mappings are displayed. If It's enabled, then disable it by running "stmsboot -d". For example, below we can see the output with Solaris mpxio enabled (and the output printed when disabling -and the reboot required-).

root@stiv-sol11:~# stmsboot -L non-STMS device name STMS device name ------------------------------------------------------------------ /dev/rdsk/c6t<san_pwwn_spa_p1>d0 /dev/rdsk/c0t60060160CD0E260000A8B1D4B789E311d0 /dev/rdsk/c6t<san_pwwn_spa_p2>d0 /dev/rdsk/c0t60060160CD0E260000A8B1D4B789E311d0 /dev/rdsk/c14t<san_pwwn_spb_p1>d0 /dev/rdsk/c0t60060160CD0E260000A8B1D4B789E311d0 /dev/rdsk/c14t<san_pwwn_spb_p2>d0 /dev/rdsk/c0t60060160CD0E260000A8B1D4B789E311d0 ........................................... root@stiv-sol11:~# stmsboot -d WARNING: stmsboot operates on each supported multipath-capable controller detected in a host. In your system, these controllers are /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0/fp@0,0 /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0 /pci@500/pci@2/pci@0/pci@a/SUNW,qlc@0/fp@0,0 /pci@500/pci@2/pci@0/pci@a/SUNW,qlc@0,1/fp@0,0 /pci@400/pci@2/pci@0/pci@e/scsi@0/iport@1 ...... If you do NOT wish to operate on these controllers, please quit stmsboot and re-invoke with -D { fp | mpt | mpt_sas | pmcs} to specify which controllers you wish to modify your multipathing configuration for. Do you wish to continue? [y/n] (default: y) WARNING: This operation will require a reboot. Do you want to continue ? [y/n] (default: y) The changes will come into effect after rebooting the system. Reboot the system now ? [y/n] (default: y)

3. Check that your storage's type is managed by EMC PowerPath
For that , we're using powermt with options display options. If our storage's type is listed as unmanaged (like the clariion below), then simply mark it as managed (using powermt manage as seen below) and you're done!

root@stiv-sol11:~# powermt display options Default storage system class: all Show CLARiiON LUN names: true Path Latency Monitor: Off Path Latency Threshold: 0 milliseconds Performance Monitor: disabled Autostandby: Proximity (prox): enabled IOs per Failure (iopf): enabled iopf aging period : 7 d iopf limit : 6000 Storage System Class Attributes ------------ ---------- Symmetrix periodic autorestore = on reactive autorestore = on status = managed CLARiiON periodic autorestore = on reactive autorestore = on status = unmanaged Invista periodic autorestore = on reactive autorestore = on status = managed Hitachi periodic autorestore = on reactive autorestore = on status = managed HP xp periodic autorestore = on reactive autorestore = on status = managed Ess periodic autorestore = on reactive autorestore = on status = managed HP HSx periodic autorestore = on reactive autorestore = on status = unmanaged VPLEX periodic autorestore = on reactive autorestore = on status = managed VNX periodic autorestore = on reactive autorestore = on status = managed

root@stiv-sol11:~# powermt manage class=clariion root@stiv-sol11:~# powermt display CLARiiON logical device count=1 ============================================================================== ----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------ ### HW Path Summary Total Dead IO/Sec Q-IOs Errors ============================================================================== 3077 pci@500/pci@2/pci@0/pci@a/SUNW,qlc@0,1/fp@0,0 optimal 2 0 - 0 0 3081 pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0 optimal 2 0 - 0 0

We should also pay attention to /kernel/drv/iscsi.conf. In fact, though Solaris  can distinguish between FC and iSCSI devices, some PowerPath versions don't make this distinction for manage and unmanage. So the mpxio-disable value must be set to yes in both the fp.conf (for fp.conf, it's also done automatically with stmsboot -d) and iscsi.conf files for PowerPath to manage EMC Clariion and VNX Storage Arrays.

Oracle Virtual Machine Server 3.2 for X86 / VM Migration from OVM 2.2

That's a short post which aims to describe a v2v (virtual to virtual) migration of Oracle VM X86  from OVM 2.2 to OVM 3.2. I won't cover the installation of OVM 3.2 nor its configuration. I'll assume  that it's already done (with valid server pool/servers/storage repositories...).

Technically, the process involves archiving the source VM (on OVM 2.2), transfer this archive on a web server (thus making it available using http), import this archive as template on the destination environment (OVM 3.2) and create a new VM based on this template. I know it seems a bit cumbersome for a simple v2v, but that's how it worked for me. So, here we go.

1. Stop the source VM and archive its folder on OVM 2.2

The source directory are located under /OVS/.../running_pool on OVM2.2

[root@ovm22 running_pool]# tar -zcvf src_vm.tar.gz src_vm src_vm/ src_vm/README src_vm/System.img src_vm/vm.cfg src_vm/vm.cfg.orig src_vm/root_swap_add_src_vm.img src_vm/u01_src_vm.img

2. Transfer this archive to a Web Server Folder (or to a FTP Server)

This web server must be accessible from the Target OVM 3.2 environment.

[root@ovm22 running_pool]# scp src_vm tar.gz stivesso.local:/my_web_folder Password:

3. Import this archive as template under the destination OVM environment

This web server must be accessible from the Target OVM 3.2 environment.

For that, from the OVM 3.2 Console: Go to “Repositories” / “Choose the target repository” / “VM Templates” / "Click Import VM Templates" and add your internal webserver and file location http://stivesso.local/src_vm.tar.gz

4. When the import job is completed, just create a new VM based on this template

You can then power on the new VM...

Linux System Recovery using Symantec Netbackup and a Linux Live-CD

The aim here is to describe a Linux System Recovery using Linux Live-CD and Symantec Netbackup Backup. Before delving into technical details, we want to highlight the fact that Symantec Netbackup has a Bare Metal Recovery features that suit this type of Recovery. But this feature doesn’t support all Linux OS (most of the popular ones are supported). For instance, this procedure was used to restored a complete Redhat 9 :)  Installation (it can also be used for a P2V migration).

Let’s check what is required for such restoration.

1.   A valid Full Symantec Netbackup Backup of the system we're trying to restore(I guess that one is obvious)

2.   A Linux Live CD that can support Symantec Netbackup Client, we’re using Ubuntu 8.04 LTS in this guide (fully supported, a list of supported OS can be found here)

3.  The installation CD of the Distribution we’re restoring, as stated above, we’re restoring a Redhat 9. So a Redhat 9 Boot CD is what we need for this case. The main reason for having the Install CD of the distribution is to avoid some incompatibility (like creating a filesystem which isn’t supported by the maintenance tools –e.g fsck- of the restored system)

4.  An Internet Connection (though it isn’t mandatory, we may just need some softwares/packages to install before installing Symantec Netbackup Client)

Once these prerequisites are met, we’re ready to start our restoration. As already described above, for this post, we’re using Ubuntu 8.04 LTS as Live-CD and are restoring a Redhat 9 Installation (the procedure can be easily adapted to others Distribution).

Below, a step-by-step (from a to f) description

a.     Boot on the Redhat 9 CD in rescue mode (linux rescue at the prompt) and recreate the target filesystems (the ones which will contain the restored data). Below is what we’ll create during this step.

MountPoint	Size	Filesystem	Device
/	65 GB	Ext3	/dev/sda3
/boot	100 MB	Ext3	/dev/sda1
/var	65 GB	Ext3	/dev/sda2
Swap	10 GB	Swap	/dev/sda5

Note that the target filesystems could be re-sized during this step (as long as we give enough space for the restoration). It’s also important to create these filesystems using the operating system we’re going to restore (avoid some filesystems features incompatibilities).

We just need a shell at this stage, so we’ll choose skip during the step in the screenshot below.

The filesystems are then created using fdisk and mkfs.ext3, mkswap… (as usual)

b.     Reboot the system on the Ubuntu Live-CD and install the required software for Netbackup Client Installation.

I assume that the network connectivity for Internet Connection and Software Installation is done for the Ubuntu LiveCD System. The list of software required to install on Netbackup Client for Supported Ubuntu/Debian can be found here. For proper NetBackup client operation on Ubuntu 8.04 Server Edition (64bit) the following packages are needed:

-  ia32-libs (In Universe Repository, so universe repository was enabled)

-  xinetd

-  ssh-server/rsh-server (required for remote install only)

We’ve also added the following for our specific environment

-  nfs-common (required to mount the directory where we’re keeping Netbackup Client binaries)

-  autofs (use to automatically mount the NFS shares)

root@ubuntu:~# apt-get update root@ubuntu:~# apt-get install ssh autofs nfs-common ia32-libs xinetd

Let’s also configured autofs to use /net automount features and complete Netbackup Client Installation.

root@ubuntu:~# cat /etc/auto.master # # $Id: auto.master,v 1.4 2005/01/04 14:36:54 raven Exp $ # # Sample auto.master file # This is an automounter map and it has the following format # key [ -mount-options-separated-by-comma ] location # For details of the format look at autofs(5). #/misc /etc/auto.misc --timeout=60 #/smb /etc/auto.smb #/misc /etc/auto.misc /net /etc/auto.net root@ubuntu:~# mkdir /net root@ubuntu:~# /etc/init.d/autofs start Starting automounter: done. root@ubuntu: # cd /net/nbmaster/applis/NetBackup_7.5_CLIENTS/ root@ubuntu:/net/nbmaster/applis/NetBackup_7.5_CLIENTS# ./install Symantec Installation Script ..... Do you wish to continue? [y,n] (y) Do you want to install the NetBackup client software for this client? [y,n] (y) This package will install Linux/Debian2.6.18 client. This package will install NetBackup client 7.5. Enter the name of the NetBackup server : nbmaster Would you like to use "ubuntu" as the configured name of the NetBackup client? [y,n] (y) Client binaries are located in /net/nbmaster/applis/NetBackup_7.5_CLIENTS/NBClients/anb/Clients/usr/openv/netbackup/client/Linux/Debian2.6.18. Saving client binaries for Linux/Debian2.6.18. ...... File /usr/openv/tmp/install_trace.10360 contains a trace of this install. That file can be deleted after you are sure the install was successful

c.     After the prerequisites, we’re adding Netbackup servers entries to the /etc/hosts files, add the Ubuntu Client Entries in Netbackup Servers hosts file, Install the client and register the Ubuntu LiveCD node.

On Ubuntu LiveCD,

root@ubuntu:~# cat /etc/hosts 127.0.0.1 localhost 127.0.1.1 ubuntu #Netbackup Master Server 10.252.37.60 nbmaster nbmaster.mtncameroon.net

On Master/Media Netbackup Servers,

[esso_s@nbmaster ~]$ head /etc/hosts 127.0.0.1 localhost.localdomain localhost ::1 localhost 10.252.37.60 nbmaster nbmaster.mtncameroon.net # Live CD Restoration 10.252.32.187 UBUNTU [root@nbmaster~]# bptestbpcd -client ubuntu [root@nbmaster~]#bpplclients baunixsystem -add ubuntu Linux Debian2.6.18

d.     We can now mount the filesystems that were previously created (step a.) in the directory where we’ll push the restoration

root@ubuntu:~#mount /dev/sda3 /mnt root@ubuntu:~# ls -lrt /mnt/ total 16 drwx------ 2 root root 16384 2013-06-11 21:02 lost+found root@ubuntu:~# mkdir /mnt/{boot,var} root@ubuntu:~# mount /dev/sda2 /mnt/var root@ubuntu:~# mount /dev/sda1 /mnt/boot root@ubuntu:~# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 1.9G 15M 1.8G 1% /lib/modules/2.6.24-26-generic/volatile tmpfs 1.9G 15M 1.8G 1% /lib/modules/2.6.24-26-generic/volatile varrun 1.9G 128K 1.9G 1% /var/run varlock 1.9G 0 1.9G 0% /var/lock udev 1.9G 52K 1.9G 1% /dev devshm 1.9G 12K 1.9G 1% /dev/shm tmpfs 1.9G 80K 1.9G 1% /tmp /dev/sda3 62G 33M 58G 1% /mnt /dev/sda2 62G 33M 58G 1% /mnt/var /dev/sda1 99M 4.1M 90M 5% /mnt/boot

e.     Let’s the restoration begin (using Netbackup Java Console, specifying as source the system to restore and as destination our Ubuntu Server, choose “restore everything to different location” and fill with the name of the directory where / is mounted)

Choose the last full you want to recover from the backup history and start the / restoration.

Uncheck  the rename soft/hard link options that are enabled by default.




f.      After the restoration, we can reboot the system and fix the Specific Operating Systems Issues that will arise. In this case, we’ve to do the following:

1.     Reinstall the Grub, for that we’re going through the “linux rescue” using Redhat 9 CD. We’ll let the rescue mode trying to mount the restored system under /mnt/sysimage. If not able, mount the filesystem manually and chroot under the mounted directory

# mount /dev/sda3 /mnt/sysimage # mount /dev/sda1 /mnt/sysimage/boot # mount /dev/sda3 /mnt/sysimage/var # chroot /mnt/sysimage # /sbin/grub-install /dev/sda

2.     Modify the /etc/modules.conf to add Vmware scsi controller driver by replacing the following in /etc/modules.conf

# alias scsi_hostadapter mptbase

# alias scsi_hostadapter1 mptscsih

3.     Recreate the initrd which will now have the correct scsi modules

# mvinitrd-2.4.20-8smp.img initrd-2.4.20-8smp.img_old # mkinitrd -v initrd-2.4.20-8smp.img 2.4.20-8smp

4.     I found that my etc/fstab was set to use e2label , so label were well reassigned to each filesystem

# cat /etc/fstab LABEL=/ / ext3 defaults 1 1 LABEL=/boot /boot ext3 defaults 1 2 none /dev/pts devpts gid=5,mode=620 0 0 none /proc proc defaults 0 0 none /dev/shm tmpfs defaults 0 0 LABEL=/var /var ext3 defaults 1 2 /dev/sda5 swap swap defaults 0 0 /dev/cdrom /mnt/cdrom udf,iso9660 noauto,owner,kudzu,ro 0 0 /dev/fd0 /mnt/floppy auto noauto,owner,kudzu 0 0 # e2label /dev/sda3 / # e2label /dev/sda2 /var # e2label /dev/sda1 /boot