Oracle VM for Sparc Autosave / Practical case

Have ever faced a situation where you think you've lost LDOMs configuration (mostly after an unexpected downtime)? If that's the case, then you must know how much autosave feature on Oracle VM for SPARC is important. Indeed, the main aim of the autosave feature is to ensure that a copy of the current configuration is automatically saved on the control domain whenever the Logical Domains configuration is changed. More important, it occurs even when the new configuration is not explicitly saved on the SP.

Note that this doesn't replace the classic LDOMs configuration Backup which consists of saving the constraints information for each domain into an XML file (ldm list-constraints -x...), but may supplement it when what is needed is just to recover domain Configurations that weren't saved to the SP. 

Enough talk! Let's move forward with this practical case. We've a System with 05 configured LDOMs, and while building them, an unexpected Hardware Failure occurs (:-)). The main issue is that we haven't saved yet our LDOM Configuration on the SP. Meaning before the Outage, we had something similar to:

root@scdomain:~# ldm ls-spconfig factory-default initial [next poweron] root@scdomain:~# ldm ls NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv- UART 16 32G 6.9% 12h 21m STEVELDOM01 active -n---- 5000 16 32G 0.0% 10h 17m STEVELDOM02 active -n---- 5001 16 32G 1.0% 10h 52m STEVELDOM03 active -n---- 5002 16 32G 0.1% 10h 52m STEVELDOM04 active -n---- 5003 16 32G 0.1% 10h 52m STEVELDOM05 active -n---- 5004 16 32G 9.9% 10h 51m

And after the outage, we've something like this (with maybe some other parameters not present for the primary domain...).

root@scdomain:~# ldm ls-spconfig factory-default initial [next poweron] root@scdomain:~# ldm ls NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv- UART 16 32G 6.9% 10m

Let's move forward with the following 05 little steps to recover our LDOMs configuration.

1. As a matter of precaution, take a Backup of the actual configuration and another one of the /var/opt/SUNWldm

The /var/opt/SUNWldm folder contains the Autosave directories.

root@scdomain:~# ldm list-constraints -x > /export/home/admin/backup_faulty.xml root@scdomain:~# tar cvf /export/home/admin/sunwldom.tar /var/opt/SUNWldm


2.lists the available autosave configurations

At this stage, we should have our autosaved configuration shown as being newer than the current configuration.

root@scdomain:~# ldm ls-spconfig -r initial [newer]

3. Recover the Autosave Configuration

root@scdomain:~# ldm add-spconfig -r initial


4. Perform a full power Cycle

root@scdomain:~# shutdown -y -g0 -i5 -> start /SYS Are you sure you want to start /SYS (y/n)? y Starting /SYS

5. Check the LDOM Configuration 

root@scdomain:~# ldm ls-spconfig factory-default initial [next poweron] root@scdomain:~# ldm ls NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv- UART 16 32G 6.9% 12h 21m STEVELDOM01 active -n---- 5000 16 32G 0.0% 10h 17m STEVELDOM02 active -n---- 5001 16 32G 1.0% 10h 52m STEVELDOM03 active -n---- 5002 16 32G 0.1% 10h 52m STEVELDOM04 active -n---- 5003 16 32G 0.1% 10h 52m STEVELDOM05 active -n---- 5004 16 32G 9.9% 10h 51m

References:

http://docs.oracle.com/cd/E19604-01/821-0406/configurationmanagement/index.html

HP ILO / RHEL7 Systemd Output to VSP console

During my last post, I described the redirection of Linux Output to Serial on an Upstart Distribution based (using RHEL6). The main aim of this post is to describe the same on Systemd Distribution based (RHEL7). Before I begin with the technical matters, I have to say that I've been quite impressed by how Systemd makes this configuration so easy (no pain at all! So cool!). This isn't to make a comparison between Systemd and Traditional Init or Upstart, but you can check these two previous posts to draw your own Conclusions:  VSP/Traditional Init, VSP/Upstart.

Let's now delve in the interesting matters. The whole procedure is just about setting Kernel Options and reboot, and if the reboot can't be performed right away, just start a systemd service.

1. Set the Kernel Options boot options: 

On RHEL7 with Grub2, add "console=ttyS1" to  GRUB_CMDLINE_LINUX in /etc/default/grub file (You might also remove rhgb quiet as rhgb is for RedHat Graphical Boot and quiet is meant to hide the majority of boot messages before rhgb starts)

[root@mynode ~]# grep "GRUB_CMDLINE_LINUX" /etc/default/grub GRUB_CMDLINE_LINUX="crashkernel=auto vconsole.font=latarcyrheb-sun16 rd.lvm.lv=el/swap rd.lvm.lv=el/root vconsole.keymap=us console=ttyS1"

Changes to /etc/default/grub require rebuilding the grub.cfg file. This file location's on BIOS based machine is /boot/grub2 and for UEFI based machine, it is /boot/efi/EFI/redhat/.

On BIOS Based Machine:

[root@mynode ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

On UEFI Based Machine:

[root@mynode ~]# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

2. Reboot or start a serial-getty service on ttyS1

Now we can either reboot the System to have the Kernel Loaded with the new parameter during the reboot , or (especially if we can't afford a downtime :-)) run the following to have a getty service started right away on the ttyS1.

[root@mynode ~]# systemctl start serial-getty@ttyS1.service

That's it! So simple! Go on the console, run vsp and there's a nice prompt...

Red Hat Linux Server 7.0 Kernel 3.8.13-35.3.1.el7uek.x86_64 on an x86_64 mynode login:


Reference:
http://0pointer.de/blog/projects/serial-console.html

HP ILO / RHEL6 Upstart Output to VSP console

Following my post related to HP ILO VSP console redirection , I've received many comments related to the same Configuration on recent Linux Distribution. In this post, I'm willing to detail the same on Upstart Based Distribution (using RHEL6, but should be the same for others Upstart Based distribution ). In fact, the main configuration on old Distribution was mostly related to Init Daemon Configuration, with new distribution based on Upstart/Systemd, things are slightly different. 

Note that I won't detail the configuration of the Virtual Serial Port on BIOS/UEFI as it had already been discussed in this previous post.
If you're interested in the same configuration for Systemd Based Distribution, it's described in this post.

1. Create an init configuration file for ttyS1

[root@mynode ~]# vi /etc/init/ttyS1.conf start on runlevel [S345] stop on runlevel [016] respawn instance /dev/ttyS1 exec /sbin/agetty ttyS1 115200 vt100-nav

2. Check the init configuration and start running the agetty process

Upstart leverages the Linux inotify API to make itself aware of any changes that can happen within its configuration directory, so the creation of the ttyS1 file above is enough to have the service available and listed when using initctl. The only thing we have to do is to start the process.

[root@mynode ~]# initctl list | grep ttyS1 ttyS1 stop/waiting [root@mynode ~]# initctl start ttyS1 ttyS1 (/dev/ttyS1) start/running, process 38394

3. Test you have access to the System through vsp

</>hpiLO-> vsp Virtual Serial Port Active: COM2 Starting virtual serial port. Press 'ESC (' to return to the CLI Session. mynode login:

4. Add Serial Port to securetty to allow login as root

This is needed if we want root account to be able to log in through this serial console.

[root@mynode ~]# echo "ttyS1" >> /etc/securetty

5. Configure the Grub GRUB config file

Finally, we can configure the GRUB to have outputs of the boot process on the console, this is easily achieved by adding console=tty0 console=ttyS1,115200

[root@mynode ~]# cat /etc/grub.conf # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/sda5 # initrd /initrd-[generic-]version.img #boot=/dev/sda default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title Red Hat Enterprise Linux (2.6.32-220.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-220.el6.x86_64 ro root=UUID=0084ea5e-39e2-4994-aaa5-5abe8bf7eeb0 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM console=tty0 console=ttyS1,115200 initrd /initramfs-2.6.32-220.el6.x86_64.img

Oracle/Redhat Enterprise Linux 7 Kickstart Installation / without DHCP

RHEL7/OEL7 is out for few months now, with a lot of new features (Revamped Anaconda, Systemd...). But before really starting to enjoy all these nice features, let's perform some basic Automated Kickstart Installation. In this short post, I'm willing to describe just that type of Installation without relying on a DHCP Server (using Static Network Parameters). I'm also using an Installation tree and Kickstart file located on httpd servers and reachable from the System I'm installing. Enough talk! let's detailed this Kickstart Installation in the following 4 steps:

1. Make Installation Tree available on an httpd server:

We've RHEL7/OEL7 ISO Files and an httpd server (192.168.0.10) configured (with DocumentRoot being the classic /var/www/html). Mounting the ISO as loop device in the DocumentRoot is enough to have the Installation Tree available over httpd.

# mkdir /var/www/html/media_oel7u0 # mount -o loop /home/stiv/Downloads/Oracle_Enterprise_Linux_7.0.iso /var/www/html/media_oel7u0

2. Create the Kickstart File and make it available on the httpd system:

For that, I used as template an anaconda-ks.cfg  from another installed node and created the following kickstart.

#version=RHEL7 # System authorization information auth --enableshadow --passalgo=sha512 # Use http tree for the installation url --url=http://192.168.0.10/media_oel7u0 # Run the Setup Agent on first boot firstboot --enable ignoredisk --only-use=sda # Keyboard layouts keyboard --vckeymap=us --xlayouts='us' # System language lang en_US.UTF-8 # Network information network --bootproto=static --device=eno49 --gateway=192.168.0.1 --hostname=olnode--ip=192.168.0.20 --nameserver=192.168.0.1 --netmask=255.255.255.0 --noipv6 --activate network --bootproto=dhcp --device=eno50 --onboot=off --ipv6=auto network --bootproto=static --device=eno51 --ip=192.168.50.22 --netmask=255.255.255.0 --onboot=off --ipv6=auto network --bootproto=dhcp --device=eno52 --onboot=off --ipv6=auto # Root password rootpw --iscrypted $6$v797wiKZv7W0MK58$4drw.ys34Z4cMoR7ECwkCIbhYuY7Aoc2A3dWs3wCTTyRTt1OovohA518vsdqpEfFCKwqwfU4weCcFK6VAmdv/1 # System timezone timezone Africa/Douala --isUtc --nontp # System bootloader configuration bootloader --location=mbr --boot-drive=sda # Partition clearing information clearpart --all --initlabel --drives=sda # Disk partitioning information part /boot --fstype="xfs" --ondisk=sda --size=500 part pv.26 --fstype="lvmpv" --ondisk=sda --size=254005 part /boot/efi --fstype="efi" --ondisk=sda --size=200 --fsoptions="umask=0077,shortname=winnt" volgroup ol --pesize=4096 pv.26 logvol /var/lib/mysql --fstype="xfs" --size=150000 --name=var_lib_mysql --vgname=ol logvol swap --fstype="swap" --size=64000 --name=swap --vgname=ol logvol /home --fstype="xfs" --size=20000 --name=home --vgname=ol logvol / --fstype="xfs" --size=10000 --name=root --vgname=ol logvol /var --fstype="xfs" --size=10000 --name=var --vgname=ol %packages @base @compat-libraries @core @development @file-server @hardware-monitoring @infiniband @large-systems @network-file-system-client @performance %end

Make this file available on http (copied under httpd DocumentRoot) and test to make sure it's reachable (i.e http://192.168.0.10/olnode.ks )

3. Start the Kickstart Installation

I'm using an UEFI system, so In order to provide the right Kickstart Parameters during the boot process, I'm selecting an installation option in the boot menu and then press either the E key ( For BIOS Systems, that'll be he Tab key). A prompt is displayed which enables to edit the boot options already defined and to add new options.  In this case, I'm adding the following: 

inst.ks=http://192.168.0.10/olnode.ks ip=192.168.0.20::192.168.0.1:255.255.255.0:olnode:eno49:none

inst.ks specifies the location of the kickstart file, and ip sets statics network parameters, it must be in this form: ip=ip::gateway:netmask:hostname:interface:none. Below are some screenshots taken to illustrate this process.
Note that parameters such as hostname and Interface could be empty

4. Enjoy the Automation:

Once that's done, the last step is to Press Ctrl-x (or enter on BIOS System) and enjoy the automation...

References:
http://docs.oracle.com/cd/E52668_01/E54695/html/ol7-install-boot-options.html

Oracle VM for X86 / RHEL PVM Kickstart Installation without DHCP

Just a quick and short post to share a tip related to installation of RHEL as ParaVirtual Machine using kickstart(or any distribution that can use kickstart) on Oracle VM 3 for X86 without having to rely on a DHCP for the Network Parameters. The only prerequisite that is needed is to have kickstart file and RHEL Installation Path available on the network and reachable from Oracle VM Manager Server.

Once that's done, we create our Virtual Server as usual, except that for Network Path, we're having to add more parameters. Indeed we can use the --args to add any parameters we're willing to pass to the kernel. So, we're having something like :

--args kernel_boot_paramaters http://install_server/rhel6_4

For example, let's say that my kickstart file is located on an http server and accessible via http://install_server/my_kickstart.ks, I also need to set 192.169.0.10 as IP of eth1 interface in order to get access to this kickstart file's URL and the RHEL installation tree is http://install_server/rhel6_4. Then, i'll have the following:

--args "ks=http://install_server/my_kickstart.ks ip=192.169.0.10 ksdevice=eth1 netmask=255.255.255.0 gateway=192.169.0.1" http://install_server/rhel6_4

Note that the --args section are in quote, this is needed to clearly draw the boundary between what is kernel arguments and the installation tree's URL. 

Ansible Hosts / Install alternate upgraded Python version

Following the highlighted error below which happened while trying to configure a RHEL4 node to be manageable by ansible (and that i rightly attributed to the old Python 2.3.4 which is the default version of Python for RHEL4), i've decided to install an alternate Python version on the same RHEL node without impacting the already existing environment. Below i'm sharing the 8 steps I followed to have that working.

Note that although i'm describing this Python Installation in  relation with Ansible Integration, Step 1 to Step 6 can be used for the installation of an alternate Python Environment on any other Linux Systems.

Ansible's Error:

[stivesso@ansible-server ~]$ ansible my_ansible_node -m ping  --ask-pass

SSH password:

my_ansible_node | FAILED >> {

    "failed": true,

    "msg": "\r\nSUDO-SUCCESS-josewbugtgyoijxrgkuxihoejbjsbiuq\r\n  File \"/home/esso_s/.ansible/tmp/ansible-tmp-1406122775.05-117998270117858/ping\", line 1177\r\n    clean_args = \" \".join(pipes.quote(arg) for arg in args)\r\n

     ^\r\nSyntaxError: invalid syntax\r\n", 

    "parsed": false

}

1. Install Prerequisites on the target host

# yum groupinstall "Development tools" # yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel

2. Get an updated python package

# wget http://python.org/ftp/python/2.7.5/Python-2.7.5.tgz --no-check-certificate

3. Untar/unzip the package on the target node

# tar zxvf Python-2.7.5.tgz

4. configure with the alternate path as option (here, i'm planning to install the alternate environment in /opt/python2.7) and compile/install

# ./configure --prefix=/opt/python2.7 --enable-shared # mkdir /opt/python2.7 # make # make altinstall

5. Export Shared Library and Bin Library in PATH and LD_LIBRARY_PATH on User profile 

# tail -3 /etc/profile ## Added for python 2.7 export PATH=$PATH:/opt/python2.7/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/python2.7/lib

6. Create ld.so.conf configuration file and run ldconfig to have library loaded system wide

# cat /etc/ld.so.conf.d/python27.conf /opt/python2.7/lib # ldconfig

7. Add the following to /etc/ansible/hosts on the Ansible Control Machine for the target node

my_ansible_node ansible_python_interpreter=/opt/python2.7/bin/python2.7  

8. Test again the ping module from ansible-server...

[esso_s@ansible-server ~]$ ansible my_ansible_node -m ping --ask-pass SSH password: my_ansible_node | success >> { "changed": false, "ping": "pong" }

Linux KVM / Network Bridge over bonded Interface

This is a short post about KVM Network Bridge Configuration for Guest Domain over a bonded Interface. For this post, we're using a Redhat Enterprise Linux 6 as KVM Physical System, but the procedure can be easily adjusted for any others Linux System which supports KVM. Before delving in the Technical Configuration, let’s have a look on what we’re trying to achieve. We’re having a Physical Server with 02 Physical Interfaces (eth0 and eth1) and are planning to use these two physical Interfaces in a highly available configuration. The same interfaces are also used to support a bridge that is used by KVM DomU (Guest Hosts). Below, we can see a picture describing the details of this configuration.

The first thing to do is to complete the bonding configuration and that's pretty straightforward. We're just creating the bonding.conf file in /etc/modprobe.d (to dynamically load the bonding module, bonding's option  will be modified later on and must reflects ones' needs), then we're editing the ifcfg-bond0 network configuration files.

[root@my_dom0 ]# cat /etc/modprobe.d/bonding.conf alias bond0 bonding

Let us now configure the bond0 Interface by editing its configuration files, note the BRIDGE configuration mentioned below, this is explicitly added to link the bond interface to the bridge interface. We are also setting the bond Interface parameters here.

[root@my_dom0 ]# cat /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE="bond0" ONBOOT="yes" BRIDGE=br0 NM_CONTROLLED=no BONDING_OPTS="mode=active-backup miimon=100 downdelay=200 updelay=200"

Next thing to do is to configure the bridge Interface, as usual we’re creating the ifcfg-br0 file containing our network configuration (IP, DNS...). Note that as described above, this is the Interface that will have the IP settings used to administrate the Physical Server (Note that this is a matter of architecture/design, on a typical production server with four interfaces, we could have for example separated this management network from the Guest Domains Network...)

[root@my_dom0 ]# cat /etc/sysconfig/network-scripts/ifcfg-br0 DEVICE=br0 TYPE=Bridge BOOTPROTO=static ONBOOT=yes # IP Configuration HOSTNAME="my_dom0" IPADDR="192.168.1.10" NETMASK="255.255.255.0" DNS1="192.168.1.2" DNS2="192.168.1.3" GATEWAY="192.168.1.1" IPV6INIT="yes" MTU="1500" NM_CONTROLLED="no"

Finally, we can configure the physical interface and restart the network service (or reboot the node…)

[root@my_dom0 ]# cat ifcfg-eth0 DEVICE="eth0" HWADDR="XX:XX:XX:XX:XX:X8" NM_CONTROLLED="no" ONBOOT="yes" BOOTPROTO=none MASTER=bond0 SLAVE=yes USERCTL=no [root@my_dom0 ]# cat ifcfg-eth2 DEVICE="eth2" HWADDR="XX:XX:XX:XX:XX:X9" NM_CONTROLLED="no" ONBOOT="yes" BOOTPROTO=none MASTER=bond0 SLAVE=yes USERCTL=no [root@my_dom0 ]# shutdown –r now

Check the Bridge Configuration, should have output similar to the one below. The bridged interface is now ready to be used by the Guest Domains.

[root@my_dom0 ~]# brctl show bridge name bridge id STP enabled interfaces br0 8000.b4b52f6d1bc0 no bond0