Unbreakable Linux Kickstart HowTo

Oracle Unbreakable linux is essential same as Redhat Linux. This post I will cover how to do a kickstart on Unbreakable linux AS 4 update 6. This would apply to other versions and RedHat installations as well.
Kickstart simplifies your installation of linux servers on the network. Essentially, these are the main components to it:
1. The server gets a small boot file from floppy, CD, USB or over the network.
2. The server gets a kickstart config file as above from media or over the network.
3. Depending on what you choose as your installation media in the kickstart config file, the operating system packages are installed.
4. You can mention pre and post installation steps in kickstart file to automate processes such as user creation etc.

From my point of view a network install all the way makes more sense and that is what I would be covering here today.
Redhat website has a nice detailed howto all the methods and can be found here:


I will break it into 3 steps:
Step 1: install required services/packages.
Step 2. Configure required files.
Step 3. Starting the pxe boot

Step 1:
We need 3 services that could be hosted by a single server or multiple servers and a few config utilities:
1. DHCP service ->gives IP address to PXE boot client and tells it the file name to boot from and tftp server address.
2. TFTP server ->where the initial boot file resides, that contains info regarding install.
3. NFS server -> where the kickstart config file and installation tree is available.
4. service-config-netboot -> that would generate the pxelinux.cfg tree and boot file for the client.

1st lets get into making the installation tree available locally on the box. I copied all the .iso files under /kickstart on the server. It could be any folder you choose as long as you are willing to NFS share it.

For my setup, I had all of 3 services above on a single server. Lets get to installing or starting these services.
These would reveal if you have tftp server and dhcp server installed or not:
rpm -qa grep *tftp*
rpm -qa grep *dhcp*
rpm -qa grep *netboot*

If these services are installed, skip this section and move onto step 2.

Install all these 3 packages using your favorite method. You can do yum install .
Or you can install them from the installation CDs. I didn't know which CD contained these packages, I mounted all of them. OUL(oracle unbreakable linux 4_6 64bit) has 5 installation CDs. So I created 5 mount points and mounted them on the server:
mkdir -p /mnt/tmp1
mkdir -p /mnt/tmp2
mkdir -p /mnt/tmp3
mkdir -p /mnt/tmp4
mkdir -p /mnt/tmp5
mount -o loop Enterprise-R4-U6-x86_64-disc1.iso /mnt/tmp1
mount -o loop Enterprise-R4-U6-x86_64-disc2.iso /mnt/tmp2
mount -o loop Enterprise-R4-U6-x86_64-disc3.iso /mnt/tmp3
mount -o loop Enterprise-R4-U6-x86_64-disc4.iso /mnt/tmp4
mount -o loop Enterprise-R4-U6-x86_64-disc5.iso /mnt/tmp5

Doing the following on the above mount points to see where the required RPMs are:
cd /mnt/tmp1/Enterprise/RPMS
and
ls | grep *dhcp*
ls | grep *tftp*
ls | grep *netboot*
After locating them, do the install. You would need to install tftp server before service-config-netboot:
rpm -i dhcp-3.0.1-59.EL4.x86_64.rpm
rpm -i tftp-server-0.39-2.x86_64.rpm
rpm -i system-config-netboot-0.1.40.1-1.x86_64.rpm

Step 2:
The following files would be created/edited:
a. kickstart config file
b. xinetd.d tftp file
c. pxelinux.cfg directory and install files.
d. dhcpd.conf file

a. kickstart config file

Since our install was oracle specific. I downloaded the recommended config file from oracle's wiki website at following address: http://wiki.oracle.com/page/Linux+installation+kickstart+for+Oracle+database
You can customize this according to:
http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/custom-guide/s1-kickstart2-options.html
or you can use GUI to create one for you using service-config-kickstart(You might have to install this package if its missing).

The following config file is basically a copy of one downloaded from oracle website with some modifications like:
1. NFS install instead of url
2. Hostname etc. changed
3. No post installations here.
You should make the necessary modifications as per your requirements to this file. The default root password in this config file is password.

Kickstart config file, saved as example.cfg under /kickstart:

##############################################################################
# Uncomment the next line to enable interactive installation
#interactive

# Comment the next line to use GUI installation
text

install

# From where to get the installation disks
#cdrom
#url --url http://myserver/redhat/
nfs --server=192.168.1.10 --dir=/kickstart

lang en_US.UTF-8
langsupport --default=en_US.UTF-8 en_US.UTF-8
keyboard us
xconfig --resolution 800x600 --depth 16 --defaultdesktop gnome

# Change the hostname and peek the best eth configuration for you
#network --device eth0 --bootproto dhcp --hostname srvoracle01
network --device eth0 --bootproto static --ip 192.168.1.100 --netmask 255.255.255.0 --gateway
192.168.1.1 --nameserver 192.168.1.1, 192.168.1.2 --hostname testmachine

rootpw --iscrypted $1$uKWECPhN$Im66UG8MpWd2/kpcHoyuy/
firewall --disabled
selinux --disabled
authconfig --enableshadow --enablemd5

# Set a different timezone if not located in EST
timezone America/New_York

bootloader --location=mbr --append="rhgb quiet"


# Create a Volgroup device sda
clearpart --all --drives=sda
part /boot --fstype ext3 --size=100 --ondisk=sda
part pv.3 --size=0 --grow --ondisk=sda
volgroup vg00 --pesize=32768 pv.3

# Create the filesystems and logical volumes
logvol / --fstype ext3 --name=lvol00 --vgname=vg00 --size=2048
logvol /tmp --fstype ext3 --name=lvol01 --vgname=vg00 --size=2048
logvol /usr --fstype ext3 --name=lvol02 --vgname=vg00 --size=3500
logvol /usr/local --fstype ext3 --name=lvol03 --vgname=vg00 --size=1024
logvol /var --fstype ext3 --name=lvol04 --vgname=vg00 --size=1024
logvol /var/log --fstype ext3 --name=lvol05 --vgname=vg00 --size=1024
logvol /opt --fstype ext3 --name=lvol06 --vgname=vg00 --size=1024
logvol /home --fstype ext3 --name=lvol07 --vgname=vg00 --size=512

# Edit the swap space to be twice the size of the machine RAM, if you
logvol swap --fstype swap --name=lvol08 --vgname=vg00 --size=8192


# All packages needed by Oracle installation include Gnome desktop
%packages
@ system-tools
@ gnome-desktop
@ dialup
@ compat-arch-support
-openldap-clients
-ckermit
-wireshark
-bluez-pin
-OpenIPMI-tools
-samba-client
e2fsprogs
-screen
-xdelta
-zsh
-nmap
lvm2
sysstat
-open
kernel-smp
grub
binutils
compat-db
control-center
gcc
gcc-c++
glibc
glibc-common
gnome-libs
libstdc++
libstdc++-devel
make
pdksh
sysstat
xscreensaver


%post
##############################################################################

Need to make the folders available for NFS share:
This is the /kickstart folder that contains kickstart config file and installation .iso files.
nfs-export --dir /kickstart --perm ro --ip 192.168.1.100
nfs-export --dir /kickstart --perm ro --ip 192.168.1.10

If you skipped step 1, mount 1st iso CD under /mnt/tmp1
(e.g. mount -o loop Enterprise-R4-U6-x86_64-disc1.iso /mnt/tmp1)
And make is available for NFS share:
nfs-export --dir /mnt/tmp1 --perm ro --ip 192.168.1.10
nfs-export --dir /mnt/tmp1 --perm ro --ip 192.168.1.100

Now we need to start the tftp daemon:
b. xinetd.d tftp file

Edit the following to run tftp as daemon, you can choose to run this in standalone mode using /usr/sbin/in.tftpd and not editing this file:
/etc/xinetd.d/tftp:
###################################################################
service tftp
{
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
###################################################################

restard xinted.d:
/etc/init.d/xinetd restart

Next step is to get /tftpboot/linux-install folder ready for boot files.
c. pxelinux.cfg directory and install files.

Using the following command to generate these files under /tftpboot:
/usr/sbin/pxeos -a -i "OUL" -p NFS -D 0 -s 192.168.1.10 -K nfs:192.168.1.10:/kickstart/example.cfg -L /mnt/tmp1 OUL4
This also generates a defualt file under /tftpboot/linux-install/pxelinux.cfg, which is read by the installer at boot time. You can create one specific for your host using pxeboot command.

d. dhcpd.conf file

DHCP config file.
You can get the MAC address for the client in lot of different ways. If you PXE boot the client, it throws the client MAC address right at the console. You would need this to set the values in following file.
If you are crossing vlans/routers make sure ip helper address or equilavent is set on the routing/network device.

/etc/dhcpd.conf :
###################################################################
deny unknown-clients;
not authoritative;
ddns-update-style ad-hoc;
option domain-name "lcc.copr.pvt";
option domain-name-servers 192.168.1.1;
option subnet-mask 255.255.255.0;

allow bootp;
allow booting;

option ip-forwarding false; # No IP forwarding
option mask-supplier false; # Don't respond to ICMP Mask req

subnet 192.168.0.0 netmask 255.255.255.0 {
option routers 192.168.1.1;
}
group {
next-server 192.168.1.10; # name of your TFTP server
filename "linux-install/pxelinux.0"; # name of the bootloader program

host node1 {
hardware ethernet 00:14:4F:45:14:0E;
fixed-address 192.168.1.100;
}
}
###################################################################

Start DHCP server:
/etc/init.d/dhcpd start

Step 3. Starting the pxe boot

Change the boot order to boot from the network on the client machine and if all the configs were done correctly, you should be able to go through the install smoothly. You can watch as it goes and look for error messages if any to fix any typos etc.

Host vs Guest Memory in VMware esx server

After watching a video for high level concepts of memory management for guests, this is a summary of what I understood:

Host memory shows the total memory assigned to that guest by the host.

Guest memory means what host sees as active memory usage on the guest, this might differ from what you see from guest’s point of view. And that’s OK.

Now why do they differ and is it okay if Host memory is much higher than guest memory?

In a traditional model where there is no virtualization, Operating system get the memory directly from underlying hardware and manages the used and free memory by itself.

In a Virtualized model, OS assumes that traditional model for memory and thus doesn’t(or should I say can’t) inform the Virtualization layer about the free memory. For example, if you assign 1G to a guest and at some point guest uses, say, 800M of the memory and after some time frees say 600M of the memory. So guest is using 200M but virtualization layer is not aware of this memory being free on the guest.

Now how does host know how to free memory? This is achieved by memory balooning. Baloon driver is part of the vmware tools and it keeps marking or pinning the free memory on the guest OS and informs the host about it.

There are other ways for host to find out about free memory and it can do swapping for guest too. But host is not very eager all the time to get the exact free memory stats from the guest to free up that memory. Only when the free memory on guest OS falls below a certain threshold, ballooning or swapping is done and host frees up that memory. This is the reason lots of time host memory is higher than guest memory. And its normal if you see that.

Virtual Center 2.5 shows incorrect power state for a guest

If Virtual center is showing incorrect power state for a guest and Host(ESX 3.0.2) shows it correct, management service on the host(on which guest resides) needs to be restarted. Following command is all that is needed:

service mgmt-vmware restart

After this, close out the virtual center session and re-login. Now you should see the correct state.

Ubuntu 7.10 server and Active Directory Integration

2003 R2 has Advanced services for NIS(Unix Services) which allows to do Active directory integration with unix. Following is very detailed blog on how to go about doing this:
Linux, Active Directory, and Windows Server 2003 R2 Revisited
I won't repeat the Windows setup steps mentioned in that blog, so make sure you have the AD setup as described in the above blog. The following are the steps taken to make this work with Ubuntu Server 7.10 without TGT validation:

Install the following packages:
sudo apt-get install krb5-user libnss-ldap libpam-krb5

libnss-ldap makes you go through certain steps and make sure you pick the values as you fill out in /etc/ldap.conf below.

Edit /etc/krb5.conf file and It should look like following:

[libdefaults]
default_realm = YOURDOMAIN.PVT

krb4_config = /etc/krb.conf
krb4_realms = /etc/krb.realms
kdc_timesync = 1
ccache_type = 4
forwardable = true
proxiable = true

[realms]
YOURDOMAIN.PVT = {
kdc = YOURSERVER.YOURDOMAIN.PVT:88
admin_server = YOURSERVER.YOURDOMAIN.PVT
default_domain = YOURDOMAIN.PVT
}

[domain_realm]
.YOURDOMAIN.PVT = YOURDOMAIN.PVT
YOURDOMAIN.PVT = YOURDOMAIN.PVT
[login]
krb4_convert = true
krb4_get_tickets = false

Here you can add multiple domains and multiple kdc and admin_server depending on the structure of your AD.

Next Step is to edit the /etc/ldap.conf. This should look like following:
host YOURSERVER.IP.ADDRESS.HERE
base dc=YOURDOMAIN,dc=PVT
uri ldap://YOURSERVER.YOURDOMAIN.PVT
binddn ad_bind_user@YOURDOMAIN.PVT
bindpw ad_bind_password
scope sub
ssl no
pam_filter objectClass=User
nss_base_passwd dc=YOURDOMAIN,dc=PVT?sub
nss_base_shadow dc=YOURDOMAIN,dc=PVT?sub
nss_base_group dc=YOURDOMAIN,dc=PVT?sub
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_objectclass posixGroup group
nss_map_attribute gecos name
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute uniqueMember member

Now edit the /etc/nsswitch.conf by adding ldap for the fields passwd, group and shadow.

Next Step is to edit /etc/pam.d/common-auth. This should look like following:
#
# /etc/pam.d/common-auth - authentication settings common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of the authentication modules that define
# the central authentication scheme for use on the system
# (e.g., /etc/shadow, LDAP, Kerberos, etc.). The default is to use the
# traditional Unix authentication mechanisms.
#
auth sufficient pam_krb5.so minimum_uid=10000
auth required pam_unix.so nullok_secure



minimum_uid=10000 is one way to make sure system userids are not touched and this depends on what number uid starts on your AD setup.

This should be all that is for setup. Now lets do some testing:

1. Getting a kerboreos ticket:
kinit ad_user_name
This should prompt for a password, and should give you prompt back if successful.
You can view the ticket generated by using klist.
If you get an error message, you have to work with what it says.
One common one is Time Sync error, for that you would need to issue following:
sudo ntpdate YOURNTPSERVER
Generally, you can use your AD Domain Controller as your NTP server.
2. If this is successful, next thing is to check pam authentication.
Restart ssh or telnet, the service you gonna test AD authentication against.
sudo /etc/init.d/ssh restart.
Now, in the current window, do a tail -f /var/log/auth.log
This would shows the running auth log.
Now open a new session to ssh(or whatever you are testing) and login with AD credentials. If your setup is correct, this should be working.
Othewise, auth.log is your friend to see where it is breaking.

Intalling VMware tools in linux when guest CD doesn't mount

Incase you can't get the VMware tools install mounted as cdrom using the Virtual Center GUI, there is a easy workaround you can use.
From the guest virtual machine, sftp to the ESX server host and for default install paths go to /vmimages/tools-isoimages.
Get linux.iso from here and quit from sftp.
On the guest linux, su and then create a tmp folder(or whatever you want to call it).
mkdir tmp
Mount the iso by using the following command:
mount -o loop linux.iso tmp
then you can "cd tmp"
Run the rpm installer from here and for other linux, copy the .gz file to your home folder.
From their you should be able to install the vmware tools.

Extend the windows disk under VMware

Few of the earlier windows installations we did with 5G or 10G OS partition started filling up and we constantly needed to juggle files around to keep it alive.
There is option available though to extend these disks, as long as the disks are basic disks and not dynamic disks(I haven't tried working with dynamic disks yet).
Another thing, there are other methods available to do the same also like using other disk partition software or ny using VMware converter, please evaluate what best suites your needs.
What would you need:
1. Downtime for the virtual machine which needs its disk extended.
2. Another Windows 2003 server virtual machine(on same vmware host or able to see the disk of 1st virtual machine) which you can power off twice.
3. ssh or telnet to the vmware esx host server.

Basically, it requires following steps:
1. Shutdown the virtual machine whose disk has to be extended.
2. Increase the disk size by ssh(or telnet) to vmware server host and using the vmkfstools command.
3. Present the disk to another Windows 2003 virtual machine.
4. Extend the disk in question.
5. Shut down the extending machine, remove the disk from this machine's resources.
6. Boot up your virtual machine in question and you should have an OS drive with more space now.
Please make sure you do a backup of the original file incase something doesn't go as expected.
Lets dive deep into these steps now:
1. Shutdown the Virtual machine in question: You have to schedule downtime for the virtual machine to extend its disk.
2. Login to vmware esx server using ssh(or telnet), whatever you use.
Issue the following command to extend the disk in question:
vmkfstools -X 15g /path/VMtest.vmdk
Replace 15g with whatever the new size you want, 10g, 20g etc.
and /path/VMtest.vmdk with the full path and vmdk file name for that host.
(Should be under /vmfs/volumes/[storage_name]/[Virtual Machine Name]/)
3. Through the virtual center now, Shut down the Windows 2003 server we are going to use to extend the disk in question. Edit settings > Click on Add, then Hard Disk, Next, Use and existing disk, Browse to the disk in question and add this. After this disk shows up the resources(or hardware), boot up this machine.
4. Through the disk manager, you should be able to see this new disk with new unpartitioned disk space. Go to command line and enter diskpart.exe.
On the prompt, type list disks.
Select the disk in question by using select disk 2, for example.
Then type List volumes and it would show you all the volumes. select the volume in question by using select volume 1, for example.
Now type "extend". This would extend the volume.
Disk Manager, should show you the disk now with extended capacity.
5. Now shut down this server, Edit settings again and remove this disk(do not delete)
6. Boot up the original virtual machine now and you should have an extended OS disk now.

ESX trunking Server Side

In continuation of my last post:VMware ESX VLAN trunking I had a issue on one of the server where service console was not on a separate NIC and hence after trunking I couldn't get to it remotely via Virtual Center or ssh.
So basically, I had to use command line utilities on local console to get the service console back on network before I could create port groups for different vlans and attach my virtual servers to these. Here is how I went about doing this:
Login to ESX server as root and issue the following command:
esxcfg-vswitch -l
This command is mostly under /usr/sbin and Gives me list of currently configured port groups, their Now lets assume that service console port is under vlan 25 for vswitch0, so we would need to create a port group for that vlan and lets call it "SCVlan25", could be anything you want to call it.
So to create this new port group, issue the following commands:
Add the new port group:
esxcfg-vswitch --add-pg=SCVlan25 vSwitch0
Add it to required vlan:
esxcfg-vswitch --pg=SCVlan25 --vlan=25 vSwitch0
Now, esxcfg-vswitch -l
This would show you the added port group.

Next step is to add the service console port under this newly created port group.
Use the following command to get the listing:
esxcfg-vswif -l
Now, you would get the name of the interface from
this command, in my case it was vswif0.
Following command would be issued to put it in the right port-group:
esxcfg-vswif vswif0 -p SCVlan25
then again, "esxcfg-vswif -l" to confirm it.

Now your service console should be on the network. And you should be able to create other port groups using command line above or the Virtual Center as mentioned in the earlier post.

Another issue I ran into is that after deleting a virtual switch so that I can team the physical NICs, the other virtual switch would not show the physical NIC I wanted to add, in virtual center.
So this had to be done via command line also:
esxcfg-vswitch -L vmnic# vSwitch#
where vmnic# would be replaced by whatever NIC you want to add to that particular vswitch#.

VMware ESX VLAN trunking

We had been running VMWare ESX servers for a while without any VLAN trunking. This means, each physical NIC was part of a separate VLAN and this led to no network teaming(hence no failover) as number of NICs were be limited.
Another limitation was that we couldn't add any more VLANs as all the NICs were already used. This is when we decided its better use trunking in our environment. I got my start from following website for Cisco IOS configs:
ESX Server, NIC Teaming, and VLAN Trunking
This worked fine for the server on switches running Cisco IOS.
But since we are running some switches in hybrid config with CatOS, the following steps were taken on those switches for ESX server to work in trunking mode:

1. If you haven't created a native VLAN for ESX environment now, you would need to create that with following command:
set vlan "vlan-id" name "name"
"vlan-id" here is the vlan number you want to assign this new vlan and could be anything between normal range 1–1000 and extended range 1025–4096.
name doesn't have to specified but makes it easy to see what vlan is what.

2. Change the port to trunking mode:
set trunk "mod/port" on dot1q
or
set trunk "mod/port" on dot1q "vlan numbers"
depending on whether you want to allow all VLANs or certain VLANs on the port.

3. Set the native vlan on the port:
By default vlan 1 is the native vlan, to change it:
set vlan "vlan-id" "mod/port"
where "vlan-id" is the vlan number created in step 1 or the vlan number if it already exists in your network.

Now, on the ESX server side:
From the virtual center, go to configuration for the host and click on networking.
Then Add Networking, Choose Virtual Machine, choose the vswitch with the NIC(s), choose the name you want to call it and "vlan-id" for that port group.
You can repeat the same steps for whatever vlans you want to configure on the host.
After that you can Edit the settings for the Virtual Machine to point to their respective port-groups and should be good to go.