Steps to configure time sync on EMC celerra

EMC Celerra needs to have ntp configured specially if you are using CIFS shares. Active directory server wouldn't issue the kerberos ticket if time is not in sync. We were getting folliwng message with event id 5:

"The kerberos client received a KRB_AP_ERR_TKT_NYV error from the server host/xxx. This indicates that the ticket used against the server is not yet valid(in relationship to the server time)."

We had recently moved our time server to a new IP and these are the steps needed for the time sync:

1. Control station time: This is not critical for AD logins, but nonetheless, it can be set from 2 locations:

• From GUI: login as root to the control station, go to Manage Control stations , change the IP for the time server and save your settings.
• or, from command line: edit /etc/ntp.conf and /etc/ntp/step-tickers to have IP addresses of the ntp servers.

2. Time for the data movers: This is critical and this is what is needed for Active directory and CIFS share to work correctly. This can be set from command line(ssh into the control station as nasadmin and then su as root) using following commands:

To see the current settings on the data mover:
server_date server_2 timesvc stats ntp

To change the time servers, following 2 steps are needed:


server_date ALL timesvc stop ntp
server_date ALL timesvc start ntp enter_ip1 enter_ip1



Update: (After I kept getting alerts regarding time sync from celerra, I realized that time sync only works with server_2 or primary data mover and server_3 has to be pointed to internal control station IP, so use the following steps.)

server_date server_2 timesvc start ntp enter_ip1 enter_ip2
server_date server_3 timesvc start ntp 128.221.252.100
 (use the spaces between ip addresses if you have multiple hosts)
(and 128.221.252.100 is the standard internal IP for control station. Since server_3 can only reach that network, it would be good with that)


Give it a minute or so and you can check the new time using:
 

server_date ALL
date

And don't worry about the lag between active and passive  data movers, that is normal.
If the control station is also time synced, time between server_2 and server_3 should be same as well.

View 4 vs View 5 Performance

This post could be titled as RDP vs PCoIP on view 5 as well.

Almost 3 years I wrote about citrix vs vmware VDI here: 
http://rjalan.blogspot.com/2009/01/vmware-vdi-vs-citrix-xendesktop.html

We ended up going vmware view route as it was apropos for our use case scenario. As things change always, we have few users going for low latency links and multiple hops and we started seeing perfomance issues with view 4.
We just upgraded to view 5, and using zero client with PCoIP we are seeing almost 70-75% reduction in bandwidth usage versus regular RDP over view 5. And performance for flash and videos is considerably better. Video is still little choppy but bearable. In earlier version, it was just showing broken images being refreshed.

cool datacenter documentation software

we recently started using device42 at our company for data center documentation. So far quite impressive as it can record almost everything about servers including IP, port connected even virtual and blade support.
Coolest thing is the application or process mapping feature in the software. Once you define dependencies just to next layer, it automatically generates a visual diagram of the whole dependency chart and process flow.
Check it out at www.device42.com.

VMware view findings

In continuation to one of my earlier posts here:
http://rjalan.blogspot.com/2009/01/vmware-vdi-vs-citrix-xendesktop.html

We concluded the following for Vmware view:

Pros:

1. Connection Server that can be placed in DMZ a good add on. Opens up desktops just using https on the front end.
2. View Manager with View composer –
a. Save space for OS images as it references master image
b. Makes it easier to update all the client desktops using recomposing and updating only master image.
c. New desktop provision is very easy and quick.
3. Single sign on works well. Supposed to work with tokens also.

Cons:

1. All the local LAN connections pass through View Manager all the time, so it’s a thick connection broker.
2. All the secure connections for DMZ connection server flow through this server as well. This is obvious though because all the RDPs are connected to https through this for end users. But that demands a server that can handle all these requests simultaneously.
3. Multimedia is not any better than normal RDP.
4. No real settings for bandwidth control for RDP. This might be a bandwidth hog depending on your needs. Through active directory group policy, we was able to make following changes:
a. End client(initiator) user can get these settings if in Active Directory: color depth, desktop background, themes, cursor shadow, compression. But in case, end user is connecting from a home computer etc., this settings can not be enforced.
b. For VM desktop, only 2 settings: Desktop background and color depth. These don’t make any major difference in the bandwidth utilization.

Poor Man's Failover on ESXi servers

HA can't do without VI license, but manual failover you can.
If you are running production, you still might want to get support for ESXi.

What you would need:
1. 2 ESXi servers running identical CPUs and have Shared Storage (FC or iSCSI).
2. Virtual Infrastructure Client(VIC)(Free included w/ Free ESXi).
3. Remote CLI installed on any other server or your desktop. Although, all the steps could be performed by VIC as well.


Steps to do the manual failover:
For documentation, I would use "testVM" as my VM guest running RHEL and 2 ESXi servers are serverA and serverB.
1. Install testVM on serverA. You would use VIC for this.
2. register the machine with other host, in this case, serverB. I prefer using remote CLI. Following command is needed:
C:\Program Files\VMware\VMware VI Remote CLI\bin>vmware-cmd.pl --server serverA --username administrator -s register "[SharedStorage] testVM/testVM.vmx" ha-datacenter Resources
Enter password:
register() =1
Here ha-datacenter is the datacenter name and Resources is the resource pool. You can specify your own resource pool name if desired.

Other way of accomplishing the same thing is by going to configuration of serverB from VIC. Go to storage, click on shared storage disk, right click and browse. Then browse to testVM, and right click on .vmx file and click "Add to Inventory".

2. using VIC shutdown testVM on serverA, poweron testVM serverB and choose always keep on the serverB when testVM is powering up.

3. now you can go back and forth easily between two hosts using VIC or remote CLI.
For remote CLI:

C:\Program Files\VMware\VMware VI Remote CLI\bin>vmware-cmd.pl --server serverA --u
sername administrator "[SharedStorage] testVM/testVM.vmx" stop
Enter password:
stop() = 1
C:\Program Files\VMware\VMware VI Remote CLI\bin>vmware-cmd.pl --server serverB --u
sername administrator "[SharedStorage] testVM/testVM.vmx" start
Enter password:
start() = 1

Vmware VDI vs Citrix Xendesktop

Few months ago we did a bakeoff between Vmware VDI and Citrix Xendesktop solution.
Few things that are worth a mention for differences between these:
1. Citrix ICA protocol for remote desktop connectivity is better with multimedia apps than RDP 5.x.
2. Both RDP and ICA are bandwidth hogs, ICA might be slightly better over low latency link. Average bandwidth usage over WAN without any policing/acceleration technology was 300kbps with both ICA and RDP.
3. Citrix Presentation server has a great option of saving the disk space for multiple cloned desktops and also upgrading/patching only one master image and refreshing all desktops from it.
4. Citrix Access gateway for secure connectivity from outside seems like a separate purchase.(didn’t test it)

We didn't go ahead with the whole project at that time.
Now Vmware has announced availability for VMware view that addresses point # 1,3 and 4 above. We are currently doing a bake off of vmware view and I will soon post how it went.

Unbreakable Linux Kickstart HowTo

Oracle Unbreakable linux is essential same as Redhat Linux. This post I will cover how to do a kickstart on Unbreakable linux AS 4 update 6. This would apply to other versions and RedHat installations as well.
Kickstart simplifies your installation of linux servers on the network. Essentially, these are the main components to it:
1. The server gets a small boot file from floppy, CD, USB or over the network.
2. The server gets a kickstart config file as above from media or over the network.
3. Depending on what you choose as your installation media in the kickstart config file, the operating system packages are installed.
4. You can mention pre and post installation steps in kickstart file to automate processes such as user creation etc.

From my point of view a network install all the way makes more sense and that is what I would be covering here today.
Redhat website has a nice detailed howto all the methods and can be found here:


I will break it into 3 steps:
Step 1: install required services/packages.
Step 2. Configure required files.
Step 3. Starting the pxe boot

Step 1:
We need 3 services that could be hosted by a single server or multiple servers and a few config utilities:
1. DHCP service ->gives IP address to PXE boot client and tells it the file name to boot from and tftp server address.
2. TFTP server ->where the initial boot file resides, that contains info regarding install.
3. NFS server -> where the kickstart config file and installation tree is available.
4. service-config-netboot -> that would generate the pxelinux.cfg tree and boot file for the client.

1st lets get into making the installation tree available locally on the box. I copied all the .iso files under /kickstart on the server. It could be any folder you choose as long as you are willing to NFS share it.

For my setup, I had all of 3 services above on a single server. Lets get to installing or starting these services.
These would reveal if you have tftp server and dhcp server installed or not:
rpm -qa grep *tftp*
rpm -qa grep *dhcp*
rpm -qa grep *netboot*

If these services are installed, skip this section and move onto step 2.

Install all these 3 packages using your favorite method. You can do yum install .
Or you can install them from the installation CDs. I didn't know which CD contained these packages, I mounted all of them. OUL(oracle unbreakable linux 4_6 64bit) has 5 installation CDs. So I created 5 mount points and mounted them on the server:
mkdir -p /mnt/tmp1
mkdir -p /mnt/tmp2
mkdir -p /mnt/tmp3
mkdir -p /mnt/tmp4
mkdir -p /mnt/tmp5
mount -o loop Enterprise-R4-U6-x86_64-disc1.iso /mnt/tmp1
mount -o loop Enterprise-R4-U6-x86_64-disc2.iso /mnt/tmp2
mount -o loop Enterprise-R4-U6-x86_64-disc3.iso /mnt/tmp3
mount -o loop Enterprise-R4-U6-x86_64-disc4.iso /mnt/tmp4
mount -o loop Enterprise-R4-U6-x86_64-disc5.iso /mnt/tmp5

Doing the following on the above mount points to see where the required RPMs are:
cd /mnt/tmp1/Enterprise/RPMS
and
ls | grep *dhcp*
ls | grep *tftp*
ls | grep *netboot*
After locating them, do the install. You would need to install tftp server before service-config-netboot:
rpm -i dhcp-3.0.1-59.EL4.x86_64.rpm
rpm -i tftp-server-0.39-2.x86_64.rpm
rpm -i system-config-netboot-0.1.40.1-1.x86_64.rpm

Step 2:
The following files would be created/edited:
a. kickstart config file
b. xinetd.d tftp file
c. pxelinux.cfg directory and install files.
d. dhcpd.conf file

a. kickstart config file

Since our install was oracle specific. I downloaded the recommended config file from oracle's wiki website at following address: http://wiki.oracle.com/page/Linux+installation+kickstart+for+Oracle+database
You can customize this according to:
http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/custom-guide/s1-kickstart2-options.html
or you can use GUI to create one for you using service-config-kickstart(You might have to install this package if its missing).

The following config file is basically a copy of one downloaded from oracle website with some modifications like:
1. NFS install instead of url
2. Hostname etc. changed
3. No post installations here.
You should make the necessary modifications as per your requirements to this file. The default root password in this config file is password.

Kickstart config file, saved as example.cfg under /kickstart:

##############################################################################
# Uncomment the next line to enable interactive installation
#interactive

# Comment the next line to use GUI installation
text

install

# From where to get the installation disks
#cdrom
#url --url http://myserver/redhat/
nfs --server=192.168.1.10 --dir=/kickstart

lang en_US.UTF-8
langsupport --default=en_US.UTF-8 en_US.UTF-8
keyboard us
xconfig --resolution 800x600 --depth 16 --defaultdesktop gnome

# Change the hostname and peek the best eth configuration for you
#network --device eth0 --bootproto dhcp --hostname srvoracle01
network --device eth0 --bootproto static --ip 192.168.1.100 --netmask 255.255.255.0 --gateway
192.168.1.1 --nameserver 192.168.1.1, 192.168.1.2 --hostname testmachine

rootpw --iscrypted $1$uKWECPhN$Im66UG8MpWd2/kpcHoyuy/
firewall --disabled
selinux --disabled
authconfig --enableshadow --enablemd5

# Set a different timezone if not located in EST
timezone America/New_York

bootloader --location=mbr --append="rhgb quiet"


# Create a Volgroup device sda
clearpart --all --drives=sda
part /boot --fstype ext3 --size=100 --ondisk=sda
part pv.3 --size=0 --grow --ondisk=sda
volgroup vg00 --pesize=32768 pv.3

# Create the filesystems and logical volumes
logvol / --fstype ext3 --name=lvol00 --vgname=vg00 --size=2048
logvol /tmp --fstype ext3 --name=lvol01 --vgname=vg00 --size=2048
logvol /usr --fstype ext3 --name=lvol02 --vgname=vg00 --size=3500
logvol /usr/local --fstype ext3 --name=lvol03 --vgname=vg00 --size=1024
logvol /var --fstype ext3 --name=lvol04 --vgname=vg00 --size=1024
logvol /var/log --fstype ext3 --name=lvol05 --vgname=vg00 --size=1024
logvol /opt --fstype ext3 --name=lvol06 --vgname=vg00 --size=1024
logvol /home --fstype ext3 --name=lvol07 --vgname=vg00 --size=512

# Edit the swap space to be twice the size of the machine RAM, if you
logvol swap --fstype swap --name=lvol08 --vgname=vg00 --size=8192


# All packages needed by Oracle installation include Gnome desktop
%packages
@ system-tools
@ gnome-desktop
@ dialup
@ compat-arch-support
-openldap-clients
-ckermit
-wireshark
-bluez-pin
-OpenIPMI-tools
-samba-client
e2fsprogs
-screen
-xdelta
-zsh
-nmap
lvm2
sysstat
-open
kernel-smp
grub
binutils
compat-db
control-center
gcc
gcc-c++
glibc
glibc-common
gnome-libs
libstdc++
libstdc++-devel
make
pdksh
sysstat
xscreensaver


%post
##############################################################################

Need to make the folders available for NFS share:
This is the /kickstart folder that contains kickstart config file and installation .iso files.
nfs-export --dir /kickstart --perm ro --ip 192.168.1.100
nfs-export --dir /kickstart --perm ro --ip 192.168.1.10

If you skipped step 1, mount 1st iso CD under /mnt/tmp1
(e.g. mount -o loop Enterprise-R4-U6-x86_64-disc1.iso /mnt/tmp1)
And make is available for NFS share:
nfs-export --dir /mnt/tmp1 --perm ro --ip 192.168.1.10
nfs-export --dir /mnt/tmp1 --perm ro --ip 192.168.1.100

Now we need to start the tftp daemon:
b. xinetd.d tftp file

Edit the following to run tftp as daemon, you can choose to run this in standalone mode using /usr/sbin/in.tftpd and not editing this file:
/etc/xinetd.d/tftp:
###################################################################
service tftp
{
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
###################################################################

restard xinted.d:
/etc/init.d/xinetd restart

Next step is to get /tftpboot/linux-install folder ready for boot files.
c. pxelinux.cfg directory and install files.

Using the following command to generate these files under /tftpboot:
/usr/sbin/pxeos -a -i "OUL" -p NFS -D 0 -s 192.168.1.10 -K nfs:192.168.1.10:/kickstart/example.cfg -L /mnt/tmp1 OUL4
This also generates a defualt file under /tftpboot/linux-install/pxelinux.cfg, which is read by the installer at boot time. You can create one specific for your host using pxeboot command.

d. dhcpd.conf file

DHCP config file.
You can get the MAC address for the client in lot of different ways. If you PXE boot the client, it throws the client MAC address right at the console. You would need this to set the values in following file.
If you are crossing vlans/routers make sure ip helper address or equilavent is set on the routing/network device.

/etc/dhcpd.conf :
###################################################################
deny unknown-clients;
not authoritative;
ddns-update-style ad-hoc;
option domain-name "lcc.copr.pvt";
option domain-name-servers 192.168.1.1;
option subnet-mask 255.255.255.0;

allow bootp;
allow booting;

option ip-forwarding false; # No IP forwarding
option mask-supplier false; # Don't respond to ICMP Mask req

subnet 192.168.0.0 netmask 255.255.255.0 {
option routers 192.168.1.1;
}
group {
next-server 192.168.1.10; # name of your TFTP server
filename "linux-install/pxelinux.0"; # name of the bootloader program

host node1 {
hardware ethernet 00:14:4F:45:14:0E;
fixed-address 192.168.1.100;
}
}
###################################################################

Start DHCP server:
/etc/init.d/dhcpd start

Step 3. Starting the pxe boot

Change the boot order to boot from the network on the client machine and if all the configs were done correctly, you should be able to go through the install smoothly. You can watch as it goes and look for error messages if any to fix any typos etc.