“Multipathed” local root disk

By default DM-multipath will attempt to create a multipath disk device for every disk it finds. It is relatively easy to end up with a system where local disks also get presented as multipathed devices even though they only have a single path. If the local disk is also the root disk, it is impossible to remove the multipathed device mapping without following some specific steps. It’s not immediately apparent what has happened as the multipathed root disk only shows up after a reboot.
While it would likely be harmless to leave the disk multipathed, it’s confusing for admins and clutters the device mapper tree unnecessarily.

The safest way to avoid this issue is to enable the option “find_multipaths” (new in RHEL 6).

mpathconf --find_multipaths y

This adds three conditions to whether DM-multipath attempts to create a multipathed disk:

  • A multipathed device has been present before (the files /etc/multipath/* are maintained by multipathd).
  • Two or more paths to the same WWID exist.
  • A multipathed device has been manually created.

By default, “find_multipaths” is set to “no” meaning all disks named /dev/sd* are candidates for multipathing whether they are local or otherwise.

Another RHEL 6 default is multipathing in the initramfs so that boot-from-SAN works as expected. If boot-from-SAN is not required, omitting multipathing from the initramfs is a “belt & braces” approach to avoiding multipathed local disks.

cat >>/etc/dracut.conf <<EOT

# Omit multipath from initramfs to prevent local disks being "multipathed" on boot.

# Rebuild initramfs.
dracut -v -f

Continue reading “Multipathed” local root disk

Persistent NIC naming in RHEL 6

As Linux contstructs its device tree dynamically at boot, RHEL 6 uses udev to ensure NICs are given the same name every time.
The MAC address of every NIC is stored along with a name “ethX” in the file /etc/udev/rules.d/70-persistent-net.rules.
If the MAC has been seen before, udev names the device accordingly. If not, a new entry is added with the next available “ethX” name.

This can cause difficulties when using virtual machine templates or replacing hardware. What was expected to be “eth0” may be renamed on boot to “eth4”.

In the case of a card replacement, /etc/udev/rules.d/70-persistent-net.rules can be modified in a text editor.
If none of the previous MAC addresses need keeping, the file can simply be deleted. It will be recreated on boot if it is missing.

Moving files from Linux to Windows – recursive symlinks

After restoring a Clonezilla backup of an NTFS partition in a Linux virtual machine, attempting to copy the files onto a Windows computer kept running into issues with recursive directories.

For example, /mnt/Users/Chris/AppData/Local/Application Data is a symlink to /mnt/Users/Chris/AppData/Local. WinSCP just failed with mysterious “file not found” errors but robocopy.exe showed what was happening when trying to mirror the files:

C:\>robocopy \\linux-vm\ntfs-restore\Users\Chris C:\clonezilla-restore\ /mir
2013/05/19 21:12:07 ERROR 3 (0x00000003) Scanning Source Directory \\clonezilla-vm\ntfs-restore\Users\Chris\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\

The system cannot find the path specified.
Waiting 30 seconds...^C

Presumably there’s a maximum length allowed for a path and when that is reached, the copy blows up.

Windows seems to love these recursive loops – every user seems to have an “Application Data” folder that ends up pointing to itself. C:\ProgramData\Application Data is also recursive.
Continue reading Moving files from Linux to Windows – recursive symlinks

Moving files from Linux to Windows – forbidden filenames

Windows doesn’t allow colons in filenames which can cause interesting problems when trying to copy files between Linux and Windows. The files will probably show up in Explorer but it will be impossible to do anything with them.
There seems to be no way to rename files with colons from within Windows so the renaming needs to happen on Linux first. If the problem files are on a USB drive, a virtual machine will work perfectly well on the Windows host to rename the files.

RHEL/CentOS don’t support NTFS by default but is available through EPEL if needed with yum install -y ntfs-3g ntfsprogs.

Once the files are available on Linux, get rid of the colon:

find . -name '*:*' -exec rename : _ {} \; 2>/dev/null

Disk stats in SAR

In RHEL5, SAR does not capture individual disk I/O metrics by default.
To enable disk performance figures, simply edit /etc/sysconfig/sysstat and so that the -d option is passed to sadc.

echo "SADC_OPTIONS=-d" >>/etc/sysconfig/sysstat

SAR is run by /etc/cron.d/sysstat.

If SAR is not installed, run yum install -y sysstat.

NetBackup client

Here are some useful checks and reminders when setting up NetBackup clients on Linux/Unix. It is assumed NetBackup is fairly modern as only TCP ports 1556 and 13724 are considered.

  1. Ensure NBU master server & media servers are listed in /etc/hosts.
  2. Modify /usr/openv/netbackup/bp.conf as appropriate.
  3. Open incoming ports on NBU client (if a firewall is running).
  4. Ensure NBU client is listed in /etc/hosts of all master & media servers.
  5. Test connection to master & media servers.
  6. Test connection to client from master & media servers.

The following commands check network connectivity to all master & media servers (tested on Linux and Solaris).

# Test NBU connectivity to media & master servers.
[ $(uname -s) == "Linux" ] && AWK=awk; [ $(uname -s) == "SunOS" ] && AWK=nawk;
NC="nc -v -z -w 5"
for NBU in $($AWK -F '=' '/^(MEDIA)?SERVER / {print $2}' /usr/openv/netbackup/bp.conf); do
	$NC $NBU 1556
	$NC $NBU 13724

For IPtables firewall rules, I prefer to add a new chain so that it’s clear the opened ports are for the NetBackup client.

# Save existing ruleset
cp -p /etc/sysconfig/iptables{,.$(date "+%F_%T")}

# Add NetBackup ports
iptables -A NETBACKUP-INPUT -p tcp -m multiport --dports 1556,13724 -j ACCEPT


service iptables save

Adding /usr/openv/netbackup/bin to the environment PATH gives easy access to useful commands like bpps.

Facter external facts

Facter 1.7 adds the capability to define external facts from source files (YAML, JSON and plain text) as well as parsing output from executables. This feature means knowledge of Ruby isn’t required to add custom facts. Anything that can output in the form fact=value can contribute a custom fact.

See http://docs.puppetlabs.com/guides/custom_facts.html#external-facts for more details.

Unfortunately the mainstream version of Facter is still 1.6.
Here is a custom fact for Facter 1.6 that emulates some of the functionality of external facts.

This loops through /etc/facter/facts.d. It runs anything with execute permission and parses the output; it also parses the contents of plain text files with the extension “.txt”. These external facts have a high weight so they take preference if the same fact has already been defined.

# Fact: ext_facts
# Purpose:
#   A partial emulation of 'ext_fact' present in Facter 1.7.
#   Allow executables to generate facts and text files to be parsed.
#   Scripts/executables *must* output to STDOUT in the form:
#     key1=value1
#     key2=value2
#   Text files should contain facts in the same format.
# Caveats:
#   Makes absolutely no attempt to validate the scripts/excutables.
#   Facts can get overwritten and the last one wins.

require 'facter'

ext_facts_dir = '/etc/facter/facts.d'

if File.exist?(ext_facts_dir)
  Dir.entries(ext_facts_dir).each do |ext_file|
    next if ext_file == '.' or ext_file == '..'
    if File.executable?(ext_facts_dir + '/' + ext_file)
      Facter::Util::Resolution.exec(ext_facts_dir + '/' + ext_file).each do |line|
        key = line.split('=', 2)[0].strip
        value = line.split('=', 2)[1].strip

        Facter.add(key) do
          has_weight 10000
          setcode { value }
        #puts key + " : " + value
    if File.extname(ext_file) == '.txt'
      File.readlines(ext_facts_dir + '/' + ext_file).each do |line|
        if line =~ /^(.+?)=(.+)$/
          var = $1; val = $2

          Facter.add(var) do
            has_weight 10000
            setcode { val }

# vim: ts=2 sw=2 et:

Password aging & expiry

A quick crib sheet of my most used commands for working with password aging & expiry of local user accounts.

Disable password expiry (service accounts and things that really don’t want to get locked out i.e. root):

# Linux (RHEL variants)
chage -M 99999 username

# Solaris
passwd -x -1 username

Force user to change password on next login:

# Linux (RHEL variants)
chage -d 0 username

# Solaris
passwd -f username

Coloured BASH shell prompt

Adding colour to the BASH prompt can add information or context. There are many possibilities but this example simply colours the username of the standard RHEL prompt; root is printed in red, other users are printed in green. This serves as a useful visual indicator to be careful when running commands as root.

if [ $(basename $SHELL) == "bash" ]; then
  if [ $(id -u) -eq 0 ]; then
    export PS1='[\[\e[00;31m\]\u\[\e[00m\]@\h \W]\$ '
    export PS1='[\[\e[00;32m\]\u\[\e[00m\]@\h \W]\$ '

Put this code in /etc/profile.d/ps1.sh and set permissions to 0644. No hashbang first line is required.
Continue reading Coloured BASH shell prompt

Local VMware Tools OSP repository

The VMware Tools are highly recommended when running virtualised servers in a VMware environment. VMware Tools OSP (http://packages.vmware.com) are equivalent to the standard tools available through VMware Player/Server/ESX but are packaged in Red Hat’s native RPM format. This makes it easy to set up virtual machines to install and update VMware Tools from a YUM repository. An added bonus of the OSP Tools is that they don’t require reinstalling after a kernel update (the standard Tools have modules that are tightly bound to the kernel that is running so need reinstalling whenever the kernel is changed).

Whilst it is possible to configure servers directly against VMware’s OSP repository, this article extends the Kickstart/YUM server to mirror the VMware Tools locally.

Continue reading Local VMware Tools OSP repository