AIX and its "stanza" structure

Most *nix systems arrange their system data in delimited rows of text files. Pretty simple, eh? Grep for the data you are searching on, then awk out the field you want.

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin

Not so AIX. This OS favors the "stanza" structure, in which data is presented in this form:

root:
        password = XXXXXXXXXXXXX
        lastupdate = 1239276751
        flags = 
 
joeuser:
        password = XXXXXXXXXXXXX
        lastupdate = 1239044995
        flags =       

Ah well, to each their own right? So how do you quickly parse data out of an AIX file? I use this method...probably not the best but it works.

sed -n '/joeuser/,/:/p' /etc/security/passwd

This will return the stanza for joeuser up to and including the next user name below him (:), which you can then grep for whatever field you want.

"I'm root, and you're not!"

We once came across a very unusual problem: root could log in to this system, get a shell, do anything that root normally does. Non-root users however couldn't log in, those that were logged in could not run any commands including the basics, ls, cd, nothing. Only shell built-ins such as echo worked, and only if you were already logged in.

Our support contact finally told us to look at permissions for the root of the file system:

root@mahler:/# ls -ld /
drwx------ 21 root root 4096 2009-01-22 19:27 /
root@mahler:/# 

A big "WTF!" Clearly, someone fat fingered a chmod.

By now you have guessed the solution:

# chmod 755 /

Repetitive tasks 101

It happens all the time, but recently I was helping with some troubleshooting. The request was for me to issue a telnet command from our UNIX system so that the network people could see the packets and figure out where they were being dropped.

After about the 20th time, I was getting tired of hitting [up-arrow, enter]. So I entered a script at the command line of this system:


# while :
> do
> telnet 192.168.1.50 &
> sleep 10
> pkill telnet
> done


I advised them that the system would repeat the telnet request every ten seconds, kill it, and launch a new one. Repeatedly. For as long as it took.

Then I went and got coffee.

Whew...very strange ARP problem on Solaris 10

File this one under "Current Events"!

If you are running Solaris 10 update 4 or higher, you may notice Ethernet addresses in Address Resolution Protocol (ARP) cache that do not match the actual address. Here's a good discussion of what's happening.

http://forums.sun.com/thread.jspa?threadID=5327921&start=0

The problem is that dual-homed Windows boxes with Broadcom NICs on the same subnet are hosing the ARP table on your Solaris system. Sun says its drivers are not the problem, and they adhere closely to the RFC 826. Regardless, it is a headache for system administrators.

The correct solution is to patch the Windows boxes. But there may be other workarounds, too. Check the comments section of this post in the weeks ahead.

Kickstarting Linux installs

If you're not familiar with kickstart, it's a way of passing all or some of the parameters to all those questions they ask you at install, greatly speeding up the install process.

We used to do kickstart installs with floppy drives. Simple enough, you write a ks.cfg file to an ext[2-3] formatted floppy disk, boot off your install CD and enter at the boot: prompt:

boot: linux ks=floppy

Lo and behold, servers started showing up without floppy drives, negating the kickstart advantage.

That is until we finally found out how to use a USB thumb drive to accomplish the same thing. Drop the ks.cfg in the root directory of the thumb drive and enter:

boot: linux ks=hd:sda1:/ks.cfg

That's assuming sda1 is your USB device. It could be something different. For instance if you have two internal (logical) raid devices, the device will probably sdc1, and so forth.

The system is running out of swap space, and I have to add more NOW!

Here's how to add swap on the fly. Find a disk partition on your system with enough space to hold a swapfile...let's say in this example, 700 Mb. Then cd to that directory.

Let's start with Solaris which is just a couple of commands:

# mkfile 700m ./swapfile
# swap -a ./swapfile
# swap -l

Instant relief! Now let's tackle Linux which is just a little more involved.


# dd if=/dev/zero of=swapfile bs=1024 count=655350
655350+0 records in
655350+0 records out
# mkswap ./swapfile
Setting up swapspace version 1, size = 671072 kB
# swapon ./swapfile
# free

It's important to note that this additional space will not be mounted as swap on the next reboot unless you add the appropriate commands to the system's startup scripts. But these commands WILL get you through a period of heavy virtual memory usage.

I have to grab every file from a web site!

I hope you're not going to sit there at your browser and click away till you get every file. Find yourself a *nix system with the widely-available wget command installed, change to a directory with a lot of space, and make it easy on yourself!

$ wget --mirror http://www.mysite.com

The --mirror option will follow all links to other files on this site.

You may be in a situation where you need to grab all the numerically-named files in one directory, say from file1.html to file3000.html, even if they are not linked.

$ /bin/bash
$ for i in $(seq 1 3000)
> do
> wget http://www.mysite.com/numberfiles/file$i.html
> done

Simple integer math in modern shells

If you're accustomed to doing simple math functions in Bourne shell, you probably recall the need for the clumsy expr command, like this:

AVERAGE=`expr $MYCOUNT / 365`

That kind of fractured syntax, with the spaces having to be in the right place, is what makes shell math too complex when compared to higher-level languages like Perl.

But if, like 90% of admins these days, you are using a more advanced shell like Korn or Bash, you can make your math life a lot simpler and more intuitive. Just use the double parentheses operator.

((AVERAGE=$MYCOUNT/365))

No spaces are needed (they're optional) and it's easier to read and analyze, especially when you are doing a large number of calculations inside a script.

I want to setuid root on my shell script.

No, you don't. Trust me, you really really don't want to do this. It's a security hole a mile wide.

Now having said that, some shells do have options that allow the observance of the setuid bit. Most shells do not have that option.

But if you absolutely must run something setuid root, I would much rather you wrap the commands of your shell in a small C program, using the system() call for instance, compile it, and then setuid on that compiled binary.

#include<stdio.h>

main(int h)
{
        return (system("/run/whatever/here"));
}

Save this file with a .c extension, then compile this with gcc:

gcc myprog.c
mv a.out mynewexe

...then setuid on the compiled executable, mynewexe. If you must. Be careful. By the way, you ever heard of sudo?

I can't create a file and df shows plenty of space!

This file system has probably run out of inodes. You can check this by:

Linux and AIX: df -i

Sun: df -F ufs -o i

Ten bucks says all your inodes are in a subdirectory named mail. Once you find the offending subdirectory, even this may not work:

rm *

because you may exceed the number of arguments rm can handle. You can put out the fire by whacking the oldest 500:

ls -t | tail -500 | xargs rm

Then use a similar strategy to clean up the rest of the files.

Automate Linux password changes

We all know, or should know, that "expect", Don Libes' tool to automate interactive programs, is the right tool to automate password changes across disparate platforms.

But did you know that some versions of the Linux passwd(1) command have an option to accept new passwords from stdin? There are cases where this would really come in handy, for instance when creating a bunch of new accounts.

for f in user1 user2 user3 ... usern
do
/usr/sbin/useradd $f
echo "newpassword" | /usr/bin/passwd --stdin $f
done

Check the man page of passwd(1) on your system to ensure the --stdin option is available to you.

What do all those TCP states mean in netstat?

You see them in UPPER CASE when doing a netstat -a command. The state of your TCP connections is very useful in determining what is going on network-wise on your system.

LISTEN
Means the socket is listening for connections (duh). Look to the left for the port/protocol that's listening.

ESTABLISHED
Indicates an active, communicating session (double duh).

TIME_WAIT
Session is done. You may sometimes see these accumulate but they'll eventually time out.


This is the majority of what you'll see when you do netstat -a. But the next group are very transitory and normally should only last a millisecond or two. If you see these persistently, you can be sure there's some problem with the communications partner, or a firewall, or something else at the logical layer.

CLOSE_WAIT
The connection is closed but waiting for an official close that may or may not come.

SYN_SENT, SYN_RECEIVED
The connection is just starting up with the three-way TCP handshake and you've caught it right in the middle...what are the chances?

FIN_WAIT, CLOSING, LAST_ACK, FIN_WAIT_2
Same thing but the connection is closing. These happen momentarily and something is wrong if you see them consistently.