Finally some news from IBM on the upcoming version 6 of AIX! IBM just launched an open beta program and apparently welcomes anyone to take part in testing the release of AIX. Judging by the new features being announced in this soonish-to-arrive relase, IBM appears to be making an effort picking up the ball that it dropped on actually developing its flagship OS instead of just incrementally adding hardware support. The list of to-be-added features looks quite interesting indeed:

• Workload Partitions
• Live Application Mobility
• Role Based Access Control
• Trusted AIX
• Encrypting filesystem
• AIX Security Expert LDAP integration
• Secure by Default installation option
• Graphical Installation
• Network Installation Manager support for NFSv4
• Concurrent AIX kernel update
• Dynamic tracing
• Enhanced software first failure data capture

The laundry list of feature looks quite impressive, but hardly original since overwhelmingly they seem to be reminding me of a me-too effort counteracting the features debuted in Solaris 10. Workload partitions sound much like Solaris Zones. Role Based Access Control, well, Solaris had it since version 8. Trusted AIX, hmmm, Solaris Trusted extensions anyone? 'Secure by Default installation option', just announced in Solaris 10 Update 3. 'Dynamic tracing' or probevue as IBM calls it, well, sure as hell sounds like DTrace. 'Enhanced software first failure data capture' is an answer to Solaris FMA if you ask me. So is it just me or IBM is really scrambling to beef up AIX and make it look at least a little less embarrassing compared with Solaris 10? Did IBM finally realize that the Linux train is not going to overtake Solaris and by not actively advancing their own flagship OS (AIX) they're in danger of falling hopelessly behind? Well, its about freaking time they realized that. Welcome back IBM! I'm glad to see you finally treat AIX as a first class product instead of a soon-to-be-retired horse waiting to be swapped out for a penguin.

See the full announcement at the following url:

http://www-03.ibm.com/servers/aix/6/preview.html

Outside of a whole raft of "celebrity" features (i.e ZFS, Dtrace, SMF, FMA, etc, etc.) added into Solaris 10, I'm still discovering nice little changes that leave a good feeling of added improvement. One such nicety I discovered today is the LookupClientHostnames option available for the SSH daemon. Before this option was available the Solaris SSH daemon would always try to resolve the address of a connecting client to a name, which would lead annoying delays when ssh is used interactively before giving you a command prompt or worse yet limit performance of the applications that actually rely on ssh/scp. That was one of the reasons we would just uninstall the ssh daemon shipping with Solaris and compile OpenSSH from source. Well, now it looks like we don't have to bother with OpenSSH again, just setting

LookupClientHostnames no

in /etc/ssh/sshd_config will do the trick. Happy ssh'ing!

It is kind of funny how people refer to the stuff stored in /etc/shadow as "encrypted passwords" (on HP-UX it is notably still in /etc/passwd most of the time). If you know even tiny little bit about cryptography, you should know that it is completely incorrect. People in the know also say it simply because the overloaded use of word "encrypted" became too commonplace. Well, the passwords in the /etc/shadow are not encrypted, they are hashed - when a new password is created by the user, the password changing utility will generate a hash of that clear text password (usually using Unix crypt, MD5, or SHA1 hashing algorithm) and store it along with the uid of the user in the /etc/shadow file. So here comes a practical question that gets asked a lot, how do I know that a particular hash in the /etc/shadow corresponds to particular clear text password? Well, using openssl utility it is actually a fairly trivial exercise. For instance lets pretend that we've got hash from the shadow file and we want to know if it actually corresponds to a password we already know. Assume that the password hash that we've got is "7im3hh5KvvmQQ" and we want to know if it corresponds to password "secret" or "foobar". Since the password hash appears to be generated by Unix crypt algorithm, we should now that the first 2 characters are used as a salt and so if we want to generate the correct hash we need to borrow those two characters. Using openssl utility the task of generating the hashes is quite trivial:

openssl passwd -crypt -salt 7i secret
7iX0ElmF8n4oA

Well the generated hash 7iX0ElmF8n4oA does not match our hash in question 7im3hh5KvvmQQ, which means only one thing - "secret" is not the clear text password that was used to generate it. Let's try "foobar" as a password:

openssl passwd -crypt -salt 7i foobar
7im3hh5KvvmQQ

Well, what do you know the hashes match and so we know that the password behind the hash was "foobar".

I'm sure you won't be using this hash everyday, but sometimes it may come useful debugging certain authentication problems.

The other day I had a junior sysadmin approach me asking for Solaris system with a storage array attached to "practice using ZFS". Apparently he wanted a few drives to be available on the system to be put togeather into a pool to get his feet wet with ZFS administration. Well, he didn't get his wish with getting a dedicated storage array for "sysadmin practice" and not strictly for the reason of unjustified use of hardware reasources, but rather because you don't need a storage array with a number of disks to practice ZFS administration -- ZFS will be just as happy if you give it files as devices in the storage pool. All you need is some spare free storage space for "virtual drives" and you're fully in business of playing with pretty much everything ZFS has to offer. For instance if I wanted to see how RAID-Z array would behave with double parity, which requires more than two drives I can just simulate the drives with three files (of course you can have as many as you want):

Create 3 files 100MB each simulating the correspondingly sized disk drives

# mkdir /data
# cd /data
# mkfile 100m disk1
# mkfile 100m disk2
# mkfile 100m disk3

Now create RAID-Z pool using these "drives":

# zpool create testpool raidz2 /data/disk1 /data/disk2 /data/disk3

Now a zpool with the name testpool should be available and happily mounted under /testpool -- you've got yourself a working ZFS pool without waisting any additional physical resources. Now you can create additional filesystems on that storage pool and test all the neat tricks ZFS has to offer:

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
testpool 286M 200K 286M 0% ONLINE -
# zpool status
pool: testpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz2 ONLINE 0 0 0
/data/disk1 ONLINE 0 0 0
/data/disk2 ONLINE 0 0 0
/data/disk3 ONLINE 0 0 0

errors: No known data errors

# zfs create testpool/foo
# zfs set quota=50M testpool/foo
# zfs create testpoo/bar
# zfs set quota=100M testpool/bar
# zfs create testpool/foo
# zfs set quota=50M testpool/foo
# zfs create testpool/bar
# zfs set quota=100M testpool/bar
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
testpool 173K 158M 36.6K /testpool
testpool/bar 32.6K 100M 32.6K /testpool/bar
testpool/foo 32.6K 50.0M 32.6K /testpool/foo

To see how ZFS would cope with a failure of one drive and replacing the failed drieve with a spare we can do the following:

1. Let's write some junk into one of the drive effectively corrupting the device:

# dd if=/dev/random of=/data/disk3 bs=1k count=100

2. Lets scrub the pool to make ZFS identify the failure immediately:

# zfs scrub testpool

3. Now we can see that disk3 is effectively toast:

# zpool status
pool: testpool
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: scrub completed with 0 errors on Wed May 2 12:38:46 2007
config:

NAME STATE READ WRITE CKSUM
testpool DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
/data/disk1 ONLINE 0 0 0
/data/disk2 ONLINE 0 0 0
/data/disk3 UNAVAIL 0 0 0 corrupted data

errors: No known data errors

Yep, disk3 is corrupt alright, but RAID-Z should be taking care of that and all of our data should still be available.

4. Now let's create a spare "drive" that we will use to replace the failed disk3 component:

# mkfile 100m disk4

5. Let's replace the failed "disk" with a newly created spare and see what happens:

# zpool replace testpool /stuff/disk3 /stuff/disk4
# zpool status
pool: testpool
state: DEGRADED
scrub: resilver completed with 0 errors on Wed May 2 12:42:17 2007
config:

NAME STATE READ WRITE CKSUM
testpool DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
/data/disk1 ONLINE 0 0 0
/data/disk2 ONLINE 0 0 0
replacing DEGRADED 0 0 0
/data/disk3 UNAVAIL 0 0 0 corrupted data
/data/disk4 ONLINE 0 0 0

errors: No known data errors

So we see that reslivering of a new drive was successfully completed and we now have got 3 normally operating "drives" in the pool -- we're back to normal.

4. After just a little bit of time ZFS will complete the replacement procedure and the failed "drive" will be out of the pool completely - everything is running like nothing happened:

# zpool status
pool: testpool
state: ONLINE
scrub: resilver completed with 0 errors on Wed May 2 12:42:17 2007
config:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz2 ONLINE 0 0 0
/data/disk1 ONLINE 0 0 0
/data/disk2 ONLINE 0 0 0
/data/disk4 ONLINE 0 0 0

errors: No known data errors

It goes without saying that using files as devices should be used only for training and experimenation and certainly not in production, since there is little to be gained from using ZFS in this fashion. Other than that give it a whirl and I'm sure you'll want to deploy ZFS in productionon using real hardware after that - ZFS is an absolutely beautiful piece of technology.

In the last post I briefly pointed out the virtues of Sun Java Enterprise System stack. As a case in point here is a good review of Sun Java Web Server 7 (part of JES stack) being contrasted with Apache:

Sun Java Web Server Delivers a Jolt

Apparently Sun Java Web Server will give you at least twice the performance compared with Apache. All that being a more polished and easier to use product. And oh yeah, don't forget the security - Sun Java Web Server is one of the most secure web server products on the market with just a handful of security issues throughout the rather longish lifetime of product dating all the way back to Netscape days.

I came to love the Sun's Java Enterprise System stack recently, especially the latest JES 5 release, which bundles very impressive products in one easy to install pack. If you're not familiar with JES, it is stack of middleware/server products for satisfying 90% of the server software needs of pretty much any organization out there -- the J2EE Application Server (think WebLogic/WebSphere replacement), the Web/Proxy Server (throw Apache out), Directory Server (a much nicer and *much* more powerful alternative to ActiveDirectory or OpenDirectory), and a Messaging Server (who needs bloated and overpriced MS Exchange anyway). What is really good about the Sun's Java Enterprise System stack is that it is composed of very mature and very enterprise-class products -- all products in the stack are used to support the largest loads you can throw at them and they are apparently being very successful at doing so at the majority of Fortune 100 companies. As an added bonus you get all these products for free! Yes, for free. Sun extracts its revenue from the software via support and not through licensing, so if you don't need support, you can use all of the products to any extend you choose. On the other hand if you decide that you need support, Sun is always there to sell it to you -- perfect combination if you ask me.

Get more info about Java Enterprise System at http://www.sun.com/jes