Many consider the number of available jobs dealing with supporting OS as a more or less reliable indicator for popularity of that OS in the marketplace. I decided to do some checks of my own on the number of currently available jobs for major Unix OS flavors, BSD, and Linux. I used Monster.com in the United States as a source of available jobs, since the USA market represents the largest IT job market in the world and therefore should produce a more realistic picture of popularity for each OS. Searching simply by using the OS names and noting the number of hits I tabulated the results as follows (in the order of popularity):

OS Type	Number of Jobs
Linux	7564
Solaris	3136
AIX	1706
HP-UX	651
FreeBSD	97
Tru64	32

So if these numbers are anything to go by, Linux currently holds the lead with more than twice the number of jobs of its nearest competitor - Solaris scoring at 3136 jobs currently on the market. After Solaris the distance between competitors widens significantly with AIX capturing about half the number of jobs available for Solaris and HP-UX having a rather smallish share of 651 jobs (1/5 of Solaris jobs available on the market). FreeBSD and Tru64 tail the rankings with 97 and 32 jobs respectively. The Tru64 numbers are particularly sad to see, since they indicate a close-to-extinction situation with this once mighty server OS. On a brighter note, if you're looking for a career in Unix system administration or software development for Unix, Linux and Solaris skills would be the most sought after in today's Unix job market.

Even though Novell and Microsoft signed their little agreement allowing Microsoft to distribute Linux, it is not all singing and dancing between these two vendors. In this report Novell goes as far as saying that SuSE Linux Enterprise Desktop is actually better than the new fangled Microsoft Windows Vista. It is an eye brow raising claim on any count, since to this date all major Linux vendors were relatively sheepish with putting their desktop offerings squarely against Windows. Well, it looks like Novell gathered enough proverbial balls to make such a claim and quite honestly it makes me respect Novell more then ever. It takes fighting spirit to look a 600 pound gorilla squarely in the eye and claiming that you deserve your fair share of the desktop market. Even though the Linux pundits have been proclaiming every year for the past five or so years "the year of Linux desktop", Linux consistently failed to capture the hearts of desktop users with the exception of a relatively small number of technophiles being consistently relegated to the server. Though I wouldn't be true to myself saying that Linux achieved the same level of usability and refinement on desktop as Windows, I have to say SuSE Linux came pretty damn close. At least in the enterprise environment (think call centers) I wouldn't have any qualms replacing Windows with SuSE. If you don't have a whole lot multimedia stuff to deal with, which is something you really shouldn't be doing in the call center type environment, SuSE Linux is more than capable replacement.

On top of pure features and capabilities, the pricing advantage SuSE Linux Desktiop has over Windows Vista should be a pretty good nudge to move in the Linux direction. Consider this, a per-desktop bundle of Windows Vista and Office 2007 comes in at around $700 to $800 dollars per user, which is nothing to sneeze at. Compare this to an all-singing-all-dancing Linux bundle coming in at $125 per user. Run these numbers on a floor-full of desktops that need to be installed and you should undoubtedly see some pretty serious savings if you go with Linux -- Novell is taking a pretty aggressive stance.

I hope Novell captures its share of the corporate desktop with its offering simply because they deserve it with this solid offering. But again, outside of the purely technical parity that Novell has almost managed to achieve with Microsoft, what I really like seeing is that a major player started seriously pitching Linux as a Windows replacement on the desktop.

Power and conciseness of Perl is quite impressive and I get reminders of that every time I have to solve a seemingly complicated problem, which is then solved by just a few lines of Perl. Case in point today I had process the logs and feed the resulting data to a reporting utility we use in house. And this is where I hit a snag - the log files as you're well aware are in chronological order (oldest record first, the newest records last) whereas the reporting utility expects the order of records fed to it in just the opposite order. In other words I had to reverse the order of all lines in the huge data file. Since there is no seemingly obvious utility that can perform this task for you I was ready to concoct a utility of my own in C. Just in time I reminded myself that may be I can accomplish the same task with Perl. Well, as you would have guessed it there is a function in Perl aptly named "reverse" that can perform the task for me. After looking at the ways to put the utility together the solution came out embarrassingly simple:

perl -e 'print reverse <>;' filename_to_be_reversed.txt

That's it! The solution to the problem is quite humbling and made me respect Perl even a little be more.

Getting something for nothing is something very irresistible by nature, you feel like you have to do it just because it is there. In computing (ok, overwhelmingly in desktop computing anyway) the holy grail of getting something for nothing is overclocking -- jacking up frequency of the CPU beyond its official spec limit. Overclocking is the favorite pass time for many geeks out there. The x86 CPU's lend themselves pretty well as subjects for overclocking and some folks managed to achieve some pretty impressive feats with increasing the clock speeds. The RISC space on the other hand is pretty much completely devoid of such activity. The biggest reason is that the RISC machines are not to be taken lightly most of the time, since quite frequently they are in some sort of a mission critical production setup and so there wasn't a whole lot of ground for people to experiment unless of course you're willing to pull your ass on the line -- overclocking is not the safest activity to be done on a mission critical gear and is likely to void some support contracts if used, even when no damage is done to the hardware. But in the recent years with a seeming deluge of very cheap RISC workstations in a form of Sun UltraSparc-powered Ultra workstations many geeks out there can experiment with these once-untouchable-by-common-folk machines. As it comes out with the most of Sun workstations the overclocking process is dead easy -- you don't even have to flip any jumpers to increase the clock speed, since everything can be done in the PROM. Not surprisingly the OpenBoot PROM commands that allow for overclocking are not very widely advertised by Sun.

The boxes that are known to be overclocked are the good old Ultra 5/10/30/60 machines. And the overclocking on them can be done as simple as dropping the machine to the OpenBoot PROM (the ok prompt) and issuing the following command:

also hidden XXX at-speed

where the XXX is the hex number designating the clock speed of the CPU in Mhz. You have to play around with numbers that will be accepted by the hardware and do some checks by issuing the ".speed" command at the OBP prompt -- make sure the new clock speed is accepted before saving the settings into NVRAM. So far the 440Mhz CPU's in Ultra 10 machines have been known to be overclocked well to 480Mhz (1E0 in hex); and the 360Mhz CPU's in Ultra 60 can be overclocked to 370Mhz (172 hex). These are not exactly mind blowing increases in the CPU clock speed and therefore will not suddenly turn ye olde Ultra 10 into a speed demon, but hey, getting something for nothing always feels real good.

For all of you out there who cut your teeth on Linux, but would like to expand to more industrial strength Unix like Solaris, here is a great Linux to Solaris administration guide that outlines the most fundamental differences in administering Solaris as opposed to Linux in every day life:

Linux to Solaris Administration Guide

Most notably the guide touches on such truly innovative things in Solaris that have no alternatives in Linux as DTrace, Zones, ZFS, and Predictive Self Healing.

If I could get payed a penny every time someone encountered an OutOfMemoryException from Java and wrecked his/her brain why Java says there is not enough memory on a seemingly adequately configured machine, I would be a millionaire. The reason many administrators seem to get stumped with this error message is a complete lack of understanding how Java Virtual Machine works. People making this mistake fail to realize that Java applications are inextricable from the concept of "sandbox" they are running in. Sandbox is a virtual environment that is created by the Java Virtual Machine (JVM) for the running application. The JVM presents the application with its own view of platform it is running on for the sake of improved security and manageability, so the amount of memory the application is allowed to "see" and use is also configurable. The amount of heap memory available to the application running under a particular instance of Java Virtual Macines is rationed by the JVM on startup -- the maximum amount of heap memory that can be given to an application is determined by the configuration parameter passed to Java runtime. The Java runtime users arguments passed to it via -Xms (minimum heap size) and -Xmx (maximum heap size) parameters to configure itself on startup and setup the minimum and maximum boundaries of amounts of memory the application can use. If neither -Xms nor -Xms parameters are provided to the runtime, the following defaults for the maximum amount of memory the application can address are taking effect on JVM 1.4.2 for instance:

• Windows: Half the real storage with a minimum of 16 M and a maximum of 2 G
• AIX, and z/OS: 64 M
• Linux: Half the real storage with a minimum of 16 M and a maximum of 512 M
• Solaris: 64 M
• HP-UX: 64 M

So if a Java application was launched without any -Xmx parameter increasing the default heap size, the amount of memory the application will be able to address on Solaris, HP-UX and AIX are measly 64M. Once the 64MB limit on memory utilitzation is reached by the Java application, the OutOfMemoryException will be thrown and the Java Virtual Machine will shut itself down, since the allocated resources are exhausted and there is no way the application can continue running unless the JVM is allowed to use more memory.

Of course the path to resolution of this issue is not to start scrambling to add more physical memory to the machine running the Java application in trouble (even though I've seen this happen more than once), but rather configure you JVM to allow more heap memory to be dedicated to the Java application. Since most of the Java applications on Unix are started via a shell script that launches the JVM and passes it the necessary configuration parameters, you would have to identify and dissect the startup script to see how the -Xmx parameter is set (even though some applications allow you change the -Xms and -Xmx parameters via some sort of a user interface). Once you've identified the way -Xmx parameter is set in your application, you can modify the script or use administrative front-end increase the amount of heap memory the application will be allowed to use and hopefully the OutOfMemoryException will go away.

Provided you sufficiently increased the value of -Xmx parameter and the workload on the application remains the same, the OutOfMemoryException should permanently disappear. If you on the other hand you discover that the application still runs out of the memory and it just takes a little longer to reach that point than before, you might be dealing with a memory leak problem in your application. At this point you might be saying "hold your horses, I thought Java applications are memory-leak proof!". Well, yes that is correct from the point of view of the way memory leaks take place in C/C++ application, but the Java applications can still balloon in size (leak memory) through bad practices by allocating objects and holding on for too long no allowing for garbage collection. So memory leaks are still possible in Java applications, only in a different way as we used to with C/C++ and with mostly less disastrous consequences.

To make the long story short, if you want take just one message from the explanation above, remember just one thing -- if you run into an OutOfMemoryException emanating from Java, by all means don't rush into thinking that somehow Java ate all of memory on your system.

After Sun and Intel announced that they will be leaving some of their antagonism behind and actually help each other out -- with Sun now planning to sell Xeon-based systems and Intel helping Sun to optimize Solaris to run better on Xeon, to me it seems to have made an interesting precedent for a possibility of a similar deal with IBM. I think if Sun plays its cards right, there is a good chance of IBM agreeing to help Sun port the Solaris OS to Power platform. I see it as win-win deal for both parties -- Sun gets wider acceptance of Solaris and ability to run on yet another platform, IBM in this reciprocal relationship gets wider exposure and potentially higher volumes for its excellent Power line of CPU's that duly deserve more sales. What is even more interesting is that Solaris kernel has already been made to boot on Power -- a project by the name of Polaris made some good strides in making Solaris to run on IBM chip. If IBM is really serious about promoting Power as "Open" platform, they should definitely take the bait on this one.

Do you need to take a screen shot on a machine that does not have any fancy screen capture utilities installed? No Gimp and not even ImageMagick installed? Well, every X11 installation on Unix platform comes with a handy xwd utility that can take care of screen or window capture just as easy. For instance on Solaris all of X Window binaries are installed under /usr/openwin/bin directory, so just run xwd as follows:

/usr/openwin/bin/xwd > /tmp/screenshot.xwd

Point the cross hairs to the window you want take a screen shot of and you are done! Remember though that xwd utility captures the screen shots into its own xwd format, but the format is easily readable by most graphics packages (most notably Gimp and ImageMagick can do it).

If it it's green, it's biology
If it stinks, it's chemistry
If it has numbers, it's math
If it doesn't work, it's technology

Anonymous

If you read my previous blog entries you should know of my aversion to typing things more than once - all repetitive tasks should be automated. This applies not only to doing things at the command line, but also while editing documents. Fortunately vi (and Vim particular) lend themselves very well to automating things while performing the edits. On particularly useful feature on this front is the macros. The macros work by recording the actions using the qcharacter in order to start the recording into the register called character. You stop the recording the pressing the q character again. To give you a more hands-on example, consider the following use-case when I always employ macros - the header file inclusions in C/C++. For instance I've got the following list of header files that need to be included in the C source file:

stdlib.h
stdio.h
string.h
fcntl.h
errno.h
poll.h
sys/socket.h

Obviously all these header files need to be included as follows in the C source:

#include <stdlib.h>
#include <stdio.h>
.....

Which means repeatedlly typing the same "#include" directive and the angle brackets to wrap up each of the header file names. Of course with ViM there is a much more elegant solution to this problem:

1. Assuming you have ViM running, we'll start off with the clean list of header files above
2. Enter the qa command to start recording the macro into the register a
3. Position the cursor at the beginning of the first line
4. Insert the in "#include <" string before the "stdlib.h" by typing i#include <
5. Press Esc to change to the comand mode and jump to the end of the line by pressing $
6. Add the ending angle bracket by entering i>
7. Press Esc to change to the command mode
8. Jump to the beginning of the next line by pressing j and then 0
9. Stop recording the macro by entering q

Now with macro recorded and ready to go you can execute it any number of times in one go. So assuming I positioned the cursor at the beginning of "stdio.h" line, I can execute the macro on the remaining 6 lines by simply entering 6@a in the command mode and viola! - the macro will run on the remaining 6 lines adding the #include directives and wrapping the header files into angle brackets:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
#include <poll.h>
#include <sys/socket.h>

Sweet! I love ViM!

Enterprise applications on Unix nowadays equate to J2EE apps most of the time. I would be hard pressed to find any major enterprise application running on Unix, which wouldn't have at least part of it implemented in Java. And administering Java applications (and troubleshooting them) invariably entails dealing with class files, which are packaged in jars (Java Archives) most of time. And dealing with jar archives are both blessing and curse - blessing because you're not confronted with hundreds of class file each time you change into a directory and a curse because finding classes becomes a much harder task. In many cases, especially when the application starts throwing the ClassNotFoundExceptions you need to track down if a particular class is actually in the application's CLASSPATH, lack of which would generally cause the above mentioned ClassNotFoundException. And since the classes are all neatly packaged into jars, most of the time I would be staring at the bunch of jars without any idea where the class in question would be hidden. Well, with a bit of scripting magic this task actually becomes pretty trivial. For instance if I was troubleshooting a J2EE application that would throw the ClassNotFoundExceptions related DiskFile class, without the knowledge of application internals it would be hard to say why the class is missing from the CLASSPATH -- in the words which jar file containing the the class is not getting picked by the app. I could go uncompressing each jar in application's directory structure and then look for the DiskFile class in the expanded directories, but that would be wasting too much time. Instead a few lines of shell scripting will answer the question in no time:

bash-3.00$ for jar_file in `find . -type f -name "*.jar" -print`; do
> jar -tvf $jar_file | grep DiskFile > /dev/null 2>&1;
> if [ $? -eq 0 ]; then
> echo "DiskFile class is packaged in $jar_file";
> fi
> done
DiskFile class is packaged in ./server/lib/commons-fileupload-1.0.jar

Done! The DiskFile class is packaged in the commons-fileupload-1.0.jar archive. Now the next step is to check the configuration file and find out why the commons-fileupload-1.0.jar is not getting picked up by the JVM. Quite simple, and you don't have to be a Java guru to resolve this sort of problem.

I sorely miss the proc tools found on Solaris and Linux each time I have to deal with HP-UX - the tools pfiles, ptree, pkill, pgrep are just too valuable to go without. Here is just one hack I found that can effectively do the same function as ptree (well, sort of):

bash-3.00# UNIX95=1 ps -Hef

The output of all processes on the system will grouped by parent process id's with children processes nicely indented below their parents. Even though it is not even nearly as good as using a genuine ptree utility, since you may have to brose the longish list to find all the parents of the process you're interested in, it still has definite usefulness. Please remember not to export the UNIX95 variable since if exported this variable will effect the way other programs you may start from command line will behave after that point.

Here is a good question you can ask a potential candidate for a Solaris admin position during an interview -- what is the standard device name for a CD-ROM or DVD-ROM on Solaris? More than likely you will here "it depends" or something alone the lines of "/dev/dsk/c0t60d0" from those who spent too much time around older Sun systems. For some reason not too many admins seem to know that "/dev/sr0" link always points to the removable media device on a Solaris system. This fact comes particularly handy when you need to write a script than can reliably access the CDROM regardless of the type of machine it is run on. So if you didn't know this fact, create a mental note for /dev/sr0.

If you frequently add and remove devices on your Solaris machine, which especially frequent when you have Solaris running on your laptop and using a number of USB devices is a day-to-day reality. Big systems with a large number of LUN's attached also suffer from the same problem, when LUN's are moved or removed. As time goes you're may be left with an un-tidy device tree that is polluted with device links, which no longer used or exist. If you are a frequent visitor to your device tree simply because you need to manually identify device name while troubleshooting why some device doesn't work, then having a polluted device tree is simply cumbersome and confusing. To clean up the device tree simply run the devfsadm in the clean-up mode:

devfsadm -C

The devfsadm utility will run through all of the links created under /dev and will remote the un-needed ones.

On the subject of Trusted Extensions, there is nice little demo on the new Sun Security Learning Center site demonstrating how the Trusted Desktop works in Solaris for the uninitiated:

See the Trusted Java Desktop System - the world's first integrated multi-level GNOME-based GUI in action.

The basic point to be taken from this demo is how the Trusted Desktop separates access to the data between compartments/labels making it nearly impossible to "downgrade" (make it more publically available and thus less secure) any piece of data accessed and manipulated within this desktop environment. I'm sure this technolgoy will be quick to find home within the domain of three-letter agencies around the world.

Want to play around with the new fangled features Solaris Trusted Extentions? Here is a new document posted on the on the OpenSolaris web site that describes step-by-step how to install the Trusted Extensions software on the laptop computer:

Installing the Solaris 10 11/06 Trusted Extensions Software on a Laptop Computer

Here is an interesting bit of IBM AIX trivia I found in Wikipedia AIX entry. As it comes out large part of the AIX kernel was written in PL/I at least in the AIX v2 days. For a Unix operating system it is actually quite unusual, since overwhelmingly the Unix kernels are usually written in a mixture of C and Assembly. I wonder if the same PL/I bits are still there in AIX 5L 5.3. It would be really interesting to have a look at the source code of that beast, which makes me wonder if IBM will ever open-source the AIX kernel like Sun did with OpenSolaris. Hmmm...

Editing large files is always a painful experience especially if you have to refer two or more parts of the document at the same time. Most often I deal with these sorts of situations when I have to debug a problem with an application and study the source code. This is when it is nice to have the "function declaration" view and the "function implementation" view or "calling function" view and the "callee function view". You get the idea. Since I do most of my source code viewing and editing in vim, I had to find a way to do that the vim way. As it came out the solution was right in front of me and was as simple as entering "split" or "vsplit" in the command code. Doh! So anyway, if you want to edit you file in a split view being able to access two parts of the same file at the same time, just open your file and enter "split" in the command mode:


When you need to jump between frames to have active focus on one frame or the other, just press Ctrl+W followed by the up or down arrow depending on in which direction you want to switch the focus. Neat, eh?

But what if having documents side-by-side instead of stacked on top of each other is more like your cup of tea? Easy, just enter "vsplit" command instead of "split" in the command mode and you get two frames side-by-side:

If you want to switch between frames, just do a similar trick as before - Ctrl+W followed by the right or left arrow depending on the direction you want to switch focus to.

When you're done with having two frames in the view, just enter a "hide" command in the command mode to remove any one of the active frames.

Did I mention I love ViM?

Alone with a couple of high profile announcements made by Sun Microsystems yesterday regarding Niagara and Rock, Sun also mentioned a project code-named Neptune. It appears that Neptune is an ASIC that will enable the SPARC powered machines to support 10GB Ethernet. Apparently the Neptune will work in concert with with the multicore Niagara and Rock chips that will offload some network specific processing functions to this ASIC. This is actually quite exciting and should definitely generate quite a bit of interest from the network equipment vendors. The likes of Cisco and Nortel should definitely raise their eyebrows at this announcement. Both Niagara and Rock are throughput oriented processors, which are ideally suited for processing the network-related workloads (processing packets is probably as throughput oriented as you can possible get). The combination of throughput oriented chips and the network specific ASIC should make the Niagara/Rock+Neptune a killer combination as foundation for network gear. I'm expecting to see some Solaris/SPARC powered routers and load balancers pretty soon.

I'm not the greatest typist -- I make a lot of mistakes in the process, but typing is something I do a lot of being a Unix sysadmin and spending at least an entire working day interfacing with the command line. And being a bad typist can be a frustrating exercise especially if you have to deal with long pathnames in shell -- when you realize that you got it wrong you have to either bring back the previous command from history and correct the error or torture yourself with re-entering the entire path again. With former being much more preferable to the latter of course -- bring the command back from history is a good time-saver, but still requires some keystrokes to get the command and arguments back and then position the right position for editing. Fortunately there is an even faster way of correcting yourself bin bash shell -- the caret trick. All you have to do after you realized that the last command is fudged is to enter "^badstring^goodstring" at the command prompt and viola the last command line will be corrected by replacing the mistyped characters by the correct ones! As a hypothetical example lets assume that I want to check permissions on an Apache configuration file located under /etc/apache directory and named httpd.conf, but for reasons of absentmindedness I entered the file as httpd.config:

bash-3.00$ ls -alh /etc/apache/httpd.config
ls: /etc/apache/httpd.config: No such file or directory

Of course ls command comes back saying that the httpd.config file doesn't exist and you realize that the command line has to be corrected. All you do is this:

bash-3.00$ ^config^conf
ls -alh /etc/apache/httpd.conf
-rw-r--r-- 1 root root 47k Jan 20 13:47 /etc/apache/httpd.conf

Entering ^config^conf causes the shell to replace the "config" string with "conf" in the last command of the command history and invoke the command again. That's it! And no need to waste the valuable keystrokes to dig in history or god forbid re-enter the command again!

Unix is platform of choice for relational databases, so if you're a Unix admin regardless of particular flavor you're supporting, be it Solaris, AIX, HP-UX, or Tru64, there is a good chance that a lot of times you double as a DBA. Being a Unix DBA there are lot of times when you have to interface your shell scripts with the database, in other words quite frequently you need your scripts to capture the standard output from SQL statements issued against the database. If you're running Oracle and therefore use sqlplus, there are a few ways of doing this. Some use the sqlplus EXIT command to pass the output via a status code, some redirect the output to a temporary file and then read it into a variable. My favorite way of capturing the query output to shell variable is by assigning it directly into the variable with the use of some here text:

#!/usr/bin/bash
rowcount=`sqlplus -s user/password << EOF
SET PAGESIZE 0
SELECT COUNT(*) FROM mytable;
EXIT
EOF`

echo "The row count on mytable is $rowcount";

Did you ever run into such a situation on a Solaris machine when there is something wrong during boot and the system refuses to go any further then single-user mode and mounts the root file system read-only? Since most of the time the root cause of issue is some incorrect configuration somewhere on the root file system (usually issues with /etc/vfstab), the read-only mode used by the OS as a protection mechanism locks you out from the solving the problem on the spot. So the usual path to resolving the problem is to groan and boot Solaris from media (CD/DVD) into single user mode, mount the root slice and fix the stupid issue. If you have physical access to the system and/or have media conveniently placed in the media tray, then it isn't much of a problem, but if the system is located somewhere in the sticks and involves hours of driving (or flying) to get to this sort of thing can quickly turn into a career threatening experience. Well, fear not, there is a smarter way of resolving this issue. The trick is as simple as forcing the OS to remount the root slice in read/write mode without distirbing the /etc/mnttab, which is where the "-m" option comes in:

# mount -m -F ufs -o remount,rw /

As a result you get yourself read/write access to the root filesystem and you can go on your merry way of fixing the issue that brought the entire boot process to a screeching halt. Simple, yes, but this trick has a potential to save your reputation if the circumstances are right...

I bet a lot of times when you're working at the command line you run into situations when you type a lengthy argument for one command and then having to re-type the same argument again for the next command. It can become really annoying if the argument for the commands is some lengthy path and you tend to mistype things in the process causing a fair amount of frustration. For instance, this is how it would look like if you wanted to check for presence of a file with an imaginary but annoyingly long path name:

$ ls /opt/SUNWfoo/share/bar/baz/foobar.xml
/opt/SUNWfoo/share/bar/baz/foobar.xml
$ vi /opt/SUNWfoo/share/bar/baz/foobar.xml

So you have to type the same long path name twice, which is itself an offensive concept for pretty much any admin doing his work the Unix way. Of course as an alternative you can bring back the first command line from history and replace 'ls' with 'vi' and save yourself from re-typing that awful path, but that too would involve some extra keystrokes. Well, with bash shell we can reduce the number of keystrokes even further by using the "!$" shorthand. In bash the "!$" combination represents the last argument of the previous command line (which would be the awful path name of course). So using this trick we can revisit the scenario above as follows:

bash-3.00$ ls /opt/SUNWfoo/share/bar/baz/foobar.xml
/opt/SUNWfoo/share/bar/baz/foobar.xml
bash-3.00$ vi !$

Here, just two keystrokes saved a bit of typing and potentially quite a bit of frustration that could be accumulating over the work day...

Sun just rolled out release 11/06 of Solaris 10 out of the door. Even though the release was 3 months late, the wait was well worth it. There are several interesting features that 11/06, but a couple that truly deserver attention are the LDoms or Logical Domains in Sun speak and the Trusted Extensions.

The LDoms is a long awaited Hypervisor-based virtualization feature that is targeted squarely at the UltraSparc T1 CPU powered T1000 and T2000 machines in Sun's stable. What LDoms basically does is it allows to partition the server hardware (CPU, memory, IO, and storage) into virtual servers. The maximum number of the virtual servers that can be split off is limited to the number of hardware threads that the particular Niagara CPU is capable of, which is a maximum of 32. Contrasted with the Containers technologies, which has become more colloquially known as Solaris Zones (even though not quite correct), the LDoms allows each virtual server to run its own instance of an OS -- in other words you can have up to 32 independent instances of Solaris 10 running on the machine. Please note that I said Solaris 10, since older versions of Solaris will not be supported with Logical Domains. The reason is that the OS should be able to run on a newly fangled platform architecture type -- the sun4v -- and unfortunately the older Solaris releases are simply not capable of that. For the hardware to support the LDoms features, a new version 6.4 of firmware will be need, so if you already have a Sun Fire T1000 or T2000 servers in your datacenter, just upgrade the firmware and you should be in the LDoms virtualization business.

This Logical Domains feature can truly bring out both Sun Fire T1000/T2000 and Solaris 10 as truly cost-effective solutions to a large number of jobs out there. Having a comparatively low end server that is being capable of being virtualized into 32 independent server OS instances is no little feat and will open plenty of opportunities for consolidation, especially in the web tier. The Sun T1-based server with LDoms can easily spell doom for many and many x86 servers out there by consolidating and reducing the number of servers needed in the first place, rack space, power, and cooling. I have to say Niagara servers just became even stronger weapon in Sun's hands.

Another truly significant feature making a debut with 11/06 release is the Trusted Extensions. Trusted Extensions is Sun's attempt to bring Trusted Solaris and the mainstream Solaris OS into one release stream. As opposed to Trusted Solaris being a separately developed OS like in the old days, the Trusted Extensions is just an add-on product which is when installed on top of mainstream Solaris install transforms the OS into the Trusted version. Converging two products into one makes a lot of sense for Sun - instead of developing and maintaining two separate OS versions, Sun now has to develop and maintain just one while keeping all Trusted technology essentially as an add-on. This is both more effective in a sense of having shorter time-to-market and more efficient in terms of costs associated with R&D and product maintenance. Trusted Extensions bring over all of the features previously found only in Trusted Solaris - the MAC (Mandatory Access Control) approach to all aspects of the operating system with label security starting from filesystem and networking to desktop and printing. Sun is targeting to achieve EAL4+ certification across three protection profiles CAPP, RBACPP, and LSPP. Achieving EAL4+ with LSPP will truly set Solaris 10 apart from the competition, since it will the only mainstream OS capable of such a feat, making it essentially the most secure OS on the planet.

An organization in Europe called MERIT just released a lengthy report with a very lengthy name Economic impact of open source software on innovation and the competitiveness of the Information and Communication Technologies (ICT) sector in the EU. What I found particularly interesting about the report is the relative comparison of the biggest contributors to FLOSS (Free/Libre/Open Source Software). The table below outlines the contributions made by each of the top 10 vendors:

From the table above it appears that Sun Microsystems is the undisputed champion of contributing software to the Open Source movement committing more then 51 thousand person-months to the cause. What is even more interesting is how far behind Sun the "Linux champions" are -- IBM and RedHat. Sun dwarfs IBM's contributions by 3 1/2 times and RedHat's by more than 5 times in terms of person-months contributed! RedHat certainly screams the loudest about being the "champion" of Linux and Open Source in general, but the contributions most definitely don't measure up against the real champion - Sun Microsystems. I have to say, hats off to Sun and keep up the good work!

Once in a while you run into timestamps that are expressed as Unix Epoch time, which is basically a number of seconds that expired since 00:00 Jan 1, 1970 GMT. Unless you're some monstrous geek that make complex computations on the fly, I don't have to explain that dealing with timestamps expressed in Epoch format is not exactly intuitive. To perform the conversion quickly in the command line you can do the following perl one-line to extract the day, month, and year in the Epoch time stamp value piped to perl:

bash-3.00$ echo 1168472595 | perl -e 'use Time::localtime; $tm=localtime(<>); printf("%d/%d/%d\n", $tm->mday, $tm->mon+1, $tm->year+1900);'
11/1/2007

In the example above I'm interested in what the epoch second value of 1168472595 corresponds to in human readable form. The result in this examples comes out at Jan 11, 2007. If you need to extract hours, minutes, and seconds from the time stamp, it be done simply by modifying the printf statement statement to accommodate for $tm->hours for hours, $tm->min for minutes, and $tm->sec for seconds. Very simple and you can get the results without leaving the command line.

Just came across a very good guide to using bc containing some very neat hacks. Some of the really useful hacks that really caught my attention are the ones dealing with performing conversions between binary, hex, and decimal using bc. For instance converting between hex (FFF) and decimal can be done as simply as this:

bash-3.00$ echo 'ibase=16;obase=A;FFF' | bc
4095

Basically all you need to do is to indicate to bc what the input and output bases are -- the input base is 16 and the output base is 10 (A in hex). As a tip don't forget to capitalize the letters in the hex number representations, otherwise bc will complain with "syntax error on line 1, teletype" error.

Converting to hex from decimal is a similar type of affair:

bash-3.00$ echo 'ibase=A;obase=16;4095' | bc
FFF

Converting to binary can be done in the same fashion by setting the ibase or obase to 2:

bash-3.00$ echo 'ibase=A;obase=2;4095' | bc
111111111111

Read more about using bc in the following blog entry:

Command Line Calculations Using bc

All files installed via Solaris packages are registered in a text database called /var/sadm/install/contents. This database can be very handy at quickly locating any files that were installed on the system. Simply grep for the names of the files you're interested in against this database and chances are you'll get the answer you were looking for. For instance, if I'm interested if Expect is installed on a particular Solaris system, probably the quickest way to do it is:

# grep expect /var/sadm/install/contents
/opt/sfw/bin/autoexpect f none 0555 root bin 7638 40650 1089271963 SFWexpct
/opt/sfw/bin/expect f none 0555 root bin 192096 49442 1089271962 SFWexpct
/opt/sfw/bin/expectk f none 0555 root bin 196360 9915 1089271962 SFWexpct
/opt/sfw/include/expect.h f none 0444 root bin 12929 62958 1089271963 SFWexpct
/opt/sfw/include/expect_comm.h f none 0444 root bin 2275 50634 1089271963 SFWexpct
/opt/sfw/include/expect_tcl.h f none 0444 root bin 1786 20006 1089271963 SFWexpct
.........

Which will get me all file names installed on the system that contain string "expect" in them. Not very efficient, but certainly can get the job done. If you want to get a little better than that, just a tiny sprinkle of split and grep will do the trick:

# cat /var/sadm/install/contents | cut -d " " -f 1 | grep "expect$"
/opt/sfw/bin/autoexpect
/opt/sfw/bin/expect
/usr/sfw/share/zsh/4.2.1/functions/TCP/tcp_expect

Much better...

Need to add some line numbers to your file? Perhaps to make the source code more readable or for some other reason. Well, with perl it is as simple as this:

bash-3.00$ perl -pi -e '$_=sprintf "%3d %s", $., $_' myfile

Keep in mind that the modification will be done in place (well, that's the beauty of it), so don't forget to make a copy of the file if you want to keep the original version of the file.

Got an ISO file and need to access some files on it, but don't want to burn a copy? This problem can be easily solved on Solaris using lofi loopback device. All you need to do is the following:

1. Locate the your iso file and create a lofi device off of it using logiadm utility:

# lofiadm -a /myisofile.iso
/dev/lofi/1

2. Note the device name generated by the lofiadm command designating the iso file (in this case /dev/lofi/1).

3. Mount the lofi device as hsfs filesystem on your designated mount point and you're in business -- you can browse the iso as a filesystem:

# mkdir /mnt/lofi
# mount -F hsfs /dev/lofi/1 /mnt/lofi

4. When you're done you can unmount the filesystem and remove the lofi device:

# umount /mnt/lofi
# lofiadm -d /dev/lofi/1

Enjoy!

The task of making an ISO copy of a CD or DVD is quite trivial on Solaris and you don't have to have any extra utilities outsize of what is provided with Solaris OE. So assuming you already loaded your CD or DVD in the tray, here is a sequence of steps you need to follow:

1. Stop vold (Volume Management daemon for removable media) daemon:

# /etc/init.d/vold stop

2. Run dd command to copy the raw contents of the drive to a file:

# dd if=/dev/sr0 of=cdcopy.iso

Done! BTW, as a bit of Solaris trivia you should know that /dev/sr0 always designates a CD or DVD drive on Solaris systems.

When working on text documents we mostly perform selections in horizontal manner -- selecting series of subsequent words, lines, or paragraphs. But once in a while comes a situation when you need perform a vertical selection to perhaps remove a text column in a tabulated report or may be remove a series of characters at the beginning of the line representing comments (i.e "#" in shell and some scripting languages or "//" in C++ or Java). Well, in ViM this task can be easily accomplished. All you need to do is press a Ctrl-v combination, which switches the editor into a visual selection mode. After the Ctrl-v combination is pressed you can use your navigation keys and adjust the selection to your liking:

After the selection is performed, you can delete (d) or yank (y) the text to be pasted (p) somewhere else. Very neat and can save you quite a bit of time and frustration.

Wow, it looks like the newly released benchmarks on the new 1.4GHz Niagara T2000 quite handily beat the pants off the competition (running Solaris 10 of course):

• Sun Fire T2000 delivers 24% faster on SPECweb2005 than the 4-core HP DL380G5 with 3.0GHz Intel Xeon 5160 (WoodCrest), while consuming 2.5X less power, and upto 3x higher SWaP.
• The Sun Fire T2000 is 13% faster on SPECweb2005 performance than the 4-core Dell PowerEdge 2950 with 3.0GHz Intel Xeon 5160 (WoodCrest).
• The Sun Fire T2000 is 208% faster on SPECweb2005 than the 4-core IBM p5 550 with 1.9GHz POWER5+

It looks like the Sun T2000 is the fastest single socket web server money can buy! Great job Sun! More on the benchmark results read the following story:

NEW 1.4GHz Sun Fire T2000 World Record 1-socket SPECweb2005 & World Record Perf/Watt

Counter to all the Linux-is-cheaper-than-Unix hype, here is a very interesting perspective if there is any point or motivation that a move to Linux from Solaris can offer (well, clearly it doesn't):

Unix Remains Pretty Spry for 'Dead'

If you don't feel one hundred percent comfortable reading your man pages with "more" and/or "less" pagers, you're not alone. When reading the man pages I always catch myself trying to use the vi commands to scroll the pages or search for a particular sting only to discover that the pager is displaying the man page and the commands do not work. Well, with just a little bit of piping and the use of the "col" utility you can use ViM to read your good old man pages, just do it as follows and you'll be reading your man pages in the comfort of good old vi:

$ man man_page | col -b | vim -

I love ViM!

HP just announced that HP-UX 11i v3 is being prepared for launch. This is very good news indeed, but to me it brings both the feeling of joy and disappointment at the same time. Joy because obviously we're going to see some new features after almost a three-year drought during which HP didn't release anything worth a mention on HP-UX, so "yay!" on that. And the feeling of disappoints quickly follows the feeling of joy as soon as you realize how hopelessly HP-UX is lagging behind Solaris and even AIX. Apparently the HP management drones concentrated too much on their Wintel line of servers and completely forgot about the mission critical platform division. The mission critical platform division seems to have suffered only failures after HP and Compaq merged: Tru64 was completely erased even though it was a stellar OS, TruCluster that was supposed to get ported from Tru64 to HP-UX was mysteriously canceled, I lost the count of delays with 11i v3, and the list of failures goes on... It looks like HP-UX lost all edge it ever had -- about 5 years ago it was easy to say that HP-UX was the easiest to manage Unix OS that was very competitive with any other flavour of Unix out there. Now HP-UX is trailing the competition led by Solaris at the forefront and AIX slightly behind. And beating Solaris that blitzed the competition with such truly stellar technologies as Dtrace and ZFS will not be a trivial task. HP has been pretty shy with publicizing the upcoming features in 11i v3 and the folloing v4 and v5 releases, but I really hope HP picks up the ball they dropped on HP-UX before it's too late. HP really have to work hard to close the competitive gap with Sun and IBM. I really hope HP will not do the same with HP-UX as SGI has done with IRIX.

Anyway, apologies for the doom and gloom -- it is just disappointing to see something you like just withering away. Read more about the announcement on HP-UX 11i v3 on ITJungle website:

HP readies HP-UX 11i v3 for Launch

If you're mostly a Solaris admin (I'm anyway) and an odd RS/6000 machine appears in your shop, you don't have a whole lot places to go for the information on AIX. The amount of publicly available information on RS/6000 and AIX especially in the printed form is pretty scant, so you're not exactly spoilt for choice in this area. So if you found yourself in exactly this sort of situation and itching to get your hands on any document that would outline the differences in administration between Solaris and AIX to become productive as soon as possible, then IBM RedBooks is your friend. Here are two excellent guides I found on RedBooks specifically for Solaris administrators needing some knowledge of AIX and RS/6000:

AIX Reference for Sun Solaris System Administrators

and

Solaris to AIX Migration: A Guide for System Administrators

With all of the holiday activity I seem to have missed a very interesting product announcement by Sun Microsystems. It looks like a new version of Sun Cluster software was announced in the late December. Sun Cluster is now at version 3.2 and it looks like the list of new features added looks quite impressive indeed. The most prominent features that captured my attention are the following:

• Cluster support of SMF in Solaris 10 -- finally Sun Cluster will be able to take advantage of the SMF feature in Solaris 10
• Quorum Server -- no longer to you have to fight the SCSI reservation conflicts on your shared storage. You can now setup a quorum server and not rely on shared storage at all! I've been waiting for this feature for a long time.
• HA ZFS -- this is the first Sun Cluster release that can use ZFS in highly available configuration. Lack of this feature probably postponed many installations from adopting ZFS and now all stops are out!
• Live Upgrade is now supported with Sun Cluster. Before this release Live Upgrade wasn't supported on Sun Cluster installations, which I bet was a major grudge with many sysadmins. Another long awaited feature made it into the product.

Also as a bit of trivia you should know Sun Cluster features the largest number of cluster agents out of any clustering product on the market.

Sun Cluster 3.2 is available for download for free at the following URL:

http://www.sun.com/software/solaris/cluster/get.html

More information about Sun Cluster is here:

http://www.sun.com/software/solaris/cluster/index.xml

I think a side-effect of being a Unix sysadmin is the obsession with extracting more efficiency and saving time doing all admin-related tasks. Unix sysadmins are notorious for automating just about everything and making command lines as terse as possible to save some key strokes. Here is another little time saver I came across in bash shell. When you rename and copy files, how do you do it most of the time? More than likely if you change an extension on the file it looks something like this:

$ mv foobar.jpg foobar.jpeg

You just committed a mortal sin of typing "foobar" twice! Well, not quite a sin, but if this sort of behavior is repeated enough times over the course of your working day, you end up waisting a lot of time just re-typing the same things, especially if you have to deal with unwieldy long names. Small things add up, you know...

Well, a more elegant and sure as hell more efficient solution to this is to use the following expansion shorthand in bash (yes, that's another reason to switch to bash!):

bash-3.00$ cp foo.{jpg,jpeg}
bash-3.00$ ls foo.jpeg
foo.jpeg
bash-3.00$

You saved yourself some typing right there. The same thing can be done on any part of the string, so there is plenty of latitude in using this trick.

Here is a nice little guide to setting up a two node Solaris cluster using Sun Cluster 3.1 on www.sun.com. Pretty concise, but fairly comprehensive:

Two Node Cluster How-to Guide

Even though CDE is slowly fading away in its relevance there are still plenty of people out there are are willing to persevere with the unapologetic clumsiness of this desktop environment. Most choose to stick with CDE simply because of its relative "lightness" compared to the likes of GNOME and KDE, which are memory and CPU gluttons in comparison. Having a light window manager certainly helps if you have a very old workstation that may not be up to the task to run GNOME or KDE and you would be surprised how many sysadmins out there are using close to 10 year old workstations on their desktops (I'm not kidding). So there is no deficit of people still using CDE on their old Sun Ultra 1/5/10/30/60's or old HP B/C series workstations. Most people I've seen facing the CDE desktop almost immediately start itching for prettying up the desktop to fight the apparent and unapologetic dullness and clumsiness. Almost the very first question they ask is how do I change to a more appealing background from the dull bunch that ships with CDE. There many methods to do that, but the one that I favour the most is using ImageMagic and sessionetc file. So if you've got a workstation that has CDE and the ImageMagic is installed, you're in business. It is basically a three step operation:

1. Place your would-be wallpaper image in some accessible location. Since ImageMagic is capable of reading pretty much any image format imaginable, the file format can anything you want (jpeg, gif, png, you name it). The only word of caution is the heavier the image, the longer it will take for ImageMagic to load it and therefore the desktop session startup times will be slower.

2. Open Style Manager - Backdrop and choose "No Backdrop" like so:


3. Change directory to your sessions directory under .dt and create a sessionetc file with executable permissions that will use ImageMagic display utility to set the image into the root window:

bash-3.00$ cd $HOME/.dt/sessions
bash-3.00$ cat > sessionetc
/usr/sfw/bin/display -window root path_to_image_file
^D
bash-3.00$ chmod +x sessionetc

4. Restart your CDE session and you should have your custom background making the experience of using CDE a little more bearable. Enjoy!

I found a nice little guide to installing AIX, which seems like a very good reference for any AIX newbie. Check out this article on the developerWorks web site:

How to install AIX 5L

If you manage Solaris systems, chances are that you grew to love the proc tools shipping with the OS (pgrep, pfiles, pstack, pwdx, pmap, pldd, etc.). The success of the p-tools was so significant that the tools were later emulated on Linux and became a staple of Linux sysadmin's toolbox. With the release of AIX 5.2 IBM got its act together and also released a counterpart to the Solaris p-tools on AIX. Here is the table that outlines the differences and similarities between Solaris and AIX proctools:

Solaris 2.9	AIX 5.2
pflags [-r] [pid | core]	procflags [-r] [pid]
pcred [pid | core]	proccred [pid]
pmap [-rxIF] [pid | core]	procmap [-F] [pid]
pldd [-F] [pid | core]	procldd [-F] [pid]
pstack [-F] [pid | core]	procstack [-F] [pid]
pfiles [-F] [pid]	procfiles [-nF] [pid]
ptime command [argument]	Not available

To find more about AIX proc tools, please see the article on IBM developerWorks website:

AIX Performance Tools Update, Part 2

Once in a while you run into situations when you need to know what network ports some particular process has open. Most of the time you would need that to troubleshoot a condition when some process is occupying a port of interest to you and you need to quickly find out which process is that. Pretty much on all Unix platforms (Linux, Solaris, HP-UX, AIX, etc.) this can be easily accomplished with lsof utility as simply as:

# lsof -i :port

But lsof utility does not ship with the core distributions of the above mentioned OS's and is a third-party add-on, so in a lot of cases you come in contact with systems that don't have lsof installed. On Solaris on the other hand ships with a pfiles utility that can perfom a similar function, but only in a less elegant manner. But hey, it gets the job done! So for instance if on a Solaris host I want to find out what is the offending process occupying the port 389, I would use pfiles with the following bit of scripting magic:

# cd proc
# for i in `echo *`; do
> pfiles $i | grep AF_INET | grep "389$" > /dev/null 2>&1
> if [ $? -eq 0 ]; then
> echo $i;
> fi
> done
# 1638

Voila! The offending process id is 1638. The bits of scripting above just iterate over the list of all process ids on the system (listing the contents of the /proc directory) and use pfiles on each pid to grep out the port of interest.

Wow, Apple finally removed the covers off the iPhone ending the long standing speculations on how this product would look like and how it would function. The phone surely looks nice and has some very interesting features like a very big screen with a touch driven user interface, a full-blown Safari browser, WiFi connectivity, and whole lot more. I think a pretty big enabler for this technology is the Mac OS X, which forms the software foundation for this phone and brings a host of software and features already perfected on the desktop. Mac OS X is Unix, which means iPhone is the first Unix-based phone! Sweet!

The phone looks absolutely awesome and I think it will be as much of a success as iPod. I definitely want one. Good on you Apple!

You can find more info on Apple iPhone at the following location:

http://www.apple.com/iphone/technology/specs.html

Once in a while you come across a situation when you need to replace occurrences of some strings in a file with different strings. An old school solution that immediately comes to mind is to pipe the contents of the file through sed, store the new corrected contents in a temporary file, and then move the temporary file to the original file effectively replacing the old contents with new ones. For instance if we wanted to replace all occurrence of "foo" with "bar" in in our little file in one fell swoop, we would usually do this:

bash-3.00$ cat myfile
foo
bar
foo
baz
bash-3.00$ cat myfile | sed 's/foo/bar/' > myfile.tmp
bash-3.00$ cat myfile.tmp
bar
bar
bar
baz
bash-3.00$ mv myfile.tmp myfile

It is not a bad solution as it does the job well if you've got no more than a few files to deal with, but there is a better and more elegant way of doing the same thing that will save you some typing. If you have Perl installed on your machine (well who doesn't nowadays), you're in business. With Perl it is as simple as doing the following:

bash-3.00$ perl -pi -e 's/foo/bar/' myfile

Done! Perl effectively performs the replacement operation in-place, so you don't have to. Neat eh? And as an added bonus you can perform the operation on multiple files at the same time, just add additional filenames at the end of the command line. Bam, done!

If you've got a good number of systems under your control and the systems have seen a few rounds of hardware upgrades, chances are that you can't tell what the SIMM/DIMM layout is on each machine. And if you're planning an upgrade things may get a little tricky without resorting to shutting down the machines and popping them open. Well, Tom Schmidt made our lives a little easier with the memconf script that can do a marvelous job on the majority Solaris, HP-UX and Linux systems at identifying the SIMM/DIMM layout. For instance running it on my Sun Blade 1000 workstation, I get the following very detailed output that tells me how banks are occupied:

# /usr/local/bin/memconf -v
memconf: V1.57 15-Dec-2003 http://www.4schmidts.com/unix.html
hostname: blade
banner: Sun-Blade-1000 (UltraSPARC-III)
model: Sun-Blade-1000
Sun development name: Excalibur
Solaris 10 6/06 s10s_u2wos_09a SPARC, 64-bit kernel, SunOS 5.10
1 UltraSPARC-III cpu, cpu freq: 900MHz, system freq: 150MHz
CPU Units:
==================================== CPUs ====================================
E$ CPU CPU
CPU Freq Size Implementation Mask Status Location
--- -------- ---------- --------------------- ----- ------ --------
0 900 MHz 8MB SUNW,UltraSPARC-III 5.11 on-line +-board/cpu0
Memory Units:
Memory Segment Table:
-----------------------------------------------------------------------
Base Address Size Interleave Factor Contains
-----------------------------------------------------------------------
0x0 4GB 4 BankIDs 0,1,2,3
Bank Table:
-----------------------------------------------------------------
Physical Location
ID ControllerID GroupID Size DIMMs Interleave Way
-----------------------------------------------------------------
0 0 0 1GB 2x512MB 0
1 0 1 1GB 2x512MB 1
2 0 0 1GB 2x512MB 2
3 0 1 1GB 2x512MB 3
Memory Module Groups:
-------------------------------------------------------------
ControllerID GroupID Labels Status DIMMs
-------------------------------------------------------------
0 0 chassis/system-board/J0100 512MB
0 0 chassis/system-board/J0202 512MB
0 0 chassis/system-board/J0304 512MB
0 0 chassis/system-board/J0406 512MB
0 1 chassis/system-board/J0101 512MB
0 1 chassis/system-board/J0203 512MB
0 1 chassis/system-board/J0305 512MB
0 1 chassis/system-board/J0407 512MB
empty sockets: None
total memory = 4096MB (4GB)
=-=-=-=-=-=-=-=-=-=

You can grab the memconf utility for free at the following location:

http://myweb.cableone.net/4schmidts/memconf.html

Imagine the following situation, you've got a number of machines in the cluster or a web farm that are similarly configured and therefore many configuration steps or changes need to be carried out on each machine in pretty much exactly the same manner. What do you do? The simplest and perhaps most inefficient way is to do it each machine manually cursing your way through that you have to do the same thing multiple times. A smarter and much more efficient way is to use a tool like multixterm. Multixterm allows you to drive multiple terminals as if they were one -- effectively every single character you type in the master terminal gets reflected in multiple terminals. So you can login to multiple machines at the same time (provide the login passwords are the same of course), use any command utilities, and even edit files across multiple machines at the same time! What a time saver!

To give a bit more background on multixterm, it is a tool that is written entirely in Expect and ships with the expect software distribution. Now Expect is a language and a program that "talks" to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be. Many sysadmins are closely familiar with expect, but small and very effective utilities like multixterm, kibitz, xkibitz, passmass just stay out of the spotlight in relative obscurity.

So back to multixterm. If you have Expect installed on your *nix or Linux machine, you should already have multixterm handy. As an example of how multixterm can be invoked to access multiple machines via ssh, you can invoke it as follows:

bash-3.00$ multixterm -xc "ssh -l root %n" hostname1 hostname2 hostname3

Following the invokation of the command you will have two xterms representing the login terminals for each of the machines and a small multixterm window with a field highlighted in green and saying "stdin window". Any character you type in to the field will be send to all terminals:

If need to update ten machines, it can be accomplished in the same amount of time as just one! Bam, done!

Using nohup command to start a process that would ignore the SIGHUP signal is a daily reality for any sysadmin building scripts that should continue running even after the shell that started the process closes. You would generally do that your command/script as "nohup ". But what if you started and backgrounded a long running process only later to realize that you forgot to use the nohup command? Usually that would mean killing the process and restarting it with nohup, but on Solaris (>=9) you can simply mend this situation by issuing the following nohup command:

# nohup -p pid

And from that point on the target process identified by pid will happily ignore the SIGHUP signal and continue running even if the parent shell was terminated. Neat eh?

A lot of sysadmin newcomers to HP-UX are surprised to discover that HP-UX does not ship with any system call tracking utility a la truss on Solaris or strace on Linux out of the box. Debugging and root cause analysis can be quite a painful experience without this short of tools. Fortunately HP finally made a tool called tusc available to the HP-UX faithful, which is modeled after Solaris truss and does the job of tracing system calls on HP-UX extremely well. Perhaps the only drawback is that tusc is not officially supported by HP and is provided as-is. The usage for tusc is as follows:

tusc [options] command [args ...] | pid [pid ...]

and if run against a pid of a process in question it give a very nice trace of all system calls invoked by the process under the observation as in this example with running tusc against a Sun Directory Server process:

# tusc 1609
( Attached to process 1609 ("./ns-slapd -D /opt/sun/DS5.2p4/slapd-foo -i /opt/sun/DS5.2p4/slapd-foo") [64-bit] )
select(16, 0x800003ffff7f0f38, NULL, NULL, 0x800003ffff7f6740) [sleeping]
select(0, NULL, NULL, NULL, 0x800003ffff4852c0) .......... [sleeping]
select(0, NULL, NULL, NULL, 0x800003ffff444310) .......... [sleeping]
ksleep(PTH_CONDVAR_OBJECT, 0x80000001000d0ae0, 0x80000001000d0ae8, 0x800003ffff4033e8) [sleeping]
select(0, NULL, NULL, NULL, 0x800003ffff3c22d0) .......... [sleeping]
select(0, NULL, NULL, NULL, 0x800003ffff381690) .......... [sleeping]
select(0, NULL, NULL, NULL, 0x800003ffff340310) .......... [sleeping]
......

You can download tusc shell archive from the following location on the HP ftp site:

ftp://ftp.hp.com/pub/catia/Utils/tusc7.4.shar

or alternatively from the Utah University Porting and Archiving Center for HP-UX (great site for HP-UX tools btw):

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/tusc-7.8/

For a bit more in-depth information on tusc, please refer to the following HP doc.

A neat little feature I never new existed in bash is being able to iterate over a sequence of number in a more or less C-esque manner. Coming from Bourne/Korn shell background creating an elegant iterator is always a slight nuisance, since you would come up with something like this to iterate over a sequence of numbers:

i=1;
while [ $i -lt 10 ]; do
i=`expr $i + 1`;
.....
done

Well, not exactly the most elegant solution. With bash on the other hand it can be done as simple as:

for((i=1; $i<10; i++)); do
....
done

Simple and to the point.

I guess I'm a bit of a late adopter when it comes to software -- I tend to switch to the newer versions of software when the dust settles a little bit around the new release. The same thing applies to one of the most frequently used tools in my Unix toolbox -- the ViM editor. I've happily plugging alone with version 6 of the editor, even though version 7 has been released about half a year ago. So after a quick download and compile I have ViM 7 at my disposal. The feature that immediately struck the cord is the tabs -- you can now have multiple documents open in the same editor have the ability to switch between them simply clicking on a different tab:


Openning a new tab is as simple as either choosing File->Open Tab... or just entering the :tabnew command in the command mode. ViM rules! vi has been my long standing favourite for years and ViM 7 just strengthens the case.

Download ViM 7 from http://www.vim.org/download.php

An interesting tidbit of grep trivia some of you out there in the Unix trenches might appreciate -- working on a script today I was bashing my head trying to work out an elegant solution how to postprocess a line that *preceded* the lines that matched the search pattern in the text file. Obviously you would have to store/cache each preceding line in the script to accomplish that, which destroyed the simplicity I was looking for and resulted in a rather clumsy solution. As I was using the xpg4 version of grep shipping with Solaris all the time I was pretty oblivious to the fact that the solution to this problem could be as simple as just using GNU grep instead! Lo and behold GNU grep (ggrep as it is called on Solaris) can perform context greping and output the lines immediately before the matching line, all you have to do is to use -B option to do that. Here is a snip from ggrep man page in relation to the context options:

-A NUM, --after-context=NUM
Print NUM lines of trailing context after matching
lines. Places a line containing -- between contiguous
groups of matches.

-B NUM, --before-context=NUM
Print NUM lines of leading context before matching
lines. Places a line containing -- between contiguous
groups of matches.

-C NUM, --context=NUM
Print NUM lines of output context. Places a line con-
taining -- between contiguous groups of matches.

In other words as a rather trivial example if I wanted to grep out the line immediately before the user uucp in the /etc/passwd file, I could do the following:

bash-3.00$ ggrep -B 1 "^uucp" /etc/passwd | grep -v "uucp"
lp:x:71:8:Line Printer Admin:/usr/spool/lp:

Well, live and learn...

Well, here it goes... The mandatory 'Hello World!' post that is a hallmark of pretty much any blog site out there. I'm a Unix systems engineer/administrator working the most gorgeous of all cities - Sydney, Australia (well, in my view anyway). I have fondness for for all things *nix out there and this is what I have decided to dedicate this blog to -- the Unixes of the world. Even though I'm rather closely associated with Sun Microsystems, I have no lost love for pretty much any other flavour of Unix there is in existance -- I have just as much fondness for AIX, HP-UX, Linux, IRIX and many others as for my good old friend Solaris. So I'm going to try covering as much of Unix space as possible in this blog with no bias for any one vendor.