Who knew it was so easy?  After Dreamhost recently started dumping some system-profiling logs into my home directory, in order to show me how many CPU minutes my processes were using per day, I realized that I’d like to be able to do this myself on my other servers.  It turns out that it’s really easy to do.

A little googling brought me to the sa(8) man page.  After reading about sa and its friends in the "See Also" section of the manual, I understood what to do.

First you need to install the GNU Accounting Utils, probably from a package named acct.  On Debian for example, I just did "aptitude install acct"; on Gentoo, "emerge -vabk acct" did the trick.  This installed the accton and sa programs (among others) on my system.

Now we’re ready to go.  Run "accton /var/log/acct" and it will log all the processes that run on the system from now on.  (Yes, the log grows pretty quickly, so watch it.  Run "accton" with no arguments to stop logging.)

The accton command produces a binary logfile to help keep the size down, and the sa command reads this binary file to produce human-readable output.  We’ll run "sa -u /var/log/acct" to produce a list of all the processes that have run, along with the amount of CPU time they consumed.

Finally, the good part: the handy sa-analysis.pl script reads the huge list produced by sa, and then displays a nice short list of each process that has run, along with its cumulative CPU usage.  Save the script as /usr/local/bin/sa-analysis and chmod it a+x, then run this command:

sa -u /var/log/acct >/var/log/acctlist && sa-analysis /var/log/acctlist

You should see a nice table like this:

Process         CPU seconds      user   machine   count  average
foo.cgi            395.7600   32.272%    1.649%     511    0.774
random.cgi         327.3200   26.691%    1.364%    2052    0.160
mysqld             145.8200   11.891%    0.608%     231    0.631
someother.cgi      115.9300    9.453%    0.483%     333    0.348
foodb.cgi           80.7400    6.584%    0.336%     231    0.350
iamwinning.cgi      61.3500    5.003%    0.256%     270    0.227
courierpop3d        14.7000    1.199%    0.061%      32    0.459
httpd               11.7800    0.961%    0.049%      95    0.124
rdjpgcom            11.6000    0.946%    0.048%     500    0.023
mrtg                 9.8600    0.804%    0.041%       8    1.233
[...]
accton               0.0000    0.000%    0.000%       1    0.000
locale               0.0000    0.000%    0.000%       2    0.000
less                 0.0000    0.000%    0.000%       1    0.000
mv                   0.0000    0.000%    0.000%       1    0.000
qmail-local          0.0000    0.000%    0.000%       5    0.000
sh                   0.0000    0.000%    0.000%       2    0.000
----------------------------------------------------------------
Total:            1318.8300  100.000%    5.495%    7325
Average per day:  1318.8300    1 days
CPU percentage assumes 24000 cpu seconds per day total.

And presto!  You now know which processes on your system are eating all your CPU time.  Of course, the GNU Accounting Utils are very flexible and can produce many other kinds of reports, so dig into the man pages to find out more.