Regular maintenance report

And here we go with the second installment of the systems maintenance report. While I can list a few excuses (vacation, general busy-ness both pre- and post-vacation, inability to patch some systems because of production schedules), I have to admit that it is not easy to keep a weekly schedule of maintenance for every server! I'll try to do better in the future, although I may move this article to a bi-weekly schedule.

I’m trying a cleaner, bullet list format this time out.

Windows 2003R2 server (domain controller)

• Updates:
• Eventlog analysis:
• Disk space:
• Portscan:
• Antimalware:
• Other:

CentOS 4 (application server)

• Updates: 199 updates (244M to download) were called for! It's been about 10 weeks since this system was last updated, but, uhm, golly. Gee. That's a lot of updates. This just shows the importance of keeping up on regular maintenance! Here is the monstrous list of updated items:
  • Installed: kernel.i686 0:2.6.9-55.EL kernel-smp.i686 0:2.6.9-55.EL openib.i386 0:1.1-7
    Dependency Installed: tcl.i386 0:8.4.7-2
    Updated: NetworkManager.i386 0:0.3.1-4.el4 NetworkManager-gnome.i386 0:0.3.1-4.el4 OpenIPMI.i386 0:1.4.14-1.4E.17 OpenIPMI-libs.i386 0:1.4.14-1.4E.17 OpenIPMI-tools.i386 0:1.4.14-1.4E.17 SysVinit.i386 0:2.85-34.4 a2ps.i386 0:4.13b-41.6 acl.i386 0:2.2.23-5.3.el4 alsa-utils.i386 0:1.0.6-6 aspell.i386 12:0.50.5-4.EL4 attr.i386 0:2.4.16-3.1.el4 audit.i386 0:1.0.15-3.EL4 audit-libs.i386 0:1.0.15-3.EL4 autofs.i386 1:4.1.3-199.3 binutils.i386 0:2.15.92.0.2-22 bluez-utils.i386 0:2.10-2.2 centos-release.i386 6:4-4.3 chkconfig.i386 0:1.3.13.5.EL4-1 comps.i386 2:4.5CENTOS-0.20070506 coreutils.i386 0:5.2.1-31.6 cpio.i386 0:2.5-13.RHEL4 cpp.i386 0:3.4.6-8 cracklib.i386 0:2.8.9-1.3 cracklib-dicts.i386 0:2.8.9-1.3 crash.i386 0:4.0-3.9 cups.i386 1:1.1.22-0.rc1.9.20 cups-libs.i386 1:1.1.22-0.rc1.9.20 curl.i386 0:7.12.1-11.el4 dapl.i386 0:1.2.1-7 dbus.i386 0:0.22-12.EL.9 dbus-glib.i386 0:0.22-12.EL.9 dbus-python.i386 0:0.22-12.EL.9 dbus-x11.i386 0:0.22-12.EL.9 desktop-file-utils.i386 0:0.9-3.el4 device-mapper.i386 0:1.02.17-3.0.1.el4 dhclient.i386 7:3.0.1-59.EL4 dhcpv6_client.i386 0:0.10-17_EL4 diskdumputils.i386 0:1.3.25-1 dmraid.i386 0:1.0.0.rc14-5_RHEL4_U5 dosfstools.i386 0:2.8-18 e2fsprogs.i386 0:1.35-12.5.el4 elfutils.i386 0:0.97.1-4 elfutils-libelf.i386 0:0.97.1-4 enscript.i386 0:1.6.1-33.el4 evolution-data-server.i386 0:1.0.2-14.el4 file.i386 0:4.10-3.0.2.el4 findutils.i386 1:4.1.20-7.el4.3 firefox.i386 0:1.5.0.12-0.1.el4.centos firstboot.noarch 0:1.3.39-6.centos4 freetype.i386 0:2.1.9-5.el4 ftp://ftp.i386/ 0:0.17-23.EL4 gamin.i386 0:0.1.7-1.2.EL4 gcc.i386 0:3.4.6-8 gd.i386 0:2.0.28-5.4E gdm.i386 1:2.6.0.5-7.rhel4.15 glibc.i686 0:2.3.4-2.36 glibc-common.i386 0:2.3.4-2.36 glibc-devel.i386 0:2.3.4-2.36 glibc-headers.i386 0:2.3.4-2.36 glibc-kernheaders.i386 0:2.4-9.1.100.EL gnome-applets.i386 1:2.8.0-8.el4 gnome-panel.i386 0:2.8.1-5.el4 grep.i386 0:2.5.1-32.3 grub.i386 0:0.95-3.8 gtk+.i386 1:1.2.10-36 hal.i386 0:0.4.2-6.EL4 hotplug.i386 3:2004_04_01-7.8 httpd.i386 0:2.0.52-32.ent.centos4 httpd-manual.i386 0:2.0.52-32.ent.centos4 httpd-suexec.i386 0:2.0.52-32.ent.centos4 hwdata.noarch 0:0.146.28.EL-1 initscripts.i386 0:7.93.29.EL-1.centos4 iproute.i386 0:2.6.9-3.EL4.7 iputils.i386 0:20020927-19.EL4.5 jpackage-utils.noarch 0:1.7.3-1jpp.1.el4 kbd.i386 0:1.12-2.el4.4 kernel-smp-devel.i686 0:2.6.9-55.EL kernel-utils.i386 1:2.4-13.1.99 krb5-libs.i386 0:1.3.4-47 krb5-workstation.i386 0:1.3.4-47 kudzu.i386 0:1.1.95.22-1 less.i386 0:382-4.rhel4 libIDL.i386 0:0.8.4-1.centos4 libacl.i386 0:2.2.23-5.3.el4 libattr.i386 0:2.4.16-3.1.el4 libgcc.i386 0:3.4.6-8 libibverbs.i386 0:1.0.4-7 libmthca.i386 0:1.0.3.1-7 libpng.i386 2:1.2.7-3.el4 libpng10.i386 0:1.0.16-3 librdmacm.i386 0:0.9.1-7 libsdp.i386 1:1.1.0-7 libselinux.i386 0:1.19.1-7.3 libstdc++.i386 0:3.4.6-8 libxml2.i386 0:2.6.16-10 libxml2-python.i386 0:2.6.16-10 linuxwacom.i386 0:0.7.4_3-EL4.5 logrotate.i386 0:3.7.1-6.RHEL4 logwatch.noarch 0:5.2.2-2.EL4 lvm2.i386 0:2.02.21-5.el4 mailx.i386 0:8.1.1-37.EL4 man.i386 0:1.5o1-10.rhel4 man-pages.noarch 0:1.67-12.EL4 mdadm.i386 0:1.12.0-2 metacity.i386 0:2.8.6-2.11 mkinitrd.i386 0:4.2.1.10-1.1 mod_ssl.i386 1:2.0.52-32.ent.centos4 module-init-tools.i386 0:3.1-0.pre5.3.4 net-snmp-libs.i386 0:5.1.2-11.EL4.10 net-tools.i386 0:1.60-37.EL4.9 netdump.i386 0:0.7.16-10 nfs-utils.i386 0:1.0.6-80.EL4 nfs-utils-lib.i386 0:1.0.6-8 nscd.i386 0:2.3.4-2.36 nss_ldap.i386 0:226-18 ntp.i386 0:4.2.0.a.20040617-6.el4 ntsysv.i386 0:1.3.13.5.EL4-1 numactl.i386 0:0.6.4-1.39 openldap.i386 0:2.2.13-7.4E openldap-clients.i386 0:2.2.13-7.4E openssh.i386 0:3.9p1-8.RHEL4.20 openssh-askpass.i386 0:3.9p1-8.RHEL4.20 openssh-askpass-gnome.i386 0:3.9p1-8.RHEL4.20 openssh-clients.i386 0:3.9p1-8.RHEL4.20 openssh-server.i386 0:3.9p1-8.RHEL4.20 openssl.i686 0:0.9.7a-43.16 pam.i386 0:0.77-66.21 pam_ccreds.i386 0:3-3.rhel4.2 parted.i386 0:1.6.19-16.EL pciutils.i386 0:2.1.99.test8-3.4 php.i386 0:4.3.9-3.22.5 php-ldap.i386 0:4.3.9-3.22.5 php-pear.i386 0:4.3.9-3.22.5 pilot-link.i386 1:0.11.8-9.RHEL4 policycoreutils.i386 0:1.18.1-4.12 popt.i386 0:1.9.1-22_nonptl procps.i386 0:3.2.3-8.6 psacct.i386 0:6.3.2-39.rhel4 python.i386 0:2.3.4-14.4 python-elementtree.i386 0:1.2.6-5.el4.centos python-sqlite.i386 0:1.1.7-1.2.1 quota.i386 1:3.12-6.el4 rdist.i386 1:6.1.5-38.40.2 rhnlib.noarch 0:2.1.1-3.el4 rhpl.i386 0:0.148.5-1 rpm.i386 0:4.3.3-22_nonptl rpm-libs.i386 0:4.3.3-22_nonptl rpm-python.i386 0:4.3.3-22_nonptl rpmdb-CentOS.i386 2:4.5-0.20070506 samba.i386 0:3.0.10-1.4E.12.2 samba-client.i386 0:3.0.10-1.4E.12.2 samba-common.i386 0:3.0.10-1.4E.12.2 sed.i386 0:4.1.2-6.el4 selinux-policy-targeted.noarch 0:1.17.30-2.145 sendmail.i386 0:8.13.1-3.2.el4 setools.i386 0:2.3-4 shadow-utils.i386 2:4.0.3-61.RHEL4 shared-mime-info.i386 0:0.15-10.1.el4 sqlite.i386 0:3.3.6-2 squid.i386 7:2.5.STABLE14-1.4E system-config-language.noarch 0:1.1.8-4 system-config-lvm.noarch 0:1.0.23-1.0 system-config-printer.i386 0:0.6.116.10-1 system-config-printer-gui.i386 0:0.6.116.10-1 system-config-samba.noarch 0:1.2.21-1.el4.1 system-config-users.noarch 0:1.2.27-0.EL4.4 tcsh.i386 0:6.13-9.el4.1 traceroute.i386 0:1.4a12-24.EL4.1 ttmkfdir.i386 0:3.0.9-20.el4 unzip.i386 0:5.51-9.EL4.5 up2date.i386 0:4.5.5-6.el4.centos up2date-gnome.i386 0:4.5.5-6.el4.centos usbutils.i386 0:0.11-7.RHEL4.1 usermode.i386 0:1.74-2 usermode-gtk.i386 0:1.74-2 util-linux.i386 0:2.12a-16.EL4.25 vixie-cron.i386 4:4.1-47.EL4 vnc-server.i386 0:4.0-11.el4 vte.i386 0:0.11.11-11.el4 words.noarch 0:3.0-3.2 xorg-x11.i386 0:6.8.2-1.EL.18 xorg-x11-Mesa-libGL.i386 0:6.8.2-1.EL.18 xorg-x11-Mesa-libGLU.i386 0:6.8.2-1.EL.18 xorg-x11-deprecated-libs.i386 0:6.8.2-1.EL.18 xorg-x11-font-utils.i386 0:6.8.2-1.EL.18 xorg-x11-libs.i386 0:6.8.2-1.EL.18 xorg-x11-tools.i386 0:6.8.2-1.EL.18 xorg-x11-twm.i386 0:6.8.2-1.EL.18 xorg-x11-xauth.i386 0:6.8.2-1.EL.18 xorg-x11-xfs.i386 0:6.8.2-1.EL.18 xscreensaver.i386 1:4.18-5.rhel4.14 xterm.i386 0:192-7.el4 ypbind.i386 3:1.17.2-13 yum.noarch 0:2.4.3-3.el4.centos zsh.i386 0:4.2.0-4.EL.4.5
    Replaced: kernel-ib.i386 0:1.0-1
  • After the update and reboot were complete, I changed /etc/motd to read: CentOS 4.4, all updates as of June 10, 2007
  • Since there were kernel updates again, I had to run the /usr/bin/vmware-config.pl script again, as mentioned before.
  • No problems were observed.
• Eventlog analysis: a quick look over all logs using egrep -iR error\|fail /var/log |less produced, well, a lot of errors. But most were relevant to Vmware trying to start on a kernel it had not been configured for. Also, this method brings up all errors in all logs. Which means that on a regular maintenance cycle, you end up seeing the same old events again and again. I need to work out a better mechanism to either sort by date, or archive logs I have already looked at.
• Disk space: plenty of disk free (via df -h)
• Portscan: netstat -l --numeric-ports showed 12 listening ports, all of which I am aware of and comfortable with.
• Antimalware: NA
• Other: nothing to report

Windows Small Business Server

• Updates: 1 update, KB926505, did not need reboot. (Note that this system has been receiving more regular maintenance than some of the others on today's list).
• Eventlog analysis: There were several minor items I repaired. These had actually been recurring in the logs (at each reboot) for some time; as this was a slow week, I took the time to find and apply fixes.
  • (event ID, source or filename listed; fix applied)
  • 2003, TermService; KB932813 resolved the issue
  • 1016, ifoctrs.dll; KB288077 resolved the issue
  • 1016, EXOLEDB; (no fix found; apparently one of the other fixes did resolve the issue though)
  • 2003, aspperf.dll; resolved by the 288077 fix above
  • 4314, WINSCTRS; uninstall WINS service (which is disabled anyhow)
  • 32026 and 32068, MS Fax. No fax devices actually installed, so I disabled the Fax service.
  • On a test reboot, none of the above events recurred in the log. Yay for cleaner eventlogs!
  • Finally, at reboot the following services complain about AD being unavailable: dsrestor, MSExchangeDSAccess, and MSExchangeAL. This is basically because AD is not fully up and accessible to these services during the first minute or so of bootup; it can be safely ignored.
• Disk space: plenty of space on all disks
• Portscan: not done
• Antimalware: (Norton) All log histories clean, and signatures are dated today.
• Backup: all backups succeeded.
• Other: nothing to report.

All told, I spent an extra 45 minutes cleaning up eventlogs on this system, but this is a good excercise to do when you can. The more you can cleanup 'ordinary errors', the more the extraordinary errors will stand out and grab your attention (as they should).

Windows XP (various systems actually)

• Updates:
• Eventlog analysis:
• Disk space:
• Portscan:
• Antimalware:
• Other:

Ubuntu 6.06LTS (desktop system)

• Updates: Again, 10 weeks since last update, and there are 55 updates to do. Dog my cats! I'll list them here, since apt-get uses a more dense paragraph form. The updates completed without any notable incidents, but Ubuntu did require a reboot after these updates were complete.
  • app-install-data-commercial evolution evolution-data-server evolution-plugins firefox firefox-gnome-support gimp
      gimp-data gimp-python libaudio2 libcamel1.2-8 libebook1.2-5 libecal1.2-3 libedata-book1.2-2 libedata-cal1.2-1
      libedataserver1.2-7 libedataserverui1.2-6 libegroupwise1.2-9 libexchange-storage1.2-1 libfreetype6 libgimp2.0
      libkrb53 libnspr4 libnss3 libpq4 libsmbclient libsnmp-base libsnmp9 libx11-6 libxfont1 linux-headers-2.6.15-28
      linux-headers-2.6.15-28-386 linux-image-2.6.15-28-386 locales openoffice.org openoffice.org-base
      openoffice.org-calc openoffice.org-common openoffice.org-core openoffice.org-draw openoffice.org-evolution
      openoffice.org-gnome openoffice.org-gtk openoffice.org-impress openoffice.org-java-common
      openoffice.org-l10n-en-us openoffice.org-math openoffice.org-writer python-uno rdesktop samba samba-common
      smbclient ttf-opensymbol xserver-xorg-core
• Eventlog analysis:
• Disk space:
• Portscan:
• Antimalware:
• Other:

Windows Vista (my own main workstation)

• Updates: 8. (Note this system has been updated at least twice since last report; I was just too lazy to document!)
• KB905866 Windows Mail Junk filter
• KB929123 Windows Mail Cumulative Update
• KB931213 Vista Security Update (permissive ACLS on user info stores)
• KB890830 Malicious Software Removal Tool
• KB933566 IE7 Cumulative Update
• KB936825 Vista Update (issues with SD)
• updates for Office and Outlook as well (1 each)
• Eventlog analysis: IE7 has failed twice in the last week. I suspect that this is because of web pages with a lot of client-side scripting in them, but I can't really be sure. Hi-ho.
• Disk space: plenty available
• Portscan: netstat -a|findstr LISTENING reports 20 listening ports, but this is misleading. Really there are ten listening ports, duplicated on both the IPv4 and IPv6 interfaces. I discovered a few ports I wasn't familiar with. Sadly, TCPView 2.4 (the latest) works on Vista, but cannot get process properties the way it can on earlier versions of Windows. I tried netstat -ab  but, even in a cmd session running with Admin privs, was rewarded with the message: The requested operation requires elevation. So I tried using 'Switch User' to login as local Administrator, and then netstat -ab worked ... mostly. It was unable to get owner info for the ports, but I learned that they were mapped to:
  • 24800 - Synergy
  • 49152 - wininit.exe
  • 49153 - Eventlog
  • 49154 - nsi (Network Store Interface Service)
  • 49155 - Schedule
  • 49156 - lsass.exe
  • 49182 - services.exe
• Antimalware: none installed. As you may recall, in my last maintenance session, I tried a couple of web scanners. Time to try again!
  • safety.live.com - I clicked on 'Protection'. The website informed me that I'd be using a beta full-service scan for Vista. I had to allow the install of a local component, which forced a UAC screenpop. The install completed in a minute or so, then I pressed a 'Launch' button that caused a popup window in which I could choose options. I chose 'Complete Scan'. However, remembering the follies from last time, I did use the 'Customize' option to make sure that only my C: drive would be scanned. Got another UAC screenpop. A 'downloading tools' session followed, for about xx minutes during which I did have an active progress bar. The download took a little over 7 minutes, but this time the scan did begin. I didn't time the scan itself - I walked away. When I checked back about 3 hours later, the scan had completed. The utility also offered to clean some extra registry items and remove some unneeded temp files; I allowed it to do so. End result: 10 weeks ago it failed, but now it works well.
  • housecall.antivirus.com -
• Other:

Windows 2000 (application server)

• Updates:
• Eventlog analysis:
• Disk space:
• Portscan:
• Antimalware:
• Other: