30. April -- Samstag
On Call
Clean up a deadlocked imapd/procmail problem.
On Call
Clean up a deadlocked imapd/procmail problem.
Power and Cooling: meetings and discussions
credit card: talk with John, perl modules required, found modules -- installed RPM perl-libwww-perl
Giraffe: check on disk from Ed.
Tru: space... how much to add a drive... well, there isn't room for another drive, so $8K to $16K ...
moose: needed to move a ugrad to get rack1c back below 97% full
On Call
General stuff: permissions and group membership on giraffe
Calendar: Another person with offline calendar/address book questions
Recruitment: reference chasing
Space: rack2h overfull again! Moved a ugrad who has placed a lot of research data in their account in the past 48 hours to another filesystem. And then rack1e decided it wanted in on the party and I moved another ugrad as I didn't find anyone who appeared to be abusing the system.
On Call
Banner: Oracle database problem. JAS/MJG working on it. No page was received.
sendpage: it hung -- modem needed a power cycle -- at 16:30 yesterday afternoon. Discovered and cleared (cycle everything) at 8:30 this morning (OUCH!). Mike had written a first pass at a script to check this last fall. Started running the script. Need to tweak it for better checking later. Modified it to be more pager service friendly -- send a single page with the details and only page the people who are actually on call.
Mink: httpd was not being automatically started, so John & Sharon couldn't add people to the calendar server.... I really have problems with key system services not starting properly when a machine is booted.
LDAP: Set up a netid as a consultant while the AOE paperwork is processed.
Space: Moved an undergraduate account between filesystems to free up space.
On Call
Environmentals: Meeting with Sal and associates to discuss the problems with head and power in the Waterman machine room. Recommendations: CIT to fund environmental study of how the room should be laid out and planning for growth, Sal to find 15 Tons of temporary cooling.
WebCT: load Solaris 10 on cottonmouth for experimentation
SecurID: Continue updating/configuring cobra for experimentation
General: group membership updates on giraffe for Nancy Beck; space concerns on tru for Barb Hogel (/u06 99% full).
DNS: Added names for HP 3500 printers and removed citink.
On Call
LDAP: porcupine hung -- slapd_db_checkpoint and slapd deadlocked (apparently) -- and caused monitoring to backup. Thank goodness the loadbalancers weren't fooled and pulled it out of the ldap.uvm.edu list. Resolved by kill -9'ing both processes and reloading the database from wolverine.
LDAP: added checks that saslauthd continues to work on the replica servers. Discovered that it was not working on porcupine at about 8:30 and started it back up, but the checks were not running so it was simple dumb luck that I happened to find it.
Recruitment: Final interview... never had an interviewee declare themselves unqualified and withdraw their application during an interview before.
SecurID: Install Solaris 9 on COBRA to experiment with version 6 of the server.
Installed RedHat Kernel security fix to seven systems: Calendar servers, LDAP servers, and Footprints.
Two hours...
LDAP: The SleepyCat DB backend database ran out of locks which caused the application of the necessary modification commands to fail. Mike paged me and I was able to determine that others have seen this and so increased the number of available locks from 1000 to 2000 and cycled all the servers. I applied the modifications and then proceeded to run the remaining pieces of the nightly update (and the Active Directory feed) by hand.
WebCT: diamondback monitoring re-enabled
General: bash login problem debugged for JSR.
LDAP/Calendar: Install OL 2.2.24 on trout
eMail: watch IBM deal with the ds6800
Recruitment: Conduct two interviews.
Fibre Fabric: Assisted with the removal of the SanDial and the installation of the second McData switch -- we're an ALL McData switch shop now... well except for the two Brocades and SanDials sitting in storage...
WEBCT: diamondback Solaris OS install from FLAR image of Moccasin, and newest maintenance applied.
RSS: put seven most recent blog postings on home page.
DNS: entry for icecast.
eMail: new storage arrived.
Recruitment: Interview
SPACE: Moved an undergrad between filesystems to free up space on moose
EMAIL: relocated penguin4 from the sandial to the mcdata (preparation for sandial going away)
RHN: applied updates as identifed by RHN on carcajou, mink, and caltest
WEBCT: Attempt installation of diamondback... it failed. ARGH!
LDAP: upgraded last two replicas so all are the same again.
EMAIL: Added a new mapping for echoVermont folks.
LDAP: update broke... two hours to fix it... I love vacation days!
Wow, here I am... everyone else is doing this blogging stuff, so here I am too. We'll see if I am diligent and keep this up or not....