Frank's Activity Log

Admin

LDAP

• Join errors -- 1 of 2 fixed. Banner being down I think is preventing the second from being delivered to LDAP.
• Privacy flags

Backup

• Examine save group failures
• Recycle expired tapes

Footprints

• 129287: email notifications of new requests being sent to former agents. Resolved -- Footprints stores this list in the MRprojects file and does not clean it up when you remove an agent.

Posted by at 7:11 AM | Permalink

Admin

LDAP

• Fix join errors from this morning
• Rant at openldap developers for calling 2.4 stable with all the issues still floating around on 2.4.18.

Backup

• Oops... too many clients defined... who was supposed to keep count this week.... hmm, looks like it was my turn. Darn!
• EMC 31051592 -- Yes, I am really running 7.4sp5 and I really do still have nsrck's that hang (well, I did before I re-defined the 94 clients that had been deleted).
• Test environment.... romulus and remus are back and they have a VTL this time.

Posted by at 7:04 AM | Permalink

Admin

LDAP

• Fix join failures from yesterday
• Privacy flag requests
• Fix join failures from today

Backup

• EMC 31359116 -- savegrp core -- message back from the SME indicates that they got the core without the binary to examine. That's strange because I packaged the core and the savegrp binary in the zip file that I uploaded. I have verified that the file present on powerlink has the core and the binary inside it (downloaded and unpacked it here). Still, they looked at something because the SME believes this is LGTsc32510 and LGTsc32658. I have requested that my support tech provide me the descriptions of LGTsc32510 and LGTsc32658.

Posted by at 7:04 AM | Permalink

Admin

Backup

• EMC 31192910 -- Continuing to work on the sev 1 issue -- Windows server 2008 DR doesn't work.
• EMC 31051592 -- Reopen -- get LGTsc30901 facts -- created June 27th, hot fix available August 4th -- but EMC support people I dealt with didn't know of it, couldn't find it when I called in the issue on August 21 and it is still not listed in the powerstink knowledgebase

LDAP

• Nightly update failures from Saturday dealt with

Posted by at 10:51 AM | Permalink

Admin

Backup

• EMC 31051592: no nsrck's hung this morning.
• EMC 31099632: no word from EMC -- have to refresh their memory tomorrow.
• Scouting for word on whether 7.4sp5 is safe or not -- appears it is good
• Tapes and Disks picked up by SecurShred for destruction.
• Start to lay out design for scripted staging.
• Investigations into why backups on reindeer are so much slower than they should be.
• Replaced batteries in tape fire safes.

LDAP

• Privacy flags
• Thoughts about how to separate the SOR and WP information
• Research how students find out their id number

Posted by at 6:50 PM | Permalink

Admin

Backup

• EMC 31051592: Forced to level 1 to get call back, two hours on the phone with support, re-defined 42 clients that had been removed over the course of the last 15 years, nsrim -X now runs without causing nsrck to hang. Acid test will be whether there is a hung nsrck or not tomorrow morning.
• EMC 31099632: Support thinks it may be an SQL sp3 hotfix -- which COMIS is leary of because they are not seeing any indications that they are having this issue. I agree. Sent support searching for why the VSS backup is ignoring the directive to skip .mdf files.

Posted by at 6:47 PM | Permalink

Admin

Backup

• Manually clean tape drive, EMC 23189374... still broken since 14. April 2008!
• EMC 31051592:
  • kill off the nsrck that started and hung at midnight.
  • Discover the "undef" volume problem is a side effect of this daily process not working.
  • Continue manual nsrck -L6's against all clients in /nsr/index
• EMC 31099632 -- open, sev 2 -- med104 failing VSS SYSTEM SERVICES:\ backup due to SQL
• Weekly cloning script finished. So, once the nightly backups finally finish, I'll be able to recycle networker on the server to see if that clears up the issue with 31051592.

Posted by at 9:38 AM | Permalink

Admin

Backup

• EMC 31051592: worked for an hour on the phone with Wallace on Friday and ran commands on Sunday afternoon. The "nsrck -m" ran without error. However, the "nsrim -X" appears to have hung up trying to work on a client that was removed in March of 2008. Will spend more time working with EMC this morning. Much work on this issue today, no resolution.
• Tape/Drive destruction assembly

LDAP

• Alumni email-for-life additions

Posted by at 9:14 AM | Permalink

Admin

Backup

• 31051592: Sev 2 -- nightly nsrim/nsrck pair starts up (nsrim forks and starts nsrck) but nsrck never finishes... one was running for 17 days. Does not use any cpu time, just sits there.... EMC is trying to force me to upgrade to the brand new 7.4.5 release.
• Volumes going "undef" instead of "expired" -- 5 more this morning, recycled them manually
• Replace broken handle on fire safe.

LDAP

• Privacy Flags

Posted by at 10:10 AM | Permalink

Admin

Backup

• EMC SR 30744500: Discovered man nsrck -MXv (long string of client names) running yesterday when I shutdown networker for monthly maintenance. Could those (some running from as far back as August 1, be related to the ongoing corruption problems in daemon.raw?
• Found more tapes that have volretent value of "undef" instead of "expired". These appear to be tapes that had previously had the volretent set to the ssretent value instead of the clretent value. At least I will assume that until I find one that was never on that list of tapes. Opened issue for tracking in sharepoint -- issue #104
• EMC SR 30964166: Working with Greg using norton1 as our guinea pig to verify that ASR recovery with AFTD's involved is bad news -- discovered that storage node ordering is critical to having it actually work.
• Notify Cambridge that I would like info about the "dual power" option for the Qualstar XLS

LDAP

• Privacy Flags
• Alumni Email-For-Life updates
• Double check that backup of data directory is working

Posted by at 11:04 AM | Permalink

Admin

Backup

Monthly Backup Maintenance

• Qualstar XLS: Upgraded firmware from version 2 to version 3. Process took 7 hours.
• AFTD: fsck'd /disk1 on storage nodes 5, 7, and 8.
• LTO4 drives: Upgraded firmware to 94D0 to address a code "A" issue with rewinds.
• Rebooted all storage nodes and NSR Server.

LDAP

Privacy flags

Accounts

Automated Purge Process test (dry) run.

Posted by at 6:30 PM | Permalink

Admin

Calendar

OS Maintenance applied and Calendar database maintenance run

Backup

• rebooted gnu - read-only file system error -- more fallout from the Day of Disaster!
• Changed nsrports range on rh4-build. Discovered the real problem in NW7.5 is that some piece of nsrexecd is hard coded to open port 7937 -- so if 7937 is in the range of ports nsrexecd is allowed to use some other piece of nsrexecd is 60% likely to get it before the piece that was written incorrectly. -- Amazing!!! Tell the client code it can not use port 7937 and it still does use it.... and this pile of sh*t passed quality assurance testing! wow!!!
• Restart of 'UVM Fullsave 2' for above two issues

Posted by at 7:10 AM | Permalink

Admin

Disaster Day

The postmortem remains to be done and will require the assistance of IBM and EMC (VMWare), but we've had a fun day of it. It started on Thursday (while I was out) when working with IBM about an issue that had caused the replication connection between two SAN systems to break. IBM determined that there was an Efix specifically for this condition. It was tested on the target SAN with a VMWare cluster connected to it without incident during the day. Therefore, somewhere between 9 and 10 PM Thursday evening, the first part of the Efix was applied to the source SAN. Well, the production VMWare cluster got really pissed and several guest systems lost access to their disks. I am not sure, but I suspect that all guests saw some interruption to their activities. Not only did these systems lose their disks, but VMWare appeared to have lost access to the files that represented their disks. I'm not sure of what things were tried in the next 3.5 hours, but I was paged at 1:30 am Friday morning to deal with recovering our primary LDAP server from backups (essentially to perform a bare metal restoration). We spent a couple of hours during which time we discovered that our support contract with EMC/VMWare had expired -- guarantees from the sales people that we'd be sent an automatic renewal notice to the contrary, we had no support contract. It took time to convince the support structure that we were not bums and would be contacting sales to negotiate a renewal and finally VMWare was working to help resolve the issue.

At about 2:30 I started the bare metal recovery of the LDAP server (create a new VMWare guest and recover all the files from the NetWorker backup system) and I was amazed that it had completed in less than two hours. It wasn't perfect, because the backup had been done before the previous day's batch update and it was missing some information that was needed. I retrieved the LDAP database from one of the replica servers and got the LDAP server back up and running. I didn't realize that I was missing the previous day's batch run data until after I'd manually kicked off the days batch run, which had lots of errors because it was comparing today's authoritative feed data to that from two days ago instead of just yesterdays. To correct this, I've updated the nightly batch script so it issues a NetWorker save of the batch update data directory when the nightly batch process is complete. So, as long as the batch process runs to completion, the backup system will have the information.

VMWare did come through eventually, with not one but two filesystem dd patches to correct the corruption and the files that were unavailable were all eventually able to be seen and their guests could be started again. Which was good.

Sadly, the incident was not free of problems. We discovered that the ASR recovery process on the Windows 2003 printer server tripped over a bug in NetWorker. An issue is open with NetWorker about that -- it was opened as a Sev 1, but has been lowered because VMWare was able to get the VMFS corruption fixed and we were able to get the printers server running again before the 1 hour call back time on the Sev 1 had expired. We will built a test Windows system that we will do an ASR recovery on and work through this problem with EMC/NetWorker. In short, the problem is that in our environment we send Full saves to one set of disk and incrementals to another set of disk. Those different sets of disk are mounted on different backup servers (storage nodes). NetWorker's ASR recovery process is reading the information from the full save and then demanding that the incremental disk be removed from the storage node it is attached to and moved to the same storage node the full save disk is attached to. That's just not physically possible with direct attached disk.

The final notice that all was recovered was sent out by Geoff at 3:50pm Friday afternoon.

Posted by at 3:00 PM | Permalink

Admin

LDAP:

• Update delEntry routine to remember the PeopleSoft emplid value

Posted by at 9:46 AM | Permalink

Admin

Backup:

• Monthly maintenance day:
  1. Qualstar XLS832700 firmware update -- FAILED
     • X-Y board is original and needs to be replaced to be certain it will work with the new firmware. Qualstar is researching replacement options (maintenance agreement coverage)
  2. NetWorker updated to 7.4.4.6 to fix several open issues
  3. OS Maintenance applied to server and dedicated storage nodes
  4. DD460's replaced with DD690
• Several 09xxxxL4 tapes returned to the scratch pool.
• Replaced worn out LTO cleaning tape with a new one.

Posted by at 10:14 AM | Permalink

Admin

Backup

Today is the start of the server migration from Solaris to Linux. This is a move that is not officially supported by EMC. Their stance is that you just start over and if you want the tapes from the previous operating system you run the scanner command to import them into the new server. I should note that while EMC officially does not support migrating your NetWorker server from one operating system to another, that support tacitly admits that it is possible to do. We have hired Cambridge Computers to assist with the process. They provided the procedure and three hours of phone support (one hour before the task and two hours during) to assist in making this move.

Posted by at 6:44 AM | Permalink

Admin

Calendar

Quarterly DB Maintenance

Backup

Several AFTD's were getting very full, one reached 99%. Manually staged savesets off the disks (AFTD's) to tape volumes.

Posted by at 6:42 PM | Permalink

Admin

FootPrints

Upgraded to v9.0.3. One fallout, the old group.html files did not work with this version. Took several hours on Thursday to fix them.

Posted by at 6:40 PM | Permalink

Admin

Backup

• Licensing for Linux migration: still confusion and EMC rep is out on vacation today.
• New tapes relocated to room Tape Library is in from my office.

LDAP

• Nightly update failed in an infinite loop that did not set off any monitoring alerts. Three hours of work to design, implement, test, and deploy the repaired logic. I really didn't need to do that to myself this morning.

Posted by at 11:36 AM | Permalink

Admin

Backup

• Adjusting management scripts for differences between Linux and Solaris paths
• More work chasing the license issues
• Requested an update on the status of EMC SR 29279000
• Opened EMC SR 29683538 (ibis1 index messup) as a Sev 1

LDAP

• Setting privacy flags
• Audit sync of replicas and primary server databases (zero errors found)

Posted by at 9:38 AM | Permalink

Admin

Backup

Getting licensing worked on for the server replacement in two weeks. It's absolutely amazing that none of the licenses that were purchased before EMC bought Legato are to be found, but we pay maintenance on these same licenses year after year. I begin to think the licensing is going to be the biggest challenge of this move.

LDAP

Still watching the OpenLDAP mailinglists for a decrease in the number and severity of issues with replication. It is so amazing to keep seeing 2.4.x be labeled as the most stable release and then the same replication problems be discussed all over again like they are new issues.

Posted by at 8:06 PM | Permalink

Admin

• Backup
  • SR 29141696: EMC NETWORKER NOT RESPONSIVE
    1. all commands (nsradmin, nsrjb, mmino) just hang
    2. Nothing new in daemon.raw since 4:47 (that wasn't notification of a User authentication session timing out)
    3. gave EMC 30 minutes to respond before forcefully stopping networker
    4. restarted networker at 7:40am
    5. Waited for daemon.raw update flood to finish (signifies networker has finished starting)
    6. run "nsrim -X" to clean up the cruft
    7. check the jukebox to see that reality matches what networker believes
    8. announce problem/recovery to major customers
    9. It's been over an hour and no call from EMC. Screw them -- time for breakfast.
• LDAP
  • Privacy Flags

Posted by at 7:37 AM | Permalink

Admin

Accounts

• Send first email to accounts that are up for removal on June 1.

Backup

• More work clearing off the DD460 that has filled up.
• More hand holding of 7.4.4

Posted by at 11:31 AM | Permalink

Admin

Backup

• Maintenance Day
  • SR 26876892: hotfix applied and later removed due to errors being logged about authentications and media emergencies
  • Tape drive 14 moved to a new storage node -- jbedit failed to do its job because drive 14 was still associated with the jukebox. Eventually wound up deleting the entire jukebox definition and redefining it. That requires a re-inventory of the jukebox. Inventory with 13 drives doing the work takes 1 hour for every 100 tapes. The jukebox holds 2500 tapes -- so if we ever fill it this kind of disaster will take a day to inventory.
  • Also discovered that the old SR 21312956 had come back. Apparently, one or more installs of NetWorker have replaced the lus.conf file and it was again not configured properly for the HBA we have installed and was showing 16*16 jukeboxes because it is just plain stupid!. I applied the old fix again and am now back to just one jukebox.
  • Defined a new AFTD on the new storage node and it is performing very slowly.

LDAP

• Privacy flags
• Emergency manual merging of two accounts discovered to belong to a single person

Posted by at 5:04 PM | Permalink

Admin

Calendar

Quarterly Maintenance Cycle

Posted by at 6:52 AM | Permalink

Admin

LDAP

• bug in setting attributes from Banner found and fixed
• Privacy flags
• Asking questions about RPMs on AIX

Backup

• Work on EMC SR # 26876892 -- once more prove they have a bug
• Sev 1: Restore broken: EMC SR #27898812 -- Turns out 7.4.4 absolutely wants the jukebox's read hostname left blank.

Posted by at 2:48 PM | Permalink

Admin

LDAP

• Privacy flags
• Fix a few messed up mail aliases and fix the logic bug that caused them
• Work on fixing preferred first name is middle name issue (Dan Dan Quale)
• More beating the head against the rock of cyrus-sasl on AIX

Backup

• Define Shibboleth servers as clients
• Start cleaning up a few disabled clients

Calendar

• db_vista ERROR -24 shows up.... what is that? SR opened
• Customer unable to use portal to access -- assist with investigation

Posted by at 2:27 PM | Permalink

Admin

LDAP

• Nightly Update errors (six of them) -- the entire gamut of possible problems!
• Discussion about "F" status fed from Registrar system -- determine conditions for it to disappear from extract.

Backup

• Power cycle LTO4 drive showing "A" to clear it for now

Posted by at 7:15 AM | Permalink

Admin

Backup

• Cloning -- move filer into tape config file
• Tape Drive -- another LTO4 with "A" code. That new firmware was supposed to fix that. Reported to Qualstar and sharepoint issue opened.

LDAP

• Audit to see if anything is missing on replicas today -- all clean
• Add some new EchoVermont.org email addresses
• Still fighting with cyrus-sasl on AIX 5

Posted by at 9:48 AM | Permalink

Admin

Backup

1. Remove Friday staging cron job
2. Set up override to do Full on NetApp Filer tonight
3. Split Fullsave staging configuration into two pieces (to not stage more disks that there are tape drives available)

LDAP

1. More research into the portal update delay -- resolved
2. More struggling with cyrus-sasl 2.1.22 on AIX5 (Oy!!!)

Posted by at 10:59 AM | Permalink

Admin

Backup

• Weekly/Monthly cloning started
• Filer set to backup directly to tape -- 2TB filesize limit on EXT3

LDAP

• Preferred name code released to production

Posted by at 11:35 AM | Permalink

Admin

Backup:

• 26155462: BugTraq -- advised EMC to close the issue, it was resolved
• 26575630: EMC called -- I think we're having a communication problem. I told her that the tape drive has an issue but that NetWorker has a logic problem because it took the entire library offline instead of the problem tape drive. I told her that five times. We'll see what she puts in the case notes.
• 23189374: Updating EMC powerlink notes to indicate that IBM drive dump analysis indicates that TapeAlert "Clean now" is never being posted. Why is NW reporting that it is seeing it?
• Replace tape drive that repeatedly presents "A" code

Posted by at 10:07 AM | Permalink

Admin

Calendar:

Quarterly DB Maintenance and Patch install

Backup:

26575630: Jukebox again made not ready by the same tape drive having an issue. Was able to get jukebox back this time by cycling it into service mode and back to enabled. Have left the tape drive in "service" mode.

Posted by at 10:04 AM | Permalink

Admin

Backup:

• EMC 23189374: Discovered a tape drive that needed to be cleaned and a "TapeAlert Unsupported" message in daemon.log. Dumped the drive and requested analysis of what TA code it had sent. Notified EMC of new development.
• Qualstar SCD "A": 131002927 -- repeat performance

LDAP:

• Nightly update issues -- human intervention required in IdM

Posted by at 9:17 AM | Permalink

Admin

Backup:

• Clones -- COMIS clones migrated to the read hostname for the jukebox overnight. cloning code needs some tweaks
• EMC: 23189374 -- had to manually clean another tape drive
• Qualstar SCD "A": -- Another drive showing the condition. Is this humidity related?
• EMC: 26441724 -- NFSv4 mount backup issues... more EMC testing

LDAP:

• Alumni email addresses

Posted by at 7:17 AM | Permalink

Admin

LDAP:

• Alumni email updates
• Privacy Flags

Backup:

• EMC: 26441724 -- Filesystems have to be listed in /etc/fstab to be backed up. Good info... so what does that have to do with NetWorker trying to back up NFS mounts it should not be??? IDIOTS!
• EMC: 23189374 -- Request for update and let them know I had to manually clean two more tape drives this morning
• Announce 6pm-8pm work on INBOX backup disk array
• Research XLS documentation to see if it is capable of performing tape drive cleaning (automatically) -- manuals seem to indicate it is a manual process (in physical mode). Send note off to support to see if there is an automatic cleaning option in the hardware.

Calendar:

• Announce Saturday downtime to
  • OS Upgrade
  • DB Maintenance
  • Calendar fixes

Posted by at 11:28 AM | Permalink

Admin

LDAP:

• Privacy flags
• Hey -- no update problems last night... what's up with that?
• Continue planning of LDAP redesign
• Continue planning of LDAP groups taking over from DCE groups

Backup:

• Notify folks to start their plans to upgrade to 7.4sp3 clients ... or at least 7.3sp4!
• Monitor the staging and it's overly aggressive use of tape drives
• Relabel expired tapes
• Update EMC powerlink issues.... keep them active
• Continue planning for 7.4sp3 server upgrade

Posted by at 8:54 AM | Permalink

Admin

LDAP

• Notify catcard office of broken feed
• Alumni email forwarding
• Privacy flags

Backup:

• Clones started and staging of NDMP incrementals

Posted by at 11:45 AM | Permalink

Admin

Backup:

• nsrexecd crashed at 11:32pm last night.
  • EMC SR 26184182 opened at 12:40am.
  • nsrexecd seems to have stayed operational overnight.
  • EMC unwilling to examine core file -- want me to move to new release. I'm not moving until they provide LGTsc09318 for something other than 7.3.3. They say there are three known issues with nsrexecd that are fixed in newer versions. Big deal -- do they know that these other issues are the same "Hey, I called strlen with a bad pointer and walked off the end of my memory" issue? ... they might, after all, calling strlen with a bad pointer seems to be NetWorker's favorite way to kill itself
• med28 has a third address -- update the DNS and the client definition.

Posted by at 8:06 AM | Permalink

Admin

Backup:

• Escalate EMC SR 20292396 -- AIX recover
• Clean up a client that was removed
• Walk a customer through restoring a clobbered web page
• Look at cleaning up the 2 month old tapes that should have already recycled (again)
• Open EMC 26173518 -- Sev 1 -- recover attempting to mount AFTD on tape drive and then a staging command that attempted to use that tape drive caused 14 tapes to become marked "unlabeled" before I found and killed the recover

General:

• Recover account that should not have been deleted

Posted by at 8:57 AM | Permalink

Admin

LDAP:

• Alumni email forwarding
• Nightly update manual interventions
• PS Privacy flags

Backup:

• COMIS: five servers had VSS failures -- suspect another bad maintenance batch
• EMC: 23189374 -- no explanation for what GUI_INVENTORY_ADD_E=YES env variable is really supposed to accomplisth -- will need to reboot server to complete setting
• EMC: 26003402 -- had to explain how to use mminfo to get a list of ssid's that I want to clone to the support tech.... Oy Vey! Hello IBM -- Please call me... what's the price for TSM these days???
• Staging valid SSIDS off tapes that should have expired months ago (missed a full)
• Open SR 26155462 about DOS

General:

• Recover sent-mail folder for customer

Posted by at 9:16 AM | Permalink

Admin

LDAP:

• Privacy Flags
• Nightly Update manual interventions
• Update code issue: Another instance of Perl's GREP not working as documented
• Track down some alumni email forwarding issues

Backup:

• Clones/Staging complete for the week!
• Troublesome client disabled until it becomes available again

Footprints:

• Convert a customer to an agent in HelpLine project

Calendar:

• VMWare caused crash, LDAP database got corrupted -- required manual recovery process

Posted by at 8:37 AM | Permalink

Admin

Backup:

• Clones -- new routines tested and weekly clone run started
• Space -- email AFTD filled up... emergency procedure A (delete oldest cycle (nsrmm), free up space (nsrim), really free space (filesys clean))

General:

• Assist as possible with giraffe issue

LDAP:

• Alumni - set forwarding addresses
• Nightly Update -- process manual interventions for weekend updates

Posted by at 9:45 AM | Permalink

Admin

Backup:

• Catalyst T3-4 group failure. Investigation turned up NFS mount to another server that was not running NFSD anymore. A forced unmount of the /mnt/cdrom filesystem and then the save group restarted and ran to successful completion.

LDAP:

• Process the errors from the nightly update (PS feed of student employee didn't find the matching student due to missing info in the student record)

General

• Giraffe: Xi-Text printers in "halted" state after system crash last night -- all had to be started
• AIX LOGLV's -- found guideline that says 1 loglv per 1 trillion bytes of filesystem... added 10 more loglvs!!!

Posted by at 7:28 AM | Permalink

Admin

LDAP:

• Nightly Update Error processing

Backup:

• Clean tape drive that NW knew needed it but failed to do so (EMC: 23189374)
• Monthly maintenance day:
  • EMC procedure esg98263)
  • DataDomain hardware repair

Posted by at 8:45 AM | Permalink

Admin

LDAP:

• Nightly Update errors
• Alumni email updates
• OpenLDAP 2.4.12 available

Backup:

• Cloning procedure testing
• Update NetWorkerReporter system
• EMC: 26003402 -- They don't understand the question sigh

Posted by at 9:26 AM | Permalink

Admin

LDAP:

• Alumni email forwarding
• Banner Update Issues
• Work on SOR/WP separation design

Backup:

• Re-design/implement cloning algorithm

Posted by at 8:20 AM | Permalink

Admin

• LDAP:

  RHEL5 OpenLDAP 2.4

  Continued work on creating the RPM that will enable OpenLDAP 2.4 to be installed without replacing Red Hat's OpenLDAP 2.3.27 build

Posted by at 8:33 AM | Permalink

Admin

WrightLine

Chasing a quote. Almost ready to see if anyone else sells parts that fit WrightLine racks

LDAP:

• Alumni life-time forwarding additions
• RHEL5 RPM -- development

Posted by at 11:20 AM | Permalink

Admin

LDAP:

• Privacy flags set
• Affiliation values examined
• Planning for OpenLDAP 2.4

Backup:

• Clones started
• Tape wrangling

General:

• Reports: update script to find correct file

Posted by at 1:03 PM | Permalink

Admin

• Rack Wrangling: 1 rack and 3 doors moved to and installed in Mann
• LDAP: cleanup some out-of-sync info in NeededIDs file -- they were created and renamed because the info was wrong within a day
• LDAP: privacy flags
• Backup: stuck tape

Posted by at 1:09 PM | Permalink

Admin -- this whole week

Backup:

• Working on moving Filer data off 460 to make more room for INBOX backups
• Surplus Qualstar TLS-412300
• Looking at 7.4 upgrade and Windows 2008/Exchange 2007 support

LDAP:

• Modify daily report to CatCard office
• Privacy Flags
• Alumni email accounts

Vendors:

• WrightLine: quote changes
• Datadomain: work on 65889 and 65941
• DD and CCS: talk about 460 upgrade/exchange program

Posted by at 9:41 AM | Permalink

Admin

• LDAP:
  • Privacy Flags
  • Nightly Update error -- 30% change in banner feed -- should have expected that!
  • Logic error in nightly update -- same person in two SOR feeds caused a "Manual Merge Required" error when the second feed was processed... cause found and fixed.
  • wiki entry updates
• Backups: Discussion with vendor about dedupe hardware replacement possibilities

Posted by at 8:14 AM | Permalink

Admin

• LDAP: Privacy Flags and Update Errors
• Backup:
  • Destruction of tapes and hard drives
  • Staging off all data off SATABeast #3
• Vendors:
  • WrightLine: chase quote
  • DataDomain: Accept meeting proposal

Posted by at 1:36 PM | Permalink

Activity Log

• Backups:
  • Cloning started
  • EMC 20205070: mark it resolved
  • EMC 25182634: new -- port LGTsc09318 to 7.4 SP3 when it is available
  • EMC 25177072: Windows 2008 VSS error -- missing file on Windows!
  • EMC 20292396: Provide even more documentation
  • Data Domain -- sales call

Posted by at 2:50 PM | Permalink

Admin

• LDAP: Privacy flags and Banner update problems
• Backup:
  • Nexsan and device-mapper-multipath frustration continues
    • The array I built yesterday that worked once -- I swear it did, I just can't prove it -- and then wouldn't work all afternoon -- died... two disks "failed". Therefore, today is being devoted to rebuilding a smaller array in the Nexsan ATABoy2F system. Hopefully, I'll be able to get back to testing soon.

Posted by at 1:48 PM | Permalink

Admin

• LDAP: dce group closer to real-time
• Backup:
  • weekly cloning started
  • check up on several groups that failed
  • Convert the last UVM Fullsave group to do differentials on the non-full weekends
  • Find test equipment for DM-Multipath development
• Footprints: check on how incoming email does or does not get address book info filled in.

Posted by at 7:25 AM | Permalink

Admin

• Backup
  • Disk storage on VLAN 805
  • Argue with EMC about SR #23189374 and 30 step procedure that will take about 8 hours to complete
• Racks
  • More fun with WrightLine on the secure lock upgrade issue
  • Looking at secure lock upgrade in DR center
• Calendar: Discussion concerning portal project and issues with logout
• Account Removal: Dealing with extension request that got lost for 23 days!

Posted by at 2:20 PM | Permalink

Admin

• Backup -- maintenance day! All good -- no problems!

Posted by at 3:01 PM | Permalink

Admin

• Calendar:
  • two footprints issues -- people unable to log into calendar web client due to immediate error message "You have timed out from your previous session. Please sign in again". Found a note in the Oracle MetaLink Knowledgebase (Note:423501.1) which describes this exact problem after upgrading from v9 to v10. The fix is to update the unison.ini file to increase the itemextinfomaxsize from the default of 1500 to a new necessary size of 5000 (why isn't this the default in v10???) anyway... once that change is made, the calendar server and web client both need to be stopped and started.
• Backup:
  • Weekly clones started
  • Cleaned two tape drives
• LDAP:
  • Clean up nightly update errors from the weekend
  • Alumni processing

Posted by at 9:39 AM | Permalink

Admin

• Accounts: unlock 2 accounts that were unable to get data before they were locked up. Gave them a one week extension -- will lock them up again on 9 June
• Backup:
  • data preservation.... special retention
  • routine cloning run

Posted by at 3:57 PM | Permalink

Admin

• Accounts -- lock up accounts for 2008 Deletion project

Posted by at 4:56 PM | Permalink

Admin

• Backup: start cloning process
• Calendar: more email about blackberry

Posted by at 10:13 AM | Permalink

Admin

• Backup:
  • More discussion with NEXSan concerning one LUN not working
  • Issue opened with EMC about what error's mean
  • Discovered /disk3 was RO instead of RW... offline, fsck (lots of errors), online -- working!
• Calendar: More work/email with team about the one blackberry

Posted by at 4:11 PM | Permalink

Admin

• Backup:
  • SATABeast controller failure -- move cable, let backups finish, reboot system, move cable back as controller came back. Reported to NEXSan -- new controller to be shipped out
• Calendar: Upgrade to v10.1.2 -- one blackberry apparently unhappy with upgrade.

Posted by at 8:09 PM | Permalink

Admin

• Accounts: final email warning to accounts that are being removed

Posted by at 2:01 PM | Permalink

Admin

• Backup
  • Cleaned tape drive
• LDAP:
  • Merge discovered duplication (caused by id number typo on one side or the other)
• Accounts: May notification of pending deletions sent

Posted by at 8:32 AM | Permalink

Admin

• Backup:
  • start replacement of old EOL datadomain device
  • Test 7.4SP2 on 2008

Posted by at 3:08 PM | Permalink

Admin

• Backup:
  • 23189374: Re-opened because two drives need cleaning and NW has only seen one does and hasn't even cleaned that one yet
  • moving indexes off dd200 action starting

Posted by at 10:15 AM | Permalink

Admin

• Backup:
  • Add aliases for COMIS systems that need them
  • EMC: 20205070 -- request update
  • Stage disk devices
  • Tape Library failed to scan I/O ports after they were closed
  • EMC: 23326942 -- ejected LTO tape not understood
• Accounts: clean up after a "network glitch"
• LDAP:
  • Privacy flags
  • Changed the wrong entry and caused a mail problem because the load balancers declared ldap to be unavailable.

Posted by at 7:30 AM | Permalink

Admin

• LDAP: Privacy flags
• Backup: config changed to use tape barcode as volume label
• LDAP: meeting with CatCard office about daily feed

Posted by at 2:58 PM | Permalink

Admin

• LDAP: privacy flags
• Backup: COMIS systems

Posted by at 3:06 PM | Permalink

Admin

• Backup Maintenance day
  • Tape Library OS upgraded
  • Disk Unit OS upgraded
  • Systems OS Upgrades applied
  • Filesystems checked
  • EMC: 20205070 hot-fix tested on Linux, deficiency reported back

Posted by at 5:03 PM | Permalink

Admin

• Backup:
  • Prep-work for tomorrow's maintenance day
  • EMC 20205070: phone call, some movement toward resolution

Posted by at 2:01 PM | Permalink

Admin

• LDAP: Privacy flags
• Backup: Dealing with the three main open EMC issues and new configuration for the COMIS clients
• Otter did a lockup and then I accidentally power cycled it when the power strip switch was labeled backwards and I had forgotten that off was on.

Posted by at 7:52 AM | Permalink

Admin

• Backup: Had to clean three tape drives and label 12 tapes over the weekend. The COMIS Onsite media pool ran out of tapes Saturday morning, so added 12 more. Discovered Sunday morning that tape drives 10-12 were calling for cleaning and NetWorker hadn't taken care of it. Yet another issue to EMC needs to be opened.
• Calendar: figured out the web interface issue (wrong node defined in the ocas.conf file) and announced the test server to ETS Staff

Posted by at 7:32 AM | Permalink

Admin

• Backup: Discover that recycle to/from doesn't change the label on the tape. Oops! That's not what we expected or desired.

Posted by at 11:00 AM | Permalink

Admin

• System problems -- Calendar server and Footprints server. Apparently, something went wrong with the VMWare cluster during Tuesday's maintenance. It took all day to track it down. Half the day was spent proving that it wasn't the applications on the VMWare guests.

Posted by at 4:00 PM | Permalink

Admin

• LDAP: privacy flags
• Backup: clones started, tape drive replaced, failed disk drive discovered, raid checking set up
• Accounts: discussion on IdM continues

Posted by at 1:46 PM | Permalink

Admin

• General:
  • Waterman UPS battery temperature high -- called SOS to have the A/C unit checked out
• Backup:
  • Pester Qualstar about the tape that got stuck in a tape drive
  • Register RMA with Qualstar -- tape is very stuck in drive
  • Replace RMA'd power supply in NexSAN SATABeast
• LDAP: email discussion about IdM and sponsored (aka guest) accounts
• Accounts: Account Removal project ramps up
• Calendar: 10.1.2 upgrade test finished, but web access is broken -- opened new Oracle SR

Posted by at 8:11 AM | Permalink

Admin

• LDAP: privacy flags, manual update errors
• Calendar: quarterly maintenance and copy data for 10g testing
• Backup: further work with COMIS, EMC (20205070)

Posted by at 6:09 PM | Permalink

Admin

• Backup:
  • Consulting engagement with Cambridge Computer Systems
  • COMIS backup performance testing
• LDAP: IdM update for feed into Active Directory, Privacy Flags
• Calendar: Inch forward on 10.1.2 deployment

Posted by at 5:55 PM | Permalink

Admin

• Calendar:
  • More work on 18647754.6
  • Building, got to E.4 Upgrades Calendar Standalone Install doc
  • That was referenced from Upgrading from 9.0.4.2
• LDAP: special CHWB entry
• NetWorker: 20205070 conference call
• Hiring: Interview

Posted by at 3:35 PM | Permalink

Admin

• Backup:
  • 20205070: some testing -- scsi doesn't have the issue
  • clones completed -- remove full tapes
• LDAP:
  • Privacy flags
  • Luminus -- regenerate sample file
• Calendar:
  • More work on 18647754.6
  • Rebuilding apache2 ... it didn't bring in mod_ssl (/sigh)
  • Building, got to E.2.2 Planning Separate Installations in Calendar Standalone Install doc
  • That was referenced from Upgrading from 9.0.4.2
• General Admin: help helpline understand magicscript, php safemode, unix permissions

Posted by at 2:34 PM | Permalink

Admin

• LDAP: privacy flags
• Accounts: password length check broken
• Backup:
  • Storage: new SATABeast installed and connected
  • Clones: Started
  • Verbose flag turned off for save group it should not have been on for

Posted by at 2:43 PM | Permalink

Admin

• LDAP: relocate replica server to new data center -- planned 2 hours, but took 3 due to unexpected issues with out-of-date systemconfigurator package

Posted by at 3:34 PM | Permalink

Admin

• LDAP: Investigate AD feed issue
• Backup: telecon with EMC/CCS about 20205070/3215 -- EMC still has nothing
• Calendar: PHP code implemented on new system
• Hiring: Interview

Posted by at 3:33 PM | Permalink

Admin

• LDAP: merge multiple personalities into a single entry
• Backup: teleconference with CCS about upcoming visit
• FootPrints: add agent bug patched

Posted by at 11:02 AM | Permalink

Admin

• LDAP: privacy flags
• Backup:
  • cloning finished, staging started
  • Decommision server: working on wiping old ATABeast disks
  • Waiting for SATABeast to be delivered by FedEx
• Calendar:
  • Working with Support on CSW issues, upgrading to 10.1.2.x
  • SR: 18647754.6
  • Building, got to E.2.1.3 OpenLDAP Directory Server in Calendar Standalone Install doc
  • That was referenced from Upgrading from 9.0.4.2

Posted by at 2:29 PM | Permalink

admin

• Backup
  • Storage node rebuild, AFTD movement, MTU 9000
  • New clients added, old clients removed
  • Consolidation of servers in a single switch (performance boost)
• LDAP
  • Logic errors with PeopleSoft feed
  • privacy flags

Posted by at 2:26 PM | Permalink

Admin

• LDAP: fix logic bug discovered when an address was entered twice with different capitalization
• Backup:
  • sloth is no more, stornode1 reigns supreme
  • deal with cloning failure issue
• Calendar: branding question

Posted by at 3:40 PM | Permalink

Admin

• LDAP - Upgrade to fix security problem
• Backup
  • Investigate weekend backup failures -- DOH, misconfigured DNS entries!
  • kick off clones

Posted by at 7:12 AM | Permalink

Admin

• Backup: looking for opportunity for MTU tweaking, stornode1 work
• FootPrints: preparation for upgrade to v8.1 on test machine
• Accounts: OAS boxen update

Posted by at 7:21 AM | Permalink

Admin

• Wrightline LMS move from old to new data center and install of three more racks in new data center
• Backup tweaking, research

Posted by at 8:20 PM | Permalink

Admin

• Backup: stornode1, cluster config, cloning, emc wrangling

Posted by at 3:01 PM | Permalink

Admin

• LDAP: Privacy flags, bug fix, answer more questions about cellular phones being work numbers, push group change
• Backup: re-test i/o port being open causing bad problems (yup, still does); fixed inquire problem last night
• Accounts: fix hung mkacct process (another timeout problem)
• FootPrints: fix flashboard problem

Posted by at 6:10 PM | Permalink

Admin

• Backup: create NO_SUPPRESS file to see if that fixes complaint

Posted by at 6:08 PM | Permalink

Admin

• Backup:
  • Re-write clonevol shell script to only clone the last week's worth of ssids
  • More work on ATABeast wiping
  • Admin work on COMIS backups
• WWW: -- assist with deployment of new web design

Posted by at 7:25 AM | Permalink

Admin

• Aus Krank!
• Backup: start clones

Posted by at 5:24 AM | Permalink

Admin

• LDAP: privacy flags
• Backup:
  • Operator training on loading new tapes into the tape library
  • Clean off slices 5 and 6 of the ATABeast
  • Initiate cleaning off slice 7 of the ATABeast
  • Install the new licenses purchased

Posted by at 3:08 PM | Permalink

Admin

• Backup:
  • XLS eject/unload error (EMC: 20205070) seen again
  • Initiate cleaning off the 4th slice of the old ATABeast
• LDAP: privacy flags
• FootPrints: work on v8.1 upgrade on test server

Posted by at 8:38 AM | Permalink

Admin

• LDAP: privacy flags
• FootPrints: work to repair corrupted project and open issue to understand how it got corrupted
• Backup: continue work on cleaning off old ATABeast

Posted by at 6:33 PM | Permalink

Admin

• FootPrints: move to new server
  • Not smooth -- but it got there.
• LDAP: fix catcard feed
• Privacy: Set flags
• Backup: learn how to play with standalone drives.
• Calendar: more work on the new server.

Posted by at 3:22 PM | Permalink

Admin

• Backup -- orders, clones, performance analysis
• LDAP -- privacy flags, new code
• Accounts -- email message

Posted by at 3:23 PM | Permalink

Admin

• Calendar
  • Have spent this week building a VMWare guest to house the current calendar server. Plan is to migrate to the new "server" on February 9.
• FootPrints
  • Working on a project -- still getting everything working after recreating a project that got created with the wrong address book originally last week. I think I finally got the last of it fixed this morning.
  • Building VMWare Guest to become the footprints server on February 5.
• LDAP
  • Update ldap_report.pl to ignore barcode conflicts between the old values and current values attributes
  • syncrepl problems -- appear to be related to tcp KEEPALIVE kernel options. Reset them to be 300/5/5 instead of 7200/75/9 and hopefully, we'll not see this particular problem any more.
• Backup
  • Data movement from the SATABeast to Tape is still rather depressingly slow. Struggling to figure out where the problem lies.
  • COMIS backups are finally working well -- that was a tcp KEEPALIVE issue as well

Posted by at 8:34 AM | Permalink

Admin

• Backup:
  • Report another -111 error (40004 this time)
  • Analyse SATABeast performance tests from last night -- 100-150MB/s read
  • Work on parts quote
  • Tape wrangling
  • Staging completed in 26 hours -- 44.39MB/sec average (seems slow!)
• LDAP: roll out new report for CatCard office

Posted by at 7:34 AM | Permalink

Admin

• Backup:
  • EMC 21801812: examine, package, deliver logs from test performed by COMIS
  • Report drive 6 -111 (no drive exists) error to Qualstar
  • Run disk performance tests on SATABeast -- indicates good speed (140+MB/s)
• LDAP:
  • Clean up nightly update failures
• Footprints - new project creation training

Posted by at 8:21 AM | Permalink

Admin

• Backup
  • Perform manual full of system that needed a full and had been accidentally shut off over the weekend.
  • Start new style clones and monitor

UVM HOLIDAY

Posted by at 8:20 AM | Permalink

Admin

• LDAP:
  • Update failures -- human intervention required in six cases
  • Account -- an account rename that failed to remove the old entry
  • Reports -- CatCard office report needs work
• Backup:
  • VACC - set up for temporary VACC backup
  • Look over license/support renewal quote
  • Revisit FSCK that nailed us yesterday
  • Work with library on dfs backup issue

Posted by at 8:24 AM | Permalink

Admin

• Backup -- maintenance day!

Posted by at 6:23 PM | Permalink

Admin

• LDAP:
  • Privacy Flags
• Backup:
  • EMC 21312956: run inquire and lus debug commands per request for more information
  • Jukebox upgrade: put together plan for Thursday

Posted by at 6:55 AM | Permalink

Admin

• LDAP: privacy flags
• Backup:
  • EMC 21801812 (CCS 3333)
  • Thursday plans

Posted by at 2:40 PM | Permalink

Admin

• TGIF!!!
• LDAP:
  • Clean up nightly update errors for the past two nights, as I was out on vacation
  • Work on PS NetID Exception issue... need to update update code to handle this issue
• Backup:
  • Cambridge 3333: Something is hosed in EMC's MS Exchange module and we need to open an issue with EMC on this. Need data from COMIS to assist with the opening
  • Library MicroSoft DFS servers are not backing up -- well, that's the way it is documented to work....
• VMWare:
  • Upgrade vmtools on test/sandbox machines -- production systems have to wait for scheduled downtime

Posted by at 10:17 AM | Permalink

Admin

• LDAP
  • Privacy Flags
• Backup
  • Tape Wrangling...
  • stornode10 craziness... the network is mysteriously working this morning!

Posted by at 8:11 AM | Permalink

Admin

• LDAP
  • non-authoritative errors on new entries from Banner and HR
  • Open issue about NETID rename that didn't get into HR
  • Work on update handling new attributes from SORs
• Backup
  • Client configuration updates
  • systemimager config -- hosts and kerberos certificates

Posted by at 8:23 AM | Permalink

Admin

• Backup
  • Analyze performance of SATABeast as Full target -- a shade disappointing
  • 20242686 - no nsrmmd cores since Nov 23 -- EMC will close
  • Start planning for 1/17 work
  • Chase support and COMIS for webex
• LDAP - chase requested additions

Posted by at 9:35 AM | Permalink

Admin

• LDAP - update entry to have last name to prevent report error
• Backup - Examine new AFTD performace -- SWEET!

Posted by at 8:09 AM | Permalink

Admin

• Backup
  • new AFTD for full saves
  • client that doesn't like doing VSS suddenly
  • expansion plans
• LDAP
  • BCA from SIS without HR info
  • Update broke because of missing SN attribute

Posted by at 1:49 PM | Permalink

Admin

• LDAP
  • BCA's without HR backing

Posted by at 8:18 AM | Permalink

Admin

• Backup
  • Reconfigure SATABeast for use as AFTD
  • Work on performance issues

Posted by at 10:21 AM | Permalink

Admin

• Backup
  • Box up VTL appliance and send it back
  • bigasm testing of LTO4/SATABeast
  • COMIS adjustments
  • New batch of LTO4 media arrives
• LDAP
  • Recycle all ldap servers to get new schema/ACLs into service
  • One of ldap servers moves to new data center
  • CatCard office changes... more testing

Posted by at 4:13 PM | Permalink

Admin

• Backup:
  • VTL Testing
  • COMIS backups
  • Retirement of old tape library planning
• LDAP:
  • Privacy flags
  • Account Creation problem -- fix symptom
  • CatCard office feed -- look at issues

Posted by at 9:59 AM | Permalink

Admin

• Backup:
  • Cloning....
  • Plan TLS reconfiguration
  • Mailspool backups separated from client mail folders for space reasons
  • COMIS more work on Exchange backups

Posted by at 7:44 AM | Permalink

Admin

• LDAP: Banner Course Assignment with no PeopleSoft entry found
• Backup: 21079860 -- maximum active devices error
• Backup: 21051540 -- I need to put in the userid for the backup command
• Calendar: some progress on v10 standalone install
• Calendar: SyncML 503 error -- nobody knows what this timeout is...
• DNS: MX entries updated

Posted by at 11:06 AM | Permalink

On Call

• 0135: monitoring system down page -- false alarm

Admin

• Backup: reset authorization parameters because previous instructions munged them.
• LDAP: provide new rpm's to mail gateway machines
• Babysit the servers for the first-year's registration time
• FileServers -- filesystem clearing

WoW

• Getting to high levels of fishing skill can cause twitches!

Posted by at 6:58 AM | Permalink

On Call

• webfarm problem
• fileserver space issues

Activity Log

• Backup -- COMIS work
• LDAP -- DOS vector

Posted by at 3:34 PM | Permalink

Admin

• Backup
  • TLS I/O Port options
  • NW issues chasing
  • More cloning
• LDAP
  • Privacy Flag setting
  • GSSAPI authentication implemented in test

Posted by at 1:57 PM | Permalink

Admin

• Backup
  • New problem -- NW has tape stuck in LTO drive (BOO!) -- Ugh -- NW thought it was till backing up to the tape from a savegroup that finished last night.
  • Replaced bad disk in SATABeast
• LDAP
  • Update broke -- non-ascii characters in a name
  • Update broke notifications disabled.... re-enabled
  • Privacy flags set
  • OpenLDAP 2.3.39 build/test -- syncrepl issue

Posted by at 8:40 AM | Permalink

Admin

• Backup
  • Work on cloning script
  • Schedule SecurShred for hard disks/tapes
• LDAP: delta-syncrepl investigation

Posted by at 1:34 PM | Permalink

Admin

• Backup
  • Tape Wrangling
  • AFTD filled up... emergency cleaning

Posted by at 1:33 PM | Permalink

Admin

• WebCT: Reboot guest system to fix clock issue
• Accounts: re-lock account that was opened up to allow email retrieval
• LDAP: one Banner non-authoritative manually processed
• Accounts: move scripts to new server

Posted by at 7:23 AM | Permalink

Admin

• LDAP: three "non-authoritative" errors to deal with manually
• LDAP: slurpd done away with
• Accounts: one account reactivated
• LDAP: server acting funny... scheduled reboot for later tonight
• Backup: clones???
• LDAP: emeriti faculty issue -- not showing up correctly -- Aha... Banner thinks the person is still active!

Posted by at 3:32 PM | Permalink

Admin

• Backup: EMC 20292396 -- respond to requests
• LDAP: work on CatCard feed preliminary info
• LDAP: remove another slurpd feed
• Backup: tape wrangling
• Measurements: update smtp measurements

Posted by at 7:51 AM | Permalink

Admin

• Backup: TLS drive7 three flashing leds problem -- wound up rebooting storage node to clear all residual problems
• Backup: EMC 20292396: provide the ulimit -a and truss output requested to show that the memory issue is not real in this recover 7.3.3 on aix problem
• Backup: Work on cloning script
• Backup: Start manual cloning of one save group that failed to clone over the weekend

On Call

• All Quiet on the pager front

Posted by at 10:36 AM | Permalink

Admin

• WebCT: monitor system from 3:40 to 5:30 to see if yesterday's changes fixed the problem
• LDAP:
  • JOIN failures fixed
  • Feed changes requested from SOR owners
• Backup:
  • DataDomain 31715: reboot 460 to ensure it is correctly configured
  • EMC 20242686: download replacement nsrmmd binary
  • Meeting with COMIS about backups for them

Posted by at 7:29 AM | Permalink

Admin

• LDAP:
  • Meeting with Catcard office about feed change request
  • Privacy flags set
  • JOIN failures repaired
• Backup:
  • schedule reboot of dd460c with Data Domain
  • Telecon about VTLs and possible path to get to them
  • Spec new disk storage requirement
• Accounts: "finish" work on porting to new system

Posted by at 9:26 PM | Permalink

Admin

• Accounts: work on moving scripts
• Backup: replace broken storage node
• Backup: brief meeting about disk storage vs VTL

Posted by at 9:27 PM | Permalink

Admin

• Accounts: locked up the accounts that are scheduled for removal
• LDAP: manually fix the join problems from the weekend
• WebCT: power off and unplug the old SUN server! ANOTHER ONE BITES THE DUST!
• Backup: semi-automatic cleaning of tape drive worked!
• Backup: one client is causing long delays in one save group. Killed the group so it can run again tonight and the rest of the clients will successfully backup.

Posted by at 6:04 PM | Permalink

Admin

• Backup: Install Cluster Command & Control Suite on the server and storage nodes
• Backup: EMC issue 20242686 - nsrmmd crashed -- respond to requested information
• LDAP: New retirees list installed
• Accounts: work on moving scripts

Posted by at 9:59 AM | Permalink

Admin

• LDAP: privacy flags
• Backup: test a theory on 20205070.... oops... looks like NW can't tell the status of the XLS ... RATS!!!
• LDAP: upgrade two replica servers to current levels and syncrepl
• Old stuff... box up ancient Hard Drives for destruction

Posted by at 6:55 AM | Permalink

Admin

• LDAP: two privacy flags
• Backup: working 20205070 with EMC -- nsrjb thinks the XLS is ready but sjirdtag knows that it is not ready as soon as the I/O ports are opened.

Posted by at 9:46 AM | Permalink

Admin

• LDAP: more lists for CAS
• Backup: Uploaded files for 20242686 problem
• LDAP: two HR entries need manual attention to find their student entries
• LDAP: two privacy flags set

Posted by at 7:29 AM | Permalink

Admin

• Backup: more work on replacing storage node with hardware problem
• WebCT: webct terminated on old server. Now wait two weeks before destroying data
• Datadomain: More work understanding the failure of the 460 from the 8th
• LDAP: privacy flags, and check_saslauthd nagios checking
• Backup: leave voice message for vendor about degausser rental

Posted by at 6:41 PM | Permalink

Admin

• LDAP: Upgrade two replica servers to newest versions and syncrepl replication
• Backup: nsrmmd problem on storage node, open EMC 20242686 issue
• LDAP: update problem -- two HR entries for same person?
• LDAP: Generate lists for CAS

Posted by at 1:08 PM | Permalink

Admin

• LDAP: privacy flag, nightly update errors
• Backup: working with vendor on XLS/NetWorker issue

Posted by at 3:37 PM | Permalink

Admin

• Backup: kick off manual clone of failed auto clones from weekend
• Backup: clean XLS drive 1 (for the second time)
• Backup: telecon about data domain devices
• LDAP: Handle several auto update failures from the weekend
• Accounts: Send out the fourth notice of accounts going away

Posted by at 10:17 AM | Permalink

Admin

• LDAP: privacy flags, nightly update error
• Backup: Hardware repair and OS install on replacement storage node
• Backup: script to prove Qualstar has a problem

Posted by at 2:39 PM | Permalink

Admin

• LDAP: Privacy flags

Posted by at 6:39 AM | Permalink

Admin

• Backup: attempt to build new server (stornode4). It has a hardware problem!

Posted by at 6:38 PM | Permalink

Admin

• Backup:
  • CEMS systems added
  • Upgrade installed
• LDAP: Privacy Flags, nightly update errors fixed

Posted by at 6:48 AM | Permalink

Admin

• Backup:
  • Plan for tomorrow's upgrade
  • CEMS backup setup testing
  • Giraffe backups discussion
• LDAP: nightly update errors corrected, privacy flags set
• WebCT: VMWare guest is now production system!!!
• QLogic camp chair donated to Tech Park Seating Solution (TPSS)

Posted by at 6:46 PM | Permalink

Admin

• LDAP: nightly update errors ... still 16 Banner Course Assignment people who don't have matching PeopleSoft entries yet... hmmm
• LDAP: Privacy Flags... and questions
• Backup: Cloning meeting... moving to scripts away from NetWorker auto clones
• Backup: 7.3.3 upgrade plans
• WebCT: preparation for tomorrows production server change

Posted by at 12:54 PM | Permalink

Admin

• Backup: Purge savesets from email servers older than 16 weeks to free up space on AFTD
• Backup: Open issue with EMC about upgrade to 7.3.3 procedure
• LDAP: privacy flags, nightly update error fix
• Calendar: start building new server: LDAP installed
• WebCT: Nagios monitoring added

Posted by at 4:10 PM | Permalink

Admin

• LDAP: clean up nightly update errors
• LDAP: lists for A&S
• Backup: full tapes out, empty tapes in
• Backup: Old Offsite tapes for a critical < cough > recover
• Backup: Discover that all the tapes in the vault are the last 5 years!
• Backup: Recycle NetWorker -- cleaning out the /nsr/tmp directory

Posted by at 2:00 PM | Permalink

Admin

• Backup:
  • nsrmm -S ssid/cloneid -e MM/DD/YY resets clretent to realistic value
  • lots of tape moving around in TLS
• LDAP: privacy flags
• EMAIL: a person with their mail folders in the wrong place

Posted by at 10:43 AM | Permalink

Admin

• Backup: new tape layout for old tape library
• LDAP: privacy flags, nightly update error fixing

Posted by at 10:42 AM | Permalink

Admin

• LDAP: correct update issues for past week
• LDAP: Privacy flags
• Backup: CEMS setups
• Backup: 7.3.3 research

Posted by at 9:35 AM | Permalink

Admin

• Backup - CEMS planning

Posted by at 9:30 AM | Permalink

Admin

• Backup:
  • storage node replacement -- move datadomain to different storage node

Posted by at 10:35 AM | Permalink

Admin

• Backup: storage node had problems, wound up rebooting and starting replacement
• LDAP: several privacy flags needed to be set

Posted by at 10:33 AM | Permalink

Admin

• Backup
  • Testing suspected bad tapes
  • Clean up clones that have clretent > 6 months on UVM pool
  • Open issue with Legato about a perpetual override based on day of month -- and get it answered!
• LDAP
  • Privacy flags
  • Banner Course Assignment -- found 5 more matches
  • Nightly Update error -- add to Banner list

Posted by at 9:08 AM | Permalink

Admin

• LDAP: privacy flags
• Backup
  • PeopleSoft examinations
  • AIT-3 tape usage status

Posted by at 10:13 AM | Permalink

Admin

• Backup:
  • storage node crashed and burned in the night...get it back up and running
• LDAP:
  • Send out account removal warnings to accounts

Posted by at 10:46 AM | Permalink

Admin

• Backup:
  • clean that tape drive as it is finally empty this morning
  • open issue with EMC about tape that was marked as "unlabeled" when NetWorker failed to read the label
  • Fix tape problem by forcing an inventory of the tape in a different drive
• LDAP - bad entry from PeopleSoft

Posted by at 8:15 AM | Permalink

Admin

• Backup:
  • monitor -- tape drive needs cleaning
  • fix broken labels on two LTO-4 tapes

Posted by at 8:13 AM | Permalink

Admin

• LDAP:
  • Correct error from nightly update
  • Search for (and find 24) of the BCA's that were not previously in LDAP
• Backup:
  • Calculate how long the AIT-3 tape supply will last
  • Cancel staging command, it was holding up backups!
  • Review COMIS backup proposal

Posted by at 8:34 AM | Permalink

Admin

• Accounts: fix scripts that were not doing passwords correctly
• Backup: Design answer to yesterdays request about u06
• LDAP: build openldap 2.3.37 rpm for RHEL4

Posted by at 9:56 AM | Permalink

Admin

• LDAP: Privacy and nightly update errors
• Backup: Meeting with PeopleSoft folks about their needs
• WebCT: License activation and SSL certificate

Posted by at 8:53 AM | Permalink

Admin

• LDAP: Privacy Flags
• Backup: More examination of backup usage
• WebCT: work on new server

Posted by at 9:12 AM | Permalink

Admin

• LDAP: Privacy Flags
• Backup: analysis of data

Posted by at 12:18 PM | Permalink

Admin

• LDAP:
  • Privacy Flags...
  • CatCard special extract file
  • Sendmail alias maintenance
• Backup:
  • Examine the NSR_REPORTER output from last night's run
  • Consider running the NSR_REPORTER daily...
  • Open a new issue about /nsr/tmp problems in 7.3.2 with EMC
• Calendar:
  • Start building the VMWare Guest to be the new server
• WebCT:
  • Start building the VMWare Guest to be the fall server

Posted by at 9:10 AM | Permalink

Admin

• Backup -- another issue with NetWorker 7.3.2.... SURPRISE!
• LDAP -- New retiree list confused the report generator and required code modification
• LDAP -- Privacy flags

Posted by at 9:08 AM | Permalink

Admin

• Upgrade footprints service to 7.5a1
• Reboot the calendar server to pick up new SSL certificate
• LDAP errors to be cleaned up

Posted by at 6:49 AM | Permalink

Admin

• Backup:
  • Another disk on storage node not staging correctly. Work on 19234660
  • Tweak jukebox config -- specify idle timeout value of 10 instead of the default of 0 so tapes that are done being used will be automatically unmounted and available on other storage nodes
• Start office move

Posted by at 2:49 PM | Permalink

Admin

• Backup:
  • Work on issue 19234660
  • Tweak new configuration
• LDAP: catch up on privacy requests from last week and update errors that needed fixing

Posted by at 2:47 PM | Permalink

Admin

• New backup hardware install and configure as well as backup software problems just for giggles

Posted by at 2:46 PM | Permalink

TODO

• LDAP: deal with update errors from Dienstag and Donnerstag
• Backup: Firewalls and XLS delivery
• Xi-Text: Figure out HP LJ 3600n problem
• Xi-Text: Figure out HP DJ 6127 printing

Posted by at 4:17 PM | Permalink

Admin

• Backup:
  • SATABeast disks installed, powered up, connected to storage node
  • Finished loading OS on new storage nodes
  • Loaded new enablement codes into NetWorker
• Phone: lots of backlog dealt with
• Team Meeting

Posted by at 4:14 PM | Permalink

Admin

• Backup:
  • Change schedules to delay one week fulls that happen this weekend
  • Email sent to EMC Licensing about the LAC and how it isn't working
  • Download Solaris packages and patches for the new FC card to be installed on Monday
  • Printout and fill out Vendor Parking Permit apps for next week
  • Rack new equipment
  • Schedule DataDomain service for 7:30am June 25

Posted by at 7:17 AM | Permalink

Admin

• Xi-Text:
  • Suggestions for dealing with hp dj 6127 printer
  • HP DJ 6127 printer still not working
  • Attempt to define another printer, but it appears to be powered off
• Backup:
  • Check that save groups actually worked as required... they did!
  • Take delivery of new storage nodes
  • Take delivery of new storage array
  • Shipping Company picks up Data Domain chassis
  • SOW meeting
• Update samba config for new users and moving users

Posted by at 8:46 AM | Permalink

Admin

• Xi-Text support -- two printers having issues... working with vendor
• LDAP: clean up errors from while I was out
• Backup: continues to be problematic. Scheduled full-save's did not start as scheduled over the weekend, they had to be manually started when I discovered the problem Saturday afternoon.

Posted by at 12:23 PM | Permalink

Admin

• Backup:
  • Update EMC on status of open issues
  • 18976574: EMC calls back. I ignored his advice because I just didn't feel he was comprehending what I was saying and found that contrary to what I was told, by changing the old client definition to specify the filesystems instead of All and put both definitions into the same save group, only one bootstrap was created. Issue closed.
  • 18972720: EMC emails -- this is a bug, working to confirm the bug
• DNS: vpn address change (late!)

Posted by at 9:33 AM | Permalink

Admin

• Backup:
  • Sessions that have long ago completed are still listed in the Monitoring->Sessions tab of NMC... related to 19014920?
  • Groups still cloning and 100% done but no movement...
  • Give up on EMC ever calling back about 19014920 and open 19036762 because I see that it is aborting inactive jobs and then is waiting for them to complete!
  • EMC called back and had me abort (stop and then kill -9) the two save groups that were hung. Hopefully, they will work tonight.

Posted by at 7:40 AM | Permalink

Admin

• Backup:
  • Couple "clones" that are just not moving forward.
  • Three INCR's that don't clone that are 100% done but not moving forward
  • Email from EMC indicates the parallelism of the nsrserverhost is used for indexes in 7.3.2 (news to me!)... increase parallelism from 1 to 50 for nsrserverhost
  • nsrwatch... hmmm have to look that one up
  • No resolution
  • 19:45-21:45: on-site tape pool ran out of empty tapes, add 20 more

Posted by at 7:38 AM | Permalink

Admin

• Backup:
  • It's broke! NMC won't connect, gstd is running, clients cannot start saves
  • Open 19014920 ... Severity 1 (1 hour call back that takes 1 hour 15 minutes)
  • Kill -9 all the savegrp's, nsrd, and several nsrmmd's (Ugh!)
  • reboot ozzie.... it hangs.... NFS problem because of ipf being misconfigured to not allow traffic out the private ethernet interface. EMC thinks this is the problem -- it's not.
  • two "nsrim -X's" , nsrjb -HEv, nsrjb -Iv .. and we're back in business.... maybe
  • (that's 4.5 hours of work)
  • Check later... seems to be running.... at least NMC is responding and things look ok

Posted by at 6:34 PM | Permalink

Admin

• LDAP:
  • lots of privacy flags
  • Deal with issues from nightly update
• Backup: Meeting about space
• FootPrints: Test server set up and announced
• Xi-Text: Add a printer for cnhs

Posted by at 2:23 PM | Permalink

Admin

• Backup:
  • gstd crashing... 18988282
    • dbsrv9 process left hanging
    • Caused by Media->Save Set search for particular client for manual cloning
    • Default pool needed -- messed up config of new client
  • LDAP: privacy flags and a manual merge of info

  Posted by at 9:53 AM | Permalink

Admin

• LDAP:
  • Privacy Flags
  • Several update issues from over the weekend!
  • Report for CatCard office
  • Approve new feed format from Banner
• Backup:
  • Open issue with aborted clones showing up in the Media->Save Set section of NMC. Looks like it might be a bug
  • Ask PeopleSoft DBA's some questions about the RMAN problems they are having -- prior to opening an issue with EMC about the RMAN saves that are getting stuck
  • Open issue with EMC about how to merge the nsrserverhost clients that backup most things and the NetWorker Console Database into a single client
  • Empty off the final Indexes tape volume
• Hiring: Read more resumes
• General: Search for email messages that never arrived

Posted by at 12:28 PM | Permalink

Admin

• LDAP: Privacy flags
• Hiring: Resume reading
• Backup: configure new library client to replace old one removed

Posted by at 7:27 PM | Permalink

Admin

• Backup
  • Tape Libary disabled by apparent operator accident
  • Sessions getting stuck in NMC display although they really finished long ago
  • Consult on finalization of tape library order
• LDAP: Several privacy flag settings
• FootPrints: testing fails. Will need to create new virtual machine because of the embedded perl version and its hard coded paths

Posted by at 6:41 PM | Permalink

Admin

• Accounts: Send eMail message to accounts to be removed
• Backup:
  • Discovered NMC reporting created its own backup client entry! Research how to merge it into the main nsrserverhost backup entry
  • Give directions on how to change the tape orders around
• LDAP:
  • Privacy flags
  • Paper Telephone Directory test extract
  • Monitoring for catastrophic failure indications of the LDAP->AD update process

Posted by at 7:23 AM | Permalink

Admin

• Backup:
  • Keep track of new version
  • Figure out how the new NMC works for deposit/withdrawal/inventory
  • Apply build 399 to fix problems found in build 368
• LDAP:
  • Write reports to feed out the new Banner ID numbers
  • Work through why not all first year student's courses are available

Posted by at 7:21 PM | Permalink

Admin

• LDAP:
  • Replica turned off on APPMS
  • Privacy flags checked/updated per requests
  • Test server performance graphs disabled
• Backup:
  • Finish plans for today's upgrade
  • Change file server backup configuration to even out the weekends in the month

Posted by at 8:17 AM | Permalink

Admin

• Backup:
  • Tape wrangling... 47 tapes to get out and 23 to get in!
  • 7.3.2 upgrade questions -- NMC installation... read, read, read
  • Meeting about backup improvements
• LDAP:
  • Questions about getting info out from BSAD
  • Questions about changing a CN value
• FootPrints:
  • Request to purge an entry from PeopleSoft help desk project
  • Obtain license for test server
  • Start process of upgrading test server to 7.5

Posted by at 10:23 AM | Permalink

Admin

• Backup: Label more offsite tapes... hungry this weekend!

Posted by at 10:22 AM | Permalink

Admin

• Backup:
  • Label offsite tapes as we ran out
  • create new indexes volume to solve hang condition
    • Hmm... perhaps the hang condition was the RO side wasn't included in the media pool... DOH!

Posted by at 10:19 AM | Permalink

Admin

• FootPrints: duplicate for testing

Posted by at 10:18 AM | Permalink

Admin

• Backup:
  • tape wrangling
  • Library client disabled -- per request
  • eMail backup disk filling up!
  • New tape library question posed to EMC

Posted by at 11:10 AM | Permalink

Admin

• Too busy to make a blog entry!
• Backup:
  • Tape Wrangling...
  • Indexes adv_file device goes to sleep... requires networker recycle to wake up

Posted by at 11:59 PM | Permalink

Admin

• Backup:
  • Tape Wrangling -- 10 more staged clean
  • Remove three unused Media Pools
  • Start staging data off 13 more tapes
  • Found the 5 tapes that had been missing -- they were in the stack of empty tape cartridge boxes /sigh
• LDAP:
  • PeopleSoft student employee error -- found and combined
  • Privacy flags -- confused department members requesting the same privacy level they already have ... lots of looking and time wasting!
• Accounts: Yesterday was the annual 6K admitted students that didn't accept UVM's offer get purged (took all day to purge the accounts). Today was the annual expire the graduated students with lots of internal changes and pages overnight about the LDAP replica's being slow to update

Posted by at 9:38 AM | Permalink

Admin

• Backup:
  • Tape wrangling
  • Library pricing
• Accounts: 6,048 admitted students removed ... log run!

Posted by at 9:10 AM | Permalink

Admin

• Backup:
  • tape wrangling
  • library meeting

Posted by at 1:38 PM | Permalink

Admin

• LDAP: Update to clean up timeing logs instead of leaving them hanging around to trip up future updates
• FootPrints: Actually set up for 7.5 testing
• Backup:
  • Get permission to upgrade storage node

Posted by at 8:12 AM | Permalink

Admin

• Backup:
  • DD460B crashing due to kernel panic. Upgraded to newest GA kernel to fix
  • DD Case 24051: New Chassis should arrive this afternoon
• Move DHCP cgi scripts from old machine about to be killed off to scripts machine

Posted by at 11:12 AM | Permalink

Admin

• WebCT: Maintenance applied in one hour -- a record I think
• Backup:
  • testing of tape with 100GB file continues
  • Preparing for moving the DD200 to the nsrserverhost to use for Indexes
  • Emailing vendor about prices... Yeesh they are high
  • Move DD200 -- and the 10/100/1000 port suddenly doesn't work anymore
• FootPrints: Setting up for testing 7.5
• Licensing: restart license server process that had 1/3 died

Posted by at 8:14 AM | Permalink

Admin

• Calendar: calendar server disk space over used -- remove old unison directory that is no longer required
• Accounts:
  • several trials to check new procedure -- No Failures
  • Update 2007 schedule -- and notify community
• Backup:
  • clean up the dd200 in preparation for moving it to the nsrserverhost
  • change storage node list for two clients to reduce impact of suspect drive
  • Meeting with PeopleSoft support team to discuss backup

Posted by at 8:11 AM | Permalink

Admin

• Backup: clean up old volumes that have nothing on them anymore
• Accounts: Research why Net::SMTP is failing to deliver messages
• VPN: document (ha!) vpnc on Ubuntu 7.04...
• Solaris: Download and test May 11, 2007 patch set

Posted by at 11:19 AM | Permalink

Admin

• Backup:
  • Consolidate tapes
• LDAP: privacy setting
• FootPrints: install new group.html file for BCOR
• eMail: correct another "mail\" config problem

Posted by at 9:29 AM | Permalink

Admin

• Backup:
  • Meeting with vendor about possible backup hardware
  • Copy good data from one tape to another to free up under-utilized tapes for reuse

Posted by at 3:34 PM | Permalink

Personal

• Information Week sends a link to an article at Aviva entitled Blog Law: 12 Important U.S. Laws Every Blogger Needs to Know -- Good article!

Admin

• Backup:
  • Remove okapi from backup definitions -- it was removed from service over the weekend
  • Send message to PeopleSoft support folks asking about one of their machines that is not responding to the backup system
  • Ran out of blank tapes... need to empty some off to get the space
• LDAP: Set up an employee privacy level per request

Posted by at 8:04 AM | Permalink

Admin

• Backup:
  • PeopleSoft recover requires tapes from the vault -- schedule meeting to understand how this offsite media pool is expected to behave.
  • Cloning of data (manual because auto failed) continues
• Footprints: old userid that should have been removed reappears as a customer and causes problems in the PeopleSoft Help Desk project

Posted by at 8:08 AM | Permalink

Admin

• So busy I forgot to put this in on time! (yikes)
• LDAP: set a privacy level
• Backup:
  • Retrieve on-site tape that was not in the TLS to allow a student to recover their home directory after it was eaten by a computer in the library
  • Replace failed tape drive
  • Re-Cable TLS to put robot on its own SCSI bus -- trippled the performance of the first tape drive!
  • Discover that jbconfig does not work the way EMC support told me -- it refused to run because the TLS was already configured -- Luckily, we had (I needed help) tricked Solaris into assigning the two tape drives from the new bus the old device names
  • Discover that I really do not understand how nsrjb works

Posted by at 7:58 AM | Permalink

Admin

• Backup:
  • Check on status and put LT6 back into service
  • email about index storage
  • Monitor clones
  • Close EMC case about staging -- no clue why it refused to work, but we worked around the problem. Good thing we use compressing disk so creating a new adv_file was not an issue
• LDAP: Banner "BCA" entry needs special handling
• WebCT: email and maintenance schedule

Posted by at 10:15 AM | Permalink

Admin

• Accounts: Third notice to 6,559 accounts of removal on June 1
• Accounts: New "welcome email" placed in production
• Backup:
  • removal of three month old data that retention rules changed after it was saved
  • removal of old adv_files/devices that no longer have any valid data on them

Posted by at 1:53 PM | Permalink

Admin

• LDAP: help determine search being performed by webmail why it is slow
• Account: Update "welcome message" in account creation script
• Backup:
  • lots of assistance to the process
  • discussion about saving indexes

Posted by at 12:38 PM | Permalink

Admin

• LDAP:
  • Respond to email about UVM ID Number
  • Fix logic error assuming uvmexpdate would exist for expired entries
  • Update ldap entry to give it a uvmexpdate value
• Backup:
  • Generate list of index tapes that have little or NO information on them

Posted by at 11:55 AM | Permalink

Admin

• Backup:
  • Work on emptying more underutilized index tapes
  • Test (sonytape) one of the tape drives thoroughly (-x)
  • QualStar says to look through the config manual for Linux
• LDAP: code (logic) error in ldap_report.pl

Posted by at 8:39 AM | Permalink

Admin

• Backup:
  • Insert 7 tapes into library needed for a homedirectory restoration
  • Insert 5 tapes into library to compact their information onto new tape
• LDAP:
  • PeopleSoft feed included a new phone number that was illegal via LDAP syntax
  • Modified the PeopleSoft perl module to enforce the RFC2252 required syntax for telephone numbers

Posted by at 9:07 AM | Permalink

Admin

• Backup:
  • Respond to second phone call and first email from EMC about the issues I opened yesterday morning and requested be done via email
• LDAP:
  • Set two privacy flags
  • Examine error in nightly update
• Create a blog

Posted by at 7:19 AM | Permalink

Admin

• Backup:
  • drive 3 (2cbn) in service mode AGAIN... sonytape command refuses to run, info gathered for QualStar
  • Staging commands issued... but server is too busy to respond?
  • Opened SR's with EMC about staging and storage node balancing
  • Downloaded new copies of linux and solaris sonytape command from QualStar

Posted by at 8:40 AM | Permalink

Admin

• LDAP: three BCA's from Banner that do not match anything already in LDAP
• Backup: more kicking to get things to move when they should...
• Backup: Fixing things my kicking broke... defined tape drives for the clone pool because it suddenly wanted tapes mounted on storage nodes that didn't have tape drives!
• FootPrints: work a couple of issues through

Posted by at 8:18 AM | Permalink

Admin

• Another day consumed by backup

Posted by at 8:18 PM | Permalink

Admin

• Backup consumed the day

Posted by at 8:16 PM | Permalink

Admin

• LDAP: Add privacy flag to new employee from PeopleSoft
• Backup: Put two problem machines on the correct storage node
• Backup: lots and lots of infernal little time wasting problems

Posted by at 7:28 AM | Permalink

Admin

• mail... being out for four days, there was quite a bit to get through
• Backup: hmmm...
• footprints: recover INBOX from Thursday as requested

Posted by at 12:16 PM | Permalink

Admin

• Backup:
  • phone conversation with Mr. G. about philosophy
  • UVM Pool ran out of tapes -- discuss with Cheryl
• LDAP:
  • bandaid the report script to igore the reslife test account
  • Check on more non-authoritative feed info from PS and Banner
• Samba issue on one of the OAS servers -- turns out the target directory was missing (oops)

Posted by at 9:22 AM | Permalink

Admin

• Backup:
  • make a couple of tweaks to config
  • gather saveset id's for failed clone in January
• Calendar: rename a resource in the LDAP directory... should really make a tool to do this for Account Services
• LDAP:
  • Help ResLife with a failed application -- I broke it when I removed an old entry
  • Work on the UVM Directory application get ready to implement requested change

Posted by at 8:52 AM | Permalink

Admin

• Backup: everything looks good.... is that a bad sign?
• LDAP: Examining the report section... it needs some updates
• LDAP: banner affiliations information gathering
• Calendar: Iron out the 10g standalone upgrade instructions
• Calendar: Convert the nagios status check to a passive (nsca) check to keep sudo messages out of the daily log report

Posted by at 1:51 PM | Permalink

Admin

• Backup:
  • Hung tape -- find and kill the process owning the tape drive to free it up
  • Giraffe clones failed -- notify operators of 71 savesets that need to be manually cloned
  • hyperion10 system ASR failed to backup -- RPC errors --
  • Configuration changes to move toward completion of the "UVM Indexes" pool migration
• LDAP
  • Reset privacy flag on an entry that should never be set
  • Discussion about Banner affiliation for course providers

Posted by at 8:27 AM | Permalink

Admin

• Backup --
  • hyperion-db1 --- RPC errors turned out to be someone had not properly created the PTR entry in the nameserver updates.
  • Lots of tape problems
• Nagios: turn log checks into passive becuase nrpe can't handle must work first time checks
• Tech Park: Inspect electrical work in Knürr cabinets

Posted by at 6:25 PM | Permalink

Admin

• Backup
  • After much work with EMC to figure out the problem -- discover that the IBM Director Agent had been installed but the machine never rebooted until I rebooted it last Friday.
  • Remove the IBM Director Agent RPM's and reboot
  • Run up and down the hall screaming because the tape drives work again
  • Close issue with EMC
• VACC: create new accounts as requested a week ago

Posted by at 3:19 PM | Permalink

Admin

• Backup:
  • apply maintenance to QualStar
  • reboot storage node... oh no! Now it can't use tapes either!!!
  • lots of trying but no solution!

Posted by at 3:17 PM | Permalink

Admin

• Backup: Day two of two day traingin session with consultant
• Backup: Determine maintenance procedure for QualStar for Friday

Posted by at 3:16 PM | Permalink

Admin

• Backup: first day of two day training session with consultant

Posted by at 3:15 PM | Permalink

Admin

• Backup: more training... ns1 failure... firewall problem
• LDAP: Update translation table for PS EmplClass values per request from HR.

Posted by at 10:31 AM | Permalink

Admin

• FootPrints: remove Kent :'(
• Calendar: Install 9.0.4.2.13 on test system

Posted by at 9:48 AM | Permalink

Admin

• LDAP: A newly hired faculty member shows up in Banner feed, but newhires are not provided from PeopleSoft yet... argh!
• InfoEd: Set up Softerra LDAPBrowser (oh the trouble of generating an old Netscape 4.x cert7.db file!!!) to test SSL to UVM's LDAP servers from the test system.
• Calendar: patch 5117392 -- 9.0.4.2.13... read the doc

Posted by at 2:05 PM | Permalink

Admin

• LDAP:
  • Clean up a couple of account errors in the nightly update that were caused by the DCE/DFS client conversion on the fileserver yesterday
  • InfoEd: look at W/2003 Server SP1 and LDAP commands
• VMWare: convert the test calendar server from physical to virtual -- P2V!

Posted by at 12:22 PM | Permalink

Admin

• LDAP: new faculty member from Banner needs account created
• DCE: convert fileserver to dce client removing dependence on local passwd file

Posted by at 10:19 AM | Permalink

Admin

• LDAP: Banner feed change results in a new batch of "Faculty" that do not match existing employee -- special run to allow 171 to be created
• Accounts: work on scripts to remove dependence on local password file

Posted by at 9:30 AM | Permalink

Admin

• LDAP: refint overlay -- working out how to deploy
• PH2LDAP: configure the fourth ldap.uvm.edu server into the DNS RR for PH
• OS Maint: boot mac-mini into all OS's and apply all updates required

Posted by at 10:32 AM | Permalink

Admin

• IDM: fix a duplication
• LDAP: experiment with the referential integrity overlay
• DST: reboot telecom servers for DST changes

Posted by at 10:08 AM | Permalink

Admin

• Calendar: audit and update mail attributes in calendar server to match the UVM Directory for 1370 people.

Posted by at 11:05 AM | Permalink

Admin

• Calendar: Perform rudimentary audit and provide output to Account Services of calendar accounts that have no matching UVM netid
• Accounts: write update to account scripts to check for account in Active Directory to prevent conflicts
• Calendar: p5908306 was re-released yesterday... check and find that the README.html was modified, but the timezone file is the same as what I applied last Friday.

Posted by at 9:06 AM | Permalink

Admin

• LDAP...
  • bandaid the Banner feeds Faculty report problems
  • Look at ACL upgrades
• Calendar: Install Oracle Patch 5862386 -- A new timezone fix

Posted by at 2:08 PM | Permalink

Admin

• LDAP....
  • Continuing saga of the DN change proposal
  • Banner feed includes Faculty and there are 342+ non-authoritative issues to deal with
• Nagios log monitor on Solaris... got some problems... Jim points out some changes

Posted by at 12:48 PM | Permalink

Admin

• LDAP
  • adjust a couple of applications to not be dependent on the format of the general DN
  • Open ETS-TECH discussion about proposed change to DN format
  • Alert COM, CESS, and CEMS of the proposed change to be sure their applications will not be affected
  • COM responds that their applications will not be affected

Posted by at 10:00 AM | Permalink

Admin

• LDAP: Banner feed testing, again
• LDAP: DN change testing
• LDAP: back-config support for unique overlay
• LDAP: PeopleSoft data feed investigation
• WebCT: Assist with testing on linux

Posted by at 2:52 PM | Permalink

Admin

• Accounts: really old account requests password reset
• LDAP: work on DN change code
• LDAP: new UVMLDAP.pm installed -- and a couple of uses fixed for the new usage/path

Posted by at 8:02 AM | Permalink

Admin

• VMWARE Guests: Install kernel fix on rh4-build guest.
• Accounts: Start the 2007 deletion process for accounts of people who have not been affiliated with UVM for at least a year in June.
• LDAP: Begin work using Net::LDAPapi

Posted by at 10:19 AM | Permalink

Admin

• LDAP: peoplesoft graduate student employee needed to be manually found in the banner feed.
• LDAP: UVM LDAP module that uses Net::LDAP updated to find DN instead of assuming how it is constructed.

Posted by at 8:54 AM | Permalink

Admin

• EMail quota problem -- FP Issue
• OpenLDAP - 2.3.34 rpm test/build
• LDAP - new banner feed testing
• MAZE - Registration

Of Interest

• Learn to talk like Yoda

Posted by at 9:17 AM | Permalink

Admin

• RPM: testing out new build styles (--define option to rpmbuild)
• LDAP: Build Net::LDAPapi perl module which uses the C api and is much faster than the all perl Net::LDAP module(s)
• LDAP: replicas out of sync with the RW server with respect to the sendmail spammy access blocking rules. Not certain of the reason. It took forty (40) minutes to determine what needed to be fixed on each of the replica servers and make the updates. Need to start doing a periodic (aka weekly/monthly) check of this

Posted by at 12:00 PM | Permalink

Admin

• Write perl script to keep DCE group updated based on LDAP departmental employees

Posted by at 10:21 AM | Permalink

Admin

• Made it to work!
• LDAP: examining last quarter of logs for malicious activity

Interesting

• For those that have to do helpline... "Introducing the book" -- a confused customer calls for help with a book

Posted by at 8:34 AM | Permalink

Admin

• Space on SIS system... notify owner
• NetReg FP issue... sounds like a local computer problem... but we'll go with the flow for now
• LDAP: SIS feed changes driven development

Posted by at 8:50 AM | Permalink

Hmmm

• E-Mail management....
• Network problems...

Posted by at 8:49 PM | Permalink

Admin

• WebCT 4.1: Deal with issues installer found
  • mkfs with 1 inode for each 4k of filesystem
  • certified on RHEL/AS 2.1 -- faq 19, it should work on RHEL/AS 4
• LDAP: Update CatCard feed process code to drop inactive entries in the feed file

Posted by at 11:03 AM | Permalink

Admin

• WebCT: meeting to start testing CE 4.1 on RHEL4 VMWare Guest
• Document (wiki) RHEL4 kernel UVM VMFIX installation for VMWare guests
• LDAP: More work on the new catcard feed (to LDAP) processing

Posted by at 9:26 AM | Permalink

Admin

• LDAP: PeopleSoft account for Graduate Student Employee -- no matching Banner feed. Verified (via FootPrint entry) and created.
• HTTP: VirtualHost defined.
• LDAP: Privacy flag set to indicate desires of twenty-one people
• FootPrints: Recover lost INBOX from a week ago
• LDAP: More work on the new CatCard feed files

Posted by at 10:22 AM | Permalink

Admin

• LDAP: Find and merge PeopleSoft info into Banner based entry -- made more difficult because the two SOR's disagree about the student's National ID Number
• LDAP: New CatCard feed work
  • New feed data provided to CatCard
  • Start work on ability to process New feed from CatCard to LDAP
  • Oops... found an error in the CSV feed file provided

Posted by at 8:49 AM | Permalink

Admin

• LDAP: New code development for CatCard and multiple ID's
• LDAP: Old wolverine and carcajou hardware DBAN'd and unplugged

Posted by at 9:44 AM | Permalink

Admin

• LDAP: Chase catcard feed failure...
  • openssh does not like the home directory to have group write permissions
• LDAP: Chase why slapd does not start in VMWare
  • runuser works but uldap does not... what???
  • Aha! runuser only works because the test specified the shell. uldap was defaulting to the ldap user's shell (/bin/false) and failing
  • Generate openldap-2.3.33-2.uvm to fix!
• UPS: full Preventative Maintenance 5pm-7:30pm
• LDAP: Upgrade wolverine to openldap 2.3.33-2.uvm and VMWare kernel fix

Posted by at 4:15 PM | Permalink

Admin

• Put the RHEL4 uvmvmfix kernel on a few more VMWare RHEL4 guest systems
• Help with several FP issues related to email problems
• Document how to install solaris maintenance in the TSG Wiki space
• Order grounding kits from WrightLine for the racks in Tech Park

Posted by at 11:26 AM | Permalink

Admin

• Chase the gremlins left over from the SAN hardware failure of Montag.
• Build RHEL4 kernel with fix for RedHat Bugzilla 197158
• Install that new kernel on carcajou to verify it actually functions

Posted by at 8:59 PM | Permalink

Admin

• LDAP: renamed account still in files -- somehow it wasn't properly cleaned up when the account was renamed on Friday. Clean up the files
• LDAP: new CatCard feed code development
• UPS: Schedule with vendor for major PM on Donnerstag (Thursday) 5pm(ish)
• WrightLine: replace broken monitor support arm

Posted by at 9:21 AM | Permalink

Admin

• WrightLine side panel installation at Tech Park
• Meeting with project manager about VACC move to Tech Park

Posted by at 9:19 AM | Permalink

Admin

• UPS failure... working with UVM Electrical and the service provider to get the UPS back after it claimed a blown fuse during the weekly generator test
• Blog:
  • Finally track down the problem to uidnumbers below 500 being the problem.
  • Rebuilt the apache rpm so the suexec command has the proper config to allow our uidnumbers to work

Posted by at 9:18 AM | Permalink

Admin

• Blog:
  • Replaced the "mt/mt.cgi" file with a simple perl script to basically say "hello" and it still fails -- it has to be an Apache problem
• virusscan: Updated to have DCE credentials
• CMD FP Issue -- student with lost iMovieProject
• DHCP/DNS config for echo

On-Call

• LDAP: catcard system is being upgraded and delivered a 0 entry file which caused all id card values to be wiped from LDAP. Went back to yesterday's file and restored them. --- 3.5 hours CTE

Posted by at 9:39 AM | Permalink

Admin

• New Blog... yippee!
• shutdown old (now unused) gameserver
• Reseat blade 10 to clear a spurious memory error!
• Set up console for last web server
• Blog:
  • make backup of the OLD server entries
  • delete my blog
  • create a new blog for me
  • Same Internal Error going to log in page! :(
• Listserv move ... dns and mail routing changes: CTE 1.5 hours

On-Call

Posted by at 10:51 AM | Permalink

Admin

• CGI server -- undefined machine check error!
  • 888 102 207 0c0
  • Power cycle
  • ERRPT calls out L2Cache0 FRU: 09P5763 P1
• Blog server -- can't get in to update my blog
  • Something wrong in MT with the server move
  • Affects 10 blogs out of the 387 I tested
• Moose filesystems moved
  • Update regular (10 minute interval) cronjob that checked filesystems to ignore the ones that are no longer there
  • Update the account creation scripts to not attempt to use a filesystem that no longer exists
  • Update gnu/elk/gnunew to use the central syslog server instead of moose
  • Move nameserv home directory to consolidate space
• Check dce group ownership for a customer
• LDAP: webmail permissions problem
  • Update acl's to allow new webmail server access
  • Update schema for Banner feed change
  • Roll out OpenLDAP 2.3.33
  • CTE: 1.5 hours
• WrightLine: chasing side panel problem

Posted by at 10:55 AM | Permalink

Admin

• Report inconsistent network performance on VPN from Waterman Cafe

Posted by at 8:21 AM | Permalink

Admin

• LDAP: cat card support development

Notes

• http://www.howtoforge.com/perfect_setup_ubuntu_6.06

Posted by at 8:16 AM | Permalink

Admin

• LDAP: find a graduate student to match with the PeopleSoft entry that came over without the proper information in it
• LDAP: test feed to new listserve machine
• Virtual Host: setting up a new virtual hosting environment

Posted by at 8:43 AM | Permalink

Admin

• Calendar SDK
  • SDK Support Forum entry unanswered after 24 hours!
  • Opened SR with Oracle
  • While waiting for Oracle to respond to SR, found 9.0.4.2.13 patch and downloaded it.
  • 9.0.4.2.13 included new CAPI libraries -- that fixed the problem!
  • Close SR, update Forum entry with fix

Posted by at 3:36 PM | Permalink

Admin

• Tech Park: Meeting/Tour
• Calendar: SDK CAPI_FetchEventsByRange returning CAPI_STAT_DATA_ICAL_COMPVALUE (AKA: I don't like something but I'm not going to tell you what!)
  • Turns out it is the Holidays... but deleting one and recreating it causes more problems.
  • Temp fix to exclude Holidays, Day Events, and Day Notes from extract

Posted by at 3:34 PM | Permalink

Admin

• LDAP: Banner adds NID to entry and we discover a duplicate with an expired entry... abfuscate the expired entry and leave the new one alone (neither one has ever been activated)
• LDAP: Look at test Banner extract that includes faculty -- oops, multiple views of the same record! How to deal with that???
• MileStone: This is the 400th blog entry!

Posted by at 8:31 AM | Permalink

Admin

• LDAP: check out systems after long break
• VMWare: guest clocks

Posted by at 8:08 AM | Permalink

Admin

• LDAP: moving test LDAP server into VMWare
• SecurID: apply security maintenance to the servers

Posted by at 12:02 PM | Permalink

Admin

• WebCT: maintenance
• LDAP: work on moving test server into VMWare

Posted by at 12:01 PM | Permalink

Admin

• Calendar: prepare for the timezone fix
• WebCT: Prepare for the maintenance day
• LDAP: Announce ldap update server move for 27 Dec

Posted by at 2:09 PM | Permalink

Admin

• LDAP: manual merge -- PS ssn change triggers conflict with old account

Posted by at 9:18 AM | Permalink

Admin

• LDAP: Identity Management work continues

Posted by at 8:23 AM | Permalink

Admin

• LDAP: Identity Management...
• Accounts: password resets and account activations messed up... performing manual resets while others fix the problem
• WebCT: testing solaris maintenance on test machine

Posted by at 10:13 AM | Permalink

Admin

• LDAP: Identity Management work... oy this is complex
• Accounts: working on "consultant procedure"
• WebCT: get 12/7 Solaris maintenance pack

Posted by at 10:12 AM | Permalink

Admin

• Telcom: reboot servers to avoid Oracle 284 day bug
• LDAP: Identity Management coding (more work on the join)

Posted by at 1:36 PM | Permalink

Admin

• LDAP: IdM coding... the join ...

Posted by at 1:55 PM | Permalink

Admin

• zebra: moved power to different UPS so the failed battery can be replaced
• Accounts: re-wrote the mkacct script (adduser and usertypes2.pl) to be able to create accounts that are not on the local filesystems
• LDAP: updated to handle the Business Unit being on the PeopleSoft job attribute... not being used for anything yet

Posted by at 2:41 PM | Permalink

Admin

• LDAP: ldaprw replacement copying
• Accounts: interim plan for storage
• Tech Park Power: PDU discussion
• FP: Issue 23211 Proj 4 -- email delivery check

Posted by at 11:48 AM | Permalink

Admin

• LDAP: one person, two UVM CatCard barcode numbers...

Posted by at 8:48 AM | Permalink

Admin

• LDAP: work on new ldaprw machine
• Tech Park: more discussion on power
• SecurID: Enhancement Request ID 44306 (RHEL3/ACE6.1.1 supported on VMWare ESX3) opened

Posted by at 11:13 AM | Permalink

Admin

• Tech Park: meeting about power
• LDAP: work on new ldaprw machine
• SecurID: investigate apparent unclean shutdown

Posted by at 1:39 PM | Permalink

Admin

• LDAP:
  • build openldap 2.3.30 RPMs
  • Install and test on carcajou

Posted by at 10:41 AM | Permalink

Admin

• SecurID: solaris servers turned off and service moved to new virtual Linux servers

Posted by at 11:04 AM | Permalink

Admin

• Securid: Lots of prep work for move
• LDAP: upgrade servers to latest OS fixes

Posted by at 9:03 AM | Permalink

Admin

• LDAP: Fix a bug in IdM stuff ... duplicate attribute value for some reason
• WebCT: reset application password for application administrator
• SecurID: reboot replica server... it had hung processes... damned RSA!
• LDAP: BenchMark a virtual server (~94 ops/s) and a blade server (~157 ops/s)
• SecurID: Download AuthMgr 6.1 code for Linux. Oops -- doesn't support RHEL4, open another issue with RSA

Posted by at 9:18 AM | Permalink

Admin

• LDAP: apparent data entry error (which resulted in a duplicate ID number) reported to Registrar
• LDAP: IdM work -- another bug found in code
• VMWare: data collection scripts installed on two LDAP servers to see if they are good candidates for virtualization
• Accounts: temporary creation script changes to reduce size of current problem

Posted by at 10:58 AM | Permalink

Admin

• Oracle: samba password change on test machine
• LDAP: manual merge of two accounts -- one expired
• VMWare: discuss with LRG staff about submitting requirements
• WebCT: discuss Linux replacement of Solaris server

Posted by at 9:28 AM | Permalink

Admin

• LDAP: investigate a manual merge demand... a bug..
• Clean up old desktop machine for reuse.

Posted by at 9:27 AM | Permalink

Admin

• Systems: RHEL3-U8 (i386) DVD creation

Posted by at 9:44 AM | Permalink

Admin

• LDAP: Review list of possible duplicates. Found and fixed two. Found two pairs that might be and requested assistance from Registrar and HR offices
• LDAP: HR determined these were duplicates as well -- corrected.
• LDAP: Update Active Directory feed to send departmental accounts over as well

Posted by at 9:09 AM | Permalink

Admin

• LDAP: Two "non-authoritative" failures. Found existing entries that matched and merged the information in.

Posted by at 8:31 AM | Permalink

Admin

• Blogs: create a couple for people
• LDAP: IdM

Posted by at 9:09 AM | Permalink

Admin

• LDAP: update complains about an account having been removed. Tracked it down to a manual removal that did not update all the files. Fixed the files.
• LDAP: IdM
• eMail: recover old INBOX files

Posted by at 8:38 AM | Permalink

Admin

• LDAP: multiple accounts for a single person discovered -- manually fixed and issue opened with PeopleSoft to alert them to the change

Posted by at 8:14 AM | Permalink

Admin

• LDAP: IdM work

Posted by at 10:04 AM | Permalink

Admin

• Email: Recover inbox for customer because HelpLine staff blew it
• LDAP: IdM processing

Posted by at 8:16 AM | Permalink

Admin

• LDAP: Inform PeopleSoft they have the birth year wrong on a person
• LDAP: Research why an account is having issues with SharePoint -- expired for a year
• LDAP: PeopleSoft non-authoritative employee... Found match in Banner feed

Posted by at 8:29 AM | Permalink

Admin

• LDAP: another new Consultant
• WebCT: finally found the correct cacerts file to stick the UVM CA Cert into
• LDAP: Register for EduCause Grouper/Signet camp

Posted by at 12:09 PM | Permalink

Admin

• Monitoring: cspage address movement starts
• LDAP: A new consultant
• LDAP: student employee needs to be tracked down
• off-campus ssh/vpn discussion

Posted by at 12:07 PM | Permalink

Admin

• LDAP: IdM
• Monitoring: move rrdtool things to new server (go live)
• VMWare: kick-off meeting... we're production now!

Posted by at 12:04 PM | Permalink

Admin

• LDAP: verify 7001 codes showed up in the PeopleSoft extract
• General: rrdtool measurements movement to new machine

Posted by at 10:10 AM | Permalink

Admin

• LDAP: four "non-authoritative" creations from PeopleSoft
• eMail: Catch up from three sick days

Posted by at 10:34 AM | Permalink

Admin

• LDAP:
  • A PeopleSoft entry that had not been created because it was a student that didn't have a match in the Banner entries was changed to be an employee. Remove the former version to allow the automatic creation of the account tomorrow morning.
• Calendar:
  
  
  
  • Working on resizing the /users filesystem on carp so a /u01 filesystem can be created
  
  
  • 5858029.993: Inform oracle that the unirmold is still running on the test machine and allow the SR to be closed.
  
  

Posted by at 8:52 AM | Permalink

Admin

• LDAP:
  
  
  
  • Banner adds ID number that matches former student to current student. Manually updated. Looks like it is a case of legal name change and the former account was never activated and is not in use.
  
  
• Calendar:
  
  
  
  • gather info on unirmold failure from test server to open SR with.
  
  
  • SR 5858029.993 opened "unirmold command fails with 0x13201"
  
  
  • Oracle responds with methods to determine the calendar user that has the bad entries and removing them.
  
  
  • After fixing that calendar user's entries, a full unirmold is kicked off again
  
  

Posted by at 7:19 AM | Permalink

Admin

• LDAP: fishercat database corrupted (txn_checkpoint errors). Perform the nuke and repave operation.
• LDAP: handle faculty member and staff member requesting home information be removed from the on-line and paper directories.
• DNS: update entry per request

Posted by at 1:44 PM | Permalink

Admin

• LDAP: logic error in dealing with null (empty/missing) titles from PeopleSoft
• Calendar: unidbfix and up2date
  • up2date took 30 minutes
  • unidbfix process took three (3) hours

OnCall

• only pages were that one of the in-mailgw machines had a high load

Posted by at 11:53 PM | Permalink

Admin

• LDAP: handle Banner/PeopleSoft update exceptions
• WebCT: Assist with v6 problem report

Posted by at 11:33 AM | Permalink

Admin

• LDAP: hand merging new PeopleSoft entries in
• LDAP: IdM testing
• WebCT: v6 patch install and SSL testing... it fails!

Posted by at 11:32 AM | Permalink

Admin

• blog creation
• LDAP: check update error -- not a problem, was a duplicate entry that never got created originally and is now removed from the feed.
• LDAP: Work on new IdM code

Posted by at 9:47 AM | Permalink

Admin

• LDAP: working on IdM
• Oracle: add an account to four machines per AIS request
• Accounts: fix a vacation config problem identified by Account Services

Posted by at 11:53 AM | Permalink

Admin

• WebCT 6 -- new patch to avoid the SYS account
• LDAP: update expired entry so PeopleSoft entry will properly match up
• LDAP: three graduate students Banner info update reveals multiple accounts exist for them.

Posted by at 9:42 AM | Permalink

Admin

• LDAP: Open issue with Registrar about current student being assigned the SSN of a former student
• LDAP: Close FP 21672 because the automatic audit of department names has been added to the nightly update code.
• VOYAGER: enable SAR reporting to track performance issue

Posted by at 9:27 AM | Permalink

Admin

• LDAP - lots of slurpd reject files --
  1. Two (2) copies of slurpd were running overnight
  2. the in-mailgw replica's must be out of sync with the master
• Zoo replacement -- working on script to convert ssh.com's config to openssh's
• LDAP - working on new code to audit OU values for employees every day to catch when department codes text values are changed.

Posted by at 8:13 AM | Permalink

Admin

• LDAP: a manual merge request from Saturday was found to be that PeopleSoft had entered a change for the SSN of a student employee and the new SSN is that of the employee's brother (another student). Sent the details to HR, Registrar, and CatCard to verify that these two (brother and sister) haven't given the wrong SSN's everywhere.
• LDAP: a graduate student/employee with the all-9's SSN is manualy entered into the system having found the Banner record that matches the information.

Posted by at 9:07 AM | Permalink

Admin

• Restart httpd server to remove expired timesheet ssl certificate to stop the daily email about it
• LDAP: handle one multiple account problem and two non-authoritative employee problems (found matches in the Student feed)
• Footprints -- a little clean up work
• Calendar: recover a calendar for a co-worker who made a mistake with a treo
• LDAP: IdM code work... Eeek, it almost worked!

Posted by at 9:11 AM | Permalink

Admin

• Accounts: one more duplicat eradication effort
• Tech Park -- PDU and WrightLine quotes
• LDAP: manually update department names for 37 entries... FP Issue=21672 Proj=4
• LDAP: Add to TODO list to automatically update entries for new department names
• Calendar: dig up directions on Event Calendars

Posted by at 8:34 AM | Permalink

Admin

• Tech Park wrightline quotes
• LDAP: IdM changes... poc
• Accounts: duplicat eradication

Posted by at 8:33 AM | Permalink

Admin

• Calendar: obtain and share doc for "Event Calendars"
• LDAP: Find and manually merge four employee records with their Student affiliation
• Provide telephone directory input file

Posted by at 9:54 AM | Permalink

Admin

• LDAP: Not errors in this morning's update!
• LDAP: IdM development work
• FootPrints: Check out ones that haven't been updated in two weeks and send a we're closing this soon message to one

Posted by at 9:32 AM | Permalink

Admin

• Discussions about netid formats
• Discussions about employee codes and which ones are really students
• Provide access to the Macintosh version of the Oracle Calendar SDK to Wes and Mason

Posted by at 12:27 PM | Permalink

Admin

• LDAP - more errors??? what is up with this!?!?
• LDAP - Update reporting code for NID conflicts to give enough detail to forward problem to responsible folks
• LDAP/Accounts - Miscellaneous Account procedures

Posted by at 8:53 AM | Permalink

Admin

• create blogs as requested
• LDAP: install code to handle new xxC (xx month part-time faculty) codes

Training

• Day 4 of VMWare ESX installation class

Posted by at 8:15 AM | Permalink

Training

• Day 3 of the VMWare implementation class

Posted by at 8:12 AM | Permalink

Training

• Day 2 of the VMWare ESX 3.0 implemention

Posted by at 8:11 AM | Permalink

Admin

• LDAP: Two positively unknown entries show up from PeopleSoft extract -- they turn out to be students. Information manually merged into LDAP directory.
• FootPrints: IE hanging in spell check... apply UniPress patch 67580... that should fix it
• Blogs: create as requested

VMWare ESX

• First day of a four day class on it

Posted by at 8:10 AM | Permalink

Admin

• LDAP: multiple dual personality problems
• LDAP: extract Arts and Sciences mailinglist info

Posted by at 9:26 AM | Permalink

Admin

• LDAP: more multiple netid problems

Posted by at 12:45 PM | Permalink

Admin

• LDAP: logrotate on barnowl was dying because /var/log/cups directory did not exist. Created the directory (damned systemimager process anyway) and use -force to get it to rotate the logs for the first time in its short life
• Penguin3: hardware issue -- it needed to have the power removed and restored before it would power up. No idea why it was so messed up that the power button was refusing to work -- just sat there and blinked at me.
• Accounts: dealing with multiple netid's problems. One is clearly the same person. Second and third is not so clear -- Banner disagrees with itself

Posted by at 8:19 AM | Permalink

Admin

• LDAP: added fourth replica to ldap.uvm.edu -- first production blade

Posted by at 8:44 AM | Permalink

Admin

• LDAP: building new replica system
  • db loaded
  • processes configured
  • nagios monitoring updated
  • measurements updated
• LDAP: manual merge of two records
• LDAP: BANNER faculty questions
• BLOGS: Creation as requested

Posted by at 8:58 AM | Permalink

Admin

• Calendar: messed up account work
• A&S: generate list of faculty/staff for their mailing lists
• LS20: RHEL4-U4-i386 -- RDAC driver...

Posted by at 8:40 AM | Permalink

Admin

• VMWARE: jumpstart meeting...
• LS20: more work on RHEL4-U4-i386 image
• LDAP: PeopleSoft questions
• Calendar: got a user that can't log in... restored account... ldap messed up

Posted by at 8:39 AM | Permalink

Admin

• LS20 build RHEL4-U4-i386 systemimager image

Posted by at 8:38 AM | Permalink

Admin

• LDAP: Upgrade to RHEL4U4
• BLOG: recreate a blog
• WebCT: test Solaris maintenance package

Posted by at 8:32 AM | Permalink

Admin

• build RHEL4-U4-i386 blade minimal image

Posted by at 12:06 PM | Permalink

Admin

• Calendar:
  • 5663822.993 - Apple X11 1.0 doesn't have the issue that I'm having with Apple X11 1.1 -- notified that lbxproxy will work around the problem and asked how we could supply a test environment to Apple
  • 5663822.993 - Oracle closes issue without my permission
  • 5691449.992 - Opened to address why Oracle closed 5663822.993 without answering question of how to help Apple
  • Oracle reopened 5663822.993 and gave me permission to provide the installer to Apple for debugging.
• Accounts: Converted an account to ERP Consultant status
• Create blogs as requested for tomorrow's CTL class on using blogs

Posted by at 8:24 AM | Permalink

Admin

• Calendar
  • 5671476.992 - Update that the workaround of removing the "update 3" text from /etc/redhat-release worked
• FootPrints: close out old footprints and ones that look to have been resolved

Posted by at 8:14 AM | Permalink

Admin

• Mail: fix folders on server for special customer
• Temp: Adjust A/C's in machine room to redirect cool air where needed
• LDAP: IdM development
• Calendar:
  5671476.992 - Oracle says that RHEL4u3 gets recognized as RHEL3 by their scanner (ie they have a bug) and to fix it remove the "update 3" info from the /etc/redhat-release file.
  • 5663822.993 - Oracle says that I should use VNC instead of X11. Right, that would require installing X11 server packages and VNC packages on the linux system that is going to be the OCS server. Bzzzt! Try again, folks!
• OS: RHEL4U4 ISO images for i386 and x64 downloaded

Posted by at 8:51 AM | Permalink

Admin

• Calendar: 5671476.992 -- Oracle agrees that no compat-libstdc++-devel rpm is needed. So, I've asked how to get runInstaller to know that so it stops complaining all the time of a "failure"
• FootPrints: recover another inbox
• WebCT: fsck filesystem for app owner
• LDAP: IdM code development

Posted by at 7:45 AM | Permalink

Admin

• Calendar:
  • 5663822.993 update -- I've confused the Oracle support person by stating that I did not install the X11 server on the machine that will be the OCS server. I have now updated the issue to explain that the X11 server is not required to run on the same physical machine as the X11 client and that their runInstaller program has a problem when the X11 server is a Mac running X11 on OS X 10.4.7. We'll see if they come back with any way to get the runInstaller program to give an error log or output of any kind in the shell that invoked it.
  • 5671476.992 opened -- compat-libstdc++-devel rpm doesn't exist for RHEL4 but the runInstaller continues to complain about it missing at every turn
• Accounts: Adjusting aliases per requests from Account Services
• FootPrints:
  • Update footprints entries that have not been updated in a week to see what can be closed and what is still a problem
  • Inbox recovery from the weekend

Posted by at 9:49 AM | Permalink

Admin

• Calendar: OCS 10.1.2 install -- still not working... missing an RPM that doesn't exist
• Calendar: Updated install SR with info about the favor they asked (run installer on local machine -- but no X server there!)

Posted by at 7:56 AM | Permalink

Admin

• WebCT: OS maintenance and App maintenance applied
• Oracle: Work on installing OCS -- oops... it doesn't work with Mac X11

Posted by at 9:58 AM | Permalink

Admin

• FootPrints:
  • Recover inbox from two days ago
  • Give one week close warning to an old one
  • Reassign one where customer has not returned call for a week to helpline for followup
• IdM: research what "NIST CAP" is
• eMail: gather info on a user who is constantly losing email

Posted by at 9:07 AM | Permalink

Admin

• Assist webteam with rebooting google appliance 2 and retrieving google appliance 1
• WebMail: Lots of hung http processes on servers
• Backup: recover two different copies of inboxes
• LDAP: new release of OpenLDAP built... testing
• FootPrints: closing old issues
• LDAP: one of the replica servers goes load happy -- customer using very badly formed searches that hit a lot of unindexed attributes found and blocked until can work with them on issue... FP 20708 opened

Posted by at 7:54 AM | Permalink

Admin

• Accounts: dcecp replacements finished, put in service
  • bug in set_password_account -- fixed
  • create a fifth one: set_description_account
• LDAP: catcard feed stale
• Webmail: hung processes cleaned up
• Monitoring: Determine why / filling up didn't cause a page, check_disk -x seems to cause / to be ignored!
• Calendar: restore a deleted calendar account
• Calendar: emergency reboot of server before it crashes and takes everything with it

Posted by at 7:03 AM | Permalink

Admin

• EchoVermont.org -- web server switch
• Accounts -- Another account that has the password encoded wrong
• Accounts -- Audit finds 8 other accounts with similar problem... fixed
• Accounts: dcecp replacement scripts
  • deactivate_account
  • reactivate_account
  • set_expdate_account

On Call

• cat301 - nuisance pages (flapping)
• copperhead - ipf failure
• groundhog/ldap - disk filled up

Posted by at 8:51 AM | Permalink

Admin

• LDAP: Notify CatCard that the feed file is stale again
FootPrints: Clean up a lingering issue that appears to have died
• Monitoring: Naupy testing
• LDAP: re-notify Registrar's office and HR office about National ID differences in three people's entries

Posted by at 9:22 AM | Permalink | Comments (1)

Admin

• Calendar: SR 5564690.992 -- respond to Oracle's suggestion
• FootPrints: Issue=20139 Proj=4 -- No real idea of what happened?
• Calendar: Experiment with smtpmailprogram and smtpmailpath options to prevent the test server from sending out meeting notice emails
• Mail: research mail delivery problem -- missing file... search for file
• FootPrints: work on closing out old issues

Posted by at 7:23 AM | Permalink

Admin

• LDAP: new group management to catch HR renaming departments
• LDAP: notify catcard office that the feed is broken
• Calendar: 10g install work

Posted by at 1:30 PM | Permalink

Admin

• FootPrints: review the oldest issues and see if some of them can be closed
• LDAP: New retiree file from HR
• LDAP: groups/department names changes
• IDs: talk with Georgetown rep about 1.0 and 2.0
• LDAP: test jobcode=0999 removal from PS feed

Posted by at 8:06 AM | Permalink

Admin

• LDAP: no drips no runs no errors! YAY
• SSL: jabber cert expired ... oops... installed good one

Posted by at 9:27 AM | Permalink

Admin

• LDAP: several uid issues to merge/fix
• LDAP: Consultants -- do we really still have 81 consultants working on ERP?
• DISK: Saturn's got one slice of a disk that claims to be bad... tell metastat to shut up and reuse that slice... works ok so far...
• ACCOUNTS: expiration of lifetime accounts meeting

Posted by at 9:40 AM | Permalink

Admin

• lab server reboot to rearrange filesystem mounts and samba shares...
  • Two reboots required because one of the fibre disks changed its path again
• HR - Affiliated Org Employee issue

Posted by at 8:11 AM | Permalink

Admin

• Calendar 10g -- RHEL4 installed, upgraded, and required extra rpm's installed

Posted by at 8:07 AM | Permalink