PS1 IPP Czar Logs for the week 2011.08.29 - 2011.09.04

(Up to PS1 IPP Czar Logs)

Monday : 2011.08.29

• 8:00 : Roy : Everything down from summit, standard processing looks ok
• 9:00 Mark: set MD01.GR warps, MD10.GR WSdiffs to cleanup. should be mostly finished with GR cleanup now.
• 13:00 another offnight diffim with large number of detections (pub_id=345096, diff_id=158270) running on ippc07 for 15hr (ippc09 prior for 29hr). killed ppMops process and set to drop

  pubtool -dbname gpc1 -updaterun -pub_id 345096 -set_state drop
  

Tuesday : 2011-08-30

• 07:00 Bill: There are lots of faults due to files wanted from ipp013 and ipp014 which are not available. Set diff.revert off to stop the madness.
• 07:00 created label for phot.jtrp to keep them out of the way. Chips are faulting and reverting repeatedly due to missing files.
• 07:37 update pantasks has more faults than successes (probably due to the down nodes). Restarted pantasks to get a clearer picture of what's happening.
• 08:00 Roy: Everything copied from summit. Same MD diff failures as yesterday: waiting for rsync to finish copying to ipp014, which should resolve the issue

Wednesday : 2011-08-31

• 10:01 Serge: Memory usage increase on ippdb00 and ippdb01. Processing stopped around 9:00 and restarted at 10:00
• 10:30 Bill: reran camRun 255845 to replace corrupted mask file that was causing a destreak failure
• 11:09 CZW: Made change to ipp user's .tcshrc file to select a nebulous server directly, instead of using the randomization via DNS. I tested it yesterday on my own account, so I don't expect issues.
• 11:45 CZW: Unstuck stuck exposure: regtool -revertprocessedexp -exp_id 384925
• 14:45 Serge: All nightly processing finished but the weird o5804g0186o / o5804g0204o diff case
• 16:50 Bill set chip, warp, and diff runs that were in state "error_cleaned" back to "goto_cleaned" (These runs were stalling some postage stampm jobs)

Thursday : 2011-09-01

• 07:05 Bill: Postage stamp server has finished all pending requests. Set all data from ps_ud labels to be cleaned up. Set all pstampDependents that were in state new to full. This will allow us to try and update any components that were temporarily "not available" or "gone" due to temporary disk outages.
• 07:10 Bill: queued STS diffs for 2011-08-26 and 2011-08-28
• 08:17 Bill: dropped STS magicRuns 195005 and 195083 which fail due to corrupt differrence images. They cannot be remade because the input warps have been deleted.
• 09:45 Serge: Ran cam stage again for cam_id 257085 (corrupted cam.mk fits file)
• 11:35 Serge: chip.off and warp.off. LAP prevents nightly science execution

Friday : 2011-09-02

Saturday : 2011-09-03

Sunday : 2011-09-04