PS1 IPP Czar Logs for the week 2013.12.02 - 2013.12.08

(Up to PS1 IPP Czar Logs)

Monday : 2013.12.02

Bill is czar today

• 10:47 Set all data with label like ps_ud% to be cleaned. Restarted cleanup pantasks.
• 13:30 Dropped the 6 STD.nightlyscience chipRuns from 11-27 and 11-29 for exposures with filter = 'OPEN' We don't process those because we have no flat detrends
• 13:35 set all pantasks to stop in preparation for full restart. (The only one doing anything currently is cleanup)
• 16:15 set STS.rp.2013 distribution bundles to be cleaned
• 19:05 MEH: a reminder when restarting any pantasks, hosts may need to be manually turned off to account for DVO/PSPS/mysql use on machines until mysql etc can be restarted or jobs/nightly processing is likely to get jammed -- currently looks like ipp048--ipp053, 039, 057 may still be problem childs.
  • ipp053 holding onto registration job >3ks, chip jobs >2ks so far..
  • turning these ipp048--ipp053 off

Tuesday : 2013.12.03

• 08:35 Bill: Serge reports that ipp is stuck. It appears that ipp057 is down and some registration jobs are stuck trying to access files there power cycling it
• 16:10 MEH: in addition to manually taking nodes out of processing if DVO/PSPS overloading, they may also need to be put into neb repair (more of an issue now that there is open space on many of the similar data nodes
• 16:20 MEH: tweak_ssdiff to get the SSdiff out for QUB

Wednesday : 2013.12.04

• 09:38 Bill: dropped chip_id 919320 and 919321 because they will fail to run. The filter for these STD exposures is OPEN and we don't have flats for that case.
• 12:25 MEH: ipp065 needed to be taken out of stack as well.. tweak_ssdiff for whatever MD stacks are available now
• 17:50 MEH: Heather finished restarting mysql on ipp048-ipp053, ipp065 so putting back into pantasks
  • ipp039 may need to have an eye kept on
• 18:00 CZW: I've set stsci13.X to repair in nebulous. Something is broken with NFS, and because of this, the log is full of "lockd: cannot monitor ipp046" entries. This prevents any new files from being written, as we lock files to ensure they're written correctly. This was forcing nearly all jobs to fail, and suggests that stsci13 needs to be rebooted to regain its sanity.

Thursday : 2013.12.05

mark is czar

• 00:40 MEH: something was slowly reducing processing until ~midnight -- registration got ~20 exposures behind, camera stage taking ~2ks, rates ~20 exp/hr. removed LAP label and lap.off @midnight and things slowly getting back to normal, LAP label back in and lap.on but such a backlog of chips/warp will have to wait and see if degrades again.
• 04:50 MEH: again found LAP overusing processing, >80 3PI chips to do, registration behind -- LAP label out.
• 08:00 MEH: nightly finished. cleared five fault 5 WS diffim (501812, skycell.1157.030; 501836, skycell.1156.007; 501847, skycell.1157.030; 501868, skycell.1156.007). cleared two distRun faults hanging around from 12/2. LAP chip.off for a bit to clear some warps for stacks, then chip.on.

Friday : 2013.12.06

Saturday : 2013.12.07

Sunday : 2013.12.08