PS1 IPP Czar Logs for the week 2013.09.23 - 2013.09.29

(Up to PS1 IPP Czar Logs)

Monday : 2013.09.23

mark is czar

• 07:30 MEH: nightly done, regular stdsci restart
• 09:30 also doing restart of summitcopy, registration, distribution, publishing, stack for the week
• 20:44 MEH: looks like 2 exposures of STD obs_mode in OPEN filter.. (o6559g0084o, o6559g0085o) -- chip stage has no FLAT detrend and will set to drop in a bit

Tuesday : 2013.09.24

mark is czar

• 07:4 MEH: nightly finished, regular restart of stdsci
• 10:00 MEH: cleaning up som eLAP stuff..

  -- fault 5
   stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 5
  -- fault 4 after several revert etc
   stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 4
  -- fault 2 -- broken warps.. some required chip updates
  perl ~ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 742361 --skycell_id skycell.0740.096 --redirect-output --dbname gpc1
  
  chiptool -setimfiletoupdate -chip_id 791999 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 748664 --skycell_id skycell.1442.095 --redirect-output
  chiptool -setimfiletoupdate -chip_id 795325 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 751839 --skycell_id skycell.2247.076 --redirect-output
  chiptool -setimfiletoupdate -chip_id 783099 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 739674 --skycell_id skycell.0739.019 --redirect-output
  --> chip_id are >> than currently updating id, may want to use a different label to get them updated -- then will manually clean chips 
  
  

• 12:05 MEH: ps2iq overloading ippc18 with outputs, all processing and such stalled. we need a policy of not running intensive programs involving ippc18 (on ippc18 and/or to the homedir). if found, will be killed
• 12:50 MEH: cleanup counters wacky, doing a restart
• 17:40 MEH: chip.off to catch up warps and push stacks before restarting stdsci and nightly data

Wednesday : 2013.09.25

Thursday : 2013.09.26

• 07:38 Bill: Lots of timeouts for postage stamp tasks. Looks like a database problem. Also quite a few in the update pantasks. Restarting them both
• 14:22 Bill: stdscience could use a restart. Set to stop

Friday : 2013.09.27

• heather 9am: sine waves on ganglia, restarting stdsci
• heather 12pm: something happened overnight to ipptopsps? Any idea? someone reboot a machine or something?

Saturday : 2013.09.28

• 07:05: Serge reports that registration is backed up.
  • Bill looked at the registration pantasks and saw that the status page was a mess. Many many job timeouts. Restarted pantasks and burntool started running.
  • At about the same time Heather repaired the fault.
• 07:18 Bill: stdscience needs to be restarted. Set to stop
• 07:34: restart complete. Didn't wait for some LAP camera runs to finish. Removed lap label for a bit to avoid running those jobs twice.
• 07:55: all back to normal
• 10:06: Bill dropped a warp skyfile which repeatedly pops assertion failure with warptool -updateskyfile -skycell_id skycell.2008.031 -warp_id 842911 -set_quality 4 -fault 0
• 14:00 MEH: stsci13 appears down around 1200.. forgot should've taken out of neb when noticed so can at least do nightly science tonight if no one around to reboot with the special raid issues
  • probably could restart stack, pstamp to clean stalled jobs -- will need to clean up hanging jobs and mounts.. MOPS will need PSS in morning so needs to be done if stsci13 not rebooted before then
• 15:50 MEH: clearing enough update+pstamp so that MOPS can get their stamps for today, some QUB ones still hanging around

Sunday : 2013.09.29