PS1 IPP Czar Logs for the week 2013.09.23 - 2013.09.29

(Up to PS1 IPP Czar Logs)

Monday : 2013.09.23

mark is czar

• 07:30 MEH: nightly done, regular stdsci restart
• 09:30 also doing restart of summitcopy, registration, distribution, publishing, stack for the week
• 20:44 MEH: looks like 2 exposures of STD obs_mode in OPEN filter.. (o6559g0084o, o6559g0085o) -- chip stage has no FLAT detrend and will set to drop in a bit

Tuesday : 2013.09.24

mark is czar

• 07:4 MEH: nightly finished, regular restart of stdsci
• 10:00 MEH: cleaning up som eLAP stuff..

  -- fault 5
   stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 5
  -- fault 4 after several revert etc
   stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 4
  -- fault 2 -- broken warps.. some required chip updates
  perl ~ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 742361 --skycell_id skycell.0740.096 --redirect-output --dbname gpc1
  
  chiptool -setimfiletoupdate -chip_id 791999 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 748664 --skycell_id skycell.1442.095 --redirect-output
  chiptool -setimfiletoupdate -chip_id 795325 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 751839 --skycell_id skycell.2247.076 --redirect-output
  chiptool -setimfiletoupdate -chip_id 783099 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 739674 --skycell_id skycell.0739.019 --redirect-output
  --> chip_id are >> than currently updating id, may want to use a different label to get them updated -- then will manually clean chips 
  
  

• 12:05 MEH: ps2iq overloading ippc18 with outputs, all processing and such stalled. we need a policy of not running intensive programs involving ippc18 (on ippc18 and/or to the homedir). if found, will be killed
• 12:50 MEH: cleanup counters wacky, doing a restart
• 17:40 MEH: chip.off to catch up warps and push stacks before restarting stdsci and nightly data

Wednesday : 2013.09.25

Thursday : 2013.09.26

• 07:38 Bill: Lots of timeouts for postage stamp tasks. Looks like a database problem. Also quite a few in the update pantasks. Restarting them both
• 14:22 Bill: stdscience could use a restart. Set to stop

Friday : 2013.09.27

• heather 9am: sine waves on ganglia, restarting stdsci
• heather 12pm: something happened overnight to ipptopsps? Any idea? someone reboot a machine or something?

Saturday : 2013.09.28

• 07:05: Serge reports that registration is backed up.
  • Bill looked at the registration pantasks and saw that the status page was a mess. Many many job timeouts. Restarted pantasks and burntool started running.
  • At about the same time Heather repaired the fault.
• 07:18 Bill: stdscience needs to be restarted. Set to stop
• 07:34: restart complete. Didn't wait for some LAP camera runs to finish. Removed lap label for a bit to avoid running those jobs twice.
• 07:55: all back to normal
• 10:06: Bill dropped a warp skyfile which repeatedly pops assertion failure with warptool -updateskyfile -skycell_id skycell.2008.031 -warp_id 842911 -set_quality 4 -fault 0

Sunday : 2013.09.29