PS1 IPP Czar Logs for the week 2013.09.23 - 2013.09.29

(Up to PS1 IPP Czar Logs)

Monday : 2013.09.23

mark is czar

• 07:30 MEH: nightly done, regular stdsci restart
• 09:30 also doing restart of summitcopy, registration, distribution, publishing, stack for the week
• 20:44 MEH: looks like 2 exposures of STD obs_mode in OPEN filter.. (o6559g0084o, o6559g0085o) -- chip stage has no FLAT detrend and will set to drop in a bit

Tuesday : 2013.09.24

mark is czar

• 07:4 MEH: nightly finished, regular restart of stdsci
• 10:00 MEH: cleaning up som eLAP stuff..

  -- fault 5
   stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 5
  -- fault 4 after several revert etc
   stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 4
  -- fault 2 -- broken warps.. some required chip updates
  perl ~ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 742361 --skycell_id skycell.0740.096 --redirect-output --dbname gpc1
  
  chiptool -setimfiletoupdate -chip_id 791999 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 748664 --skycell_id skycell.1442.095 --redirect-output
  chiptool -setimfiletoupdate -chip_id 795325 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 751839 --skycell_id skycell.2247.076 --redirect-output
  chiptool -setimfiletoupdate -chip_id 783099 -set_label LAP.ThreePi.20130717 -dbname gpc1
  /home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 739674 --skycell_id skycell.0739.019 --redirect-output
  --> chip_id are >> than currently updating id, may want to use a different label to get them updated -- then will manually clean chips 
  
  

• 12:05 MEH: ps2iq overloading ippc18 with outputs, all processing and such stalled. we need a policy of not running intensive programs involving ippc18 (on ippc18 and/or to the homedir). if found, will be killed
• 12:50 MEH: cleanup counters wacky, doing a restart
• 17:40 MEH: chip.off to catch up warps and push stacks before restarting stdsci and nightly data

Wednesday : 2013.09.25

Thursday : 2013.09.26

• 07:38 Bill: Lots of timeouts for postage stamp tasks. Looks like a database problem. Also quite a few in the update pantasks. Restarting them both
• 14:22 Bill: stdscience could use a restart. Set to stop

Friday : 2013.09.27

• heather 9am: sine waves on ganglia, restarting stdsci
• heather 12pm: something happened overnight to ipptopsps? Any idea? someone reboot a machine or something?

Saturday : 2013.09.28

• 07:05: Serge reports that registration is backed up.
  • Bill looked at the registration pantasks and saw that the status page was a mess. Many many job timeouts. Restarted pantasks and burntool started running.
  • At about the same time Heather repaired the fault.
• 07:18 Bill: stdscience needs to be restarted. Set to stop
• 07:34: restart complete. Didn't wait for some LAP camera runs to finish. Removed lap label for a bit to avoid running those jobs twice.
• 07:55: all back to normal
• 10:06: Bill dropped a warp skyfile which repeatedly pops assertion failure with warptool -updateskyfile -skycell_id skycell.2008.031 -warp_id 842911 -set_quality 4 -fault 0
• 14:00 MEH: stsci13 appears down around 1200.. forgot should've taken out of neb when noticed so can at least do nightly science tonight if no one around to reboot with the special raid issues
  • probably could restart stack, pstamp to clean stalled jobs -- will need to clean up hanging jobs and mounts.. MOPS will need PSS in morning so needs to be done if stsci13 not rebooted before then
• 14:30 MEH: stacks cleared, but stack.revert.off since many faults from warps only on stsci13...
• 15:50 MEH: clearing enough update+pstamp so that MOPS can get their stamps for TODAY, some QUB ones still hanging around
• 16:30 MEH: dumped and restarted cleanup.. so little space, need any we can recover at this point.. there will be stalled ipp_cleanup.pl, won't kill and don't want to do -9 right now..
• 17:10 MEH: in case no stsci13 before tonight, still trying to get things running. registration stuck on dark imfile -- cleaned and restarted registration and summitcopy
• 17:15 MEH: will leave stdsci last as it will be a major PITA to clean out
• 16:10 MEH: neb having all kinds of fun timouts/faults in summitcopy, 900+ connections in nebdb.. time to start clearing all out before nightly starts..
• 20:00 MEH: good thing bad weather, things still screwy.. LAP label to be removed from stdsci, clearing/killing chip_imfile.pl and warp_skycell.pl
  • won't/don't really want to kill camera_exp.pl unless have too, not sure what state in and issues will cause
• 20:40 MEH: registration rawExp somehow borked for o6564g0013d.660484 and why not finishing 13d, loaded exp_id but NULL/UNKNOWN values for everything else.. cleared with regtool -dbname gpc1 -revertprocessedexp -exp_id 660484
• 20:50 MEH: nightly darks finished downloading and registering, stack running, pstamp finishing up QUB, cleanup running, stdsci running w/o LAP label
  • also looks like a lot of somethings cleared accessing nebulous as well, now down to order 100 connections

Sunday : 2013.09.29

• 11:00 MEH: looks like some MD09 chip/warps stalled in stdsci
• 11:10 MEH: LAP stacks were kept running last night and mostly caught up with whatever warps available
  • now nightly done, LAP warps could be tried for whatever chips/cam files available
  • camera should probably remain off until jobs can be checked to avoid any strange update cases