(Up to PS1 IPP Czar Logs)

Monday : 2012-07-30

• 01:24 CZW: Registration was stalled as per Mark's email. My resolution was to dump the previous exposure input table, and move on without it. This sometimes causes segfaults for unknown reasons. As this is an early exposure, I don't think the loss of the input table is that significant. It looks like it's unstuck things, so we should catch back up soon.

  /home/panstarrs/ipp/psconfig/ipp-20120626.lin64/bin/funpack -S /data/ipp058.0/nebulous/ef/54/2283421506.gpc1:20120730:o6138g0063o:o6138g0063o.ota14.fits > /tmp/burntool.502313.XY14.26Z7.fits
  /home/panstarrs/ipp/psconfig/ipp-20120626.lin64/bin/burntool /tmp/burntool.502313.XY14.26Z7.fits  out=/data/ipp058.0/nebulous/7c/b1/2283600282.gpc1:20120730:o6138g0063o:o6138g0063o.ota14.burn.tbl tableonly=t persist=t
  /home/panstarrs/ipp/psconfig/ipp-20120626.lin64/bin/regtool -dbname gpc1 -updateprocessedimfile -exp_id 502313 -class_id XY14 -burntool_state -14 -set_state full
  

• 08:59 Serge: fixed 0-size/lost instances gpc1/20100518/o5334g0071o/o5334g0071o.ota14.burn.tbl (LAP)

• 9:00 onwards (HAF): more unsticking of exposures:

  1. find the stuck exposure
  2. funpack it
  3. redo burntool without history
  4. copy old table out of the way and insert new
  5. redo regtool:
  6. repeat as necessary
  
  example:
  
  funpack -S `neb-locate --path gpc1/20120730/o6138g0175o/o6138g0175o.ota14.fits` > o6138g0175o.ota14.fits
  burntool o6138g0175o.ota14.fits out=o6138g0175o.ota14.burn.tbl tableonly=t persist=t
  neb-mv gpc1/20120730/o6138g0175o/o6138g0175o.ota14.burn.tbl gpc1/20120730/o6138g0175o/o6138g0175o.ota14.burn.tbl.bak
  neb-insert gpc1/20120730/o6138g0175o/o6138g0175o.ota14.burn.tbl o6138g0175o.ota14.burn.tbl --copies=2
  regtool -updateprocessedimfile -class_id XY14 -burntool_state -14 -set_state full -exp_id 502425 -dbname gpc1
  regtool -updateprocessedimfile -class_id XY14 -fault 0 -exp_id 502425 -dbname gpc1
  
  This was done for 3 stuck exposures.
  
  I hate XY14... 
  

• 13:26 pm (haf) restarted stdsci
• 15:55 Serge: Changed chip.pro so that all labels but LAP.ThreePi.20120706 are reverted now.
• 16:25 Serge: Restarted stdscience after I miserably crashed it
• 16:33 MEH: tweaking stdscience to run stackstack diffs from last night server input tweak_ssdiff

Tuesday : 2012-07-31

• 9:14 (haf): bad weather last night...
• 10:17 (Serge): Recovered two missing otas: gpc1/20101110/o5510g0331o/o5510g0331o.ota26.fits and gpc1/20101108/o5508g0412o/o5508g0412o.ota26.fits. Note: To allow (manual) replication to atrc, I had to set ippb00 as up for 10 seconds or so.
• 11:40 (Serge): Restarted (pantasks) replication. Set ippb0[01].[12] disks as up e.g. neb-host --volume ippb01.2 up
• 15:00 (Serge): Stopped (mysql) replication on ippdb03
• 15:23 CZW: Restarted stdscience to kick the rate back up.
• 23:10 MEH: would really prefer not to have a tag change in the middle of a deepstack set... taking compute3 out of processing to just do deepstacks (2x left in stdscience should be ok).

Wednesday : 2012-08-01

• 10:28 (Serge): Started rsync of ippdb02:/export/ippdb02.0/mysql to ippc63 (screen session "mysql_rsync" as root)
• 13:55 CZW: Restarted stdscience to kick the rate back up.

Thursday : 2012-08-02

Bill is czar today

• 00:14 MEH: deep stacks finished, returning compute3 to stdscience (6x) and stack (1x)
• 09:40 Bill: registration got stuck with failed burntool on 502898 XY14. Followed Heather's most excellent instructions on how to fix and burntool is now proceeding..... with yesterdays failures.
• 10:10 Today's problem was actually a summit copy job that crashed leaving the entry in the book. Restarted summit copy and registration pantasks and now we're off and running
• 10:30 stdscience is sluggish, needs daily restart.
• 10:35 ipp066 down, 20120802T1035-ipp066-crash

Friday : 2012-08-03

Saturday : 2012-08-04

Sunday : 2012-08-05