PS1 IPP Czar Logs for the week 2013-12-16 - 2013-12-22

(Up to PS1 IPP Czar Logs)

Monday : 2013-12-16

• 10:20 Bill: restarted stdscience with STS.rp.2013 label added.
• 13:13 Bill: I'm running a psphot full force test using compute3 and wave4 and since there are only 14 runs this should not be very visible.
• 16:59 CZW: restarted replication pantasks to use the rawcheck.pro module to scan all raw OTA data and replicate and cull as appropriate. Use rawcheck.show.date and rawcheck.set.date DATE to monitor progress.
• 17:50 MEH: running local trunk staticsky+skycal on MD04, taking 1x c3 from stack since LAP processing will be behind STS for a day or so

Tuesday : 2013-12-17

• 12:29 CZW: STS has finished warp stage, so I stopped stdscience to rebuild psLib to pick up the changes to the psMixtureModel code. Restarting stdscience now.

Wednesday : 2013-12-18

• 07:50 Bill: turned chip revert off. There are 6 chips with missing burntool tables that I want to fix.
  • 08:08 It was actually only one missing burntool table (--exp_id 178815 --class_id XY34). The other faults were due to ipp050 misbehaving temporarily cannot instantiate logfile and other similar nebulous errors). Revert back on.
  • 08:09 Fixed a broken pstamp dependency. diff_id 504256 (from just last week) was marked as state cleaned but several skycells' data state was 'full'. The dependency checker does not expect this state. Worked around this by changing the state of the run to 'update'
• 08:45 Bill: restarted pstamp and update pantasks their pcontrols were spinning
  • 08:58 ...and since the restart 6596 apparently backlogged jobs have run.
• 22:50 MEH: STS warps finishing up, LAP updates well underloaded -- stdsci needs its regular restart. did distribution as well

Thursday : 2013-12-19

mark is czar

• 08:00 MEH: chip.revert.off to fix some burn.tbl for STS -- it appears all burn.tbl are 0 and missing as normal case BUT primary and secondary are both on the ippbXX machines ONLY.. if this is common for all then may be a problem..
  • all have rogue file to copy over, so primary and secondary still only on ippbXX

    o5354g0426o 	XY34 	178903 	925065 
    neb://ipp045.0/gpc1/20100607/o5354g0426o/o5354g0426o.ota34.burn.tbl
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ippb01.2/nebulous/7a/c2/929738415.gpc1:20100607:o5354g0426o:o5354g0426o.ota34.burn.tbl
          0                     NON-EXISTANT file:///data/ippb02.0/nebulous/7a/c2/929746459.gpc1:20100607:o5354g0426o:o5354g0426o.ota34.burn.tbl
    -->
       300606 | /data/ippb02.2/nebulous/7a/c2/929746459.gpc1:20100607:o5354g0426o:o5354g0426o.ota34.burn.tbl
    
    
    o5354g0388o 	XY06 	178865 	925027 
    neb://ipp023.0/gpc1/20100607/o5354g0388o/o5354g0388o.ota06.burn.tbl
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ippb01.2/nebulous/11/ac/929738221.gpc1:20100607:o5354g0388o:o5354g0388o.ota06.burn.tbl
          0                     NON-EXISTANT file:///data/ippb02.0/nebulous/11/ac/929749698.gpc1:20100607:o5354g0388o:o5354g0388o.ota06.burn.tbl
    -->
       252446 | /data/ippb02.2/nebulous/11/ac/929749698.gpc1:20100607:o5354g0388o:o5354g0388o.ota06.burn.tbl
    
    
    o5354g0381o 	XY13 	178858 	925020 
    neb://ipp024.0/gpc1/20100607/o5354g0381o/o5354g0381o.ota13.burn.tbl
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ippb01.1/nebulous/99/e1/929738267.gpc1:20100607:o5354g0381o:o5354g0381o.ota13.burn.tbl
          0                     NON-EXISTANT file:///data/ippb02.2/nebulous/99/e1/929746247.gpc1:20100607:o5354g0381o:o5354g0381o.ota13.burn.tbl
    -->
       235117 | /data/ippb02.1/nebulous/99/e1/929746247.gpc1:20100607:o5354g0381o:o5354g0381o.ota13.burn.tbl
    
    
    o5354g0366o 	XY13 	178843 	925005 
    neb://ipp024.0/gpc1/20100607/o5354g0366o/o5354g0366o.ota13.burn.tbl
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ippb01.2/nebulous/1a/2c/929738232.gpc1:20100607:o5354g0366o:o5354g0366o.ota13.burn.tbl
          0                     NON-EXISTANT file:///data/ippb02.2/nebulous/1a/2c/929746241.gpc1:20100607:o5354g0366o:o5354g0366o.ota13.burn.tbl
    --> 
       222820 | /data/ippb02.1/nebulous/1a/2c/929746241.gpc1:20100607:o5354g0366o:o5354g0366o.ota13.burn.tbl
    
    

• 10:57 CZW: Mark pointed out an exposure that was stuck with chipRun.state = 'error_cleaned' and warpRun.state = 'update'. This blocks LAP somewhat, as it waits for a warp that is waiting for a chip that isn't updating because chiptool refuses to update data that isn't in state 'cleaned'. This has happened on occasion before, and my solution has been to issue the commands generated by the output of /data/ippc18.0/home/watersc1/PV2.LAP.20130717/fixes/scan_chips_for_errorCleaned.sql. An example output is listed below. The run_update_for_chip_id.pl script issues the chiptool/camtool/warptool commands to set things to be updated.

   chiptool -updaterun -set_state cleaned -chip_id 574277
   chiptool -updaterun -set_state cleaned -chip_id 574396
   chiptool -updaterun -set_state cleaned -chip_id 574444
   chiptool -updaterun -set_state cleaned -chip_id 837183
   /home/panstarrs/watersc1/bin//run_update_for_chip_id.pl 574277
   /home/panstarrs/watersc1/bin//run_update_for_chip_id.pl 574396
   /home/panstarrs/watersc1/bin//run_update_for_chip_id.pl 574444
   /home/panstarrs/watersc1/bin//run_update_for_chip_id.pl 837183
  

• 16:35 MEH: regular restart of stdsci before nightly
• 23:20 MEH: something stalling LAP.., also ~6 LAP id stalled waiting for pending update and will look into later

Friday : 2013-12-20

mark is czar

Saturday : 2013-12-21

Sunday : 2013-12-22