PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2015.09.14

Tuesday : 2015.09.15

  • 10:35 EAM : restarting pantasks; cleared out bad diffs (missing cmfs) for SNIa

Wednesday : 2015.09.16

  • 07:45 EAM : ipp017 crashed, rebooting it now (no console messages)
    • on boot, ipp017 gives a console error 'keyboard error' and requires the user to type F1. this needs to be fixed so it will boot automatically.
  • 15:10 MEH: modifying nightly stack pantasks for LIGO now, no MD likely in near future
  • 21:10 MEH: high RH @summit, using stdsci nodes for MOPS test runs
    • 23:35 nodes back to stdsci

Thursday : 2015.09.17

  • 06:00 MEH: will use old, uncleared WS diffs (after the cleaned files are updated) to test pantasks using the modified ipp-20141024 ops tag and the WS diff problem
  • 14:00 MEH: even with the modified ops tag fix for cmf, there are still 829 skycells in 324 exposures lost due to missing FITS that needed to be cleared after the extensive update
  • 20:30 MEH: secondary pantasks for WS diff and mod ops tag seems to be running fine along side main stdscience
  • 23:00 MEH: restarting pstamp

Friday : 2015.09.18

  • 08:30 MEH: re-sending back to cleanup faulting chip updates for MOPS due missing files and neb-entries from PV3 LAP and MD09 data (mdc, burn.tbl, psf...). wondering how often this is happening and things getting silently cleared
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id  482627 
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1196201 
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1267165 
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1385649 
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1252016 
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1358946 
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1386707 
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1386708
  • 14:10 MEH: after clear, will restart all nightly pantasks and include the changes for the WS diffs to be run in a secondary pantask (ippmops:stdscience, 3x c2 nodes)
  • 18:55 MEH: ask Gavin fixed the timezone/localtime problem on ipp010 earlier in the week (saving bad file in ipp010 root directory for future use). tested and looks ok like others, can be added back into processing

Saturday : 2015.09.19

  • 03:15 MEH: restarting pstamp
  • 05:55 MEH: clearing WS diffs stalled still even w/ modified ops tag due to missing FITS files -- 316 skycells lost from 112 exposures (only 152 exposures for night so >50% would be stalled if not cleared..)
  • 17:40 MEH: ipp017 down again -- power cycling, back up -- happening fairly often, taking out of regular processing (already neb-host repair)

Sunday : 2015.09.20

  • 02:25 MEH: ipp017 down again @0137 -- had to do 2 power cycles, nothing on console
  • 04:50 MEH: ipp017 down again @0400 -- instead going to leave down and off
  • 11:00 MEH: previously updated data from 20150912 to clear the large WS set has been returned to cleaned
  • 13:50 EAM: I rebooted ipp017 @ 8am and rsynced off the ff slice master located there (to ipp095). it crashed ~12:45 and I've powered it down via console.