PS1 IPP Czar Logs for the week 2015.08.31 - 2015.09.06

(Up to PS1 IPP Czar Logs)

Monday : 2015.08.31

  • 06:30 Bill: set quality = 66 for 8 diffskyfiles for skycell.1486.051 The template stack's (3187917) cmf is missing.
  • 06:45 EAM: I've removed all but the w-band LAP labels from ~ipp/src/ippMonitor/czartoo/czartool/CzarDb.pm and restarted czarpoll and roboczar

Tuesday : 2015.09.01

Wednesday : 2015.09.02

Thursday : 2015.09.03

  • 22:15 EAM: restarting all ipp pantasks in case we get data

Friday : 2015.09.04

  • 15:15 EAM : rebooted ippb06 which had a kernel crash (load of 400)
  • 15:50 MEH: clearing WS diffs from x.20150903 -- had turned off cleanup earlier in day (init.day doesn't seem to do cleanup on state new diffs which is good) but not necessary
    • diff.revert.off
    • batch execute per diff_id, skycell_id with fault 2 for %.20150903 -- use quality 66 established by Bill to distinguish from normal manual 42 in case want to know which were this case
      difftool -dbname gpc1 -updatediffskyfile -fault 0 -set_quality 66 -diff_id "$1" -skycell_id "$2
      
    • 571 skycells will be missing, 24 OSS, 34 3PI, 8 SNIa exposures delayed out of 101 exposures total.. -- will need to be manually sent to cleanup later
  • 21:25 MEH: will manually queue the new LIGO obs_mode in morning along with clearing the WS diff faults --

Saturday : 2015.09.05

  • 07:30 MEH: cleared remaining WSDiff faults
  • 09:00 MEH: using nightly processing nodes for label LIGOtest
  • 11:15 MEH: LIGOtest finished using nightly processing nodes for now
  • 15:10 MEH: restarting at least stdsci pantasks -- with all the diff faults from missing data products, the counter can rapidly increase over night (of if data left uncleared)
  • 16:15 MEH: Serge needs 20150905 data to not get cleaned up until he fixes the MOPS servers -- cleanup stop for now
    difftool -dbname gpc1 -updaterun -data_group OSS.20150905 -set_label OSS.20150905.mopshold
    chiptool -dbname gpc1 -updaterun -data_group OSS.20150905 -set_label OSS.20150905.mopshold
    
  • 21:50 MEH: warp repeat fault 4 from cannot build growth curve
    warptool -dbname gpc1 -updateskyfile -set_quality 42 -fault 0 -skycell_id skycell.0864.025 -warp_id 1618894  
    warptool -dbname gpc1 -updateskyfile -set_quality 42 -fault 0 -skycell_id skycell.0862.030 -warp_id 1618925 
    warptool -dbname gpc1 -updateskyfile -set_quality 42 -fault 0 -skycell_id skycell.0953.043 -warp_id 1618949  
    
  • 23:45 MEH: clearing 1587 skycells unable to make WS diff

Sunday : 2015.09.06

  • 00:45 MEH: another diff fault (4, cannot build growth curve)
    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.1044.069 -diff_id 1197676 
    
  • 08:00 MEH: setting quality 66 for skycells unable to make WS diff -- 1843 skycells lost, 328 exposures cleared