PS1 IPP Czar Logs for the week 2011.12.19 - 2011.12.23

(Up to PS1 IPP Czar Logs)

Monday : 2011.12.19

Mark is czar

• 09:00 need to remove ipp064 (raid rebuild), 065, 066 (memory swap) from processing for work to be done by Haydn. Also set neb-host down to avoid any possible hang-ups with the hardmounts now.
• 12:00 Haydn reported ipp063 back online with 48GB RAM, ipp065 & ipp066 rebooted twice and also report 48GB RAM. ipp064 RAID is rebuilding and Cindy will install Gentoo when finished however sometimes it shows 40 and 48 GB RAM so will need to be fixed after RAID finishes.
• 12:20 oops, ipp064 was up with ipp065 for a few minutes so a few files updated or were added to nebulous from LAP. need to set the ipp064 data temporarily on ipp065 to repair with

  neb-host --volume ipp064.0 repair (option needs to be added to help/man?)
  

• 12:30 MD06 chips+warps set to update for comparing how many lost with ipp064 and attempt refstack building again this week for the next OC starting in January otherwise will need to queue up full new run of condor.
• 12:40 working through stalled LAP, restarting stdscience and distribution to rotate logs and refresh things.
• 16:00 half of LAP running again after re-updating many error_cleaned chips
• 20:30 fixing stalling LAP chips due to the FITS with incomplete extensions from ipp064. replicated ones to ATRC seem okay, so cp likely faulty one to .bad and cp good copy

  cp /data/ipp064.0/nebulous/46/b5/365746896.gpc1:20100722:o5399g0214o:o5399g0214o.ota26.fits /data/ipp064.0/nebulous/46/b5/365746896.gpc1:20100722:o5399g0214o:o5399g0214o.ota26.fits.bad
  ...
  cp /data/ippb00.2/nebulous/46/b5/1158188565.gpc1:20100722:o5399g0214o:o5399g0214o.ota26.fits /data/ipp064.0/nebulous/46/b5/365746896.gpc1:20100722:o5399g0214o:o5399g0214o.ota26.fits
  cp /data/ippb02.1/nebulous/cb/80/1160615386.gpc1:20100821:o5429g0180o:o5429g0180o.ota26.fits /data/ipp064.0/nebulous/cb/80/407468336.gpc1:20100821:o5429g0180o:o5429g0180o.ota26.fits
  cp /data/ippb00.0/nebulous/78/7e/936702266.gpc1:20100821:o5429g0194o:o5429g0194o.ota26.fits /data/ipp064.0/nebulous/78/7e/407501214.gpc1:20100821:o5429g0194o:o5429g0194o.ota26.fits
  cp /data/ippb02.0/nebulous/b6/9b/1158190731.gpc1:20100722:o5399g0241o:o5399g0241o.ota26.fits /data/ipp064.0/nebulous/b6/9b/365752348.gpc1:20100722:o5399g0241o:o5399g0241o.ota26.fits
  cp /data/ippb02.1/nebulous/7b/da/896024650.gpc1:20100917:o5456g0093o:o5456g0093o.ota26.fits /data/ipp064.0/nebulous/7b/da/451746379.gpc1:20100917:o5456g0093o:o5456g0093o.ota26.fits
  

• missing file

  neb://ipp024.0/gpc1/20100822/o5430g0244o/o5430g0244o.ota13.fits
        0                     NON-EXISTANT file:///data/ippb02.2/nebulous/4c/cd/920273147.gpc1:20100822:o5430g0244o:o5430g0244o.ota13.fits
        0                     NON-EXISTANT file:///data/ippb02.2/nebulous/4c/cd/920291539.gpc1:20100822:o5430g0244o:o5430g0244o.ota13.fits
  chiptool -dropprocessedimfile -set_quality 42 -chip_id 365090 -class_id XY13 -dbname gpc1
  regtool -updateprocessedimfile -set_ignored -exp_id 210989 -class_id XY13 -dbname gpc1
  

Tuesday : 2011.12.20

Mark is Czar

• 07:00 the little nightly science finished, LAP hung-up by few faulted chips, MD06 chip/warp updates continue.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD