PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.01.16

• MEH: MOPS test chunk running ipps, upper ippx, ippc nodes w/ data to ipp100+
• MEH: setting nightly processing up for another night of QUB targeted followup -- no changes should be made to nightly processing

Tuesday : 2017.01.17

• MEH: setting nightly processing up for another night of QUB targeted followup -- no changes should be made to nightly processing

Wednesday : 2017.01.18

• MEH: QUB targeted followup needs ippx, ipps nodes as well as nightly nodes in AM and mid-afternoon -- finished
• MEH: OSS WSdiff skipped in distribution, doing large number of WSdiff updates in normal ps_ud_QUB -- using ipps, ippx048-088

Thursday : 2017-01-19

• 16:15 CZW: Started running trunk/tools/raw_MD5_check.pl on ippb04. This script uses 12 client runs to calculate on disk MD5 sums for raw GPC1 OTA data in each nebulous sub-sub directory (/data/HOST.VOL/nebulous/aa/bb/FAKE_ota55.fits has an MD5 sum recorded in file /data/HOST.VOL/nebulous/aa/bb.OTA_md5sums). This will cause some added load on this machine, and on the other b node machines when I start running it on them (ippb05 today, the others either later tonight or tomorrow). A quick check suggests total runtimes on the order of a week and a half.
• 22:10 MEH: ipp121 kernel panic blocking nightly processing for while now (90 exposures stuck in chip stage) and not checked on... neb-host repair and take out of any pantasks -- moving forward some now...
  • ipp121 appears to not be up any longer (data mount was accessible at some level before) -- will power cycle soon if no one claims doing so, otherwise stalled mount mess...
  • ipp121 back up (boots w/in 5 minutes so not doing ram check for the 260GB...) -- nightly processing is moving again

Friday : 2017.01.20

• MEH: special QUB needs ippx, ipps nodes as well as nightly nodes in AM and mid-afternoon
• 15:15 CZW: Started more md5sum scans on ippb03, ippb05, ippb06. Should have little impact on other processing.
• MEH: setting nightly processing up for another night of QUB targeted followup -- no changes should be made to nightly processing

Saturday : 2017.01.21

• MEH: QUB targeted followup needs ippx, ipps nodes as well as nightly nodes in AM -- finished
• MEH: setting nightly processing up for another night of QUB targeted followup -- no changes should be made to nightly processing
• MEH: manually managing large number of QUB pstamp image updates

Sunday : 2017.01.22

• MEH: IfA-wide AC failure -- powering down ipp003,001,022 as less critical to ipp002,ippops1,ippops2
  • last gpc1 dump was successful ~0000 last night before ippdb machines went down last night