== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2017.01.16 ===
 * MEH: MOPS test chunk running ipps, upper ippx, ippc nodes w/ data to ipp100+
 * MEH: setting nightly processing up for another night of QUB targeted followup -- no changes should be made to nightly processing 
=== Tuesday : 2017.01.17 ===
 * MEH: setting nightly processing up for another night of QUB targeted followup -- no changes should be made to nightly processing 

=== Wednesday : 2017.01.18 ===
 * MEH: QUB targeted followup needs ippx, ipps nodes as well as nightly nodes in AM and mid-afternoon -- finished
 * MEH: OSS WSdiff skipped in distribution, doing large number of WSdiff updates in normal ps_ud_QUB -- using ipps, ippx048-088
=== Thursday : 2017-01-19 ===

 * 16:15 CZW: Started running trunk/tools/raw_MD5_check.pl on ippb04.  This script uses 12 client runs to calculate on disk MD5 sums for raw GPC1 OTA data in each nebulous sub-sub directory (/data/HOST.VOL/nebulous/aa/bb/FAKE_ota55.fits has an MD5 sum recorded in file /data/HOST.VOL/nebulous/aa/bb.OTA_md5sums).  This will cause some added load on this machine, and on the other b node machines when I start running it on them (ippb05 today, the others either later tonight or tomorrow).  A quick check suggests total runtimes on the order of a week and a half.
 * 22:10 MEH: ipp121 kernel panic blocking nightly processing for while now (90 exposures stuck in chip stage) and not checked on... neb-host repair and take out of any pantasks -- moving forward some now...
  * ipp121 appears to not be up any longer (data mount was accessible at some level before) -- will power cycle soon if no one claims doing so, otherwise stalled mount mess...
  * ipp121 back up (boots w/in 5 minutes so not doing ram check for the 260GB...) -- nightly processing is moving again
=== Friday : 2017.01.20 ===
 * MEH: special QUB needs ippx, ipps nodes as well as nightly nodes in AM and mid-afternoon
 * 15:15 CZW: Started more md5sum scans on ippb03, ippb05, ippb06.  Should have little impact on other processing.
 * MEH: setting nightly processing up for another night of QUB targeted followup -- no changes should be made to nightly processing 
=== Saturday : 2017.01.21 ===
 * MEH: QUB targeted followup needs ippx, ipps nodes as well as nightly nodes in AM -- finished
 * MEH: setting nightly processing up for another night of QUB targeted followup -- no changes should be made to nightly processing
 * MEH: manually managing large number of QUB pstamp image updates
=== Sunday : 2017.01.22 ===
 * MEH: IfA-wide AC failure -- powering down ipp003,001,022 as less critical to ipp002,ippops1,ippops2
  * last gpc1 dump was successful ~0000 last night before ippdb machines went down last night
 * MEH: was QUB night, will restart pantasks now that things appear to be back online
  * ipp121 not booting -- must be set neb-host down until fixed -- hopefully wont bork summitcopy since power glitch interrupted download o7775g0046f-o7775g0049f but only flats