== PS1 IPP Czar Logs for the week 2016.08.01 - 2016.08.07 ==
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.08.01 ===
 * 04:44 MEH: registration pantask crash ~01:30, restarting.. -- catchup rate reached ~80 exp/hr which is good
 * MEH: spontaneous reboot nodes ipp008,012,013,014,016,018 in cab1 still a problem -- ipp013 was in normal stdsci loading and others just light summit+registration -- take all back out of processing, ipp013 takes a long time to reboot so leave in repair, others leave up to data space
 * MEH: nightly processing hosts update -- pantasks_hosts.input -- restoring commented out nodes w/ past problems that should be ok now (guessing mostly since notes are very uninformative)
 * MEH: data targeting summary -- manually modified in builds ~ipp/psconfig/ipp-20141024.lin64/share/pantasks/modules/ipphosts.mhpcc.config  and ~ippqub equivalent
  * previous was ipphosts.mhpcc.config.20160730avoidfullv2
  * current is ipphosts.mhpcc.config.20160730avoidfullv2
  * attempted to set current data load ~60% on ipp100-104 -- io on those machines looks to be in~60-80/out~20-40 MB/s during nightly processing (w/o ipptopsps etc running)
  * chips to ipp005-066, skycells (warp+diff) 054-104 (mostly 100-104), raw to 017/028,054-066,074/76/77,100-104 (mostly 100-104)
  * could add skycell set to ipp005-031 from ipp100-104 to help lower ipp10x data load or maybe from ipp067-097. not sure if ipp005-031 can handle with the space available (warps are kept on disk longer than chip+diff and skycells include warp+diff where large number of diffs are uncompressed)
  * likely need more raw to ipp100-104 from ipp054-066 soon, and later when 17/28,74/77 fill up
=== Tuesday : 2016.08.02 ===

=== Wednesday : 2016.08.03 ===

=== Thursday : 2016.08.04 ===

=== Friday : 2016.08.05 ===

=== Saturday : 2016.08.06 ===

=== Sunday : 2016.08.07 ===