== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2017.02.27 ===
 * MEH: re-deploying ipp-20170121 as the ops tag -- online
  * night 20170217 with buggy ipp-20170121 used is okay except for warp->diff updates where the correction for the 0.1pix offset won't be applied since it is missing the ADDITIONAL_WCS_ORDERS value in the .mdc file
 * MEH: ippc120 will be used by MOPS at the ITC for stamps once nightly processing starts there
=== Tuesday : 2017.02.28 ===
 * MEH: another case of WSdiffs queuing once the diffs are cleaned (but warps are still full) -- so not a fluke and a bug in new tag somewhere... ~ippqub/stdscience_ws diff.off until mess cleaned up -- 
  * looks like changes to difftool.c/difftool_definewarpstack.sql during LAP.PV3 WSdiff processing has broken something
  * WSdiff will need to be manually queued until problem fixed.. -- turning survey WSdiff off -- survey.stage warpstack.diff off
=== Wednesday : YYYY.MM.DD ===

=== Thursday : 2017.03.02 ===
 * MEH: Haydn working on ipp078, ipp121 and powerdown required -- in addition to removing from processing, neb-host should also be dropped to repair (and really down while machine is down) to avoid odd file states while work being done -- also good to not only take out of processing in ~ipp/ippconfig/pantasks_hosts.input but also to actually take it out of ACTIVE processing ./check_system.sh hostoff ipp078
  * ipp078 -- replace troublesome OS mirror disk -- problem with mirror on sdb and root on sdc (backwards) and booting off sdb and sdc stopped working -- both SSD seem to be working ok currently so leaving as is until something fails -- however, going to be leaving out of processing and neb-host repair until monday...
  * ipp121 -- again went unresponsive to ssh connections but nfs could still read the disk -- rebuild root disks and fixed 1G back to 10G connection -- 
   * problems connecting -- /etc/ssh files needed to be restored
   * exports not working -- /etc/exports file needed to be restored
   * ganglia not working -- /etc/ganglia/gmond.conf file needed to be repointed to ippc19/10.10.20.17
   * automounts not working -- /etc/autofs/auto.data old version and newer version from ipp122 copied over (since generic file)
   * leaving in neb-host repair and still out of processing until at least monday to see how it goes

=== Friday : 2017.03.03 ===
 * MEH: Haydn replaced BBU in ipp083, ipp088, ipp092 -- neb-host repair until tomorrow (disks mostly full so not a big issue)
 * MEH: Haydn working on ipp078 again -- neb-host repair and out of processing again until tomorrow
 * MEH: re-targeting data again off ipp067,082-097, ipp100,102,103,104.0 to reduce the level of random
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===