== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2017.02.27 ===
 * MEH: re-deploying ipp-20170121 as the ops tag -- online
  * night 20170217 with buggy ipp-20170121 used is okay except for warp->diff updates where the correction for the 0.1pix offset won't be applied since it is missing the ADDITIONAL_WCS_ORDERS value in the .mdc file
 * MEH: ippc120 will be used by MOPS at the ITC for stamps once nightly processing starts there
=== Tuesday : 2017.02.28 ===
 * MEH: another case of WSdiffs queuing once the diffs are cleaned (but warps are still full) -- so not a fluke and a bug in new tag somewhere... ~ippqub/stdscience_ws diff.off until mess cleaned up -- 
  * looks like changes to difftool.c/difftool_definewarpstack.sql during LAP.PV3 WSdiff processing has broken something
  * WSdiff will need to be manually queued until problem fixed.. -- turning survey WSdiff off -- survey.stage warpstack.diff off
=== Wednesday : YYYY.MM.DD ===

=== Thursday : 2017.03.02 ===
 * MEH: Haydn working on ipp078, ipp121 and powerdown required -- in addition to removing from processing, neb-host should also be dropped to repair (and really down while machine is down) to avoid odd file states while work being done -- also good to not only take out of processing in ~ipp/ippconfig/pantasks_hosts.input but also to actually take it out of ACTIVE processing ./check_system.sh hostoff ipp078
  * ipp078 -- replace troublesome OS mirror disk -- problem with mirror on sdb and root on sdc (backwards) and booting off sdb and sdc stopped working -- both SSD seem to be working ok currently so leaving as is until something fails -- however, going to be leaving out of processing and neb-host repair until monday...
  * ipp121 -- again went unresponsive to ssh connections but nfs could still read the disk -- rebuild root disks and fixed 1G back to 10G connection -- 
   * problems connecting -- /etc/ssh files needed to be restored
   * exports not working -- /etc/exports file needed to be restored
   * ganglia not working -- /etc/ganglia/gmond.conf file needed to be repointed to ippc19/10.10.20.17
   * automounts not working -- /etc/autofs/auto.data old version and newer version from ipp122 copied over (since generic file)
   * leaving in neb-host repair and still out of processing until at least monday to see how it goes

=== Friday : 2017.03.03 ===
 * MEH: Haydn replaced BBU in ipp083, ipp088, ipp092 -- neb-host repair until tomorrow (disks mostly full so not a big issue)
 * MEH: Haydn working on ipp078 again -- neb-host repair and out of processing again until tomorrow
 * MEH: re-targeting data again off ipp067,082-097, ipp100,102,103,104.0 to reduce the level of random
=== Saturday : 2017.03.04 ===
 * MEH: ipp078 seems to be having problem again with sdc mirror disk, Haydn will look into it next week -- neb-host up but light on processing 
 * 18:20 EAM: for the record:
{{{
it looks like the weather will be good tonight, so we should get some data.   I have therefore stopped the mysql replication on the gpc1 copy on ipp114.  This will allow me to use the data collected tonight as a test case tomorrow and/or monday to run the ITC IPP in parallel to the MRTC-B IPP.

I am not going to stop the nebulous copy on ipp115 until tomorrow.
}}}
=== Sunday : 2017.03.05 === 
 * MEH: Weryk reports pstamp isn't providing chips properly now -- looks like issue in new tag/trunk possibly related to WCS changes
 * MEH:ipp121 seems ok for past few days -- neb-host up since really need the 2 disk partitions with space