PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.02.27

• MEH: re-deploying ipp-20170121 as the ops tag -- online
  • night 20170217 with buggy ipp-20170121 used is okay except for warp->diff updates where the correction for the 0.1pix offset won't be applied since it is missing the ADDITIONAL_WCS_ORDERS value in the .mdc file
• MEH: ippc120 will be used by MOPS at the ITC for stamps once nightly processing starts there

Tuesday : 2017.02.28

• MEH: another case of WSdiffs queuing once the diffs are cleaned (but warps are still full) -- so not a fluke and a bug in new tag somewhere... ~ippqub/stdscience_ws diff.off until mess cleaned up --
  • looks like changes to difftool.c/difftool_definewarpstack.sql during LAP.PV3 WSdiff processing has broken something
  • WSdiff will need to be manually queued until problem fixed.. -- turning survey WSdiff off -- survey.stage warpstack.diff off

Wednesday : YYYY.MM.DD

Thursday : 2017.03.02

• MEH: Haydn working on ipp078, ipp121 and powerdown required -- in addition to removing from processing, neb-host should also be dropped to repair (and really down while machine is down) to avoid odd file states while work being done -- also good to not only take out of processing in ~ipp/ippconfig/pantasks_hosts.input but also to actually take it out of ACTIVE processing ./check_system.sh hostoff ipp078
  • ipp078 -- replace troublesome OS mirror disk -- problem with mirror on sdb and root on sdc (backwards) and booting off sdb and sdc stopped working -- both SSD seem to be working ok currently so leaving as is until something fails -- however, going to be leaving out of processing and neb-host repair until monday...
  • ipp121 -- again went unresponsive to ssh connections but nfs could still read the disk -- rebuild root disks and fixed 1G back to 10G connection --
    • problems connecting -- /etc/ssh files needed to be restored
    • exports not working -- /etc/exports file needed to be restored
    • ganglia not working -- /etc/ganglia/gmond.conf file needed to be repointed to ippc19/10.10.20.17
    • automounts not working -- /etc/autofs/auto.data old version and newer version from ipp122 copied over (since generic file)
    • leaving in neb-host repair and still out of processing until at least monday to see how it goes

Friday : 2017.03.03

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD