PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.02.27

  • MEH: re-deploying ipp-20170121 as the ops tag -- online
    • night 20170217 with buggy ipp-20170121 used is okay except for warp->diff updates where the correction for the 0.1pix offset won't be applied since it is missing the ADDITIONAL_WCS_ORDERS value in the .mdc file
  • MEH: ippc120 will be used by MOPS at the ITC for stamps once nightly processing starts there

Tuesday : 2017.02.28

  • MEH: another case of WSdiffs queuing once the diffs are cleaned (but warps are still full) -- so not a fluke and a bug in new tag somewhere... ~ippqub/stdscience_ws diff.off until mess cleaned up --
    • looks like changes to difftool.c/difftool_definewarpstack.sql during LAP.PV3 WSdiff processing has broken something
    • WSdiff will need to be manually queued until problem fixed.. -- turning survey WSdiff off -- survey.stage warpstack.diff off

Wednesday : YYYY.MM.DD

Thursday : 2017.03.02

  • MEH: Haydn working on ipp078, ipp121 and powerdown required -- in addition to removing from processing, neb-host should also be dropped to repair (and really down while machine is down) to avoid odd file states while work being done -- also good to not only take out of processing in ~ipp/ippconfig/pantasks_hosts.input but also to actually take it out of ACTIVE processing ./check_system.sh hostoff ipp078
    • ipp078 -- replace troublesome OS mirror disk -- problem with mirror on sdb and root on sdc (backwards) and booting off sdb and sdc stopped working -- both SSD seem to be working ok currently so leaving as is until something fails -- however, going to be leaving out of processing and neb-host repair until monday...
    • ipp121 -- again went unresponsive to ssh connections but nfs could still read the disk -- rebuild root disks and fixed 1G back to 10G connection --
      • problems connecting -- /etc/ssh files needed to be restored
      • exports not working -- /etc/exports file needed to be restored
      • ganglia not working -- /etc/ganglia/gmond.conf file needed to be repointed to ippc19/10.10.20.17
      • automounts not working -- /etc/autofs/auto.data old version and newer version from ipp122 copied over (since generic file)
      • leaving in neb-host repair and still out of processing until at least monday to see how it goes

Friday : 2017.03.03

  • MEH: Haydn replaced BBU in ipp083, ipp088, ipp092 -- neb-host repair until tomorrow (disks mostly full so not a big issue)
  • MEH: Haydn working on ipp078 again -- neb-host repair and out of processing again until tomorrow
  • MEH: re-targeting data again off ipp067,082-097, ipp100,102,103,104.0 to reduce the level of random

Saturday : 2017.03.04

  • MEH: ipp078 seems to be having problem again with sdc mirror disk, Haydn will look into it next week -- neb-host up but light on processing
  • 18:20 EAM: for the record:
    it looks like the weather will be good tonight, so we should get some data.   I have therefore stopped the mysql replication on the gpc1 copy on ipp114.  This will allow me to use the data collected tonight as a test case tomorrow and/or monday to run the ITC IPP in parallel to the MRTC-B IPP.
    
    I am not going to stop the nebulous copy on ipp115 until tomorrow.
    

Sunday : 2017.03.05

  • MEH: Weryk reports pstamp isn't providing chips properly now -- looks like issue in new tag/trunk possibly related to WCS changes
  • MEH:ipp121 seems ok for past few days -- neb-host up since really need the 2 disk partitions with space