PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2018.07.23

• MEH: Haydn wants to reboot ippc19 to see if fixes errors --
• MEH: start running PS2 WSdiffs during night -- stdscience default set unwant 5... needs to be bumped up to 30 or so

Tuesday : 2018.07.24

Wednesday : 2018.07.25

• MEH: unhandled camera quality fault again due to no stars -- o8324g0525o -- manually set quality 42 and state full
• MEH: PS2 grid angle reduce to +-3 from +-10 degree in gpc2/psastro.config
• MEH: PS2 WSdiffs running during night may need more nodes --
• MEH: PS1 zEuclid observations during full moon for next 4 nights -- looks like comment string overloaded and missing visit # -- will need to manually queue WWdiffs, changed warp label to Bright.<field>visit1 for the first in list of exposures with warp label Bright.nightlyscience and so on, then queue by warp label visit1/visit2, visit3/visit4

Thursday : 2018.07.26

• MEH: manually setup PS1 zEuclid WWdiffs, manually publish single field zEuclid WWdiff to MOPS -- tried to do stage publishing in groups since not able to currently parse comment string, tedious so doing in bulk seemed to work out ok
• MEH: non-critical bias faults not cleared

  2 summit faults
    exp_name         registered      fault
  c8325h0018b   2018-07-26 07:51:23   110
  c8325h0019b   2018-07-26 07:59:53   110
  

Friday : 2018.07.27

• MEH: PS2 OSS pointing wrong for o8326h0033o for target_id, nightly auto-queue still made an empty WWdiff with no overlapping skycells.. have to manually queue v2-3
• MEH: manually queue PS1 zEuclid WWdiffs again and manually publish all to MOPS -- after clearing multiple warp faults due to problems with ippx nodes out of disk space...
• MEH: the ippx nodes running PS2 nightly processing were a minefield waiting to bork processing -- manually forced log rotation /var/log/messages (>14G..) and Heather helped clear out the excessive mysql data so now they should be more reliable...
• MEH: added PS2 masking issues to IPP_for_PS2
• MEH: ippc19 having issues again -- Haydn rebooted but had to power cycle, still another disk problem but binaries working again -- ganglia reporting some ippb nodes down, Gavin fixed

Saturday : 2018.07.28

• MEH: Angie will try to modify the comment string for zEuclid moon observations to the format Euclid r0000d30S.8326.10b.z ps1_29_1244 Day 1 visit 1 that should help make queuing the WWdiffs easier -- leaving auto-queuing off for Bright.nightlyscience WWdiff regardless
• EAM: updated xattr for ippb00 - ippb03 to 5 as these machines are now retired (should prevent nebdiskd from checking on them); changed .ssh known_host entries for ippb18, b21, b22 for ipp user so nebdiskd can get their df values

Sunday : YYYY.MM.DD