== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2018.07.23 ===
 * MEH: Haydn wants to reboot ippc19 to see if fixes errors -- 
 * MEH: start running PS2 WSdiffs during night -- stdscience default set unwant 5... needs to be bumped up to 30 or so
  
=== Tuesday : 2018.07.24 ===


=== Wednesday : 2018.07.25 ===
 * MEH: unhandled camera quality fault again due to no stars -- o8324g0525o -- manually set quality 42 and state full
 * MEH: PS2 grid angle reduce to +-3 from +-10 degree in gpc2/psastro.config 
 * MEH: PS2 WSdiffs running during night may need more nodes -- 
 * MEH: PS1 zEuclid observations during full moon for next 4 nights -- looks like comment string overloaded and missing visit # -- will need to manually queue WWdiffs, changed warp label to Bright.<field>visit1 for the first in list of exposures with warp label Bright.nightlyscience and so on, then queue by warp label visit1/visit2, visit3/visit4 

=== Thursday : 2018.07.26 === 
 * MEH: manually setup PS1 zEuclid WWdiffs, manually publish single field zEuclid WWdiff to MOPS -- tried to do stage publishing in groups since not able to currently parse comment string, tedious so doing in bulk seemed to work out ok
 * MEH: non-critical bias faults not cleared
{{{
2 summit faults
  exp_name         registered      fault
c8325h0018b   2018-07-26 07:51:23   110
c8325h0019b   2018-07-26 07:59:53   110
}}}

=== Friday : 2018.07.27 ===
 * MEH: PS2 OSS pointing wrong for o8326h0033o for target_id, nightly auto-queue still made an empty WWdiff with no overlapping skycells.. have to manually queue v2-3
 * MEH: manually queue PS1 zEuclid WWdiffs again and manually publish all to MOPS -- after clearing multiple warp faults due to problems with ippx nodes out of disk space...
 * MEH: the ippx nodes running PS2 nightly processing were a minefield waiting to bork processing -- manually forced log rotation /var/log/messages (>14G..) and Heather helped clear out the excessive mysql data so now they should be more reliable...
 * MEH: added PS2 masking issues to [wiki:IPP_for_PS2]
 * MEH: ippc19 having issues again -- Haydn rebooted but had to power cycle, still another disk problem but binaries working again -- ganglia reporting some ippb nodes down, Gavin fixed 

=== Saturday : 2018.07.28 ===
 * MEH: Angie will try to modify the comment string for zEuclid moon observations to the format {{{Euclid r0000d30S.8326.10b.z ps1_29_1244 Day 1 visit 1}}} that should help make queuing the WWdiffs easier -- leaving auto-queuing off for Bright.nightlyscience WWdiff regardless
 * EAM: updated xattr for ippb00 - ippb03 to 5 as these machines are now retired (should prevent nebdiskd from checking on them); changed .ssh known_host entries for ippb18, b21, b22 for ipp user so nebdiskd can get their df values

=== Sunday : YYYY.MM.DD ===