PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.08.28

Tuesday : 2017.08.29

  • MEH: ipp076 neb-host up until red, more clean up some past data products has freed up ipp102.0 as well
    • adding ipp115.0 now that neb replication no longer there
  • MEH: ipp032 and ippc120 returned to IPP from past/possible MOPS use -- putting ippc120 into normal processing group ci1, ipp032 neb-host up for random nightly data

Wednesday : YYYY.MM.DD

Thursday : 2017.08.31

  • MEH: Weryk using nodes c23-c63, x065-x088, s00-s14 for tracklet linking again -- primarily CPU use, nice'd played nice with normal stack and other processing previously but keeping of of primary nightly science and data nodes
    • past 24 hours seems ok except for c23-c28 -- ppSub and stamp jobs stalled, possibly on file access -- as the c23-c25 (c0) and c26-c28 (part c2) are not part of normal processing (as-needed only), commenting them out in ~ippitc/ippconfig/pantasks_hosts.input
    • may be related to rsync stalling between compute nodes Weryk noticed yesterday when distributing 28GB file -- specific example ippc27 data to ippc42 (rate rapidly drops to few KB/s)

Friday : 2017.09.01

  • MEH: adding nodes to push through the large number of QUB stamps -- double allocation during day plus s3+4+5+6+7 added
  • 16:25 CZW: Restarting ippitc pantasks after clearing the /tmp/nebulous_server.log file on ippc73. The other nodes (ippc70-ippc75) are getting somewhat full, so I'll take another look at them next week.

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD