PS1 IPP Czar Logs for the week 2016.12.05 - 2016.12.11

(Up to PS1 IPP Czar Logs)

Monday : 2016.12.05

• MEH: large set QUB/MOPS warp updates/reprocessing continues and targeting ipp100+, running ~ippmops/stdscience using ipps and c2 (normally used for QUB WS and stacks) -- if pantasks is stopped, then you must notify me directly as it can block MOPS+QUB stamps

Tuesday : 2016.12.06

Wednesday : 2016.12.07

• MEH: ipp118-122.x were not completely setup and/or checked for ipp use, link for /local/ipp broken -- fixed, adding as group s7 in pantasks_hosts.input, adding to nightly science targeting and heavily loading CPU and IO with MOPS/QUB processing until weather clears
• MEH: ippcXX nodes lately have been having NFS hangs that stall MOPS pstamp request (and QUB if there had been any data) -- restarting nfs/rpc.statd seems to clear -- czar needs to keep eye on
• MEH: stamps/updates for LAP.PV3 showing ongoing faults due missing files from stsci shuffle -- some reporting ipp118 but looks like due to IP being reused
• MEH: gpc1 dumps to ipp001 have been 0 size for past couple days it appears due to /data/ipp001.0 being full -- adding an email warning crontab in ipp@ipp001, running cleanup script and rm'ing incomplete dumps so not to contaminate archive...
  • will set one up in ipp@ippc18 to also monitor homedir
• MEH: Haydn to replace ipp122 10G card and wanted to replace ipp083 BBU -- out of processing and neb-host down
  • ipp122 back up but cannot log into while disks mount ok -- leave out of processing and in repair -- then later was able to log into again, gmond restart and neb-host up
  • ipp083 unclear, leave in repair until morning -- BBU will be replaced in next few days, neb-host repair when nightly starts as it is in WT state

Thursday : 2016.12.08

• MEH: no nightly data tonight, leaving MOPS/QUB processing at day rate

Friday : 2016.12.09

Saturday : 2016.12.10

Sunday : 2016.12.11