PS1 IPP Czar Logs for the week 2015.10.12 - 2015.10.19

Extra/Non-standard Processing

Continuing HAF suggestion to improve communication, so we know what's going on better -- a list in the czar pages additional (non-standard) processing - so that we all know what's going on.

Daily Czaring:

• currently there is a modified ops tag running diffs (WS labels only) as ippqub (was ippmops) under ~ippqub/src/stdscience_ws on ippc06 (was ippc29) -- if problems (Njobs>100k, power loss on ippc06 etc), it will need to be restarted like a normal nightly processing pantasks

  ./start_server.sh stdscience_ws
  

  • even with the modified ops tag, there are still various files MIA in nebulous and as such requires the daily czar to check and clear them as has been discussed before

MD processing:

• ippmd/stdscience running WS diffs w/o writing images -- using ippx065-x096 (hosts_xmd) -- stop as necessary, but always communicate doing so

(Up to PS1 IPP Czar Logs)

Monday : 2015.10.12

• 05:00 EAM: ipp023 crashed, rebooting it.
• 11:00 MEH: normal restart of nightly pantasks
• 18:00 MEH: plan is not to switch back to ippc18 as home disk unless ippc19 has problems -- so add new ipp user Archive_c19.1 symlink for pantasks logs and start the rsync+bzip to clear up space
• 20:20 MEH: clearing stalled warp -- cannot build growth curve (psf model is invalid everywhere)

  warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.0947.044 -warp_id 1628049 -fault 0
  

Tuesday : 2015-10-13

• 16:00 CZW: I'm restarting ipp/replication to run rawcheck.pl commands to shuffle raw data to ippb06.X.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD