PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

Continuing HAF suggestion to improve communication, so we know what's going on better -- adding a list in the czar pages for additional (non-standard) processing so that we all know what's going on.

Daily Czaring:

• MOPS has repeatedly requested NO changes be made to the ~ipp ops tag without being verified by their test set -- so currently there is a modified ops tag running diffs (WS labels only) as ippqub under ~ippqub/src/stdscience_ws on ippc06 -- if problems (Njobs>100k, power loss on ippc06 etc), it will need to be restarted like a normal nightly processing pantasks (or IF ANY OTHER CHANGES made for the ~ipp pantasks then likely same needs to be made for ~ippqub)

      ./start_server.sh stdscience_ws
  

  • even with the modified ops tag, there are still various files MIA in nebulous and fault problems that requires the daily czar to check and clear them as has been discussed before
  • ps_ud_QUB has also been moved to the ippqub:stdscience_ws pantasks to support updates possibly broken by missing cmf files, chip and warp updates will also be done in this pantasks as well

(Up to PS1 IPP Czar Logs)

Monday : 2015-12-07

• 19:55 CZW: restarted summitcopy and ippqub/stdscience_ws as they were crashed, and the email thing was complaining.

Tuesday : 2015.12.08

• 02:20 MEH: nightly processing got stuck, oddly some nodes just stalling jobs in reg+stdsci for 2+ hrs.. -- ipp065, 055, 005, 069, 089 (ipp055 common only to both)..
• 03:10 MEH: registration stuck on o7364g0449o and needed manual revert, ipp087 also having load spikes and high cpu wait so put neb-host repair for a bit (BBU seems fine)
• 06:50 Bill: Registration is 137 exposures behind. Burntool status shows a faulted imfile. Reverted it and data is flowing again

  % regtool -revertprocessedimfile -exp_id 1001097
        Updated 1 rawImfile
  

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD