PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

Continuing HAF suggestion to improve communication, so we know what's going on better -- adding a list in the czar pages for additional (non-standard) processing so that we all know what's going on.

Daily Czaring:

  • MOPS has repeatedly requested NO changes be made to the ~ipp ops tag without being verified by their test set -- so currently there is a modified ops tag running diffs (WS labels only) as ippqub under ~ippqub/src/stdscience_ws on ippc06 -- if problems (Njobs>100k, power loss on ippc06 etc), it will need to be restarted like a normal nightly processing pantasks (or IF ANY OTHER CHANGES made for the ~ipp pantasks then likely same needs to be made for ~ippqub)
        ./ stdscience_ws
    • even with the modified ops tag, there are still various files MIA in nebulous and fault problems that requires the daily czar to check and clear them as has been discussed before
    • ps_ud_QUB has also been moved to the ippqub:stdscience_ws pantasks to support updates possibly broken by missing cmf files, chip and warp updates will also be done in this pantasks as well

(Up to PS1 IPP Czar Logs)

Monday : 2015-12-14

  • 16:27 CZW: Stopping processing for restart. Gene will be restarting gpc1 database as well.

Tuesday : 2015.12.15

  • 05:19 MEH: STScI is downloading SAS data from ippc26-ippc29, using 100 wget for total 400 connections to those 4 machines -- they are seeing ~200Mbps per machine. if notice any local issues, just need to let them know. net i/o in/out looks to be ~20MB/s
  • 12:26 MEH: sending some SNIa processing to cleanup. also noted old ws_nightly_update label not cleaned yet, those need to be remembered to be cleaned when finished with the update fix..
  • 19:50 EAM: czarpoll crashed due to yesterday's gpc1 restart. restarted it now.

Wednesday : YYYY.MM.DD

Thursday : 2015.12.17

  • 18:10 EAM: stopping pantaskes for a general restart.

Friday : YYYY.MM.DD

  • 23:40 EAM: stopping pantaskes for a general restart.

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD