PS1 IPP Czar Logs for the week 2015.11.30 - 2015.12.06

Continuing HAF suggestion to improve communication, so we know what's going on better -- adding a list in the czar pages for additional (non-standard) processing so that we all know what's going on.

Daily Czaring:

• MOPS has repeatedly requested NO changes be made to the ~ipp ops tag without being verified by their test set -- so currently there is a modified ops tag running diffs (WS labels only) as ippqub under ~ippqub/src/stdscience_ws on ippc06 -- if problems (Njobs>100k, power loss on ippc06 etc), it will need to be restarted like a normal nightly processing pantasks (or IF ANY OTHER CHANGES made for the ~ipp pantasks then likely same needs to be made for ~ippqub)

      ./start_server.sh stdscience_ws
  

  • even with the modified ops tag, there are still various files MIA in nebulous and fault problems that requires the daily czar to check and clear them as has been discussed before
  • ps_ud_QUB has also been moved to the ippqub:stdscience_ws pantasks to support updates possibly broken by missing cmf files, chip and warp updates will also be done in this pantasks as well

(Up to PS1 IPP Czar Logs)

Monday : 2015.11.30

• 04:53 Bill: ganglia says ipp015 has been down for over 1900 seconds. This was causing chip processing faults due to unavailable detrend files. Set ipp015 to down in nebulous.
• 06:25 EAM : rebooted ipp015 and set to 'repair' in nebulous
• 08:00 Bill : raised priority of ps_ud_MOPS so the tiny number of update requests that their postage stamp requests need get serviced during nightly processing.
  • MEH: just a note on priority -- the QUB.nightly will get raised above when a trigger for processing is made
• 16:55 CZW: Starting up ipplanl/pv3update pantasks as ipplanl. Server host is ippc11, and the label is PV3.stsci.update. This will process all of the updates needed for the stsci holes. I've assigned 4x ippx* nodes, and will adjust tomorrow if this is too much/too little/etc.
  • I'm having trouble with my update commands, so this will most likely remain stopped until tomorrow.
• 17:00 CZW: Restarting ipp pantasks.

Tuesday : 2015.12.01

• 00:26 MEH: if changes to ~ipp are made, same changes need to be made for ~ippqub... ippc02 needs to be commented out as a nebulous host as well if it is down.. -- doing and restarting ippqub:stdscience_ws pantasks..
• 14:36 CZW: Just noticed ipp046 is down. Can't ssh, ganglia shows it as down. Cycling power.
• 17:00 CZW: There was a disk space recovery spike. I moved a set of chip-stage cleanup log files out of the way, as they had "no instance found" errors (likely due to the instance being on the stsci nodes). This cleared the issues with chip cleanup, and allowed a backlog to be cleaned.

Wednesday : 2015.12.02

• 09:20 Bill: restarted postage stamp server pantasks after integrating a change to the postage stamp parser to allow it to find the correct camRun for a given release.

Thursday : 2015.12.03

Friday : 2015.12.04

Saturday : 2015.12.05

Sunday : 2015.12.06