PS1 IPP Czar Logs for the week 2012-05-14 -- 2012-05-20

(Up to PS1 IPP Czar Logs)

Monday : 2012.05.14

Serge is czar

• 00:05 Mark: wave1 hosts not very happy when system is full use, may be overloading. manually removing 1 group from stdscience to see how that helps. ipp021 was also being killed by a ppSub using >90% RAM (neb://ipp021.0/gpc1/OSS.nt/2012/05/14/RINGS.V3/skycell.1560.018/RINGS.V3.skycell.1560.018.dif.238554)
• 06:30 Serge: Nightly processing almost complete (~50 3pi exposures). ipp002 is down. Gavin rebooted it around 7:30.
• 09:00 Serge: Nightly processing complete.
• 13:50 Serge: Ganglia hiccups pattern: restarting stdscience
• 14:20 Serge: Killed 4 ppImage that were drowning ipp039
• 15:19 Serge: ipp002 has a huge load because of apache(?)
• 17:00 Bill: stopping pstamp, distribution, and publishing in preparation for rebuilding the ippRequestServer database
• 18:00 Bill: testing re-ingest was taking too long. I will change the database tomorrow. pantasks' restarted

Tuesday : 2012.05.15

• 08:15 Bill pstamp, distribution and publishing set to stop. Waiting for mysql slave on ippc19 to catch up then we will rebuild the ippRequestServer database on ippc17 using InnoDB storage engine.
• 10:11 CZW: Restarting stdscience to hopefully gain some speed and track down jobs taking longer than they should.
• 10:45 Bill ippRequestServer has been restored with InnoDB tables. New version inserted into replicated database on ippc19. Slave restarted. Now caught up. Thanks Serge.
• 14:08 Bill dropped 875 chipProcessedImfiles whose state was not full and quality non-zero even though the corresponding rawImfile was in ignored state. Warp updates get stuck if somebody asks for a skycell thinks it needs a chip in that state.

Wednesday : 2012-05-16

• 00:57 CZW: restarted stack, which apparently committed suicide between 18:00 yesterday and now.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD