PS1 IPP Czar Logs for the week 2013.07.15 - 2013.07.21

(Up to PS1 IPP Czar Logs)

Monday : 2013.07.15

• 10:25 Bill Mops reports slow postage stamp response. I restarted pstamp pantasks. It's pcontrol was spinning. Added 2 x compute3 and compute2.
• 11:05 Bill Mops is done. Updates are now the bottleneck. Restarted update pantasks. added two sets of compute 3 nodes. Removed them from pstamp
• 12:40 MEH: if 2x compute3 moved to update, need to turn off in stdsci or may overload -- loading up MD10.refstack.20130715 chip->warp now

Tuesday : 2013.07.16

• 08:49 Bill: The postage stamp server is behaving strangely today. MOPS has submitted a number of requests but each one is taking a very long time to parse into jobs. The result is we have a number of requests parsing (limited to 10 to avoid database overload) and no jobs to run. Once the requests parse they finish quickly. I'm not sure what to do? Perhaps the mysql on ippc17 is tired.
• 08:55 Bill: Had a look at ippc30 (where the postage stamp working directories live) top showed many apache processes and little free memory. Ran my program eatmem and the amount of memory cached dropped as expected. After the program finished we had 15G free and the apache processes went away in top and nfsd showed up. Restarted apache
• 09:04 Bill: Sample req_id 287529 submitted 08:03. Parsing begins 08:54. Parsing finishes 09:02 200 jobs complete 09:03. request finishes seconds later.
• 09:07 All caught up now. Strange.
• 10:40 new batch of MOPS pstamp requests. Same poor parse performance. Restarted pstamp and update pantasks.
• It appears that the speed of inserts into the pstampJob table is now a bottleneck
• 15:45 Bill restarted mysql on ippc17 and the pstamp pantasks
• 17:45 MEH: taking 4x compute3 from stdsci for local stdsci and deepstack runs

Wednesday : 2013.07.17

Thursday : 2013.07.18

Friday : 2013.07.19

Saturday : 2013.07.20

Sunday : 2013.07.21