PS1 IPP Czar Logs for the week 2014.04.28 - 2014.05.04

(Up to PS1 IPP Czar Logs)

Monday : 2014.04.28

mark is czar

• 07:00 MEH: chip.off to push remaining night warp+diffim through (lots of LAP chips..). two fault 5 OSS diffim cleared, small stamp poor stats robustness (543558 skycell.1014.058; 543560 skycell.1014.001)
• 08:10 MEH: stdsci poll underloaded, nightly finished, regular restart of pantasks to begin
  • have to cycle chip.off to mostly push backlog of LAP warps out, seems mixed and need all out to start making stacks..
• 10:20 MEH: LAP stacks now loading, will need to throw stdsci nodes into stack -- if like yesterday ~3500 stacks/400hr then mostly finished by nightly obs, but if normal night processing then leaving ~1k keeps the few stack nodes busy all night
• 10:40 MEH: ipp053 was never put back into processing after the mobo was replaced on 3/20..
• 18:40 MEH: ~800 LAP stacks remain, returning power to stdsci for nightly (after pantasks restart)

Tuesday : 2014.04.29

• 10:04 Bill: staticsky memory use is mostly staying around 30G peak. Set one set of c2 notes to on in stack.
• 11:22 Bill: ganglia reports that ippc13 has been down for 6 1/3 hours. I set the power to off and notified Haydn.
• 15:30 CZW: I've restarted the replication pantasks, pointing it at the new permcheck.pl script and the PV1 stacks. This will shuffle the backup image copy to the b storage nodes.
• 20:20 MEH: ipps05-14 machines manually allocating to staticsky until next MD run ready -- 2x should be possible; ipps01-04 3x possible

Wednesday : 2014.04.30

• 14:30 CZW: ippdb01 seems to have crashed. Nothing on the console except iostat/temperature checks someone had done that were left in the log. Power cycled, and it seems to up and responding again. I'll use this as an opportunity to restart the pantasks servers.
• 15:30 CZW: Just realized that the reason the czartool page wasn't updating was because the crash of ippdb01 probably killed the scripts that run those jobs. That was the case. Restarted.

Thursday : 2014.05.01

• 10:11 Bill: ippc26, 27, and 28 have got extremely large psphotStack processes. Set those nodes to off in the pstamp and stack pantasks.
  • 11:10 ippc27 and 28 have finished and the one on ippc26 is in the process of exiting.
  • we are out of the galactic plane for now. Enabling a second set of c2 nodes in staticsky
  • 11:40 MEH: remember to bump up the poll number, the ipps nodes will float in/out as available

• 15:20 EAM : stopped and restarted stdscience, pstamp, registration (pantasks were running slow)
• 15:25 EAM : ipp051 is down. trying to power cycle. so far, no luck.

Friday : 2014.05.02

Saturday : 2014.05.03

Sunday : 2014.05.04