| | 161 | * 22:59 CZW: noticed stacks were stuck. It looks like the problem is that the stack.summary stage runs at a particular time each night, and launched ~100 jobs. These jobs attempted to read data from stsci00, causing that host to grind to a halt trying to serve all the files. This hasn't been a problem since we generally don't do 10k stacks a day. I killed all the ppSkycell instances I could find, and that seems to have unstuck things. I've disabled stack.summary for the weekend, and will turn it back on Monday with a lower number of concurrent jobs. |
