PS1_IPP_Czarlog_20140210 – Pan-STARRS IPP

Context Navigation

Version 21 (modified by watersc1, 12 years ago) ( diff )
--

09:40 Heather stopping loaders
09:45 MEH: LAP label out of stdsci to try and catchup last nights data. ThreePi.WS.nightlyscience out as well
09:50 MEH: still seeing loader machines having trouble, putting neb-host repair and taking out of processing as it holds up summit+registration -- ipp043,048--062 many can probably be brought back up one at a time during LAP once things settle down
10:35 MEH: see MOPS stamps showing up, adding c3 to pstamp to help speed those along (should run fine with other normal processing jobs there, no ghost jobs...)
10:50 MEH: turning hosts on in summitcopy that are normally off to try and boost the downloads since many wave4 had to be turned off from fallout from loaders. will want those turned off afterwards unless monitored as they are untested for reliability.
11:15 CZW: A check of the summitcopy pantasks.stdout.log shows that all failures this morning are "Nebulous::Client::stat - unhandled fault - database error: no storage object found" for ota42. This points to an issue with ipp017, so I have set that to repair in nebulous.
11:30 MEH: ipp054 is in a bad state, probably needs to be rebooted -- so out of processing (wave4_ignore) and in neb-host repair
16:45 MEH: turning staticsky off on stare nodes so psps loaders can start using again, will take a couple hours to clear

14:00 Bill: removed right ascension limit on staticsky so that all pending jobs will run. Once this set clears out we'll start queueing new ones and when I do that I'll order them by slices.
15:00 CZW: restarted all pantasks for the daily restart. This should pull the new pantasks.pro code that properly removes CRASH-ed job pages.

12:20 CZW: I just noticed that we have a very large number of chips/warps in the processing queue. There are now 274 lapRuns active, up significantly from the ~75 a few days ago. It appears that the cleanup task is timing out for the one run still in 'full' state, and when it times out, it doesn't correctly clear out the full run, leading to many more lapRuns being launched that is expected. I've pointed ~ipp/lap/current.queue to the empty off.queue, which will prevent new runs from being launched (so we can taper back down to ~75). I'll manually clear the full run, and figure out how long to set the timeout limit to.
14:40 CZW: A number of pantasks servers are not running. I'm stopping the remaining ones for a fresh restart.
15:20 CZW: Back up and running. The job rate was remarkably stable over the past day or so. The current hypothesis is that the slow downs we've been seeing in the past after 100k jobs may have been due to crash/timeout pages filling up the book (that only requires a 1/1000 rate). With the fix to that a few days ago, we may not get slowdowns anymore. This is something to keep an eye on.
16:45 CZW: The last orphan ippsky jobs have finished, so I have restarted that pantasks as well.