== PS1 IPP Czar Logs for the week 2014.12.01 - 2014.12.07 == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : 2014.12.01 === * 09:53 Bill: restarted distribution pantasks which was sluggish with diff stage enabled. SAS distribution is all finished except for diff stage. * 13:00 Bill: ~ippsky is rerunning staticsky and skycal for sas.37 * 17:00 MEH: setting up ippx049+ for ippsky staticsky/cal to use -- manually adding when ready, ~3x should be fine for 50G ram * ippx049--x088 have been setup and added temporarily to a x2 group (need split by pdu?), after Haydn's email x089-x092 also -- see x2 group for ones still down * 22:00 MEH: nightly processing not doing well w/ ippmd and ippsky running (or something else?) -- ipp083-087 overloaded/not responding and stalling registration === Tuesday : 2014.12.02 === * 07:11 Bill: started up fullforce in ~ippsky/staticsky (staticsky and skycal are finished except for one skycell that faulted with a data error. It is running now * 07:30 MEH: registration backed up quite a bit, too many nodes in use -- ippmd stop until nightly finishes * '''looks like something may have started ~0200, nightly rate dropped greatly then as well.. and not just for stdsci''' -- looking at the storage node network, appears the red storage nodes hit a data block about that time and there it is... * 08:30 MEH: taking some xNNN nodes out of ippsky until nightly finishes.. then probably could do 4x on those for FF * 08:40 MEH: an exposure (o6993g0619o) is stalled in registration.. been a while so restart summitcopy and registration (then stdsci once nightly is finished) -- had to manually run {{{ regtool -dbname gpc1 -revertprocessedexp -exp_id 829654 }}} * 09:30 MEH: ipp056,058,060 have space and could be moved out of repair to help ease load on new storage nodes and get nightly finished * same for ipp012,013,029,030 * now get to see those load >50 -- so even when get 10G for ipp083-088 setup, the other 1G systems will get large loads? * 12:00 MEH: warp growth curve fault -- {{{ warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1241149 -skycell_id skycell.1616.069 }}} * 12:30 MEH: nightly finally finished, while waiting for storage node reboots, going to do the regular restart of stdsci and set to stop * 12:50 MEH: adding host groups s4,s5 for the new storage hosts ipp067-070,ipp071-088 respectively. will start testing running jobs on them including deep stacks and should not be allocated for any other jobs at this time * 13:15 CZW: ran {{{check_system.sh stop}}} to ensure IPP processing is off while we do 10G reboots (and hopefully scidb switch). * 14:00 MEH: not doing the scidb/ippdb01/05 switch, might as well use this standby time to restart the mysql on ippdb01 (been ~10d or so) * restarted czarpoll and roboczar on ippc11 * 15:47 Bill: found another bug in skycal. skycal will need to be rerun for SAS.37 (as does fullforce) * 16:00 MEH: Haydn & Gavin finished config and reboot of storage nodes. CZW tested r/w on storage nodes, look like ready to go. set pantasks to run and shortly after ipp077 crash -- set to neb-host down * 17:50 MEH: Gavin restored ipp077 back w/ 1G network, problem w/ 10G card? put back to neb-host repair * 18:30 MEH: ipp013 appears unresponsive.. ipp012 and some others did for brief period of time as well.. neb-host down for a bit just in case.. * console showing mem check.. looks like it power cycled.. * '''ipp008,012,013,014,016,018,037''' actually were all power cycled according to ganglia.. oh why.. * 19:00 MEH: nightly seems to be stalled.. lanl stdlocal and ippmd stop.. * restarted registration seems to have cleared the problem... was stuck on all ota like {{{ o6994g0004o XY76 0 pending_burntool neb://ipp053.0/gpc1/20141203/o6994g0004o/o6994g0004o.ota76.fits }}} * 19:10 registration finally caught up.. * 20:00 MEH: starting ippsky/staticsky processing in place of stdlocal as SAS has priority -- attempting to balance after skycal finishes to try and finish SAS by morning w/o impacting nightly processing * skycal finished, fullforced doesn't seem to be running after 20 mins.. -- c2 off, leaving 2x x2 on in case something happens -- ippsky to stop * 20:10 MEH: manually adding ipp067 and ipp072 to summitcopy to test, red and data blocked so if crash will minimize cleanup.. * 21:05 MEH: ippmd run, stdlocal run, watching load -- stdlocal turn off all x0,x1 so ~135 jobs running (ippmd ~170 jobs) -- BUT this is w/o diffims being done in stdsci apparently.. * 23:45 MEH: '''no WW diffs for MOPS being made..''' -- 3 chunks available and nothing being queued.. -- CZW noticed initday not run today due to downtime, when added diffims now queued * 00:30 MEH: warp faults -- cannot build growth curve (psf model is invalid everywhere) {{{ warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1241654 -skycell_id skycell.2295.051 warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1241730 -skycell_id skycell.2295.051 }}} * 01:00 MEH: nightly processing seems to have screeched to a halt -- ipp060 is terrible wait_cpu state, neb-host repair finally cleared. stdlocal and ippmd stop until cleared and then watch similar loading again (~135 stdlocal, ~170 ippmd) * rate just isn't great, so stdlocal and ippmd off until morning. needs more tuning and since ippmd stop, put some ipps nodes into stdsci (will remove in morning) === Wednesday : 2014.12.03 === * 8:00 HAF I cleared up some faults in registration: {{{ regtool -revertprocessedimfile -dbname gpc1 -exp_id 830390 regtool -revertprocessedimfile -dbname gpc1 -exp_id 830392 regtool -revertprocessedimfile -dbname gpc1 -exp_id 830393 }}} This allowed all but 1 exposure to finish registration ( 0471) - the obvious tricks aren't working, I need to think a bit how to fix it. * 9:53 HAF: the last exposure is done, it fixed itself (?) * 10:40 MEH: nightly xSS finished w/ diffims, turning on ippsky for SAS FF w/ 2x x2 group to start with and ippmd after removing ipps from stdsci * seems fine, adding more x2 -- 4x x2, poll 200, rate ~250-300s per, maybe finish ~8-10hr? * 11:30 MEH: all nightly finished, putting update labels back in * 12:30 MEH: stdlocal only has 2x x0+x1 loaded, adding 1x x0+x1 into ippsky as well * 16:25 CZW: Good news: I fixed the warp-stack issue that was preventing these from being queued. Bad news: there are now 1150 diffs that want to run. * 16:54 CZW: ipp034 died again. Nothing informative on the console. Power cycled. It's already in repair, so there's not much more to do when it comes back. === Thursday : 2014.12.04 === * 08:30 MEH: reducing ippsky by 2x x2 to use in ippmd processing -- SAS finished except for faults and should be enough nodes to finish those if necessary * 10:30 CZW: stsci03 is back online, and is set to repair in nebulous. * 10:35 MEH: MSS diffim fault 5 needed to be dropped so MOPS can finish their processing.. -- similar problem {{{ -> VectorFitPolynomial1DOrd (psMinimizePolyFit.c:633): unknown psLib error Could not solve linear equations. -- so just marked quality 42 difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 613152 -skycell_id skycell.1099.062 -dbname gpc1 }}} * 11:00 MEH: deep stack test runs on ipp068,072 to start -- will put into repair if not finished by time observations being * in doing this, discovered /local/ipp symlink not setup... will check others and fix as necessary -- 073-075,077, 083-088 as well * 11:15 CZW: I've set chipRuns with label=LAP.PV3.20140730.ipp, data_group=LAP.PV3.20140730.ipp.20141129 to label LAP.PV3.20140730 to migrate this processing to LANL. These chips are behind a large backlog locally, and as the LANL queue is largely empty, I'd rather push them there to get processing time. I may repeat this with other local data groups if this works well to clear the .ipp runs in favor of the the .bigmem ones. * 12:30 MEH: ippmd using all of x2 group -- 640 jobs running at once now * 13:00 MEH: Gavin changed the rwsize for mounts on 10G (071-076,078-082) -- will see how it goes * 13:30 CZW: full ipp/ipplanl pantasks restart. * 14:45 MEH: massive overloads on ipp083,085,088 -- put to repair and see how it goes for now -- ippc0x nodes also overload but suspect due to waiting on transactions from ipp083,085,088 * then started picking on the other 1G -- 056,058,086,084 -- so go just 10G for a bit -- * actually not clear 10G vs 1G, or rather the 1G mostly all have bad batteries and in WT not WB for raid.. even ipp056, 058 * ipp060 has good battery, but looks like problem from dmesg {{{ [2014-12-03 00:38:16] SLAB: Unable to allocate memory on node 0 (gfp=0x20) [2014-12-03 00:38:16] cache: TCP, object size: 1728, order: 1 [2014-12-03 00:38:16] node 0: slabs: 8/8, objs: 32/32, free: 0 [2014-12-03 00:45:30] mrmonitord: page allocation failure: order:1, mode:0x200020 [2014-12-03 00:45:30] Pid: 15556, comm: mrmonitord Tainted: G O 3.7.6 #1 ... }}} * 15:05 MEH: ipp072 then ipp068 high loads, suspending deep stack test === Friday : 2014.12.05 === === Saturday : 2014.12.06 === * '''revised''' -- planned downtime MRTC-B 590B Lipoa Automatic Transfer Switch (ATS) -- Saturday December 6th. They will be starting our shutdown processes at 9am and will hopefully start turning things back on at 3pm. * '''all processing stopped before ~7am''' * Haydn on site to start power down of systems @7am === Sunday : 2014.12.07 ===