PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.15

• 08:00 MEH: cleanup must be turned back on, the few number of nodes available for data is killing processing again
• 11:00 CZW: I've updated the database to prevent the exposures from the dead pixelserver from clogging up summitcopy:

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832987;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832988;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832989;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832990;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832991;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832992;
  

• 14:00 CZW: Restarting stdlocal. I'm going to see what happens with the "default" set of hosts.
  • 14:07 CZW: default set is underpowered. Turning on xnodes with xnode.hosts.on macro.
• 14:50 MEH: summitcopy stop -- camera group reporting ramping overload -- anyone else doing summitcopy type stuff?
• 14:45 CZW: I noticed more failing exposures in pzDownloadExp table, and dropped the camera controlled exposures in state 'run':

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834453;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834454;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834455;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834456;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834457;
  

• 14:55 MEH: Haydn working on ipp077, will be up/down and shouldn't be used until he gives the all-clear
  • back online -- setting to repair only so not to overload until we catchup
• 15:30 MEH: cleanup diff on only, doing double loading and poll 120 -- seems okay, will help move the larger size diffs from past few days out of the way
• 16:00 CZW: With ipp077 back online, the LANL remote stack prep commands are no longer failing. I've reverted the LAP.PV3.20140730.bigmem stacks locally, to see if adding stacks impacts the processing rate.
• 16:25 CZW: Restarting the ipp user pantasks. I will add 4x c2 to stdscience and ensure that cleanup is prioritizing the larger diffs.
• 16:55 CZW: ipp071 had a load spike, probably related to stdlocal having the stack poll set too high. I've set it to repair, and I'll move it back to up when the stack npend number drops.
• 18:20 CZW: After hearing back from Gene, I've added the stsciXX nodes to 'up' in nebulous to see if that helps with our processing throughput.
• 18:25 CZW: Load jumped on stsci machines running Gene's rsyncs. I'm hesitant to keep them up for nightly processing with this added load. Back to repair, and I'll try again tomorrow when it's easier to monitor.
• 18:30 MEH: ippx001 new kernel behavior essentially no different to old kernel version (though the overloading xNNN w/ stacks seems to work well) -- need more info on what the network to these machines is doing
• 18:35 MEH: will manually add new storage nodes to summitcopy as done for past nights -- however, before adding, seeing a large number of fault events in summitcopy, something wrong? stsci overload? stdlocal heavily loaded still?
  • ota having timeout seem to download fine -- stdlocal stop.. -- it needs to be rebalanced for nightly processing anyways
  • looks like a select set of ota for every exposure

    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d04.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d32.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d71.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d14.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d45.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d41.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d36.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d60.fits
    

  • not always o7007g0011d has a slightly different set and only 7 that timeout --
• 20:10 MEH: restarting ~ippmd/stdscience/ptolemy.rc at normal half power for nightly
  • not adding new storage nodes back to summitcopy since camera group allowing more throughput, avoid overloading tonight
• 22:10 MEH: dont see stdlocal running or fully adjusted for nightly running -- turning down to 2x:c2, 3x:x or too many jobs trying to run
• 00:05 MEH: massive faults and registration delay from db00 transcaction issue around

  141215 23:44:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
  

  • nothing specifically different running?? what else was running around this time?

Tuesday : 2014.12.16

• 07:30 MEH: ipp083 in bad state, causing the severe drop in processing rate ~0400 -- put to repair, stdlocal, ippmd, cleanup stop

  /bin/tcsh: Too many open files in system
  Connection to ipp083 closed.
  

  • seen before, sometimes clears on own, sometimes needs a reboot -- can log in and yes something went strange --

    [843651.114681] VFS: file-max limit 19809058 reached
    
    Dec 16 07:35:00 ipp083 [843346.823876] VFS: file-max limit 19809058 reached
    Dec 16 17:35:00 ipp083 portmap[15243]: warning: cannot open /etc/hosts.allow: Too many open files in system
    Dec 16 17:35:00 ipp083 portmap[15243]: error: bad option name: "^C"
    Dec 16 07:35:00 ipp083 [843346.830817] VFS: file-max limit 19809058 reached
    
    Dec 16 07:40:33 ipp083 nrpe[16177]: Daemon shutdown
    Dec 16 07:40:33 ipp083 nrpe[16179]: Network server accept failure (23: Too many open files in system)
    Dec 16 07:40:33 ipp083 nrpe[16179]: Cannot remove pidfile '/var/run/nrpe/nrpe.pid' - check your privileges.
    

  • more odd than seen before, but we probably want to fix whatever is hitting the huge file-max limit -- processing is moving more again -- cleanup back run
• 08:10 MEH: and another exposure to be manually reverted in registration

  regtool -dbname gpc1 -revertprocessedexp -exp_id 839383
  

• 09:10 MEH: should be okay to put ipp083 neb-host up again -- nightly ~finished so stdlocal back run also -- nope, back to repair, this was the cause of the problem at 4am and then things broke from there..

  Dec 16 04:22:15 ipp083 MR_MONITOR[16550]: <MRMON195> Controller ID:  0   BBU disabled; changing WB logical drives to WT, Forced WB VDs are not affected
  

• 09:15 MEH: Haydn fixed ipp077 disk yesterday, was in repair fine and processing caught up -- try neb-host up for a while
• 09:20 MEH: ipp071 had raid overload, battery looks to be charged again so try neb-host up
• 09:25 MEH: another bad fault 5 OSS WW diffim

  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 622241 -skycell_id skycell.0720.020 -dbname gpc1
  

• 09:45 MEH: need to test IDLE node case in stdlocal, stack.off for bit -- need to do other things.. stack.on but will not be modifying for daytime allocation other than bumped up 5x:x
• 11:10 MEH: ippx100 is also being deleted from normal use to be monitored w/ ippmd processing
• 11:30 CZW: stsci storage experiment on again.
• 13:15 CZW: storage nodes on in stdlocal to see if that improves rate. Nebulous connections are down and stably so so that shouldn't be a problem.
  • once stsci available it looks like x001 and x100 are more similar to ipps -- after talking to Gavin this could easily make sense, he is making a diagram for where things are plugged into where.
• 15:00 CZW: chip and warp cleanup off to get the diffs through again. We're not really in a space crunch with the stsci nodes open, but getting space cleared on the ipp07-8X storage nodes will help keep the number of available target nodes high if/when we need to take the stsci nodes out for DVO operations.
• 18:20 CZW: Leaving stsci up for the night. Will turn on warp/chip cleanup and reduce stdlocal power when I get home.
• 19:35 MEH: starting new storage node (s4+5) processing test during nightly processing -- long running jobs from ~ippmd/deepstack/ptolemy.rc

Wednesday : 2014.12.17

• 09:35 MEH: monitoring some high load/wait systems ipp085,075
• 10:30 MEH: pause to reduce all but deepstack processing for 10 minutes -- back to run -- deepstacks also suffering from network topo
• 11:55 MEH: ipp083 BBU reports okay still but it seems to still be in the WT cache state and acts as an anchor to processing -- leaving in repair
• 12:45 MEH: ipps03 unresponsive -- not too heavily loaded w/ deep stacks -- will deal with it
• 14:15 MEH: chip+warp cleanup off to focus on diffs until finished --
• 20:20 MEH: nightly processing would do a lot better if the polling was fully loaded.. doing the necessary regular restart -- also needs 4x:c2
• 21:00 MEH: cleanup diff finished, chip+warp back on
• 22:45 MEH: nightly slowly catching up. pstamp also could use a restart --

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD