PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.15

• 08:00 MEH: cleanup must be turned back on, the few number of nodes available for data is killing processing again
• 11:00 CZW: I've updated the database to prevent the exposures from the dead pixelserver from clogging up summitcopy:

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832987;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832988;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832989;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832990;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832991;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832992;
  

• 14:00 CZW: Restarting stdlocal. I'm going to see what happens with the "default" set of hosts.
  • 14:07 CZW: default set is underpowered. Turning on xnodes with xnode.hosts.on macro.
• 14:50 MEH: summitcopy stop -- camera group reporting ramping overload -- anyone else doing summitcopy type stuff?
• 14:45 CZW: I noticed more failing exposures in pzDownloadExp table, and dropped the camera controlled exposures in state 'run':

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834453;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834454;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834455;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834456;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834457;
  

• 14:55 MEH: Haydn working on ipp077, will be up/down and shouldn't be used until he gives the all-clear
  • back online -- setting to repair only so not to overload until we catchup
• 15:30 MEH: cleanup diff on only, doing double loading and poll 120 -- seems okay, will help move the larger size diffs from past few days out of the way
• 16:00 CZW: With ipp077 back online, the LANL remote stack prep commands are no longer failing. I've reverted the LAP.PV3.20140730.bigmem stacks locally, to see if adding stacks impacts the processing rate.
• 16:25 CZW: Restarting the ipp user pantasks. I will add 4x c2 to stdscience and ensure that cleanup is prioritizing the larger diffs.
• 16:55 CZW: ipp071 had a load spike, probably related to stdlocal having the stack poll set too high. I've set it to repair, and I'll move it back to up when the stack npend number drops.
• 18:20 CZW: After hearing back from Gene, I've added the stsciXX nodes to 'up' in nebulous to see if that helps with our processing throughput.
• 18:25 CZW: Load jumped on stsci machines running Gene's rsyncs. I'm hesitant to keep them up for nightly processing with this added load. Back to repair, and I'll try again tomorrow when it's easier to monitor.
• 18:30 MEH: ippx001 new kernel behavior essentially no different to old kernel version (though the overloading xNNN w/ stacks seems to work well) -- need more info on what the network to these machines is doing
• 18:35 MEH: will manually add new storage nodes to summitcopy as done for past nights -- however, before adding, seeing a large number of fault events in summitcopy, something wrong? stsci overload? stdlocal heavily loaded still?
  • ota having timeout seem to download fine -- stdlocal stop.. -- it needs to be rebalanced for nightly processing anyways
  • looks like a select set of ota for every exposure

    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d04.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d32.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d71.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d14.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d45.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d41.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d36.fits
    http://conductor.ifa.hawaii.edu/ds/gpc1/o7007g0001d/o7007g0001d60.fits
    

  • not always o7007g0011d has a slightly different set and only 7 that timeout --
• 20:10 MEH: restarting ~ippmd/stdscience/ptolemy.rc at normal half power for nightly
  • not adding new storage nodes back to summitcopy since camera group allowing more throughput, avoid overloading tonight
• 22:10 MEH: dont see stdlocal running or fully adjusted for nightly running -- turning down to 2x:c2, 3x:x or too many jobs trying to run
• 00:05 MEH: massive faults and registration delay from db00 transcaction issue around

  141215 23:44:46InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
  

  • nothing specifically different running?? what else was running around this time?

Tuesday : YYYY.MM.DD

• 07:30 MEH: ipp083 in bad state, causing the severe drop in processing rate ~0400 -- put to repair, stdlocal, ippmd, cleanup stop

  /bin/tcsh: Too many open files in system
  Connection to ipp083 closed.
  

  • seen before, sometimes clears on own, sometimes needs a reboot -- can log in and yes something went strange --

    [843651.114681] VFS: file-max limit 19809058 reached
    
    Dec 16 07:35:00 ipp083 [843346.823876] VFS: file-max limit 19809058 reached
    Dec 16 17:35:00 ipp083 portmap[15243]: warning: cannot open /etc/hosts.allow: Too many open files in system
    Dec 16 17:35:00 ipp083 portmap[15243]: error: bad option name: "^C"
    Dec 16 07:35:00 ipp083 [843346.830817] VFS: file-max limit 19809058 reached
    
    Dec 16 07:40:33 ipp083 nrpe[16177]: Daemon shutdown
    Dec 16 07:40:33 ipp083 nrpe[16179]: Network server accept failure (23: Too many open files in system)
    Dec 16 07:40:33 ipp083 nrpe[16179]: Cannot remove pidfile '/var/run/nrpe/nrpe.pid' - check your privileges.
    

  • more odd than seen before, but we probably want to fix whatever is hitting the huge file-max limit -- processing is moving more again -- cleanup back run
• 08:10 MEH: and another exposure to be manually reverted in registration

  regtool -dbname gpc1 -revertprocessedexp -exp_id 839383
  

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD