Processing Throughput Tiger Team Notes

Proposals

• Overtasking gpc1 on ippdb01: load on ippdb01 regularly +20, starting early August.
  • test turing off updates over Friday/weekend 2011.09.09-2011.09.10. didn't seem to improve rates?
  • setup representative test queries and time delay for different processing loads
• Polling times/rates: are the polling rates causing extra load and delay with DB queries unnecessarily?
  • Bill testing by changing times in update

• Each pantasks input file has a different definition of wave1, wave2, compute, etc. These should be unified into a single list that is included into each input file. This will make managing untrustworthy hosts easier. Beyond this, we should re-evaluate the distribution of hosts among the various servers to ensure we are properly allocating processing power. It appears that each unit of load corresponds to being included in one pantasks one time. This suggests that we are manually underutilizing the upgraded servers.
  • Chris implemented 2011.09.12@1500 - see ~ipp/ippconfig/pantasks_hosts.input, evaluation of improvement/issues continue

• Stage delay/timings: are there possible areas to speed up
  • is pswarp stage a significant delay in processing? Ken asks could going to lanczos2 be acceptable.

• IPP meeting 9/19 Ken asks about ways of tracking server load with processing.
  • possibly load ganglia details into czardb to pair processing poll times with ganglia reported states/status.

Action items from 2011.08.29

• Chris
  • tidy / finish disk usage stuff (regular job to be run on all storage nodes)
    • Coded, and cronjobs created. Undiagnosed bug in cronjobs is preventing completion.
  • LAP / magic bug : magic does not queue multiple runs for the same label & exp_id
    • Workaround done, will be fixed in next LAP code.
  • LAP / smf destreak bug : warp-stack runs destreak the same exp_id in multiple cam_ids; these should be merged somehow (either smf level or streak level)
    • Will be fixed in next LAP code by ensuring only one processing of an exposure exists, and delaying warp-stack diffs until they are complete.
  • apache targets via ipp/.cshrc
    • Done.
  • ROC (at low priority)
• Bill
  • streaksremove / ppMops table memory fix
  • log file rename
  • smf in pstamp / web interface
  • event monitoring
• Mark
  • most large-count bad diffs had telescope tracking glitch / optical aberration
    • examine diff stats from moments - no clear indicator, finished preparing sample set to look at 9/6
    • other ways to automatically reject these?
  • MD02 starting - still trying to finish up, izy ready as of 9/6

Notes from 2011.08.25

• Bottlenecks
  • Need to catch files with storage object but no instance.
    • Bill says that this "should" only be an issue with log files. Suggestion for that is to move old logfile to logfile.timestamp
    • That way repeated failures can be diagnosed (editors note: somewhat but not in ippMonitor ....)
  • Long running jobs
    • Need a way to have pantasks stop and fault jobs that are stuck due to
      • nfs hangs
      • alogrithm problems (should not be re-run)
    • Talked about the "Event monitoriing" hooks that Bill was working on.
  • When a nebulous apache servers does not respond it causes problems throughout the system because the requests don't get serviced
    • Suggestion to investigate more robust load balancing solution. Apache, hardware solutions, etc
    • short term perhaps target apache requests to a server based on the client node. That way errors would be localized to a subset of machines
  • ipp014
    • New RAID card on order. Turned off as processing node and for nebulous output (repair mode)
  • memory usage
    • Mark is investigating ppMops explosion (related to diff source explosion)
    • Bill is investigating higher than expected memory usage by streaksremove

• Disk space
  • We are still using more space than predicted. Chris has scripts running to evaluate.

• What about long term storage on data store. Do we need to keep the clean camera and diff runs forever?
  • Bill created a web interface to get the smfs for Dave Monet to use from Manoa. Can this be hardened and put on the pstamp server?