Processing Throughput Tiger Team Notes

Action items from 2011.08.29

• Chris
  • tidy / finish disk usage stuff (regular job to be run on all storage nodes)
    • Coded, and cronjobs created. Undiagnosed bug in cronjobs is preventing completion.
  • LAP / magic bug : magic does not queue multiple runs for the same label & exp_id
    • Workaround done, will be fixed in next LAP code.
  • LAP / smf destreak bug : warp-stack runs destreak the same exp_id in multiple cam_ids; these should be merged somehow (either smf level or streak level)
    • Will be fixed in next LAP code by ensuring only one processing of an exposure exists, and delaying warp-stack diffs until they are complete.
  • apache targets via ipp/.cshrc
    • Done.
  • ROC (at low priority)
• Bill
  • streaksremove / ppMops table memory fix
  • log file rename
  • smf in pstamp / web interface
  • event monitoring
• Mark
  • most large-count bad diffs had telescope tracking glitch / optical aberration
    • examine diff stats from moments - no clear indicator, finished preparing sample set to look at 9/6
    • other ways to automatically reject these?
  • MD02 starting - still trying to finish up as of 9/6

Notes from 2011.08.25

• Bottlenecks
  • Need to catch files with storage object but no instance.
    • Bill says that this "should" only be an issue with log files. Suggestion for that is to move old logfile to logfile.timestamp
    • That way repeated failures can be diagnosed (editors note: somewhat but not in ippMonitor ....)
  • Long running jobs
    • Need a way to have pantasks stop and fault jobs that are stuck due to
      • nfs hangs
      • alogrithm problems (should not be re-run)
    • Talked about the "Event monitoriing" hooks that Bill was working on.
  • When a nebulous apache servers does not respond it causes problems throughout the system because the requests don't get serviced
    • Suggestion to investigate more robust load balancing solution. Apache, hardware solutions, etc
    • short term perhaps target apache requests to a server based on the client node. That way errors would be localized to a subset of machines
  • ipp014
    • New RAID card on order. Turned off as processing node and for nebulous output (repair mode)
  • memory usage
    • Mark is investigating ppMops explosion (related to diff source explosion)
    • Bill is investigating higher than expected memory usage by streaksremove

• Disk space
  • We are still using more space than predicted. Chris has scripts running to evaluate.

• What about long term storage on data store. Do we need to keep the clean camera and diff runs forever?
  • Bill created a web interface to get the smfs for Dave Monet to use from Manoa. Can this be hardened and put on the pstamp server?