== Processing Throughput Tiger Team Notes ==

=== Action items from 2011.08.29 ===

 * Chris
   * tidy / finish disk usage stuff (regular job to be run on all storage nodes)
     * Coded, and cronjobs created. Undiagnosed bug in cronjobs is preventing completion.
   * LAP / magic bug : magic does not queue multiple runs for the same label & exp_id
     * Workaround done, will be fixed in next LAP code.
   * LAP / smf destreak bug : warp-stack runs destreak the same exp_id in multiple cam_ids; these should be merged somehow (either smf level or streak level)
     * Will be fixed in next LAP code by ensuring only one processing of an exposure exists, and delaying warp-stack diffs until they are complete.
   * apache targets via ipp/.cshrc
     * Done.
   * ROC (at low priority)
 * Bill
   * streaksremove / ppMops table memory fix
   * log file rename
   * smf in pstamp / web interface
   * event monitoring
 * Mark
   * most large-count bad diffs had telescope tracking glitch / optical aberration
     * examine diff stats from moments - ''no clear indicator, finished preparing sample set to look at 9/6''
     * other ways to automatically reject these?
   * MD02 starting - ''still trying to finish up as of 9/6''

=== Notes from 2011.08.25 ===

 * Bottlenecks
   * Need to catch files with storage object but no instance.
     * Bill says that this "should" only be an issue with log files. Suggestion for that is to move old logfile to logfile.timestamp
     * That way repeated failures can be diagnosed (editors note: somewhat but not in ippMonitor ....)
   * Long running jobs
     * Need a way to have pantasks stop and fault jobs that are stuck due to
       * nfs hangs
       * alogrithm problems (should not be re-run)
     * Talked about the "Event monitoriing" hooks that Bill was working on.
   * When a nebulous apache servers does not respond it causes problems throughout the system because the requests don't get serviced
     *  Suggestion to investigate more robust load balancing solution. Apache, hardware solutions, etc
     *  short term perhaps target apache requests to a server based on the client node. That way errors would be localized to a subset of machines
   * ipp014
     * New RAID card on order. Turned off as processing node and for nebulous output (repair mode)
   * memory usage
     * Mark is investigating ppMops explosion (related to diff source explosion)
     * Bill is investigating higher than expected memory usage by streaksremove

 * Disk space
   * We are still using more space than predicted. Chris has scripts running to evaluate.

   * What about long term storage on data store. Do we need to keep the clean camera and diff runs forever?
     * Bill created a web interface to get the smfs for Dave Monet to use from Manoa. Can this be hardened and put on the pstamp server?