IPP Progress Report for the week 2010.10.11 - 2010.10.15

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

Heather Flewelling

• made throughput plots for october, added to wiki: ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ProcessingThroughputPlots2010Sept
• ipp liason:
  • helped Aspin, Tomo, Gaidos, Urbaneja
  • created wiki describing how to configure IPP for ifaps1: ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/IPP_on_ifaps1
• skyprobe processing issues
  • I'm running 'ispstdscience' as heather, using ~5 or 6 c nodes
  • skyprobe raw images are being replicated, but not compressed
  • processed a 3pi chunk through cam and created a dvo database
• minidvodb rsync - automated way to transfer minidvodbs
  • minidvodbtool is finished, about half done with the tasks needed to run it in pantasks

Roy Henderson

• PSPS
  • Continued creating 3PI batches, ready for loading once merge is complete
  • investigated huge slow-down for certain batches (see below)

• IPP
  • Czartool
    • finally removed confusing cleanup-jumps from plots (with an option to put them back, if useful)
    • finished 'rate-of-processing' plots (for Ken's daily report)
    • method to figure out start/finish time for a given stage (for use with Ken's daily report)
    • added 1st derivatives plots for all stages
    • added 'hasPlateaued()' method for use by roboczar
    • issues with ippMonitor Db user permissions for czardb (fixed, though I need to look more closely at this)
  • DVO
    • found 0/360 boundary bug that was affecting ippToPsps. Applied a Band-Aid
    • investigated other minor bug with imlist command
  • Other IPP
    • wrote a quick backup script for czarDB and ippToPsps Dbs and mailing list code. Serge set up replication for mailing list Db.

• Other
  • one day off to look after ill family
  • finally got Ken's new mailing lists set up
  • Czar for two days. Plain sailing.

Bill Sweeney

• modified ipp_cleanup to deal with files that are lost on dead volumes (such as ipp020). Ran the program
• Added modes to the chip, warp, and diff tool programs to support reverting of runs that failed to clean up properly.
• Analyzed the flow of daily ipp processing and found a couple of bottlenecks that increased latency. Moved magic streak detection to the distribution pantasks. We still have some optimization to do. Right now warp processing gets delayed due to ThreePi chip processing increasing latency for OSS and MD data for which timeliness is important.
• Looked into the reasons that about 5% 3pi data is not getting fully processed through diff, magic, and distribution. Part of the problem is that the difference stage queuing is too restrictive. Visits that do not occur within 1 hour don't get diffed against each other. We should probably consider using the 'object' field in the database to insure that warp warp difference pairs are paired correctly. Especially for the great reprocessing.
• Fixed problem with difference stage postage stamp images reported by Larry.
• Modified distribution and postage stamp extraction programs to stop censoring pixels which only have the "diffraction spike" and "star core mask" bits set.
• Improved timing parameters of certain pantasks jobs that were timing out frequently due to mysql server slowness. (Root problem was that mysqld was tired and needed to be restarted, but these changes will make the system more reliable).
• Minor fixes to the postage stamp request form processing.
• Fixed some cases where the postage stamp request parser was setting incorrect error codes.

Chris Waters

• Linearity: held up due to programming bug in correction code.
• Nebulous: reworked replicate query to find all objects that have a number of instances different than the requested number of instances. These files are then replicated or culled until the numbers match. Wrote and tested disk balancing code to shift second copies of files onto the hosts with the largest disks.
• Identified and removed leftover burntooled pixel data that was missed last time we cleaned that up.