IPP Progress Report for the week 2010.11.01 - 2010.11.05

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Updated czartool page with replication status
• Added wiki page about how replication problems were solved (link from the czartool page)
• Contributed to to Heather's IPP wiki documentation
• Investigated XFS defragmentation concerns
• Tried to figure out how OSS data are published and to see where they are hiding between warp and diff stages
• Working on psLib 'make test' and trying to make it work with Hudson

Heather Flewelling

• addstar - copied ThreePi minidvodbs to ipp022, announced to users, answered user questions
• czar for 2 days (with help from SuperCzarBill
• investigated MD04 dvodb- we expect ~1000 exposures in the db, there are only ~300 because the rest are unmagicked
• investigated diff dtimes, see ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ProcessingTimesOct2010
• created processing checklist with Serge, see ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/IPP_Processing_Checklist

Roy Henderson

• PSPS
  • loading:
    • stopped 3PI, and loaded remainder of MD04
    • troubles starting 3PI loading again with primary DVO Db: corrupted. Spent some time investigating this
    • loading is now stopped pending a new 3PI DVO Db
  • investigated current state of DVO MD04 Db with Heather. Verdict: needs more exposures
  • added extra filtering to ODM-checking script: can check those only on the datastore, 'MergeWorthy', 'Merged' etc
  • communications with TOPCAT developer about a plan for the PSPS Toolbox

• IPP
  • Czar on Monday and Tuesday
    • the strange case of the missing OSS exposures
  • IPP Metrics in czartool:
    • new StageMetrics class to encapsulate metrics for a stage for a given time/label
    • new method in Gpc1Db class to get a reliable count of summit exposures for a given date
    • new method in CzarDb to get a full list of labels in use for a given time period
    • script to use above classes to generate HTML page of metrics for a given day, with plots
    • new stacked histogram showing rate of processing over time for each stage

Bill Sweeney

• Queued data for MD04 reference stack.
• Monitored processing of MD04 data. Large numbers of warps and stacks triggered bugs that had to be worked around.
• pestered the processing czars with suggestions.
• dealt with various cluster problems
• Debugged problem with magic processing. Somehow an invalid magicRun got entered in the database. Didn't find root cause.
• finished implementing background run cleanup. (This is more time consuming than it would seem. Lots of ippTools changes).
• Debugged MPIA data store problems.
• investigated some postage stamp server problems.
• one day as official processing czar.
• defined new tessellations for STS and M31.

Chris Waters

• Nebulous: limited the shuffle code that is moving second instances to large free disks to prevent causing excessive load on either the destination computers or the nebulous database server. Started work on a neb-fsck that will be used to ensure that the disk and the database agree. Miscellaneous bugfixes to neb-admin to allow shuffle and replication to run simultaneously.
• IPP-OTIS interface: added a bit of code to start sending more information than a simple binary acceptance field. Most likely result will be to send the CDF value of the current exposure's measurements so OTIS can decide at what level to retake an exposure.
• automation: unlinked the MD stacks and OSS diffs from each other, allowing each survey to proceed after checking that its own prerequisites are met. This seems to work correctly, and should get OSS diffs out a few hours earlier. Fixed some typos that may have been preventing the magic/stack blocking from running (which means that stacks are made from destreaked data). Will have to check later to see if this resolved the issue.
• burntool: moved permanently missing storage objects out of the way, and reran burntool for these dates. This should resolve the issues with update.
• diskspace: ran diskspace analysis code, noting that mdc config files are now #3 user of space. Started running compression on old darks flagged by Gene.
• linearity: code works faster after Gene's refactoring. Need to confirm that it applies the scientifically correct values.