IPP Progress Report for the week 2010.09.27 - 2010.10.01

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• IPP: Database profiling. Replication and optimization. Created nebulous_opt database on ippdb02 for optimization tests.
• MOPS: Meetings. Documentation (mainly reading but also wrote a few words on retention policy on datastore).

Heather Flewelling

• caught up on email from vacation
• ifaps1 things:
  • answered questions from: gaidos, wainscoat, tonry
  • transferred current dvo db to ifaps1
• skyprobe things
  • requested skyflats
  • checked into ipp trunk some fixes for isp (to get proper values in db, and to make checkexp more versatile)
  • verified we have downloaded all skyprobe images
• working on gene's task of plotting processing throughput for various stages:
• sick for 2 days. This coincided with czar duty and bad weather.

Roy Henderson

• PSPS
  • time with Sue investigating the source of duplicates in new merge. Sadly the source was me...
  • typo in stack schema table causing problems. Committed fix.

• IPP
  • Czar tues/wed
    • nebulous issues: stopped and restarted everything
    • finished cleanup code for czartool: now trims all Db tables from previous day to save space, then optimizes everything
    • added functionality to stop and run pantasks servers from czartool page, including 'stop all' and 'run all'
    • started work on 'roboczar': now emails (me) if a particular server has been down for a given interval (default of 30 mins)
  • ippToPsps
    • pspsSchema2xml.pl script changed to generate mapping file as well. Can be viewed from wiki so people can collaborate on resolving missing fields
    • changes to ODM-checking script: now logs loading failures
  • investigated MySQL optimization

• Other
  • flurry of mailing-list activity after Will's email regarding the outreach list

Bill Sweeney

• Updated stack processing to save the MJD-OBS value for the run in the database. (average value for input exposures). This is needed to enable image look up for detectability queries from stacks and stack-stack diffs.
• Wrote programs to examine every avaliable stack FITS header to extract the MJD-OBS value and them into the database.
• updated the postage stamp parser to look up stacks by MJD-OBS.
• Spent some time studying the way time is handled by IPP.
• Changed the postage stamp server to treat MJD-OBS values as TAI. (Previously the values of rawExp.dateobs was used. That value is UTC
• Lots of wasted time dealing with faulout of ipp020 outage. Built new stacks for the M31 skycells that were lost on ipp020.
• More work improving the reliability of the postage stamp server in response to errors due to unavailability of data. Temporary or permanent. Added fault_count columns to pstampJob and pstampDepenent tables. Stop reverting when count gets to 5. Todo: task to stop jobs in this state.
• Spent a while working with Niall to determine why he wasn't getting postage stamps from some measurements he found in DVO. It turns out he was using a "Gene Maginer private reserve" catalog made in 2009 which contains data that hasn't been run through magic. This exposed a bug in the hooks we have for allowing IfA to have un-magicked postage stamps. Fixed that.
• started working on cleanup for chip_bg and warp_bg stages.

Chris Waters

• Wrote and debugged new OSS nightlyscience handler. Identified issues with difftool handling of warp-warp diffs.
• Discovered that bathtub ring noise pattern is back after camera change. Appears confined to 2-phase devices. Although it does not seem to directly trigger magic, the average number of streaks is higher post camera change.
• Culled ~5 TB of data being used by excess copies of data.
• Non-linearity correction. Still doesn't work. Does not seem to be applying a large enough correction, and has discontinuities that cause more problems.
• Started planning JHU liaison trip.