IPP Progress Report for the week 2010.09.27 - 2010.10.01

(Up to IPP Progress Reports)

Eugene Magnier

I spent the week at Harvard for the 'photfest'. This involved folks from

both the CfA/JHU transient team and the QUB transient team. We had a specific set of skycells from MD07 and MD08 with known events. I generated new nightly stacks, and ran the difference images for those stacks against an appropriate reference stack. Mark Huber ran the photpipe code to generate difference images as well, and we spent the week comparing the results.

The most important result from 'photfest' was that the CfA/JHU group are now willing to use at least the IPP-produced difference image pixels, if we set the Medium Deep nightly stack-stack diffs to use SINGLE direction convolution on the reference stack. We agreed that this produces results essentially indistinguishable from the photpipe difference image analysis. They also convinced me that, for the stack-stack diffs where we have a deep reference stack, it is OK to always choose the deep reference stack as the convolution target -- the occasional instances of small amounts of de-convolutions are acceptable given the high S/N of the reference stack and the over-sampling.

We also agreed that the IPP-produced detection lists are close to ready (and may already be acceptable), but that more tests need to be made of the false-positive rates (after appropriate exclusions are applied) and the false negative rates.

We (mostly they -- Mark Huber, Micheal Wood-Vasey, and Ken Smith) are looking at the detections produced by the IPP, along with the feedback I gave them on the flags and exclusions, to see if (with appropriate filtering) the false positives and false negatives are acceptable and if the photometry is is consistent with the photpipe equivalent. The quick look at the last point was positive, but they are doing more checks on this. Mark Huber has promised a report on the conclusions of 'photfest'.

We also agreed that the DUAL convolution is at worst equivalent to an optimal SINGLE convolution plus an additional smoothing, and it may be the case that the DUAL convolution is acceptable or can be made acceptable. I am going to continue work on this for the near term.

I also worked closely with Eddie Schlafly to bring up up to speed on the issues related to the star/galaxy/cosmic-ray separation process. He is using exposures with good SDSS overlap to explore the application of the various measurements available (moments, various fluxes) to improve the star/galaxy/cosmic-ray identifications.

Serge Chastel

  • IPP: Database profiling. Replication and optimization. Created nebulous_opt database on ippdb02 for optimization tests.
  • MOPS: Meetings. Documentation (mainly reading but also wrote a few words on retention policy on datastore).

Heather Flewelling

  • caught up on email from vacation
  • ifaps1 things:
    • answered questions from: gaidos, wainscoat, tonry
    • transferred current dvo db to ifaps1
  • skyprobe things
    • requested skyflats
    • checked into ipp trunk some fixes for isp (to get proper values in db, and to make checkexp more versatile)
    • verified we have downloaded all skyprobe images
  • working on gene's task of plotting processing throughput for various stages:
  • sick for 2 days. This coincided with czar duty and bad weather.

Roy Henderson

  • PSPS
    • time with Sue investigating the source of duplicates in new merge. Sadly the source was me...
    • typo in stack schema table causing problems. Committed fix.
  • IPP
    • Czar tues/wed
      • nebulous issues: stopped and restarted everything
      • finished cleanup code for czartool: now trims all Db tables from previous day to save space, then optimizes everything
      • added functionality to stop and run pantasks servers from czartool page, including 'stop all' and 'run all'
      • started work on 'roboczar': now emails (me) if a particular server has been down for a given interval (default of 30 mins)
    • ippToPsps
      • pspsSchema2xml.pl script changed to generate mapping file as well. Can be viewed from wiki so people can collaborate on resolving missing fields
      • changes to ODM-checking script: now logs loading failures
    • investigated MySQL optimization
  • Other
    • flurry of mailing-list activity after Will's email regarding the outreach list

Bill Sweeney

  • Updated stack processing to save the MJD-OBS value for the run in the database. (average value for input exposures). This is needed to enable image look up for detectability queries from stacks and stack-stack diffs.
  • Wrote programs to examine every avaliable stack FITS header to extract the MJD-OBS value and them into the database.
  • updated the postage stamp parser to look up stacks by MJD-OBS.
  • Spent some time studying the way time is handled by IPP.
  • Changed the postage stamp server to treat MJD-OBS values as TAI. (Previously the values of rawExp.dateobs was used. That value is UTC
  • Lots of wasted time dealing with faulout of ipp020 outage. Built new stacks for the M31 skycells that were lost on ipp020.
  • More work improving the reliability of the postage stamp server in response to errors due to unavailability of data. Temporary or permanent. Added fault_count columns to pstampJob and pstampDepenent tables. Stop reverting when count gets to 5. Todo: task to stop jobs in this state.
  • Spent a while working with Niall to determine why he wasn't getting postage stamps from some measurements he found in DVO. It turns out he was using a "Gene Maginer private reserve" catalog made in 2009 which contains data that hasn't been run through magic. This exposed a bug in the hooks we have for allowing IfA to have un-magicked postage stamps. Fixed that.
  • started working on cleanup for chip_bg and warp_bg stages.

Chris Waters

  • Wrote and debugged new OSS nightlyscience handler. Identified issues with difftool handling of warp-warp diffs.
  • Discovered that bathtub ring noise pattern is back after camera change. Appears confined to 2-phase devices. Although it does not seem to directly trigger magic, the average number of streaks is higher post camera change.
  • Culled ~5 TB of data being used by excess copies of data.
  • Non-linearity correction. Still doesn't work. Does not seem to be applying a large enough correction, and has discontinuities that cause more problems.
  • Started planning JHU liaison trip.