IPP Progress Report for the week YYYY.MM.DD - YYYY.MM.DD

(Up to IPP Progress Reports)

Eugene Magnier

Eddie and I examined difference between his ubercal magnitudes and my dvo magnitudes after applying the ubercal zero points. We tracked down 3 main errors:

# a bug in the -64bit image read byteswap code in dvo (used only for the ubercal table) messed up mid-level bytes. fixed this in dvo # a difference of how to treat off-chip positions for flat-field corrections (dvo saturated position on the chip, ubercal chose a random (incorrect) location). This only happened 10-7 times, but ubercal will adopt the dvo solution # ubercal uses per-exposure airmasses, while dvo uses per-detection airmasses. For the test field (near 0 Dec), the difference was quite small (< 1mmag), but the difference is important at higher airmass. We compared by applying the airmass difference, but in a re-run ubercal will move to per-detection airmass values.

In the end, with the fixes and corrections, all detections agreed at the floating point limit (5e-7).

I am now working on a (largish) test sample to run all steps to generate the reference database. This includes merging the synthetic data into the dvo database, and running relphot on everything. I will use the synthetic data to test the amount of clouds as an additional check. After the reference database is done, I will also apply astrometry to the resulting database to generate a dvo db for distribution

I also ran another set of test MD image for Doug F. to further check on the faint end bias.

Serge Chastel

  • Czar on Thu and Fri
  • Added ssp database to mysqldump from ippdb01 to ipp001. Changed isp dump host (was: ipp001; is: ippdb01).

Heather Flewelling

  • isp
    • updated isp database
    • processing several days of isp on ippc13 (as I notice a day finishes, I process another several days). The plan is to do this:
      • find/catalog failure modes of isp astrometry/photometry
      • build a dvocatalog
  • ssp
    • jt transferred ssp to ipp004
    • heather created ssp database, and is in the process of ingesting the 100,000 ssp images (i'm about 95% done)
      • there are 2 copies of each in nebulous
  • jtmd stacks
    • mostly done! there are 45 missing ones that need investigation
  • helped pavlos - he can't install ipp, i'm working with him on that
  • helped larry/peter on ipp installation
  • sick half day

Roy Henderson

The bulk of my work this week was on code to speed-up the ingestion of DVO data into MySQL while not breaking the existing mechanism of using the C-API for reading DVO data on an exposure-by-exposure basis, as this will still be more efficient for parts of the sky with thin coverage. One big speed-up came from removing the need for 'SOURCE_ID', which I had been carrying unnecessarily per detection in my MySQL database.

The other development task was figuring out how to queue available data to load to PSPS by region on the sky based on the ratio of exposures I have to load and the size on disk of the corresponding DVO FITS files. An algorithm is required to decide when it is appropriate to load on-the-fly, exposure-by-exposure, and when it is more efficient to pre-ingest a region of DVO. This becomes especially confusing as I have a multi-client system where clients, thus far, worked independently of one another. This will have to change.

Other work included:

  • after a couple of weeks of testing new features, I released new version of PSVO
  • tested new STILTS release with ippToPsps

Mark Huber

  • Czar Tuesday and Wednesday, lotsa LAP and system crash headaches.
  • Sick Thursday
  • Finished ironing out final wrinkles in the TSS event/name server setup on ipp003 (crontab not setup for account/permissions) and adding more open IP access for mysql from odyssey cluster.
  • UK liaison? Looking into some oddities with the SAS2 data release.
  • Started looking at the DIFF_* and FLAGS2 for diffims in more detail for Larry and for possible feedback with some of the possible convolution issues.
  • Continuing to monitor convolution direction comparisons with recent data and add to ppSub testing plan wiki when time.

Bill Sweeney

  • Started investigating possible improvements in astrometry for the STS observations. Starting with one of the 7 pointings from 2011 built a DVO database to use as a reference. Passing this to psastro instead of the 2MASS synthetic catalog resulted in significant improvement in the astrometric error. However since all of the inputs are from very similar pointings (STS uses a very small dither), the stars are always observed in the same area of the same chips.) Comparing the solution for a given exposure with the 2 reference catalogs there appears to be a small systematic error using the gpc1 reference catalog that isn't being corrected by the IPP. Going from a 3rd order to 5th order polynomial might correct it but this would require some significant work to implement. More investigation is needed.
  • Spent some time looking into the very high number of false detections in the CNP fields. No conclusions as of yet. More work is needed. Deferred for STS astrometry investigation.
  • released some g, z, and y band m31 observations that were not previously processed sans-magic.
  • several hours investigating some update/cleanup issues that were blocking the postage stamp server.
  • short vacation

Chris Waters

  • ran a test stack set using the background continuity code -- there may still be a bug in the application.
  • vacation