IPP Progress Report for the week 2010.06.28 - 2010.07.02

(Up to IPP Progress Reports)

Eugene Magnier

Much of the team work this week has been aimed at smoothing rough edges of the processing and attempting to improve the quality of the data. We have two ongoing efforts to improve the masking and detrends: Heather is concentrating on the darks and improving the reliability of the regular detrend processing. Chris has been focusing on the issues with OTA14, OTA11, and the suspected non-linearity in edges and certain cells. Much other work is related to adding more information in the database and the ippMonitor tools to make it easier to see what is happening and what has happened with the data processing.

I've been concentrating on the extended source analysis, in particular the non-linear fitting. I spent much of my time this week re-working the Sersic fitting to use a more traditional representation of the parameters: I have been using a simplified form of the function f = Io exp(-zn). I have been finding poor convergence, especially for DeVaucouleur?-like profiles, and I am suspicious that this is due to my representation. There is strong coupling between the n and elliptical contour terms which may be exaggerated by the choice of parameterization.

Serge Chastel

* Adapted ippMonitor framework for plotting/image display

  • Details can be seen at the end of ippMonitor/def/README
  • Need to fix minor bug: 'unlink(tmp files) deletes data and image files before display in php script' => unlink tmp files removed from php scripts => Dirty temporary solution: clean from 'Admin and debug' menu
  • Query columns names from some db tables

* Roy's shadow for czar activities

Heather Flewelling

  • liason duties
    • answered Harvard/LCOGT questions
    • answered IFA questions / transferred dvo dbs etc
  • ippMonitor - more bug fixes (checked in now)
  • detrend work
    • reverts - (ippTools and ippTasks) detrends now revert in pantasks, similar to the other stages. dettool has -revertstuff -all-run to revert any failures in state 'run'
    • monitoring of detrend pantasks
  • addstar
    • monitoring of merge (it finally finished)
    • emailed Bill how to restart addstar - I should put this on the wiki

Roy Henderson

  • PSPS/IPP
    • attempts to load batches on sandbox with new DXLayer with Conrad. Various (still unresolved) issues...
    • some schema changes
    • now saving ippToPsps metadata in XML format (had been plain-text)
    • removed dependence on smf column numbers to protect against future smf format changes
    • now logging 'total detections' and storing in ippToPsps Db for auditing/debugging
    • work on undefined detection fields:
      • numPhotoRef in ImageMeta: setting to same as numAstroRef, which is true for now at least
      • numPhotoRef in FrameMeta: setting to sum of numPhotoRef over all chips
      • detectorID: changed to string in PSPS schema, using 'DETECTOR' from IPP smf
  • IPP
    • Czar on Friday:
      • Outlined Czar duties to Serge
      • added 'revert' buttons to Czartool

Paul Price

  • Processing Czar
    • Reverting failures due to filesystem glitches
    • Fixed pmSourceMatch (wasn't incrementing size for disjoint regions) and backported to fix Gene's MD04 stacks
    • ipp012 woes
  • Background restoration pipeline
    • Merged branch into trunk
    • Ran single M31 image to work out bugs
  • Added concept CHIP.TEMPERATURE to be set to the Camera's new-and-improved temperature header keyword
  • Stacks:
    • Discovered that STS stack photometry has more scatter than expected or desired
    • PSF photometry has more scatter than aperture, apparently because the PSF model isn't good
    • Found astrometric differences about 1 pix (0.2 arcsec) p-p in the warps
    • Unconvolved photometry seems to be better than convolved

Bill Sweeney

  • vacation 2 days
  • 2 days as processing czar
  • helped team queue data for cleanup and monitored it's progress.
  • debugged update failures
    • diff updates were occasionally failing. Paul suggested some changes to inputs in this mode that fixed the problem
    • For some very large requests cleanup is getting queued between the time that a postage stamp job is queued. This causes the jobs to fail
    • updates stall if the previous magicDSRun cleanup did not succeed.

Chris Waters

  • Confirmed that IPP-OTIS interface is up and running correctly.
  • Bug fixes to nebulous, including changes to neb-ls to make useful to find data.
  • Identified IPP side values for the missing columns in PSPS.
  • Checked OTA14 bad columns, and used un-detrended data to determine that columns 1, 3, and 4 can be unmasked after new detrends are created.
  • Cleaned up old diff stage data, and looked into the current diskspace usage (http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Cluster_Storage_Notes)