IPP Progress Report for the week 2012.03.26 - 2012.03.30

(Up to IPP Progress Reports)

Eugene Magnier

I found errors with the reference database that I generated: it turned out that relphot failed to set the mean mags based on the synthetic photometry, if no good PS1 photometry existed. This left large gaping holes in some places. I needed to modify relphot and re-run it, and then re-extract the reference database subset. I added code to psastro to apply JT's w-r vs g-r color trend to the data, so that w-band images can be well calibrated using this new database. Looking at the astrometric and photometric residuals across the sky: astrometry looks very solid. photometry looks good in general, but there are a number of exposures which were rejected from the analysis, and have not been adjusted to match the rest of the system. I'll need to run a clean up stage to make them consistent, but for now I think the reference database is good (because the mean magnitudes exclude those rejected exposures). We are going to re-run SAS over the weekend with the new reference database.

Serge Chastel

  • Fixed 13902 out of 16759 (about 83%) 0-size instances "on ippb02" (as detailed during the ipp meeting: the instances have been actually moved to ippb03). Some faulty instances on other hosts have been fixed as well (0-size files and wrong md5sum according to [gpc1|isp]/pzDownloadImfile tables). The complete list of the files that have been fixed can be found in ~ipp/sch/fix_ippb02/repair/good. All instances that have been fixed had a good copy somewhere on the cluster.
  • I'm now checking the 259 "possibly repairable" instances, that is those for which the size and md5sum are not set in the [gpc1|isp]/pzDownloadImfile tables.
  • There are 2598 instances that cannot be repaired: list in ~/sch/fix_ippb02/repair/others/cant_fix.txt. I need to make more work on those though. Some instances might be lost in nebulous but could still be on one of the disks.
  • Once all faulty files on ippb02 have a known status (good or bad) I will perform a new overall check to be sure that everything's fine on it from the raw images point of view. I will then *) start looking at the other products on ippb02; *) make sure there are exactly two instances of the raw images and that 1 is at mhpcc, 1 at atrc *) remove the raw images which have in more than 2 copies.

Heather Flewelling

  • kuhio day
  • sick one day
  • czar one day
  • liason
    • answered Bertrand's questions about dvodbs
    • answered Cindy's questions about installing ipp
    • answered Chris's question about sdss stripe 82 data
  • jtrp
    • u band done except for MD06 - something goofy with astrometry - have some insights from Gene but not yet investigated
    • u band has odd problem with stacks: the names are not consistent so JT's pulling scripts do not work. I need to fix this.
    • jt found a few skycells of gpc1 stacks - these affect old processings only (ie, a bad skycell out of a stack, but not flagged)
    • answered questions about cmf/smf - still need to grab smfs for JT
  • dvodb
    • sorted out timeouts on addstar tasks - adjusted timeout times and adjusted queries. Problem: staticsky query is still long, because it joins back to rawExp to get the camera for addstar, I can't figure out how to optimize this, but moving the bulk of the 'new' staticskys out of the way works.
    • separate Cam minis/SSkys minis are building for LAP.ThreePi?.20110809.V2
    • merged all the ThreePi? Minis that haven't been merged into 'ThreePi?.V3.Q/ThreePi.V3.2012.Q1'

Roy Henderson

  • sick for 3 days
  • figured out how to queue up object batches using unique DVO region IDs
  • re-factored metrics and loading code to accommodate the above
  • create, mostly empty, OB batches for testing on the PSPS side
  • reading in Photcodes.dat from DVO in order to get to mags for each objID
  • started work on stored procedure to deal with DVO object mags stored by row number rather than ID

Mark Huber

  • ppStack continuing to trace out the two remaining example problems of larger then expected target/final convolved stack PSF for making deep stacks.
  • Czar Friday and personal research/telescope proposal.

Bill Sweeney

  • Performed some experiments to see how expensive it would be to regenerate camera stage mask files that have been lost. With the current code we can limit the chips that psastro operates on but the resulting masks are different. Specifically glints from other chips on the camera are not included. To get them the full analysis is required which takes several minutes because it redoes the astrometry.
  • Finished destreak_camera_restore script which will recover space used by censored smf and mask files. Started running it over the weekend. It takes about 200 seconds per camRun so it will take a few weeks to complete. (We don't want to assign too many hosts at once to work on this because it can be stressful to nebulous)
  • psphotStack still can't deal with the galactic plane (at least near the center). Wrote a script to limit the runs to with abs(galactic latitude) > 20 for now.
  • 2 days as processing czar. One day spent several hours finding bottlenecks in processing. It turned out to be the galactic plane again. pswarp could use some work reducing memory footprint. Also removed the nodes running psphotStack from stdscience so that memory bloat doesn't block that processing.

Chris Waters

  • Finished generating noisemap detrend. There are still some minor bugs in this process, so it will need to be repeated once those are fixed.
  • Applied noisemap to SAS g footprint, and looked at false positive rate, including a comparison to the SDSS stripe 82 data. Details listed here: https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Dark_Model
  • Looked at statistics for improving the noisemap.
  • Prince Kuhio day and a day of illness prevented much more progress.