IPP Progress Report for the week 2011.04.25 - 2011.04.29

(Up to IPP Progress Reports)

Eugene Magnier

I've spent the week working on two somewhat related issues. First, I have been looking into the photometric quality of the 3pi reference database we have generated. I have been using the stellar locus as a test of the photometry. The results are somewhat mixed. For some large areas, the stellar locus test suggests that we are getting the right answer to an acceptable level of ~2%. This measurement is limited partly by the stellar locus itself and by astrophysics: I clearly see the effects of extinction in the Galactic Plane. Some other regions have stellar locus offsets which are larger than our goal. These tend to be blocks, which suggests these are the residual impact of weather. There are also some regions for which the measured zero points to not appear to have converged on a valid result; this is not expected, and suggests that the relative photometry analysis needs to be re-run for those areas. I am still examining this issue. Meanwhile, we have released the next 3pi dvo database snapshot (demo month to 2011/04/01) and I have started the process of running the reference database analysis on that snapshot.

The other issue I have been concerned with this week is the aperture correction / growth correction measured by psphot. I have noted seeing-dependent photometry in the past (clearly measured in the MD fields), and I have attributed this to the growth correction or the size of the reference aperture. I ran an experiment with 2 nights of MD data for which the seeing dependency was clear, using a set of different reference apertures. To my surprise, the size of the aperture did not affect the zero point offsets between the two nights. I have turned to simulated data (with the PS1-style power law psf) to understand the source of the seeing dependence. So far, I have not been able to make a correction to the photometry which does not retain some seeing dependence (at the ~2% level). It is possible that we will need to correct the photometry with an empirical seeing model derived from the MD field data.

Serge Chastel

  • MOPS czar
  • Replication: all databases on ippdb01 are replicated on ippc02 and on ipp001.
  • Slow log query supervision now works (see SlowLogSupervision? database on ipp004). I need to make it a daemon to update the tables contents and display them in czartool.

Heather Flewelling

  • addstar
    • addstar was moved to new ippdvo account
    • restarted minidvodbs
    • copied old ThreePi?.V1 to /data/ipp006.0/gpc1/catdirs (others live here too), and deleted the original off of ipp004 to free up space
  • JTMD stacks
    • MD02 and MD03
    • found and fixed bug in ippconfig for old data
    • found and fixed bug in detRun and dettools - detselect does not always choose the correct detrend when the use_date is null
  • MD04 stuff
    • answered questions on addRun/staticskyRun
    • added in fake addRuns for Gene's MD04 deepstack dvodb
  • isp astrometry
    • doesn't work - same runs as before, but things have changed in the recipes
  • sdss
    • downloading ra/dec/mags/etc for comparision to 3pi data
  • czar for 2 days

Roy Henderson

  • DVO -> Mysql program written and working:
    • loops through all DVO sub-dirs, pulls in all measurement and average tables, joins and inserts into master dvoDetection table
    • problems with duplicates in DVO - ignoring for now, will investigate later
    • various system problems: broke ipp005 by building big database in the root partition, moved to ippc00 (too slow) then back to ipp005 after Gavin moved Dbs to partition with some space
    • completed ingest of 167 million detection rows in about 30 hours. Query speed increased from 40 minutes for a full frame to around 20 seconds.
  • Testing of detection loading using new MySQL database:
    • path problems - csv dir not understood by ODM
    • DXLayer defaulting to 3PI rather than SCR
    • confusion over IMAGEID key: IMAGEID in smf files is not same as IMAGE_ID in DVO. Using SQL workaround.
    • crazy -999 "NULL" overflowing byte fields and breaking load
    • ImageMeta extension existing for empty Detection extension: fixed
    • duplicate detections in smf files breaking load: altered my tables to reject them and report to the log
    • nightly stacks in DVO do not have header values that were in original stack demo file. Changed code to query gpc1 for them instead
    • deep stacks not in addRun table (they were loaded manually) so I couldn't queue. Heather to the rescue...
    • found rows in StackApFlx and StackModelFit not in StackDetection: ignoring for now
  • Actual loading of detections:
    • required lots of hand-holding over the weekend, but necessary to get everything loaded in time.
    • encountered bug causing program to halt with a 'database user' prompt. Spent a lot of time diagnosing issue, then fixing it (STILTS functions not closing all their database connections)
    • investigated high numbers of bad flux, duplicate objIDs. Confirmed same result with old code.
  • made plot of magic mask fraction across lunations for Ken

Bill Sweeney

  • Spent many hours tracking down a memory corruption problem in addstar when importing SDSS data. It was very difficult to find since the problem only happened in an optimized build. Code was wiping out a saved variable on the stack.
  • Fixed problem with postage stamp web form where user errors were not being reported in a useful way.
  • Investigated the cause of some missing keywords in static sky cmf files that are needed for PSPS. Tracked the problem down to ppStack. Fixed it.
  • Analyzed results of tests of new STS fields. Did some tests of yet another STS tessellation. This one with 7 projection centers.
  • Did some debugging of some postage stamp requests that got stuck waiting for updates.
  • One day as processing czar.

Chris Waters

  • ATRC Storage: Added new storage nodes into Nebulous. Reworked Nebulous SQL to use the mountedvol.xattr field to restrict creation of new files on these nodes, and then reworked random replication to respect this as well. This prevents Nebulous from putting any data on these nodes due to a regular random process. This does not hurt the neb-shuffle code, as that explicitly targets a destination node based on the source node. They appear to be working fine, and slowly filling with second copy data.
  • Large Area Processing: Continued work on LAP code. Added the tables to the gpc1 database that are needed to manage this processing. Began initial tests and debugging.
  • Astrometry: Processed STS data through astrometry tests, and updated wiki page: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_BrightStarAstrometry_STS_Tests . It looks like a -1.5V offset in SW_H is the solution to resolve this issue.