IPP Progress Report for the week 2011.04.25 - 2011.04.29

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• MOPS czar
• Replication: all databases on ippdb01 are replicated on ippc02 and on ipp001.
• Slow log query supervision now works (see SlowLogSupervision database on ipp004). I need to make it a daemon to update the tables contents and display them in czartool.

Heather Flewelling

• addstar
  • addstar was moved to new ippdvo account
  • restarted minidvodbs
  • copied old ThreePi.V1 to /data/ipp006.0/gpc1/catdirs (others live here too), and deleted the original off of ipp004 to free up space
• JTMD stacks
  • MD02 and MD03
  • found and fixed bug in ippconfig for old data
  • found and fixed bug in detRun and dettools - detselect does not always choose the correct detrend when the use_date is null
• MD04 stuff
  • answered questions on addRun/staticskyRun
  • added in fake addRuns for Gene's MD04 deepstack dvodb
• isp astrometry
  • doesn't work - same runs as before, but things have changed in the recipes
• sdss
  • downloading ra/dec/mags/etc for comparision to 3pi data
• czar for 2 days

Roy Henderson

• DVO -> Mysql program written and working:
  • loops through all DVO sub-dirs, pulls in all measurement and average tables, joins and inserts into master dvoDetection table
  • problems with duplicates in DVO - ignoring for now, will investigate later
  • various system problems: broke ipp005 by building big database in the root partition, moved to ippc00 (too slow) then back to ipp005 after Gavin moved Dbs to partition with some space
  • completed ingest of 167 million detection rows in about 30 hours. Query speed increased from 40 minutes for a full frame to around 20 seconds.

• Testing of detection loading using new MySQL database:
  • path problems - csv dir not understood by ODM
  • DXLayer defaulting to 3PI rather than SCR
  • confusion over IMAGEID key: IMAGEID in smf files is not same as IMAGE_ID in DVO. Using SQL workaround.
  • crazy -999 "NULL" overflowing byte fields and breaking load
  • ImageMeta extension existing for empty Detection extension: fixed
  • duplicate detections in smf files breaking load: altered my tables to reject them and report to the log
  • nightly stacks in DVO do not have header values that were in original stack demo file. Changed code to query gpc1 for them instead
  • deep stacks not in addRun table (they were loaded manually) so I couldn't queue. Heather to the rescue...
  • found rows in StackApFlx and StackModelFit not in StackDetection: ignoring for now

• Actual loading of detections:
  • required lots of hand-holding over the weekend, but necessary to get everything loaded in time.
  • encountered bug causing program to halt with a 'database user' prompt. Spent a lot of time diagnosing issue, then fixing it (STILTS functions not closing all their database connections)
  • investigated high numbers of bad flux, duplicate objIDs. Confirmed same result with old code.

• made plot of magic mask fraction across lunations for Ken

Bill Sweeney

• Spent many hours tracking down a memory corruption problem in addstar when importing SDSS data. It was very difficult to find since the problem only happened in an optimized build. Code was wiping out a saved variable on the stack.
• Fixed problem with postage stamp web form where user errors were not being reported in a useful way.
• Investigated the cause of some missing keywords in static sky cmf files that are needed for PSPS. Tracked the problem down to ppStack. Fixed it.
• Analyzed results of tests of new STS fields. Did some tests of yet another STS tessellation. This one with 7 projection centers.
• Did some debugging of some postage stamp requests that got stuck waiting for updates.
• One day as processing czar.

Chris Waters