IPP Progress Report for the week 2011.04.25 - 2011.04.29

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• MOPS czar
• Replication: all databases on ippdb01 are replicated on ippc02 and on ipp001.
• Slow log query supervision now works (see SlowLogSupervision database on ipp004). I need to make it a daemon to update the tables contents and display them in czartool.

Heather Flewelling

• addstar
  • addstar was moved to new ippdvo account
  • restarted minidvodbs
  • copied old ThreePi.V1 to /data/ipp006.0/gpc1/catdirs (others live here too), and deleted the original off of ipp004 to free up space
• JTMD stacks
  • MD02 and MD03
  • found and fixed bug in ippconfig for old data
  • found and fixed bug in detRun and dettools - detselect does not always choose the correct detrend when the use_date is null
• MD04 stuff
  • answered questions on addRun/staticskyRun
  • added in fake addRuns for Gene's MD04 deepstack dvodb
• isp astrometry
  • doesn't work - same runs as before, but things have changed in the recipes
• sdss
  • downloading ra/dec/mags/etc for comparision to 3pi data
• czar for 2 days

Roy Henderson

• DVO -> Mysql program written and working:
  • loops through all DVO sub-dirs, pulls in all measurement and average tables, joins and inserts into master dvoDetection table
  • problems with duplicates in DVO - ignoring for now, will investigate later
  • various system problems: broke ipp005 by building big database in the root partition, moved to ippc00 (too slow) then back to ipp005 after Gavin moved Dbs to partition with some space
  • completed ingest of 167 million detection rows in about 30 hours. Query speed increased from 40 minutes for a full frame to around 20 seconds.

• Testing of detection loading using new MySQL database:
  • path problems - csv dir not understood by ODM
  • DXLayer defaulting to 3PI rather than SCR
  • confusion over IMAGEID key: IMAGEID in smf files is not same as IMAGE_ID in DVO. Using SQL workaround.
  • crazy -999 "NULL" overflowing byte fields and breaking load
  • ImageMeta extension existing for empty Detection extension: fixed
  • duplicate detections in smf files breaking load: altered my tables to reject them and report to the log
  • nightly stacks in DVO do not have header values that were in original stack demo file. Changed code to query gpc1 for them instead
  • deep stacks not in addRun table (they were loaded manually) so I couldn't queue. Heather to the rescue...
  • found rows in StackApFlx and StackModelFit not in StackDetection: ignoring for now

• Actual loading of detections:
  • required lots of hand-holding over the weekend, but necessary to get everything loaded in time.
  • encountered bug causing program to halt with a 'database user' prompt. Spent a lot of time diagnosing issue, then fixing it (STILTS functions not closing all their database connections)
  • investigated high numbers of bad flux, duplicate objIDs. Confirmed same result with old code.

• made plot of magic mask fraction across lunations for Ken

Bill Sweeney

• Spent many hours tracking down a memory corruption problem in addstar when importing SDSS data. It was very difficult to find since the problem only happened in an optimized build. Code was wiping out a saved variable on the stack.
• Fixed problem with postage stamp web form where user errors were not being reported in a useful way.
• Investigated the cause of some missing keywords in static sky cmf files that are needed for PSPS. Tracked the problem down to ppStack. Fixed it.
• Analyzed results of tests of new STS fields. Did some tests of yet another STS tessellation. This one with 7 projection centers.
• Did some debugging of some postage stamp requests that got stuck waiting for updates.
• One day as processing czar.

Chris Waters

• ATRC Storage: Added new storage nodes into Nebulous. Reworked Nebulous SQL to use the mountedvol.xattr field to restrict creation of new files on these nodes, and then reworked random replication to respect this as well. This prevents Nebulous from putting any data on these nodes due to a regular random process. This does not hurt the neb-shuffle code, as that explicitly targets a destination node based on the source node. They appear to be working fine, and slowly filling with second copy data.
• Large Area Processing: Continued work on LAP code. Added the tables to the gpc1 database that are needed to manage this processing. Began initial tests and debugging.
• Astrometry: Processed STS data through astrometry tests, and updated wiki page: ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_BrightStarAstrometry_STS_Tests . It looks like a -1.5V offset in SW_H is the solution to resolve this issue.