IPP Progress Report for the week 2011.04.04 - 2011.04.08

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Databases backup. All details in the wiki ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DatabaseBackups
• Mops Hemocrat
• bzip is better than gzip to compress dump files (and possibly products?)

Heather Flewelling

• continued merging minidvodbs. monday morning: 140/185.
• addRun modifications are done, but not fully tested.
• MD10 and MD09 stacks (for JT) are done
• started MD04 dvodb for Roy

Roy Henderson

• Some design, discussion and a little coding for the implementation of Daniel's query builder into PSVO. Should be ready for Boston....

• Lots of development on ippToPsps, culminating in the loading to PSPS of the first-ever stack format batch, albeit with bogus object IDs. The road to this included:
  • now using local, rather than remote, MySQL database to enable a huge speed increase when loading tables from FITS
  • wrote a method to report which columns of which tables are fully or partially NULL, and to replace those NULLS with weird PSPS '-999' pseudo-NULL value
  • many more stack fields populated after discussions with Jim/Gene
  • implemented XML config file for use with Jython code: includes all Db parameters, paths, DVO and datastore settings etc
  • wrote a python class to handle gpc1 queries, including nebulous calls to return file paths
  • 'DVO' table in database to store all IDs (objID etc). Filled with fake values and used to test load
  • jython code now interfacing with ippToPsps db, getting new batch ID, saving manifest files and publishing to datastore
  • worked with Thomas and Sue loading the stack to the ODM. A few problems encountered, but it loaded eventually
  • started work on Jython version of detection batches, though this is low priority

• Meetings and discussions about IPP processing required before Boston meeting to ensure a good PSPS demo.

• Worked through a list of exposures missing from PSPS as reported by Dave Monet. Figured out a good reason for most, if not all.

Bill Sweeney

• Completed the masktest analysis using the IPP trunk. The new code base is ready to go.
• Created new production tag. Did some testing of it that pointed out some source files that hadn't been updated (my fault as it turned out.)
• Finally finished running the ThreePi.rerun data through the system (several hundred exposures of interest to Bertrand Goldman that didn't get processed by nightly science for various reasons. This was made more difficult than it should be due to some errors I made in queuing the processing originally.
• Copied 2.7 TB three pi database to 2 2TB usb disk drives for Dave Monet. This simple task was made complicated because the dvo database had many directories with more than 32768 files on them. This doesn't work with NTFS. Some scripting took care of this.
• Modified the script that performs the static sky analysis to support analyzing a single input. This involved a miserable afternoon investigating a bug in the perl version of the metedata config file parser. Never did find a fix to the problem, but I found the root cause and a workaround.
• Modified the skycells program to optionally have East to the left. Used it to create the new ThreePi.V3 tessellation which we plan to use for 3pi processing.
• Started reprocessing the STS data with the new codebase, tessellation and psastro recipe.

Chris Waters

• Begin work on fixing detectability server.
• Created updated masks for OTA47 and OTA06, to correct specific camera changes.
• Worked on ppStack to allow quick stacks to be made without convolving inputs. Quickstacks do not seem to reject pixels as efficiently, so spent time trying to improve that. Built test quickstack for reprocessing test.
• Reprocessing test: processed a set of exposures using both warp/quickstack diffs and TTI pair diffs. After magic and destreaking, stacked the input warps for both sets into final (convolved) stacks, and compared coverage and stack quality.