IPP Progress Report for the week 2010.11.08 - 2010.11.12

(Up to IPP Progress Reports)

Eugene Magnier

I spent most of my week working on issues related to the repair of the ThreePi? database which was corrupted when /data/ipp004.0 filled. I created a tool to make repairs on a DVO database, and have been applying it to this case. The program, dvorepair, does several things:

  • it scans the full database and counts the number of detections associated with each image. This count is compared with the number of detections expected to come from each image, and any that are short are noted.
  • given a list of image IDs (as output from the previous analysis), dvorepair will locate all associated detections from the database measurement tables and delete them
  • given a list of image IDs, dvorepair will delete the images from the image table, and if all chips for an exposure are deleted, it will delete the exposure entry as well.
  • given a list of defective 'cpt' and/or 'cps' files (average objects and photometry tables), dvorepair will use the information in the (valid) 'cpm' table (detections) to re-generate a consistent set of cpt and cps tables.

These types of repairs are needed because of the ways in which the database was corrupted: some detection tables were partially written (dropping the new detections, and in other cases, the average object tables were partially written.

I've run these on the ThreePi? database, and deleted the images with incomplete detection sets -- these correspond to the images written during the step in which the disk filled up, and will be re-added after the database is internally consistent. I've also discovered another interesting issue: There are a number of images which were added to the database multiple times. This seems to be caused by an error raised by 'addstar', in coordination with an error in the addstar sequencing logic (ie, a failed exposure should not be automatically retried). Now that we have a working tool to delete images, we can identify the multiple copies and use this new tool to delete them, and then re-add them after we correct the error that caused them to fail in the first place.

I have also created a tool to scan the dvo database and check for inconsistencies. This tool should be run on the minidvodbs before they are merged, and on the full dvo database after a merge. I have also updated the addstar and dvomerge code to catch the errors which result from a disk failure, so we will be less likely to require this type of repair in the future.

Serge Chastel

  • Thursday off
  • IPP Czar on Tue and Wed
  • MOPS Czar (all week but no obs on Thu and Fri)
  • Stand-by concerning databases replication (waiting for the replication host)
  • Contacted Bill Giebink concerning the test framework. Some stuff is running on ipp002. I didn't have the time to go into the details.
  • Tessellation: I got the software/scripts on Tuesday. I ran some experiments on Wednesday and I sent some results that seemed excellent on Wednesday (better N/fov size ratio and smaller inter-cells distance). I spent Friday looking at some references for the problem to figure out if the number of observations could be decreased. Unfortunately, on Friday at 5:30PM, I was told that the metrics that I thought were to be optimized were not the right ones. I will start again on Monday... I'm missing the tool that give the number of observations disks a given point falls in.

Heather Flewelling

  • worked on wiki and ken's list. Information on how to modify the sidebar is now on the wiki: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/HowToEditTheTracSidebar
  • zeropoints - investigating why 20101108 w band zeropoints are off
    • plots of mag_inst vs mag_inst - mag_ref look very similar for both good and bad sets of exposures
    • tried making histograms of mag_inst and mag_ref to figure out what is bad (didn't work)
    • tried ordering the w band exposure by different columns to see if there was any way to sort on the good/bad zeropoints -sorting by n_stars sort of gets most of the bad exposures (bad ones have fewer stars).
    • transferring smfs for gene to investigate
  • met with eric gaidos (skyprobe)
  • merged minidvodbs 249-261 for gene's threepi dvodb reconstruction
  • investigating skyprobe relphot problems
    • it segfaults when it finds no stars - fixed
    • Using GRID_TOOFEW 0 STARS_TOOFEW 0 causes it not to segfault (but I'm not sure if it is doing anything useful)
  • ifaps1 things
    • cleaned up disk space a bit
    • downloading threepi minidvodbs -> /data/ifaps1.0/ps1data/catdirs/ThreePi.Minis/ThreePi
    • answered Tomo's questions
  • czar thursday
  • off friday

Roy Henderson

  • Czartool:
    • improvements to temp file handling (IPP czarpoll was clashing with other users)
    • added stacked histogram rate plot to czartool page
    • stages (chip, cam etc) now ordered properly on all plots
    • auto interval function for rate plots
    • fixed bugs in timeDiff functions
    • while doing metrics (below) added ability to show magic mask for a single exposure
  • Metrics:
    • encapsulated day metrics into a class (DayMetrics) that inherits from a new Metrics class handing stuff common to all metrics pages
    • now getting magic mask fraction and plotting

Bill Sweeney

  • Reorganized warptool -towarped to avoid very long query time when lots of warp runs are outstanding. The problem was with the way 'order by priority' was implemented.
  • Implemented changes to publishing system to allow OSS and 3pi detections to be processed simultaneously.
  • Created chip runs for the CNP calibration fields. Rediscovered bug in getting stars reference catalogs near the pole.
  • Created new MD04 tessellation MD04.V3 which has pixel size closer to the gpc1 camera. Queued new set of warps for those.
  • Studied chip processing in preparation for implementing replicated outputs and better handling of update errors.
  • Wrangled some exposures through the system to the rsync server for the outreach program.
  • Debugged and repaired more problems with MPIA postage stamp data store. Looks like I my have inadvertently deleted some data files. Now that I understand how the system is configured it will be easier to debug problems in the future.
  • Debugged and fixed problem on postage stamp request form.
  • One day as official processing czar. Other days was a pest for the acting processing czar. (Fix the faults!)

Chris Waters

  • Fixed error in split queuing for stacks and OSS diffs that was causing the wrong pairs to be diffed for OSS.
  • Managed lossy compression run to free up diskspace. Continued shuffling data from full to less full hosts to do the same.
  • Worked on linearity code, discovering that the multiplicative correction was hurting the output image quality in the tests due to the overscan not having the same factors applied. I reworked the linearity correction to be additive instead of multiplicative, remeasured the input data on overscan corrected images, and remade the linearity detrends. Still testing this to confirm if it works successfully.
  • Day out due to illness.