IPP Progress Report for the week 2012.02.06 - 2012.02.10

(Up to IPP Progress Reports)

Eugene Magnier

I have been working on the reference photometry database, incorporating the ubercal zero points. I have been running tests with a chunk of the sky in which I have generated a database, applied ubercal, run relphot, and have the synthetic photometry merged in. This region has good (photometric) and bad (non-photometric) data, so it is a good test. I am ready to examine the quality of the relative photometry, but I have been side-tracked by two other issues:

First, I needed to modify the synthetic photometry database to match the 'depth' (density of the sky-based partition) of the 3pi database. Unfortunately, trying to do this, I encountered some format handling bugs in the program, dvosplit, which is capable of doing this operation. I fixed the bugs, and now dvosplit is more robust. I now have a synth database in the right format to merge with the full 3pi database. I have also launched the dvomerge for these two (still running as of 2/13).

Second, I was frustrated by the long time to do large-scale database operations in DVO. I wondered if the I/O rates were substantially worse that I would expect. I ran some tests and discovered that, although there is some overhead loss to the DVO I/O layer, it looks like we are also largely limited by the disk I/O layer (probably seek time more than just bytes in or out). I decided that the only way to get any significant speed up would be to parallelize the DVO I/O layer across many machines. I have worked out how this can be done, and have written much of the supporting structure. So far I have:

  • dvodist : a new program which will distribute DVO database tables across the clusters in an appropriate fashion (and retrieve them to make a single directory database if desired)
  • setphot : the program which applies ubercal (and other) zero points now has parallel capability (mostly tested and working, but it needs some hand-shaking work).
  • relphot : the program which does relative photometry now has parallel capability coded, but not yet tested.

One nice point is that the work on these programs illustrates all of the modes of parallelization needed by DVO, and gives example code chunks to be used in the rest of the system. Thus, it will be fairly quick to parallelize most of the system. Another nice point is that the database tables are distributed, but links are left behind, so the non-parallel programs can be run as before (with some performance hit going across the network for data).

Serge Chastel

  • Still populating ipp003 with nebulous dump (20120206).
  • Ran slocate on the hosts. All index files are in <hostname>:/tmp/slocate. Note: slocate is different from one wave to another.
  • mysql partitioning sucks (e.g. user-defined functions cannot be used for hash partitioning)
  • Bad files hunt in nebulous and/or on ippb02 (with Heather):
    • Ran slocate on ippb02.[0-2]/nebulous (got ~10.7M files) -> This version runs incredibly fast. Scanning all three ippb02 disks took 22 minutes (no attempt to optimize)
    • Checked their status in nebulous:
      • ok
      • uri not present in instance table for a given ins_id: ~5M files (Note: it seems that some files are temporary) For a lot of those files, the uri in nebulous is file:///data/ippb02.1/nebulous[...] while the file on disk is file:///data/ippb02.0/nebulous[...].
      • uri not present in instance table and no ins_id but "similar" uris exist: 189k files
        Example of two similar uris:
        file:///data/ippb02.0/nebulous/00/00/1006632459.gpc1:ThreePi.nt:2011:01:23:o5584g0495o.285189:o5584g0495o.285189.ch.182968.XY34.mdl.fits
        file:///data/ipp025.0/nebulous/00/00/1230393764.gpc1:ThreePi.nt:2011:01:23:o5584g0495o.285189:o5584g0495o.285189.ch.182968.XY34.mdl.fits
        
      • No entry in nebulous but there is a file on the disk: ~349k files (e.g: gpc1/ThreePi.nt/2010/12/30/o5560g0658o.269979/o5560g0658o.269979.ch.174390.XY03.ptn)
    • Total: 5.7M "problems" See /tmp/nebulous_contents/problems.

Heather Flewelling

  • ssp
    • ingested jt's ssp data into ssp database. There are duplicates due to problems last week with nebulous and with a crash on my computer. These need to be culled.
  • isp
    • processing dates through cam (continuing). wrote a script to make this easier
    • trolling through database/images to add comment and other columns to isp database
    • trolling through database/images to check that isp raw exposures have 2 good copies. Currently 8% have bad or missing copies.
  • gpc1/nebulous
    • investigating (with serge) bad raw images in nebulous/gpc1.
  • grabbed files for larry
  • helped pavlos some (need to help him more) to get ipp running.
  • czar
    • restarted summitcopy (it had too many connections, causing problems for camera)
    • cleaned up lots of blood on thursday
  • finished jtmd stacks (2011 and earlier)
  • sick half a day

Roy Henderson

  • sick on Thursday
  • ippToPsps:
    • finally finished off code changes to load by box on the sky, configurable from the config. Still needs some tweaking.
    • finally implemented a class encapsulating the ippToPsps config file:
      • less code duplication as all classes can be passed config object and use 'getters' for the various fields
      • logger objects now created by config object, again, to reduce duplicated code across programs
      • a refresh() method means the config can be re-read at any time meaning config settings can be changed on-the-fly without a need to stop/start programs
      • config values can be printed to log on demand
      • re-factored all other code accordingly
  • PSVO
    • fixed bug in PSVO reported by Nigel and made a release
    • slowly improving comments in PSVO code, generally neatening-up the code
  • Other
    • czar Monday and Tuesday. Learned how to unstick LAP, which came in handy.
    • mailing list stuff - new list for psps working group
    • fixed bug in script that backs-up various databases - was not dumping contents, only schemas
    • updated PSPS news page after merge completed, also including IPP 3PI/OLD coverage for comparison
    • some cleanup after completed merge

Mark Huber

  • ppSub DIFF_* stats and flags2 investigations with Larry and tracing out possible failure examples, revisiting filtering of detections from diffim residuals wih Ken Smith and monitoring cases of poor subtractions.
    • condition for neighbor and/or self-match to use source s/n rather than detection peak s/n
  • updating IPP on macbook and local setup of photpipe for testing
  • DRAVG minutes
  • occasional kicking of LAP
  • sick Friday

Bill Sweeney

  • Completed testing of astrometric reference catalog for one of the STS fields. Ran 120 exposures using it. sigma_ra and sigma_dec are significantly improved: reduced by factor of about 8).
  • Investigated by detectability queries for stack-stack diffs do not work. There are some significant problems with the method used to find the exposures.
  • Queued and monitored processing of 162 M31 exposures in g, z, and y filters. Ran into a serious problem with some of the z band data. One one night the seeing was very good ~3 pixels and psphot's memory usage exploded causing machines to crash and jobs to fail. Tried running these on a machine with large swap space dedicated to these 5 chips. It took most of the weekend but the exposures completed. Need to investigate what psphot is doing. Also the data from 2009 did not come out well. There is something wrong with the recipes used for this data. Pattern correction?
  • cleaned up database mess caused by ippc17 crashing repeatedly.
  • one day vacation

Chris Waters

  • Vacation half the week.
  • Finished testing and debugging of PATTERN.CONTINUITY correction. Created new g-filter footprint stacks, and distributed them to the background group for comparison to previous footprint stacks. Results summarized here: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Continuity
  • Czar, involving kicking LAP and working around host repair work.