IPP Progress Report for the week 2012.02.06 - 2012.02.10

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Still populating ipp003 with nebulous dump (20120206).
• Ran slocate on the hosts. All index files are in <hostname>:/tmp/slocate. Note: slocate is different from one wave to another.
• mysql partitioning sucks (e.g. user-defined functions cannot be used for hash partitioning)
• Bad files hunt in nebulous and/or on ippb02 (with Heather):
  • Ran slocate on ippb02.[0-2]/nebulous (got ~10.7M files) -> This version runs incredibly fast. Scanning all three ippb02 disks took 22 minutes (no attempt to optimize)
  • Checked their status in nebulous:
    • ok
    • uri not present in instance table for a given ins_id: ~5M files (Note: it seems that some files are temporary) For a lot of those files, the uri in nebulous is ​file:///data/ippb02.1/nebulous[...] while the file on disk is ​file:///data/ippb02.0/nebulous[...].
    • uri not present in instance table and no ins_id but "similar" uris exist: 189k files

      Example of two similar uris:
      file:///data/ippb02.0/nebulous/00/00/1006632459.gpc1:ThreePi.nt:2011:01:23:o5584g0495o.285189:o5584g0495o.285189.ch.182968.XY34.mdl.fits
      file:///data/ipp025.0/nebulous/00/00/1230393764.gpc1:ThreePi.nt:2011:01:23:o5584g0495o.285189:o5584g0495o.285189.ch.182968.XY34.mdl.fits
      

    • No entry in nebulous but there is a file on the disk: ~349k files (e.g: gpc1/ThreePi.nt/2010/12/30/o5560g0658o.269979/o5560g0658o.269979.ch.174390.XY03.ptn)
    Total: 5.7M "problems" See /tmp/nebulous_contents/problems.

Heather Flewelling

• ssp
  • ingested jt's ssp data into ssp database. There are duplicates due to problems last week with nebulous and with a crash on my computer. These need to be culled.
• isp
  • processing dates through cam (continuing). wrote a script to make this easier
  • trolling through database/images to add comment and other columns to isp database
  • trolling through database/images to check that isp raw exposures have 2 good copies. Currently 8% have bad or missing copies.
• gpc1/nebulous
  • investigating (with serge) bad raw images in nebulous/gpc1.
• grabbed files for larry
• helped pavlos some (need to help him more) to get ipp running.
• czar
  • restarted summitcopy (it had too many connections, causing problems for camera)
  • cleaned up lots of blood on thursday
• finished jtmd stacks (2011 and earlier)
• sick half a day

Roy Henderson

• sick on Thursday

• ippToPsps:
  • finally finished off code changes to load by box on the sky, configurable from the config. Still needs some tweaking.
  • finally implemented a class encapsulating the ippToPsps config file:
    • less code duplication as all classes can be passed config object and use 'getters' for the various fields
    • logger objects now created by config object, again, to reduce duplicated code across programs
    • a refresh() method means the config can be re-read at any time meaning config settings can be changed on-the-fly without a need to stop/start programs
    • config values can be printed to log on demand
    • re-factored all other code accordingly

• PSVO
  • fixed bug in PSVO reported by Nigel and made a release
  • slowly improving comments in PSVO code, generally neatening-up the code

• Other
  • czar Monday and Tuesday. Learned how to unstick LAP, which came in handy.
  • mailing list stuff - new list for psps working group
  • fixed bug in script that backs-up various databases - was not dumping contents, only schemas
  • updated PSPS news page after merge completed, also including IPP 3PI/OLD coverage for comparison
  • some cleanup after completed merge

Mark Huber

• ppSub DIFF_* stats and flags2 investigations with Larry and tracing out possible failure examples, revisiting filtering of detections from diffim residuals wih Ken Smith and monitoring cases of poor subtractions.
  • condition for neighbor and/or self-match to use source s/n rather than detection peak s/n
• updating IPP on macbook and local setup of photpipe for testing
• DRAVG minutes
• occasional kicking of LAP
• sick Friday

Bill Sweeney

Chris Waters