IPP Progress Report for the week YYYY.MM.DD - YYYY.MM.DD

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Still populating ipp003 with nebulous dump (20120206).
• Ran slocate on the hosts. All index files are in <hostname>:/tmp/slocate. Note: slocate is different from one wave to another.
• mysql partitioning sucks (e.g. user-defined functions cannot be used for hash partitioning)
• Bad files hunt in nebulous and/or on ippb02 (with Heather):
  • Ran slocate on ippb02.[0-2]/nebulous (got ~10.7M files) -> This version runs incredibly fast. Scanning all three ippb02 disks took 22 minutes (no attempt to optimize)
  • Checked their status in nebulous:
    • ok
    • uri not present in instance table for a given ins_id: ~5M files (Note: it seems that some files are temporary) For a lot of those files, the uri in nebulous is ​file:///data/ippb02.1/nebulous[...] while the file on disk is ​file:///data/ippb02.0/nebulous[...].
    • uri not present in instance table and no ins_id but "similar" uris exist: 189k files

      Example of two similar uris:
      file:///data/ippb02.0/nebulous/00/00/1006632459.gpc1:ThreePi.nt:2011:01:23:o5584g0495o.285189:o5584g0495o.285189.ch.182968.XY34.mdl.fits
      file:///data/ipp025.0/nebulous/00/00/1230393764.gpc1:ThreePi.nt:2011:01:23:o5584g0495o.285189:o5584g0495o.285189.ch.182968.XY34.mdl.fits
      

  • No entry in nebulous but there is a file on the disk: ~349k files (e.g: gpc1/ThreePi.nt/2010/12/30/o5560g0658o.269979/o5560g0658o.269979.ch.174390.XY03.ptn) Total: 5.7M "problems" See /tmp/nebulous_contents/problems.

Heather Flewelling

• ssp
  • ingested jt's ssp data into ssp database. There are duplicates due to problems last week with nebulous and with a crash on my computer. These need to be culled.
• isp
  • processing dates through cam (continuing). wrote a script to make this easier
  • trolling through database/images to add comment and other columns to isp database
  • trolling through database/images to check that isp raw exposures have 2 good copies. Currently 8% have bad or missing copies.
• gpc1/nebulous
  • investigating (with serge) bad raw images in nebulous/gpc1.
• grabbed files for larry
• helped pavlos some (need to help him more) to get ipp running.
• czar
  • restarted summitcopy (it had too many connections, causing problems for camera)
  • cleaned up lots of blood on thursday
• finished jtmd stacks (2011 and earlier)
• sick half a day

Roy Henderson

Mark Huber

• ppSub DIFF_* stats and flags2 investigations with Larry and tracing out possible failure examples, revisiting filtering of detections from diffim residuals wih Ken Smith and monitoring cases of poor subtractions.
  • condition for neighbor and/or self-match to use source s/n rather than detection peak s/n
• updating IPP on macbook and local setup of photpipe for testing
• DRAVG minutes
• occasional kicking of LAP
• sick Friday

Bill Sweeney

Chris Waters