IPP Progress Report for the week 2011-07-04 - 2011-07-10

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• IPP czaring explained to Mark
• MOPS Czar (Fri -> Fri)
• IPP Czar (Friday -> Tuesday, Friday)
• MD IPP-MOPS-TEST publishing issues
• SVN server for IPP operation (to be continued)
• Hsieh mops smf
• ippdb02 crash (not recovered yet)
• Huntley support for stares
• Mailed IPP-users concerning gpc1_0/gpc1_1 future

Heather Flewelling

• czar for 3 days
  • many problems - registration/burntool/summitcopy problems on wed
  • nfs/nebulous/many machines down on thursday
  • stopped processing twice (once for dump, once for memory upgrade)
  • cleaned up lots of blood on czartool (see czar log for details)
  • mark - gave him instructions on processing MD10, some czar hints as well.
• isp
  • after discussions with ken - possible way to tag the exposures that have been obscurred by the dome - this requires more testing, I was interuppted by czarring
• dvodbs
  • ThreePi - merged ThreePi.124 - ThreePi.340
  • rsyncing over this db to ipp005
  • LAP - queued all available destreaked files
• holiday monday

Roy Henderson

• ippToPsps loading:
  • some minor troubleshooting, eg missing IQ_FW1, FW2 values in psf header causing problems for moments in ImageMeta
  • some dxlayer issues:
    • lost config used for Boston loading, csv path for loader was wrong
    • daemon quitting after a single batch. Had to keep restarting
    • Conrad to the rescue: tested and working now
  • issues finding the correct stack cmfs to load:
    • apparently we make two cmfs per stack_id. Changed SQL accordingly
    • impossible to find the full path for a given stack from GPC1, so wrote code to read all cmf headers at a given path_base until correct one is located
  • issue with skycell ID: we need a schema change, as we store as an int, but format is eg 0683.043. Mangled to 0683043 for now
  • loaded all stacks, not that there were many

• ippToPsps coding:
  • started work on some loading metrics code including:
    • queries to find numbers for pending batches, processed batches, faults etc
    • queries to monitor speed of loading, time of last published
  • sub-classed python Logger class in order to add some convenience methods, like printing separators, value pairs
  • improved a lot of the logging messages for consistency
  • code changes, as well as changes to ipptopsps Db, mean different batch types are handled polymorphically. Much cleaner
  • added a warning prompt if 'force' and 'publish' options are both enabled
  • encapsulated FITS reading code into a class. Constructor now copies file to the local file-system first to speed-up access
  • started work on jython code to query ODM via SOAP for batch status.

Mark Huber

• office shuffle + new desktop computer setup + wrapping up relocation process
• learning the ways of czaring, watching resolution of possible problems and processing flow
• simtest IPP to PSPS -
  • camera SMF setup trials
  • stack extended source analysis setup trials
  • looking into adding psphotStack
  • comparing to list from Roy of key elements
• MD10 refstk new tessellation
  • working through the wiki notes from Bill on MD06,MD07 and suggestions from Heather. working though IPP setup on production cluster.

Bill Sweeney

Chris Waters

• SAS: queued y/z SAS footprint and year1 + year2 LAP processing.
• Dark: started processing on new dark master, to allow for less downtime when dark creation fixes are in place. Ran various tests to determine that the current dark is introducing more noise than the old dark, and began work to attempt to not over-fit the dark model (which seems to be a likely source of the introduced noise).
• Diskspace: continued shuffle, added code to nebulous to attempt to ensure that files are properly written to disk. This should resolve issues with NFS errors.