IPP Progress Report for the week 2010.10.04 - 2010.10.08

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• IPP:
  • databases questions: all optimization of nebulous attempts failed (ippdb02, ipp022, tried also locally);
  • pretended to be czar on Wednesday and Thursday;
  • wrote scripts to set hosts off on all pantasks servers. Wrote also a script to check hosts status on the different servers but it needs to be improved (since it seems the information given by pantasks server is not always reliable).

• PSPS:
  • played a bit with selenium and phpunit.

• MOPS:
  • czar (with Larry)

Heather Flewelling

• ifaps1 liason (tonry, gaidos)
  • transferred dead skycell back to ipp-land.
• found that bathtub rings occurred after video cells
  • ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/BathTub20101006
  • ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/BathTub20100925
  • ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/BathTub20100801
• addstar
  • MD04 is finished
• created processing throughput plots for Sept 2010
  • ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ProcessingThroughputPlots2010Sept
  • ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ProcessingTimesSept2010

Roy Henderson

• PSPS
  • resumed batch creation, but not loading until merge/flip/copy is complete
  • meeting with Jim/Gene about stacks. Consequent schema changes
  • tracked the source of loading duplicate frames: datastore/DXLayer logic

• IPP
  • czartool
    • added option to make a useful plot of a single day
    • removed the zeros from labels table (Bill's suggestion)
    • work on a 'daily report' for Ken (not finished yet)
    • fair bit of work on a 'rate of processing' plot
  • ippToPsps
    • added database access to ippToPsps: needed for stack stuff
    • slowly populating fields for PSPS stacks
    • fun with fits binary table 'vectors'

• Mailing lists
  • figured out how to add new lists (code and Db changes)
  • wrote a backup script to dump Db etc (needed as everything is on neverland just now)
  • new sts list

Bill Sweeney

• Made several improvements to the handling of errors in update jobs queued by the postage stamp server. (The errors are caused by data that has been lost on ipp020). The update system is now much more capable of recovering for errors without manual intervention. This is an extremely important milestone for Bill as these problems have been an huge time sink and source of stress.
• Investigated the source of slow download performance seen by MOPS from the ipp data store. Two sources were found. First the perl module used for downloading has some inefficiencies. Secondly the web proxy can be a bottleneck. MOPS will now bypass the proxy and access the data store directly.
• Implemented ordering by priority for diff, destreak, and stack stages.
• Fixed bug where magicRuns were queued even though a diff run failed to generate any good quality skycells
• Processed 3 nights X 36 exposures from CFHT-MEGACAM through the IPP for the MOPS team. This went remarkably smootly given that we have not processed data from that instrument lately. Unfortuntely astrometry failed on several chips per exposure so some data was lost.
• Wrote a program to examine the nebulous database to discover all components that can never be updated because the original run information was lost on ipp020.
• Wrote a program to find all of the data listed on the datastore that is unavailable because data on ipp020 was s lost.
• Spent half a day dealing with the fallout due to the crash of the mysql database backing up the postage stamp server and data store.
• Updated pstamp-test system (ipp049) published CFHT detections for MOPS there while the DB was recovered.
• Processing czar on Friday. Helped out acting czar on other days. Did some label and list manipulations to allow the OSS diffs to jump ahead in the queues.

Chris Waters