IPP Progress Report for the week 2010.06.21 - 2010.06.25

(Up to IPP Progress Reports)

Eugene Magnier

We had a planned power outage at MHPCC on Wednesday this week. The outage went smoothly, though it took somewhat more time than we expected. I took the opportunity to re-organize our chip->host and skycell->host relationships in an attempt to make the storage usage more evenly balanced. We had not been using all of the wave 1 nodes for storage in several of the stages, with the result that those machines were nearly 10% emptier than the wave 2 and 3 nodes. It will probably take some time to get the balance back even after the adjustments.

I spent an inordinate amount of time attempting to clean up the many terabytes of test data sets taking up space on ipp004. As discussed by Heather, the disk was full, making the dvomerge have trouble. The cleanup is still not complete, but I freed up 25% of the partition. We are getting ready to receive our first set of replacement 2TB disks, just in time, as the cluster has been pushing the limits lately. This is partly driven by a number of non-standard processing batches and large sets of data not currently automatically deleted: the MD04 stack data, all diffs from 3pi and Sweetspot for Paul's photometry reprocessing, all Sweetspot images for the stacking. In the coming days, some of these big datasets can be deleted to clear up more space. The new disks will get us 90TB more free space (30TB extra per wave 1 node). Assuming this process goes well, we will schedule upgrades for another set of 3 nodes towards the end of the summer.

The plan for the MD04 deep stack was to produce an example with the standard pixel scale (0.2 arcsec), and then make an additional example in r-band with the GPC1 native scale (0.258 arcsec) for sensitivity comparison tests by Mark Huber. It turned out that the MD04 deep stack was made with the 0.258 arcsec pixel scale, so I launched the comparison in r-band using the 0.2 arcsec pixel scale. I also started the last lunation's sweetspot stack and warp-stack diff run.

I've been working this week on the extended source analysis. I added Kron magnitudes to the standard data products for PSF sources as requested by folks at Durham. I've also been testing the non-linear fitting, and attempting to use information from the Kron analysis to improve the initial parameter guesses.

Serge Chastel

  • Computer installation
  • ippMonitor
    • Histogram committed to SVN
    • Looked at the generator/definition files for the PHP scripts
    • Looked at JavaScript?+PHP+HTML to see how dynamic restrictions/displays could be added to current PHP scripts
  • Visit of IfA/MIC during Wed power outage.

Heather Flewelling

  • ippMonitor work
    • detrend verification histograms for darks are up on the test ippMonitor
    • more bug fixes
  • detrend server - sheparding detrend verifications (darks)
  • dvomerge - we moved the master to ipp037 when ipp004 ran out of disk space. merging minidvodbs from ipp004 to ipp037 was too slow, so Gene and Paul cleaned up space, then I got sick, and I moved back the master to ipp004 to restart the merge on monday (merge is restarted now).
  • gave Serge a grand tour of IfA and MIC during the cluster power outage on Wed.
  • out sick 2 days

Roy Henderson

  • PSPS
    • mysteriously wrong ra/dec for certain chips and exposures: generated test data for Jim to work with
    • changed schema: renamed moments in various tables and views, removed momentTheta from ImageMeta
  • IPP
    • changes to configuration after schema change (above)
    • now storing image flags from DVO in ImageMeta
    • wrote script to poll ODM web-service and query batch status for everything loaded, then update local Db with results. This is the basis for the recovery system.
    • Processing Czar for two days:
      • faults shown in Czartool now link to the correct log files
      • added 'last night's data' page in Czartool: quick summary of data taken at summit vs that registered at MCPCC
      • updated Processing Czar wiki page with recent changes
  • Other
    • lost time to power shutdown on Wednesday

Paul Price

  • MOPS support:
    • Discovered several problems with diffphot output which will require a re-run
    • Fixed psModules to read/write PSF clump parameters
    • Fixed psphot to produce meaningful anti-dipole stats when loading a PSF
    • Fixed psphot to read WCS, so CMF file includes celestial coordinates
    • CR_NSIGMA, EXT_NSIGMA not being set for sources because of lack of clump parameters in input PSF --- can't do anything about it, and not a big deal anyway
    • Fixed psphot to set rough classification of 'STAR' where the clump parameters don't exist
    • Fixed psphot to respect mask when doing linear fit, otherwise photometry not done for sources lying in area with advisory mask
    • Re-running diffphot (6226 diffs)
    • Fixed ippTools to respect magicked status properly
    • Publishing (magicked) diffphot results to IPP-MOPS-UPDATE on DataStore?
  • New production build: ipp-20100623
  • Background restored stages:
    • Added scripts and tasks
    • Debugging/verification run on simtest
    • Merged into trunk
    • Adding destreak/distribution
  • Cleaned out data on ipp004
  • Investigating processing failures
    • Re-ran several runs with corrupted files (this is annoying!)
    • Updated FITS table reads to deal with empty tables for removestreaks
    • Set runs to 'drop' where inputs have been cleaned

Bill Sweeney

  • Vacation

Chris Waters

  • IPP-OTIS interface:
    • Determined realistic FWHM and extinction criteria from CDFs of these values from science data taken since Demo Month
    • Added automation handler to queue new bundles every hour.
    • Created bundles for past data from before the automation take over.
  • Cleanup. Changed the cleanup handling to be less monolithic, allowing us to clean different surveys and stages at different rates. Added distribution cleanup with a 7-day retention time.
  • Began work on OTA14 masking status. Generated a large number of images to check to see if we can recover columns 0,2,3.
  • Edits to nebulous:
    • Provide a note to be set for each volume, so we have a record of their status.
    • Implement neb-migrate, to remove data from a specified volume.
    • Modifications to neb-ls to make it more user friendly.