Current Tasks for the IPP Team

(Up to Main Page)

Notes from WBS entries

Here are the tasks that I believe we need for (a) and (b):

a) Tasks required before the start of the survey:

• IPP to OTIS interface : the code is in place; we will turn this on with basic rules by the end of this week (3/12 - CZW)
• Detectability Server : two items are required
  • the program 'psphotDetect' must be finished (3/19 - EAM/CZW)
  • integration with the postage stamp server (3/26 - WS)
• Distribution System updated for Raw Data : (3/12 - WS)

b) Tasks not required for the survey start:

• Finish Extended Source Analysis Code (3/31 - EAM)
• Stack Photometry Process (4/30 - EAM)
• Stack Rejection Issues : this is divided into three main tasks:
  • demonstrate MD stacks with internal (PS1-only) astrometry (3/19 - PAP)
  • improve masking strategy following advice from Durham (3/26 - PAP)
  • define the 3pi processing sequence for stacks (4/30 - PAP + EAM)
• Full-Scale Photometry & Astrometry Relative Calibration Demo
  • this task can be done with SAS data (3/31 - CZW)
• IPP to PSPS interface : this item consists of several sub-tasks (RH):
  • create initialization tables : done
  • code to generate 'detections' table : done (but missing some fields?)
    • missing fields from SMF -> EAM
  • code to generate 'difference' table : done (but missing some fields? ingest untested?)
    • missing fields from CMF -> EAM
  • code to generate 'stack' table : done, but untested -- blocked by 'extended source' task above
  • resolve recalibration process : blocked by 'Full-Scale Photometry.. Demo'
• ISP setup for automation (4/7 : HAF)
• Distribution of Detrend Images : (4/15 : WS)
• Integrated Release of Uncorrected Data : (4/30 : WS + PAP)

Tasks for the Immediate Future

• major missing features : required for the start of the survey
  • detectability server
  • ipp->otis interface
  • distribution of raw data

• major missing features : important, but not blocking start of the survey
  • extended source analysis
  • stack photometry
  • stack rejection issues
  • ipp->psps interface
  • full-scale photometry & astrometry calibration demo
  • distribution of detrends
  • integrated release of uncorrected data
  • processing recipe for final stacks

• major demo month lessons to be addressed
  • keeping the pantasks queues filled
  • address large load spikes
  • operational rigidity
  • how to deal with down machines (on-the-fly replication?)
  • address real vs nfs failures
  • address real failures
  • more load and disk space balancing (machine limits by job?)
  • better targeting of data to nodes
  • more regular operations schedule - procesing czar
  • better tolls for examining current state (more ippMon? more other tools?)

• current operations tasks
  • nightly processing
  • Run 5b : waiting for Paul Sydney's OK
  • Run 5c
  • M31
    • Run 5b data
    • make uncorrected warps
    • make uncorrected deep stack
    • diffs relative to deep stack (needs to update magic)
    • restored warps

• current dev tasks
  • fix forced photometry
  • pswarp errors
  • handle nebulous widows in scripts (done?)
  • addstar imports from UKIDSS & SDSS
  • ISP detrends : define & set up processing
  • optmize dvo image lookups

• cluster management
  • automate deletion of extra diff images
  • automate cleanups of un-destreaked images
  • define reboot procedure

• psphot work:
  • update the CR masking code
  • add psphot stats on for dipoles
  • 'scale error' in stacks
  • limit max source detections is psphot

• write ups
  • ppSub updates
  • DVO usage

Eugene Magnier

• binned 4x4 mask for ken
• extended sources
  • set upper limits on Petrosian radii
  • make sure new structures are linked into output functions
  • generate a more complete test set & run tests
• ingest giebink stuff into svn trunk
• poor pixels flagged in detections

Heather Flewelling

• set up ISP detrends
• double-check output headers (try Chris' script)

Bill Giebink

• DIMM work
• psLib & psModules test code
• build a "realtest"

Roy Henderson

• PSPS work
• PSPS interface bugs
• demo samples
• IPP-PSPS stack output tess_id + skycell.(NNNN).MMM tess_id + skycell.MMM

Paul Price

1. Run psphot on SAS stacks (manually)
2. Automated deletion of diff convolved images
3. Implement ippdb/ippTools/ippTasks for stackphot
4. Pipeline for uncorrected data

• pay attention to stack, diff, warps failures
• add skycell_id and tess_id to headers of diff + stack outputs (incl cmfs)
• check use of FWHM for ppSub scaling
• update stack masking for unconvolved stacks (on a branch)
• write up and demo data set for ppSub / ppStack

Bill Sweeney

• send raw detrend and m31 science data to MPE
• postage stamp server
• send raw detrend and m31 science data to MPE
  • stamps from cleaned runs - write tasks for detecting update completion - 1 week
    • optimize : don't reprocess everything
  • repair un-censored deletion : 2 days
  • meet with NCU folks on pstamp web interface
  • improve access control add quotas
  • enhance web interface with new features especially parse results file and display
  • automate cleanup of old requests -
  • update user documentation

Chris Waters

• queue rawExp compressions
• burntool / nightly science bugs
• ipp to otis interface
• sas relphot & relastro

Cindy Giebink & Steph Mohr

• ipp005 bios needs to be reset (wait for Gavin)
• debug hardware problems on ipp008, 9, 14, 16, 18
• move homedirs to ippc18
• areca scripts for automatic verify
• try neb-hosts with Chris
• work with Bill Sweeney to get otis

KPAT / DRAVG Priority Issues (+ Notes)

• latency
  • we have performed a throughput demonstration which meets requirements.

• magic masking
  • the latest large-scale run had a median streak count of 8.
  • we need to make an assessment of the current state of masking by masking bit (both warp and chip)
    • Heather has made an initial assessment of the masking using ~100 individual chips and warps. If this was done for 100 chip Files (not chip Runs) and equivalently for warps, then we need more statistics to say something useful.

• stacking: noise correlation
  • we believe this has been addressed by Paul's recent work
    • we need an independent assessment if this is still an issue
  • Nigel is concerned about the photometry of psfs in the stacks
    • Nigel and Paul have both been investigating the situation. Some conclusions:
      1. the latest version of the code (eg, stack_id 25532) results in scatter in psf magnitudes which are larger than the warp-warp photometry (0.04 vs 0.03).
      2. the aperture magnitudes seem to be in better agreement, which says the pixels are probably OK
      3. Paul started to use simtest to verify the quality of the pixel photometry

• diff images: residuals and noise correlation
  • we have made some good progress on this with the dual convolution technique, and will generate a set of test images this week.
    • this is critical : as of 2009/11/23 this has not been done
  • another significant source of residuals comes from poor astrometry
    • Gene has investigated the astrometry failures and improved the analysis (outlier clipping). Also, several cases of poor astrometry resulted from stars being falsely identified as cosmic rays and the wings being used in the fit instead. this should now be fixed in psphot.
  • we also also working on improving the quality of the detection analysis and the associated photometry in the diff images
    • Gene will be making this top priority again week of 2009/11/23 (this has been on the back burner while the astrometry issues were studied)

• astrometry: some catastrophic failures (offsets>3")
  • some progress made in some extreme fields (M42). this may also result in improvements in other poor situations. otherwise under investigation.
  • Gene has included a number of fixes. there is a bit more work to make sure the warps do not use poor astrometry results.

• xtalk artifacts.
  • for the near term, we will address this by masking the worst offenders in the camera masking stage.
  • still need to pin down the xtalk rules, but we can implement the xtalk we know.
  • as of 2009/11/23, this xtalk masking has been implemented by Chris, but with only limited testing to date.
  • Mark Huber says (preliminarily) it is looking good

• other image artifacts.

• photometric calibration
  • the zero point is now being calculated from the 'blue edge' of the ref - raw magnitude distribution to reduce the impact of halo stars. The error analysis is now done using bootstrap-resampling, but this can only tell us about the random errors, not the systematic errors from field-to-field.

• star/galaxy separation
  • Eddie has made some plots of the quality of the current S/G flag: PS1_rates. The rules still need to be tweaked.

Other Near-Term Goals

• address warp failures (photometry problem on 3 warps in throughput test)
• someone needs to put together code that feeds back quality info to summit

• make white light dome flats (including w)
• make new flat-field corrections
• make new masks
• iterate on the darks