Gene 1

Oct 5
Here are some notes related to the Great Reprocessing which will be
useful for discussion today.

In terms of IPP issues, I believe the following are needed before the
reprocessing can start:

1) new hardware delivered and installed (shipped late last week,
expected in 2-3 weeks according to Freddie)

2) star/galaxy/CR separation finalized.  While I was in Harvard, I got
Eddie Schlafly up to speed on this issue.  I gave him a collection of
SMF files in the latest format that includes all of the parameters
which I am currently trying to use to make the star / galaxy / CR
separation.  These exposures overlap SDSS, so he is able to use that
as ground truth.  He is looking at the parameters I measure (Mxx, Myy,
PSF mag, Kron mag, other aperture-like mags) and attempting to define
the cuts in this multi-dimensional space that define stars, galaxies,
and CRs.  He is first working to define the a-priori probability that
a given object is consistent with a PSF.  Next, I've asked him to work
with Shaun Cole on defining the a-posteriori probabilities (star,
galaxy) given some other information (specifically galactic latitude
and longitude).  He has already made some headway in better defining
the CRs from the faint stars.  There is some subtlety here that I've
asked him to explore: is it enough to define the probability of
something being a star (PSF) using on the locations of stars in the
space, or is it necessary to include some description of the shape
distribution of the galaxies themselves.

3) decisions / implications to difference image analysis from
photfest.  The big advance over last week was that the CfA group now
agree that ppSub SINGLE direction convolution produces results
indistinguishable from the photpipe difference image analysis.  They
also convinced me that, for the stack-stack diffs where we have a deep
reference stack, it is acceptable to always choose the deep reference
stack as the convolution target -- the occasional instances of small
amounts of deconvolutions are acceptable given the high S/N of the
reference stack and the over-sampling.  We also agreed that the DUAL
convolution is at worst equivalent to an optimal SINGLE convolution
plus an additional smoothing, and it may be the case that the DUAL
convolution is acceptable as well.  In the near term, we are going to
switch the MD stack-stack diff processing to use SINGLE convolution of
the refstack, and the CfA group will start to use at least the IPP
pixels.  We (mostly they -- Mark Huber, Micheal Wood-Vasey, and Ken
Smith) are looking at the detections produced by the IPP, along with
the feedback I gave them on the flags and exclusions, to see if (with
appropriate filtering) the false positives and false negatives are
acceptable and if the photometry is is consistent with the photpipe
equivalent.  The quick look at the last point was positive, but they
are doing more checks on this.

As far as the Great Reprocessing goes, the main lesson & outstanding
point from the photfest was the remaining concern that DUAL
convolution STILL results in more smoothing than one would expect or
hope.  Since DUAL is required for the warp-warp diffs (ie, 3pi and
SS), it is important to continue to push on this.  Two possible
avenues: 1) the range of kernels which we use is not as large as CfA:
we have a set of 3 Gaussians which we scale by the FWHM, while Armin
adds an additional large Gaussian if the FWHM gets too large.  2) the
DUAL convolution could potentially be made to fall-over to SINGLE
convolution if only one of the two images is convolved.  This is
tricky to determine, but a bit of work on that may be useful.

4) address recent large variations in processing throughput (perhaps
due to nebulous mysql fragmentation?).  Serge & Roy have learned that
running the mysql optimization process may help to speed up the
nebulous interactions; Serge is investigating this and will also look
into partitioning the big nebulous table.  The throughput variations
can be a factor of 3.5!

5) updates to non-linear correction, dark correction, and resulting
mask modifcations.

6) a full processing strategy (what area first, exactly what
processing is done) and any resulting scheduling code changes (ie,
updates to the nightly science code to manage the processing).

7) astrometric and photometric reference catalog for the portion of
the sky that is to be first processed.

It would be nice, but is not required, that the 3pi static sky
analysis be finalized, and that the IPP->PSPS interface be finished.
It is also important to have a plan for the storage upgrades, but we
have enough space to start.