IPP Progress Report for the week 2010.09.13 - 2010.09.17

(Up to IPP Progress Reports)

Eugene Magnier

I've been working this past week on the difference image dual convolution process. The dual convolution process attempts to solve for kernels which convolve both input images. If only the chisquare image is minimized, there is (potentially) no unique solution because both images may be convolved to an arbitrarily large size. To add a constraint, we have been applying penalty terms in the system of equations to punish large kernels (based on a suggestion from Yuan & Akerlof). The penalty scale was very sensitive and I became dissatisfied with some of the results. I modified the system of equations to simultaneously minimize the residual image (chi square) and the second moments of the output image (actually, \Sum(x2 f2) / \Sum (f2) to avoid driving the solution to negative fluxes). This works very well for a very wide range of input test images.

In addition, I have been fixing the stack masks. The current stacking code generates masks which only use a single "bad" value for pixels without any valid input pixels. This means we lose the information about (a) which pixels are bad because of consistent issues (especially saturated stars) and (b) pixels for which some / most / all of the pixels are suspect (a category weaker than 'bad'). In the latter case, this includes the mask circles around bright stars and diffraction spikes. As a result, we were unable to detect the difference structures resulting from the wings of bright stars, at least for the stack-stack diffs. I have modified the stacking code to deal with this issue. Now, if all input pixels are 'bad', the output mask is the bitwise OR of all of the bad input pixels. If an output pixel is valid, then the output mask is the bitwise AND of all input pixels actually used. The former means that a bad pixel shows all possible causes, while the good pixels only show consistently present suspect bits. (Perhaps we should be using a fuzzy logic and raising the suspect bit if something like 80% of the accepted pixels have that bit).

Serge Chastel

  • PSPS: built small test files from existing ones, types conversion concerns
  • MOPS: meetings
  • IPP: who uses the cluster, helped Roy for mailing-lists.

Heather Flewelling

  • ifaps1
    • helped set up Roger
    • answered questions (geoff, dixon, ebeling, tonry, gaidos)
    • found skyprobe data for gaidos
  • czar 2 days
    • investigated 131.3PI.g.W+7 chunks with bill
    • tested bills's rundiffskycell.pl - works well
    • changed priority of processing - processing SweetSpots? first per ken's advice
    • restarted distribution after it died
  • planning trip to Harvard
  • attend grad student talks on panstarrs
  • addstar
    • merged final ThreePi? minidvodb on ipp004. now what?
    • restarted addstar for MD04
  • ppMerge -investigating differences between my version of ppMerge and the
    • trunk. They are different, but I'm not sure they are better.

Roy Henderson

  • PSPS
    • monitoring of loading. Some failures to look into to (some are the < -30 issue, others need to be looked at)
    • had to stop loading due to issues with DVO Dbs at IPP end
  • IPP
    • Czar on Monday:
      • re-queued MDFs from the 11th (were overlooked due to stdscience restart over the weekend)
      • some czartool tweaks (eg log 10 instead of natural log plots)
    • a lot of time spent figuring out mailing list administration. This needs some reorganization.
    • rest of time spent on stacks for PSPS:
      • located example cmf files, worked with Jim to tally IPP tables with PSPS schema
      • some refactoring of my Perl code to prepare for stacks:
        • new class to encapsulate Batch creation/deletion etc. Needed before incorporating stacks
        • batches are now subclasses of Batch, eg DetectionBatch, StackBatch etc. This minimizes code duplication while making code more readable and maintainable.

Bill Sweeney

  • Fixed problem reported with postage stamp diff stage image look ups byexp. This was causing problems with detectability queries.
  • Debugged problem with destreak -advancerun. Some timing parameters for the task needed to be changed.
  • Debugged problem with dvomerge 2mass + ps1 that MPIA was experiencing. It turns out that the 2mass dvo catalog we have distributed (years ago) is incompatible with dvo merge. I also needed to make some changes to the photcodes file that was being used.
  • debugged and resolved problem with MPIA's data store where commands behaved differently on some machines. Turned out to be some perl modules missing from the build that were installed on the system on one of the machines.
  • debugged some stuck postage stamp jobs. Turned out to be due missing burntool tables and corrupt camera runs (missing mask files) I look forward to the Great Reprocessing so we can avoid these bad data.
  • checked in the tools for repairing corrupt image data by rerunning the processing by hand.
  • Debugged the fact that some exposures have been added to the ThreePi?.V1 dvo database multiple times. It looks like we need to get more clever about error recovery for addstar and dvomerge.
  • Two days processing czar.
  • Spent quite a bit of time dealing with the special SweetSpott? OSS exposures. The desired difference pairs are incompatible with the way we queue diff runs automatically. We also had warp-stack diffs turned on for these data which caused all sorts of chaos.
  • investigated a stack failure. Looks like poor quality inputs are not being handled properly in one case. Filed ticket 1427.

Chris Waters

  • Added MD11 handler to nightly science code with MD11.V2 tesselation
  • Finished diff/inverse/destreak file cleanup, freeing >50TB of disk space. Reran cluster diskspace usage code, with updated results here: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Cluster_Storage_Notes
  • Attempted to determine why nebulous database occasionally slows down.
  • Regenerated linearity detrends, and finished bulk of linearity correction coding. Testing and debugging this week.