IPP Progress Report for the week YYYY.MM.DD - YYYY.MM.DD

(Up to IPP Progress Reports)

Eugene Magnier

We have set up 2 'tiger teams' to push on 2 specific top-level concerns: 1) why is processing throughput not where it should be? (WS, CZW, MEH) 2) what is the cause (and mitigation?) of the false detections found in the footprint stacks? (HAF, SC, RH). Both teams have met multiple times over the week and are making what seems to be good progress.

The quick answer to (1) is "various issues" : some specific hardware problems (ipp014, ippc06), some memory overload problems (streaksremove, ppMops), some bottleneck bugs in the LAP management code, ongoing disk space issues. There are a few places that we can reduce our sensitivity to hardware problems (the distributed apache/neb interface can be made responsive to machine failures; a particular type of hang-up caused by a blocked log file can be fixed). We need to keep fixing and avoiding glitchy hardware and to finish the disk upgrades in progress. The memory overload issues can be addressed by modifying the fits table I/O functions somewhat, but there is also a problem with allowing some bad diffs to go through the system. Bill and Mark are working on those issues, while Chris works on the LAP related issues. Overall, the bottom line is that, if things go smoothly, the system can run ~50 exposures / hour (1200 / day), but keeping things at that rate is the trouble.

The quick answer to (2) seems to be "faint, roughly sky-level detections". It is clear that the wide wings of the pixel distribution are not Gaussian, and these outliers are driving the faint source detections. The bulk of false positives are faint, and are not associated with specific bright defects (or real effects). But they do prefer certain parts of the camera over others. The likely bet is that these are coming from the 'streakies', the residuals of the row-by-row bias variations that remain in the images. One additional supporting piece of evidence is that the g-band has by far the largest number of these type of false positives. The psphot/SDSS test shows that the software does not tend to pick up fake things that are not coming from the data. TT2 is looking further at tests that can distinguish these faint false positives from real faint things, but hopes are not very high (IMO).

We were having lots of problems with ipp014 (which had the backup of ipp013 while under refurb). Cindy replaced ipp014's raid card, with good improvement in its behavior. Cindy and Haydn also did the second round of disk swaps (and mobo swap for ipp037). We are ready for the last one 8/31.

I also spent some time this week talking with John Tonry and Doug Finkbeiner about photometry. Doug has done a lot of work comparing PS1 and SDSS in MD fields, and the results generally look really good. There is one lingering effect that we are trying to understand better: at the faint end, the PS1 mags (IPP values) are fainter by a small amount than the SDSS mags of the same objects. The effect is between 0.2 and 0.5 sigmas, but it seems to be a consistent bias. There are actually a number of effects that may be present and can drive this difference in this same direction: color terms (faint stars are redder than bright stars on average), galaxy contamination (harder to distinguish at the faint end, and the SDSS numbers are model mags), Poisson vs Constant weighting in the psphot analysis, and error in the sky estimation.

Finally, I am more-or-less happy with the state of the Kron mags in psphot -- they are now in good agreement with sextractor and reasonable agreement with input values for synthetic galaxies. I needed to include better downweighting of nearby objects and a more robust 1st moment measurement. I am now putting the finishing touches on the model fits using those new Kron mag measurements as inputs.

Serge Chastel

  • ppMops fix
  • ippMonitor move to ippdb01 (clusterMonitor, db replication status)
  • False Positive zoology with Heather and Roy

Heather Flewelling

  • zoology (the tiger team to investigate the false positive detections)
  • addstar
    • ipp004->ipp006 rsync finished
    • restarted ThreePi? dvodb creation (on ipp006: it is about 2x faster than ipp004, but was stopped because we ran out of space)
    • finished up SAS dvodb - hunted down and investigated the missing entries (a fw missing in cam stage because of them not being destreaked, and one missing in staticsky because of a programming error)
    • started rsync (ipp006->ipp004)
  • helped roy with kent wood: gave him my script to query dvo at a specific location
  • grabbed files for JT
  • sent TD MD field centers

Roy Henderson

  • ippToPsps
    • found minor bug in queuing: now using batch-type in query to determine what has, and hasn't, been processed
    • added code to reject chips with bad astrometric solutions, the source of the duplicate object IDs in P2 batches (see last week's report)
    • investigated duplicates in stacks: some with NULL flux, some with negative flux. Simply rejecting duplicates, i.e. taking the first one only
    • attempts to use the DXLayer web-service to delete batches. Help from Gavin.
    • now logging to the database when loading to datastore fails (eg no network), and...
    • wrote code to queue batches that didn't load to the datastore
    • continued to load new LAP data (400 so far)
  • Support
    • more discussions with Kent Wood regarding his Fermi sources
    • pulled detections from the 3PI DVO database for Kent, with lots of help from Heather
  • Tiger team
    • lots of discussions and meetings
    • worked on a program to run a set of cuts against all filters for all tables. Fully configurable
    • helped write-up results on wiki
  • Other
    • attended, and made a short presentation, at Conrad's MOPS workshop on Wednesday afternoon. Prep included some minor tweaks to PSVO.
    • set up a new ps1sc mailing list

Mark Huber

  • moved office
  • MD reference stacks:
    • push through remaining MD01, verify mostly complete and distributed.
    • working through cleanup process for MD01, MD10 chips, warps, WSdiffims. tracking diskspace used for MD runs.
  • Looking into reasons for increased difference image detections in some exposures stalling in publish/ppMops.
  • Czar Thursday, Friday

Bill Sweeney

  • Where did the week go?
  • Found the reason that warp processing gets stuck from time to time. The output of warptool -towarped was formatted in a way that confused pantasks. Changed warptool to be compatible.
  • Investigated cause of streaksremove (and probably ppMops) large memory usage. Determined that the current method of storing fits table internally is extremely inefficient. Started working on a more compact representation.
  • Repaired postage stamp server whose host's disk ran out of space. This was noticed around the globe.
  • Investigated and repaired several transient processing issues.

Chris Waters

  • LAP:
    • Updated planning wiki page (http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/LAP_Science_Scheduling).
    • Fixed warp-stack diff creation, as the diffs were not being queued. This also explains why some exposures were not being published to DVO (without a diff, no magic could be performed).
    • Identified bug in magic's handling of multiple diffs using the same exposure in the same label. Working on a fix.
  • Diskspace: Reran disk usage scripts to get a current snapshot of disk usage. Wrote various accounting scripts to attempt to measure how much disk space any given processing job will consume. With these usage rates understood, we can calculate the expected disk usage based simply on a database query of the number of runs in each state. This appears to work reasonably well and give reasonably believable results.
  • ROC: Continued running repairs. This is somewhat hindered by the disk swaps happening at the same time.
  • Other:
    • Made new mask for OTA66 to remove cells that were causing large numbers of detections.
    • Processed STD exposures for John Tonry, using new constant weighted photometry reduction class.
    • Identified possible source of warp-stage slowness: The returned METADATA was formatted incorrectly, which meant only one label at a time would be processed by pantasks.