Sort of Introduction

Idiosyncratic Definitions

(Let's talk about the same things...)

We have files containing detections (the positives). They have been verified against Sloan. If they were in Sloan, they are true positives or matched. If they are not, they are false positives or unmatched. Detections come with features. Features can be numeric (ra, dec...), alphanumeric (skycell, filter...).

I don't see anything else for the moment.

Related Material

The main wikipages are:

#8: 2011-09-06

Actions

  • A-016 (Serge?): Check if the detections found on flipped images are the flipped detections
  • A-017 (Heather): Ken's plots (see ZoologyKensPlots)
  • A-019 (Roy): Investigate different cuts

#7: 2011-09-01

Attendees: Heather, Roy, Serge

  • A-015: Serge modified the cut-o-matic.py script and implemented scripts allowing to find out "optimal" cuts.
  • A-017: Heather still working on it
  • A-018: Roy can't get the results Peter&Nigel had. He will send questions to them.
  • A-016: Need to talk to Chris about it

Actions

  • A-015 (Serge) Find out "good" cut-off values for true/false detections
  • A-016 (Serge?): Check if the detections found on flipped images are the flipped detections
  • A-017 (Heather): Ken's plots (see ZoologyKensPlots)
  • A-018 (Roy): Rebuild the match experiments.

#6: 2011-08-30

Attendees: Heather, Roy, Serge, Gene, Ken

Actions

  • A-015 (Serge) Find out "good" cut-off values for true/false detections
  • A-016 (Serge?): Check if the detections found on flipped images are the flipped detections
  • A-017 (Heather): Ken's plots (see ZoologyKensPlots)
  • A-018 (Roy): Rebuild the match experiments

Closed Actions

  • A-009 (See details): Done Answer Gene's questions:
    • 2) (Heather) What is the spatial distribution of these detections?
    • 3) (Heather) What does the histogram of pixel value / sqrt(pixel variance) look like in areas / cells with large numbers of these detections? Should be a Gaussian with a width of 1, but if there is excess noise from the camera in some areas, the width will be > 1 (we underestimate the variance). (see http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyInvestigations2)
  • A-012 (Roy): Done Build DS9 region files for "matched" detections with bad flags (see Heather for details, will help answering Gene's questions). Jython tool in /export/ipp001.0/zoology
  • A-013 (?): Comments/Open Questions Done
    • 2) What has been missed from the Sloan detections? See Action A-018
  • A-014 (Serge) Done (1) Check if FITFAIL flag is set only if FITTED flag is set; (2) EXTERNALPOS, CR_LIMIT, SKY_FAILURE, SKYVAR_FAILURE, MOMENTS_SN should always be 0;

Detailed answers can be found there: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyInvestigations#AnswerstoA-014

#5: 2011-08-26

Attendees: Heather, Roy, Serge, Gene, Mark

The purpose of the meeting is to answer these three questions:

  1. How are the detection features built? a) For floating-point values; b) for binary values (flags);
  2. How can the flags be used to tell if a detection is "good" or "bad" or better, (since it implies the definition of "good" or "bad") which hints could be given to users to judge about the detection features quality?
  3. Can we make sensible cuts using x_psf, y_psf, ext_nsigma, psf_inst_flux, psf_inst_flux_sigma or some combination of those (or by using other fields)?
  • The right page to use for the detection features description is http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/CMF_PS1_V3
  • The flags that have to be used to tell that a detection has problems are: for flags: DEFECT, SATURATED, MOMENTS_FAILURE, BIG_RADIUS, AP_MAGS, SIZE_SKIPPED, ON_SPIKE, ON_GHOST, OFF_CHIP; for flags2: ON_SPIKE, ON_BURNTOOL
  • Some flags are correlated or have fixed values: FITFAIL is set only if FITTED is set; EXTERNALPOS, CR_LIMIT, SKY_FAILURE, SKYVAR_FAILURE, MOMENTS_SN should always be 0 (see A-014);
  • For question 3, the answer is yes but values have to be determined to avoid cutting too many detections in the true positives.

Answers:

  • A-013-1) How were the "unmatched" detections defined? Only the astrometry was used to classify true and false positives (e-mails from Gene + Peter Draper + Nigel Metcalf, named "Re: We need advice..." on Thu, Aug 25, 2011 at 5:47 PM)

Actions

  • A-009 (See details): Answer Gene's questions:
    • 2) (Heather) What is the spatial distribution of these detections?
    • 3) (Heather) What does the histogram of pixel value / sqrt(pixel variance) look like in areas / cells with large numbers of these detections? Should be a Gaussian with a width of 1, but if there is excess noise from the camera in some areas, the width will be > 1 (we underestimate the variance). (see http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyInvestigations2)
  • A-012 (Roy): Build DS9 region files for "matched" detections with bad flags (see Heather for details, will help answering Gene's questions)
  • A-013 (?): Comments/Open Questions
    • 2) What has been missed from the Sloan detections?
  • A-014 (Serge) (1) Check if FITFAIL flag is set only if FITTED flag is set; (2) EXTERNALPOS, CR_LIMIT, SKY_FAILURE, SKYVAR_FAILURE, MOMENTS_SN should always be 0;
  • A-015 (?) Find out "good" cut-off values for true/false detections
  • A-016 (Serge?) Check if the detections found on flipped images are the flipped detections

Pending Actions

  • A-008 (Heather):Done Once A-005 is done, check flux_sig/flux ratio for true positive
  • A-009 (See details): Answer Gene's questions:
    • 1) (Serge/Heather for plots) Done What fraction of the detections have flux / flux_sig < 5 (the nominal cutoff) or < 3 (a possible agressive cutoff)?;
  • A-010 (Roy): Done Is the false detections filter dependent? See http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyInvestigations
  • A-011 (Roy/Heather for plots): Done Classify "matched": statistics similar to the unmatched (flags, flags2, flux_sig/flux ratio histogram, anything else) See http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyInvestigations
  • A-013 (?) Comments/Open Questions:
    • 1) Done How were the "unmatched" detections defined?
    • 3) Using the flags we cut about 2/3 of the good detections: Irrelevant now

#4: 2011-08-25

Attendees: Heather, Roy, Serge, Jim

Actions

  • A-008 (Heather): Once A-005 is done, check flux_sig/flux ratio for true positive
  • A-009 (See details): Answer Gene's questions:
    • 1) (Serge/Heather for plots) What fraction of the detections have flux / flux_sig < 5 (the nominal cutoff) or < 3 (a possible agressive cutoff)?;
    • 2) (Serge/Roy/Heather) What is the spatial distribution of these detections?
    • 3) (Who?) What does the histogram of pixel value / sqrt(pixel variance) look like in areas / cells with large numbers of these detections? Should be a Gaussian with a width of 1, but if there is excess noise from the camera in some areas, the width will be > 1 (we underestimate the variance).
  • A-010 (Roy): Is the false detections filter dependent?
  • A-011 (Roy/Heather for plots): Classify "matched": statistics similar to the unmatched (flags, flags2, flux_sig/flux ratio histogram, anything else)
  • A-012 (Roy): Build DS9 region files for "matched" detections with bad flags (see Heather for details, will help answering Gene's questions)
  • A-013 (?): Comments/Open Questions
    • 1) How were the "unmatched" detections defined?
    • 2) What has been missed from the Sloan detections?
    • 3) Using the flags we cut about 2/3 of the good detections

Pending Actions

  • A-004 (Serge): Done Create the database again on ipp001 (neverland is almost full).
  • A-005 (Serge): Done Ingest the full detections on ipp001 (in an another table)
  • A-006 (Roy): Done Update http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyInvestigations with the flux_sig/flux ratio explanation
  • A-007 (Heather): Done Make histograms of flux_sig/flux ratio.

#3: 2011-08-24

Attendees: Heather, Roy, Serge, (Bill invited himself because we were noisy and gave info about processing)

  • Serge created the database (see action A-002)
  • Heather showed that using appropriate flags to cut the "detection candidates" reduced the size of the set by 8 (56000 out of 400000). See action A-003. She reported her discoveries in http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyInvestigations
  • In the 56000 that remain, a lot have a large flux_sig/flux ratio

Pending Actions

  • A-002 (Serge): Done Create a database with: all values taken from the footprint-<filter>-not-matched-psphot.fits files. Add columns filter, stack_id, skycell_id. See /data/ipp009.0/heather/heather.blog.txt .
  • A-003 (All): Done Identify false positives which have been correctly set as "having a problem" (see flags meaning at http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/cmf_columns )

Actions

  • A-004 (Serge): Create the database again on ipp001 (neverland is almost full)
  • A-005 (Serge): Ingest the full detections on ipp001 (in an another table)
  • A-006 (Roy): Update http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyInvestigations with the flux_sig/flux ratio explanation
  • A-007 (Heather): Make histograms of flux_sig/flux ratio.
  • A-008 (Who?): Once A-005 is done, check flux_sig/flux ratio for true positive
  • A-009 (Who?): Answer Gene's questions:
    • 1) What fraction of the detections have flux / flux_sig < 5 (the nominal cutoff) or < 3 (a possible agressive cutoff)?;
    • 2) What is the spatial distribution of these detections?
    • 3) What does the histogram of pixel value / sqrt(pixel variance) look like in areas / cells with large numbers of these detections? Should be a Gaussian with a width of 1, but if there is excess noise from the camera in some areas, the width will be > 1 (we underestimate the variance).
  • A-010 (heather): Is it filter dependent?

#2: 2011-08-23

Attendees: Heather, Roy, Serge

Heather wrote some doc here: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ZoologyHints

Pending Actions

  • A-001 (Heather): Done Provide FITS files where false positives can be displayed and underlined using ds9.

Actions

  • A-002 (Serge): Create a database with: all values taken from the footprint-<filter>-not-matched-psphot.fits files. Add columns filter, stack_id, skycell_id. See /data/ipp009.0/heather/heather.blog.txt .
  • A-003 (All): Identify false positives which have been correctly set as "having a problem" (see flags meaning at http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/cmf_columns )

#1: 2011-08-22

Attendees: Heather, Roy, Serge

Heather talked about the problem. She already has wikipages showing some of them:

Actions

  • A-001 (Heather): Provide FITS files where false positives can be displayed and underlined using ds9.

Previous Actions

  • None

Next meeting: as soon as A-001 has been completed.

Attachments