IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 3 and Version 4 of GrandReprocessingWishlist


Ignore:
Timestamp:
Jan 24, 2011, 8:38:15 AM (15 years ago)
Author:
eugene
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GrandReprocessingWishlist

    v3 v4  
    11== Grand Reprocessing Concerns/wishlist ==
    2 
    32
    43=== burntool ===
    54 * burntool is not complete / done for certain days. 
    65   * Is there a way to reburn these days?  If so, it needs to be documented
     6     * '''This is easy with the new registration / burntool system, but needs to be documented (CZW)'''
    77   * Can we automatically detect & reburn these, rather than have a human find them (as they fail), and requeue by hand? 
     8     * '''finding these days is a fairly easy query; automatically detecting and re-burning seems too ambitious.  (CZW)'''
    89 
    910=== documentation on the reprocessing tasks ===
    … …  
    2526   For 3Pi, it would know to go through warp, stack, ww diff, magic, destreak, dist, publish.
    2627
    27 
    2828=== autoqueuing robot ===
    2929
    … …  
    3939   * M31?
    4040   * robot should prevent queueing of things that have already been queued for the GR 
     41   '''The scope of this is potentially very wide and not well-defined.  Eg, what defines GR for this robot?  How automatic?  There is definitely room for improvement over the shell scripts in stdscience, but something only slightly more sophisticated would be sufficient for our operational needs (and not critical before GR starts)'''
    4142
    4243=== processable images ===
    … …  
    4849   * this can simplify things for future reprocessings (whether GR2 or special
    4950     request reprocessing)
     51   * '''This is a good suggestion, but it has to be done with care.  We should rely on the data quality we measure to exclude an image, and we need to take care in vetting some of those as well'''
    5052 * a suggestion for it:  add a column called "do_not_process", which contains a small int
    5153   * 0 - everything is fine (we assume all images are fine from the beginning)
    5254   * 1 - camera problem (some kind of daytime camera testing, the camera crashed during the exposure, the camera pon time was small, or some other 'camera' problem)
    5355   * 2 - flagged by the operators - (the fwhm is way too fuzzy, or they aborted part of the chunk for some reason. They are pretty good at flagging these in email)
     56     * '''parsing email and interpreting comments is very time-consuming; for most images, we have a measure of the IQ and the zero point, which we can use to reject bad images.  We should not reject an image just because its pair was not taken in a given night'''
    5457   * 3 - flagged by us - (various reasons - NONE should be because IPP can't process.  Maybe part of a chunk is missing or something)
     58     * '''we have to be careful that these reasons are not due to a recoverable software error that has since been addressed'''
    5559 * add a column called "do_not_process_notes" to add words for why the exposure was flagged
     60     * '''to avoid cluttering the big rawImfile table, we should instead define a different table that includes the note and the exp_id, class_id for the affected file'''
    5661 * add -use_do_not_process to chiptool to allow it to queue these. By default chiptool should exclude any that do not have '0' in do_not_process
    5762
    … …  
    6469 * md.stargrasp's list of GRotMDFS (Grand Reprocessing of the Medium Deep Fields) https://md.stargrasp.org/wiki/GRotMDF
    6570   Things to remember for the GRotMDF!
     71
    6672=== Fields to process ===
    6773
    6874   1. Choose an appropriately early starting date. Apr 2009? Earlier?
    69    2. Process M31 is nightly stacks?
     75      * '''April 2009 is the start date'''
     76   2. Process M31 in nightly stacks?
     77      * '''yes'''
    7078
    7179=== Image modifications ===
    7280
    7381   1. Go to 0.258"/pixel. ACTUALLY strongly prefer 0.250" in order to have comfortable integer sizes for arcsec and arcminute areas. Also this will make the beat wavelength a comfortable ~30 pixels in cases where rotations are not present. Keeping the present tesselation will lead to tile sizes of 4800 instead of 6000, file sizes of 5280 instead of 6600.
     82      * '''yes: 0.250'''
    7483   2. Use the ~300 pixel overlap (is IPP going to change the Tess. for 3Pi and other surveys?)
     84      * '''yes: need to define RINGS.V1 with 0.25 and overlap'''
    7585   3. Fix the field centers! Esp. MD07 and MD09! Tesselation field centers must be the OTIS pointing centers.
     86      * '''yes'''
    7687   4. JT's unbelievably clever idea about medians of binned images. Can this conceivably have power for helping out Magic?
     88      * '''FYI: we already produce binned-by-16 images.  Is that OK, or is 8x8 required?'''
    7789
    7890=== Processing modifications ===
    … …  
    90102   2. IPP DB needs some sort of "expired" flag for reprocessed skycells. Do not depend on clever alterations on "data_group" to do the job.
    91103   3. IPP DB needs some sort of flag that indicates whether this is a nightly stack or a reference stack. Do not depend on clever names in the ID field to reveal it.
    92    4. we need a column for the software version/configuration (a real tagged version of the production software) in the db (serge).
     104   4. we need a column for the software version/configuration (a real tagged version of the production software) in the db (serge). '''this exists: eg, chipProcessedImfile.software_ver'''
    93105 
    94106