Grand Reprocessing Concerns/wishlist

burntool

• burntool is not complete / done for certain days.
  • Is there a way to reburn these days? If so, it needs to be documented
    • This is easy with the new registration / burntool system, but needs to be documented (CZW)
  • Can we automatically detect & reburn these, rather than have a human find them (as they fail), and requeue by hand?
    • finding these days is a fairly easy query; automatically detecting and re-burning seems too ambitious. (CZW)

documentation on the reprocessing tasks

• Can we have a well defined and complete document/wiki of what the reprocessing tasks are?
  • the different stages for each survey
  • the different diffs

exposure processing tracking

• with well defined processing, we will know ultimately what should happen to each exposure. We should have a webpage to track each exposure, so that we can verify each exposure goes through all the necessary stages (unlike now, where nightly science tasks like stacks and diffs often get missed)

For example, for a MD exposure, it would know that it needs to go through warp, stack, the different diff types, magic, destreak, and dist. We should be able to tell if various stages have not been started, or if they are faulting (or other problems).

For 3Pi, it would know to go through warp, stack, ww diff, magic, destreak, dist, publish.

autoqueuing robot

• can we have a robot that automatically queues things for us (and adds them to pantasks as necessary?)
  • different robots for different surveys
  • 3pi would queue by part of sky (with a date range)
  • MD would queue by date
  • STS?
  • SS?
  • OSS?
  • M31?
  • robot should prevent queueing of things that have already been queued for the GR
  • The scope of this is potentially very wide and not well-defined. Eg, what defines GR for this robot? How automatic? There is definitely room for improvement over the shell scripts in stdscience, but something only slightly more sophisticated would be sufficient for our operational needs (and not critical before GR starts)

processable images

• We should add a column in rawImfile or rawExp for files which can't be processed, and chiptool should avoid queueing them.
  • some of these files are misfires (the camera died or other camera glitches)
  • some of these are ones that the observers flag as bad in email
  • this can simplify things for future reprocessings (whether GR2 or special request reprocessing)
  • This is a good suggestion, but it has to be done with care. We should rely on the data quality we measure to exclude an image, and we need to take care in vetting some of those as well
• a suggestion for it: add a column called "do_not_process", which contains a small int
  • 0 - everything is fine (we assume all images are fine from the beginning)
  • 1 - camera problem (some kind of daytime camera testing, the camera crashed during the exposure, the camera pon time was small, or some other 'camera' problem)
  • 2 - flagged by the operators - (the fwhm is way too fuzzy, or they aborted part of the chunk for some reason. They are pretty good at flagging these in email)
    • parsing email and interpreting comments is very time-consuming; for most images, we have a measure of the IQ and the zero point, which we can use to reject bad images. We should not reject an image just because its pair was not taken in a given night
  • 3 - flagged by us - (various reasons - NONE should be because IPP can't process. Maybe part of a chunk is missing or something)
    • we have to be careful that these reasons are not due to a recoverable software error that has since been addressed
• add a column called "do_not_process_notes" to add words for why the exposure was flagged
  • to avoid cluttering the big rawImfile table, we should instead define a different table that includes the note and the exp_id, class_id for the affected file
• add -use_do_not_process to chiptool to allow it to queue these. By default chiptool should exclude any that do not have '0' in do_not_process

diff photometry

• will there be improved diff photometry? Will we have time to implement it before the GR?

stargrasp's list

• md.stargrasp's list of GRotMDFS (Grand Reprocessing of the Medium Deep Fields) ​https://md.stargrasp.org/wiki/GRotMDF Things to remember for the GRotMDF!

Fields to process

1. Choose an appropriately early starting date. Apr 2009? Earlier?
   • April 2009 is the start date
2. Process M31 in nightly stacks?
   • yes

Image modifications

1. Go to 0.258"/pixel. ACTUALLY strongly prefer 0.250" in order to have comfortable integer sizes for arcsec and arcminute areas. Also this will make the beat wavelength a comfortable ~30 pixels in cases where rotations are not present. Keeping the present tesselation will lead to tile sizes of 4800 instead of 6000, file sizes of 5280 instead of 6600.
   • yes: 0.250
2. Use the ~300 pixel overlap (is IPP going to change the Tess. for 3Pi and other surveys?)
   • yes: need to define RINGS.V1 with 0.25 and overlap
3. Fix the field centers! Esp. MD07 and MD09! Tesselation field centers must be the OTIS pointing centers.
   • yes
4. JT's unbelievably clever idea about medians of binned images. Can this conceivably have power for helping out Magic?
   • FYI: we already produce binned-by-16 images. Is that OK, or is 8x8 required?

Processing modifications

1. There are dates that need to be burntooled (7-20-2010 is one of them). (see above)
2. Fringing of everything in z and y (ken) -- we are doing fringing, but in some cases are not applying it correctly. This needs to be fixed, and we need an upper limit for crappy fringe coefficients.
3. Holes around objects from polynominal bias correction. Fix or provide correction image please.
4. Report the flux scale in the header! E.g. e/ADU and what is the ADU level anyway?
5. Fix bitmap bits
6. The .wt.fits files sometimes lack BSCALE and BZERO, e.g. MD07.skycell.018.stk.119069.unconv.wt.fits

IPP DB modifications

1. How about a data_group that uses TJD (of observation) instead of date of processing (which nobody really cares about). Also, how about using "MD00" for "M31"? So data_group would be (for example) MD01.5123.
2. IPP DB needs some sort of "expired" flag for reprocessed skycells. Do not depend on clever alterations on "data_group" to do the job.
3. IPP DB needs some sort of flag that indicates whether this is a nightly stack or a reference stack. Do not depend on clever names in the ID field to reveal it.
4. we need a column for the software version/configuration (a real tagged version of the production software) in the db (serge).
   • this exists: eg, chipProcessedImfile.software_ver