Grand Reprocessing Concerns/wishlist

burntool

• burntool is not complete / done for certain days.
  • Is there a way to reburn these days? If so, it needs to be documented
  • Can we automatically detect & reburn these, rather than have a human find them (as they fail), and requeue by hand?

documentation on the reprocessing tasks

• Can we have a well defined and complete document/wiki of what the reprocessing tasks are?
  • the different stages for each survey
  • the different diffs

exposure processing tracking

• with well defined processing, we will know ultimately what should happen to each exposure. We should have a webpage to track each exposure, so that we can verify each exposure goes through all the necessary stages (unlike now, where nightly science tasks like stacks and diffs often get missed)

For example, for a MD exposure, it would know that it needs to go through warp, stack, the different diff types, magic, destreak, and dist. We should be able to tell if various stages have not been started, or if they are faulting (or other problems).

For 3Pi, it would know to go through warp, stack, ww diff, magic, destreak, dist, publish.

autoqueuing robot

• can we have a robot that automatically queues things for us (and adds them to pantasks as necessary?)
  • different robots for different surveys
  • 3pi would queue by part of sky (with a date range)
  • MD would queue by date
  • STS?
  • SS?
  • OSS?
  • M31?
  • robot should prevent queueing of things that have already been queued for the GR

processable images

• We should add a column in rawImfile or rawExp for files which can't be processed, and chiptool should avoid queueing them.
  • some of these files are misfires (the camera died or other camera glitches)
  • some of these are ones that the observers flag as bad in email
  • this can simplify things for future reprocessings (whether GR2 or special request reprocessing)
• a suggestion for it: add a column called "do_not_process", which contains a small int
  • 0 - everything is fine (we assume all images are fine from the beginning)
  • 1 - camera problem (some kind of daytime camera testing, the camera crashed during the exposure, the camera pon time was small, or some other 'camera' problem)
  • 2 - flagged by the operators - (the fwhm is way too fuzzy, or they aborted part of the chunk for some reason. They are pretty good at flagging these in email)
  • 3 - flagged by us - (various reasons - NONE should be because IPP can't process. Maybe part of a chunk is missing or something)
• add a column called "do_not_process_notes" to add words for why the exposure was flagged
• add -use_do_not_process to chiptool to allow it to queue these. By default chiptool should exclude any that do not have '0' in do_not_process

diff photometry

• will there be improved diff photometry? Will we have time to implement it before the GR?

stargrasp's list

• md.stargrasp's list of GRotMDFS (Grand Reprocessing of the Medium Deep Fields) ​https://md.stargrasp.org/wiki/GRotMDF Things to remember for the GRotMDF!

Fields to process

1. Choose an appropriately early starting date. Apr 2009? Earlier?
2. Process M31 is nightly stacks?

Image modifications

1. Go to 0.258"/pixel. ACTUALLY strongly prefer 0.250" in order to have comfortable integer sizes for arcsec and arcminute areas. Also this will make the beat wavelength a comfortable ~30 pixels in cases where rotations are not present. Keeping the present tesselation will lead to tile sizes of 4800 instead of 6000, file sizes of 5280 instead of 6600.
2. Use the ~300 pixel overlap (is IPP going to change the Tess. for 3Pi and other surveys?)
3. Fix the field centers! Esp. MD07 and MD09! Tesselation field centers must be the OTIS pointing centers.
4. JT's unbelievably clever idea about medians of binned images. Can this conceivably have power for helping out Magic?

Processing modifications

1. There are dates that need to be burntooled (7-20-2010 is one of them). (see above)
2. Fringing of everything in z and y (ken) -- we are doing fringing, but in some cases are not applying it correctly. This needs to be fixed, and we need an upper limit for crappy fringe coefficients.
3. Holes around objects from polynominal bias correction. Fix or provide correction image please.
4. Report the flux scale in the header! E.g. e/ADU and what is the ADU level anyway?
5. Fix bitmap bits
6. The .wt.fits files sometimes lack BSCALE and BZERO, e.g. MD07.skycell.018.stk.119069.unconv.wt.fits

IPP DB modifications

1. How about a data_group that uses TJD (of observation) instead of date of processing (which nobody really cares about). Also, how about using "MD00" for "M31"? So data_group would be (for example) MD01.5123.
2. IPP DB needs some sort of "expired" flag for reprocessed skycells. Do not depend on clever alterations on "data_group" to do the job.
3. IPP DB needs some sort of flag that indicates whether this is a nightly stack or a reference stack. Do not depend on clever names in the ID field to reveal it.
4. we need a column for the software version/configuration (a real tagged version of the production software) in the db (serge).