PSNSC_NCU requirements and notes (draft) --

NCU observations -- processing needs to be finished by early morning for MOPS to use for NEO

• obs_mode NCU -- new, possibly start Oct. 25 2016 HST (not start after 11/1 HST?)
  • 4 chunks (returns) -- NCU1B...NCU4B; NCU1A...NCU4A -- see night sequence below
    • B=color set, sequence wgwrwiwz per chunk (8 targets each)
    • A=w-band only, wwwwwwww (8 targets each)
  • 8 targets (fields) -- NCU1.1...NCU1.8; NCU2.1...NCU2.8; NCU3.1...NCU3.8 ; NCU4.1...NCU4.8 (chunk#.target#)
    • 4 above, 4 below ecliptic centered around opposition -- all fields non-overlapping
  • no dither in a night (no rotation like MD/QUB/FSS) -- offset done each night mean motion retrograde (W along ecliptic, ~0.3 degrees/night)
  • exposures per field --
    • griz --
      • A: 1/chunk * 4 chunks = 4
    • w --
      • B: 4/chunk * 4 chunks = 16
      • A: 8/chunk * 4 chunks = 32
  • 4 sequential nights, ~6 hr/night
  • exptimes wgriz: 60,120,120,120,180s
  • run queue sequence then
    • night 1: NCU1B, NCU1B, NCU1B, NCU1B -- only color queue -- return to exact position (no dither)
    • night 2: NCU2A, NCU2A, NCU2A, NCU2A -- only w queue -- return each time to position (no dither), but has target/field center motion shift from previous night
    • night 3: NCU3A, NCU3A, NCU3A, NCU3A -- only w queue -- return each time to position (no dither), but has target/field center motion shift from previous night
    • night 4: NCU4A, NCU4A, NCU4A, NCU4A -- only w queue -- return each time to position (no dither), but has target/field center motion shift from previous night
  • chunk name (QUEUEID keyword in headers) -- added filter with vitit# here
  • target name (OBJECT keyword in headers) -- NCU(chunk#.target#) -- may be enough to trigger for WW per chunk (8 visits) rather than all night (32 visits), may also trigger .multi case too over the multiple nights...
  • comment string -- needs to be descriptive and easily parsed, need to check with MOPS for WS grouping/parsing issues (spaces) --
    • B: old format: NCU1B w4 NCU1.1 -- (chunk) (filter)(visit# for w only) NCU(chunk#.target#) -- want to add "visit #" at end of comment (1-4), move filter to chunk name
    • B: new format: (chunk).(filter) NCU(chunk#.target#) visit (visit # in chunk) -- adding visit# at end not easily done in B chunk so put on chunk name instead
    • B: revised new format: (chunk).(filter).(visit# in chunk) NCU(chunk#.target#)
    • NCU1B only night 1 --

      NCU1B.w.1 NCU1.1
      NCU1B.w.1 NCU1.2
      ...
      NCU1B.w.1 NCU1.8
      NCU1B.g.1 NCU1.1
      NCU1B.g.1 NCU1.2
      ...
      NCU1B.g.1 NCU1.8
      NCU1B.w.2 NCU1.1
      NCU1B.w.2 NCU1.2
      ...
      NCU1B.w.2 NCU1.8
      NCU1B.r.1 NCU1.1
      ...
      NCU1B.r.1 NCU1.8
      NCU1B.w.3 NCU1.1
      ...
      NCU1B.w.3 NCU1.8
      NCU1B.i.1 NCU1.1
      ...
      NCU1B.i.1 NCU1.8
      NCU1B.w.4 NCU1.1
      ...
      NCU1B.w.4 NCU1.8
      NCU1B.z.1 NCU1.8
      ...
      NCU1B.z.1 NCU1.8
      
      

    • A: old format: NCU1A NCU1.1 -- (chunk) NCU(chunk#.target#) -- add "w" so similar format for B and "visit #" at end of comment (1-8), move filter to chunk name
    • A: new format make similar: (chunk).(filter) NCU(chunk#.target#) visit (visit # in chunk) visit (visit#) possible using otis repeat(?) but not for B so don't use
    • A: revised format also make similar: (chunk).(filter).(visit# in chunk) NCU(chunk#.target#)

      Night 2:
      
      NCU2A.w.1 NCU1.1
      NCU2A.w.1 NCU1.2
      NCU2A.w.1 NCU1.3
      ...
      NCU2A.w.2 NCU1.1
      NCU2A.w.2 NCU1.2
      ...
      NCU2A.w.8 NCU1.1
      ...
      NCU2A.w.8 NCU1.8
      
      Night 3:
      
      NCU3A.w.1 NCU3.1
      NCU3A.w.1 NCU3.2
      ...
      NCU3A.w.8 NCU3.8
      
      Night 4:
      
      NCU4A.w.1 NCU4.1
      NCU4A.w.1 NCU4.2
      ...
      NCU4A.w.8 NCU4.8
      
      

• Keyhole problem -- ecliptic nearly overhead for opposition (~same next lunation on other side), then opposition near plane of galaxy, then similar position on other side of galaxy, so maybe 5 months if not done in October
  • Night 1: (B chunk is color set) --

    do one whole B chunk, 
    then half a B chunk (which will be w,g,w,r), 
    then do something else for about 75 minutes until it comes out of the keyhole,
    and do the rest of that chunk (which will be w,i,w,z),
    then do the final whole B chunk.
    

  • Nights 2-4: (A chunk all w) -- do A twice, then intermission for the keyhole, then do it twice again

• chip->warp normal stdscience pantasks as with xSS products (ipp-20141024 tag)
• WSdiff runs in ~ippmops/stdscience_ws with modified ipp-20141024 ops tag to make uncompressed WSdiffs (like FSS, PSNSC_MOPS)
  • larger volume, may need more nodes -- must be uncompressed for MOPS stamps since large data volume
  • default detection @5-sigma while WWdiff is <5-sigma (3.5) and may need to be adjusted -- default plan will be at normal 5-sigma level unless tuning agreed upon to a lower level
    • test chunk should be evaluated in more detail -- see below
    • MD not a great equal comparison since exptimes were longer (but was also 5-sigma)
    • [sc] Not sure if the catalog size is not going to explode if threshold too small. WWdiff is at 2.5-sigma(not sure?). What was the value for WSdiffs e.g. MDs? We could use the values (in each rgiz) unless something has changed. Not sure about what's used for w-band.

• WSdiff published to IPP-MOPS-WS -- WWdiff possibly later to regular datastore IPP-MOPS-TEST
  • normal OSS warp-warp diffims will go to IPP-MOPS-TEST during the night
  • [sc] We will likely request WWdiff to be published to the regular datastore (IPP-MOPS-TEST) later (manual queuing/processing at the IPP level OK for MOPS)

Processing products -- Mark taking responsibility for

• chip-cam-warp -- NCU.nightlyscience -- nightly
• warp-stack diffims -- NCU.WS.nightlyscience -- nightly
• publishing of WSdiffs -- IPP-MOPS-WS -- nightly
• manually done --
  • nightly stack -- NCU.nightlyscience -- nightly, manually done to avoid any poor images
  • stack-stack diffims -- NCU.SS.nightlyscience -- nightly, likely automatic possible once manual stacks made
  • warp-warp diffims -- TBD -- NCU.nightlyscience
• stamps -- TBD

Cleanup --

• chip -- likely normal 1 day retention
• warp -- indefinite hold
• WSdiff -- indefinite hold -- uncompressed so will need to keep eye on and cleanup once MOPS is finished
• publishing WSdiff catalog bundles -- TBD/auto?
• pstamp -- manual cleanup like other ps_ud_% -- TBD

Tasks to do -- priority order now

• verify w-band stacks okay along ecliptic for opposition end October --
  • new reference stacks to be made for the expected area (mix of reprocessing+updates to minimize data load) -- NCU.ref.20161026 -- (add more details)
• WSdiff detection threshold -- 5-sigma default but may need to lower
  • preliminary check of known sources shows default 5-sigma should recover ones with WWdiff threshold ~3.5 in MOPS test chunk --
• MOPS interested in SSdiff -- published where?
• check processing rate/nodes for WSdiffs -- during the night of observations -- likely the normal 3x c2 group is sufficient but will have some nodes available to boost as needed (more c2 and/or m0+1, x3?) -- only x2+3 available