PSST QUB requirements and notes (draft) --

[[PageOutline]]

== Processing products == 
Like for MOPS, czar needs to make sure all faults are cleared daily for OSS, MSS, ESS, ThreePi products
 * chip-cam-warp -- nightly
 * warp-stack diffims -- nightly
 * datastore of WS diffim catalogs -- nightly
 * PSS and rapid updates for historical stamps (warp+stack+diff) -- 


QUB targeted observations (QUB.nightlyscience, QUB.WS/SS) should only be managed by Mark Huber -- notes on psweb.mp.qub.ac.uk/psat-lv-wiki/index.php/Pan-STARRS_pages


== Processing timescale == 
Ideal and last cutoff times for diffim catalogs to be available for QUB ingestion and processing (TBD) -- 
 * chip-cam-warp -- nightly
 * warp-stack diffims -- nightly 


== Cleanup == 
(not up to date 20180124)
 * chip -- 2 days (same as MOPS)
 * warp -- 4 days (so weekends covered)
 * WSdiff -- 4 days (so weekends covered) -- if disk space issue, can reduce this to 2 days as long as warps uncleaned the update time is minimal
 * datastore WSdiff catalog bundles --  1 month (manually done)
 * ps_ud_QUB -- manual, only Mark Huber should deal with
 * QUB.nightlyscience, QUB.WS/SS -- manual, only Mark Huber should deal with

 
== Tasks to do == 
(not up to date 20180124)
in order of possible accomplishment -- 
 * only really need image+mask (compressed so ~ok) -- like MOPS could switch off weight image for less disk use in processing 
 * gpc2 import of PV3 stack products -- requires new ops tag and then export/import of gpc1 stack rows to gpc2
 * WSdiff daily test set -- 
 * PV3 w-band stacks -- some done at LANL, likely a good amount of reorganization work to be done as well as compute and disk space required -- some exist as of 20160201, have asked QUB if they wish to partially switch over but suspect not with all the other work going on

|| [[Image(ecliptic.rp_skycellplot.php.png,300px)]] || [[Image(LAP.PV3W.20150312_skycellplot.php.png,300px)]] ||


 * gri synthetic w-band stacks for all sky -- much development work, nodes and disk space required.

 * PSS should have option to turn up power for updates when large number of historical stamps are required (method to do so is unclear/undocumented..)
 * cleanup timescales -- 3PI(7d)+OSS(2d)+SNIa(?) are all various times..
 * gpc1 on ipp0012 restored for access to what diffims should be available -- 


 * switch to PV3 stacks for grizy (done 201xxxx) -- requires new ops tag, still some bugs in it -- ~ippqub running modified old tag to use 20141024 since ipp-20170121 broken for WSdiff

 * s/n cut in diffphot -- MOPS will use WSdiff in future at lower cut ~3.5-4, will the be useful for QUB (or at least able to filter to their desired s/n cut?)

 * good_frac limit 0.2 reduce to 0 (remove constraint) to match MOPS WWdiff if possible

 * gpc1 replication to ipp003 (tcs mysql) for QUB+ use -- will be retiring machines soon so need to work in that time frame


== Processing stage time summary ==
Based on 20180122, 20180123 --
 * nightly rates limited by summitcopy, ch-cam-wrp ~60 exp/hr so keeps up with observations (generally can catchup to some degree ~80exp/hr possible?), WSdiff+distribution can match ~60-80 exp/hr without much modification 
 * dtime_script -- Nsources dependent generally, but on average (seconds) | sigma | min | max |
  * ch -- 118.48012954361 |      34.706445650573 |   51.000003814697 |               668 |  
  * cam -- 448.97415129087 |      102.86848443046 |   9.9999856948853 |              1001 |
  * wrp -- 63.084047381327 |      30.766145179551 |   12.999969482422 |               680 |
  * WSdiff -- 102.62499931014 |      31.209872126263 |   56.000019073486 |               672 | 
 * timestamps using dateobs, diffRun.registered, distRun.time_stamp disappointing ~30-50 minutes for exposure through WSdiff to make it to datastore 
{{{
| exp_name    | exp_time | dateobs             | registered          | time_stamp          | chip_id | cam_id  | warp_id | diff_id | dist_id |
+-------------+----------+---------------------+---------------------+---------------------+---------+---------+---------+---------+---------+
| o8141g0062o |      120 | 2018-01-23 05:13:03 | 2018-01-23 05:40:40 | 2018-01-23 06:04:29 | 2048378 | 2014852 | 1991114 | 1651871 | 5171373 | 
| o8141g0187o |       45 | 2018-01-23 07:12:22 | 2018-01-23 07:40:39 | 2018-01-23 08:08:56 | 2048501 | 2014975 | 1991234 | 1651997 | 5171444 | 
| o8141g0249o |       45 | 2018-01-23 08:13:04 | 2018-01-23 08:40:38 | 2018-01-23 09:21:44 | 2048561 | 2015036 | 1991299 | 1652074 | 5171500 |
| o8141g0440o |       45 | 2018-01-23 11:11:03 | 2018-01-23 11:35:39 | 2018-01-23 11:43:22 | 2048720 | 2015195 | 1991452 | 1652210 | 5171550 |  
| o8141g0445o |       45 | 2018-01-23 11:15:53 | 2018-01-23 11:40:39 | 2018-01-23 11:57:05 | 2048725 | 2015200 | 1991459 | 1652216 | 5171556 | 
| o8141g0448o |      120 | 2018-01-23 16:02:14 | 2018-01-23 16:25:41 | 2018-01-23 16:34:55 | 2048727 | 2015202 | 1991463 | 1652218 | 5171558 | 

}}}
  * manually crontab queuing WSdiffs still -- sped up somewhat -- need full night to see
 * pstamp and updates can be limiting factor as well for flagging real transients --  
  * need to adjust warp+WSdiff pixel cleanup to match QUB request window -- suspect 1 week for warp and 3 day for WSdiff would be sufficient (current 2/60hr day warp, 1 day/36hr WSdiff) 
  * splitting out QUB updates to dedicated nodes also useful -- depends on timing of the submissions and nodes to be allocated