Summary

MOPS would like a refstack for the STS fields in order to get warp-stack (WS) diffims. The previous STS.refstack.20110613,20110615 wasn't usable as it was destroyed by magic. STS.20120510 was made using a similar number of inputs from the best exposures from 2011. Not pretty but hopefully sufficient.

Status:

  • 5/8: past refstack from 6/2011 are cleaned, magic destroyed them. no magic problem anymore..
  • 5/10: test samples and selected exposures processed for chip->warp
  • 5/13: test samples for refstack finished, full refstack started
  • 5/15: refstack finished and checked. looking at diffim samples and how to publish detections to MOPS
  • 5/18: test diffim sample 1 using updated warps from past nights in 2011, early 2012
  • 5/20: test diffims finished but fault in publishing, tracing out.
  • 5/27: test sample 1 a mess, restart with test sample 2
  • 5/29: test sample 2 diffim detections published to IPP-MOPS-TEST-2, waiting for feedback if will work

Things to do/fix/test:

  • convolved versus unconvolved stack WS diffims
  • target PSF limits in convolved stacks - rerun with selected target PSF?
  • skycell overlap in refstack and diffims - want to restrict inputs PP1 into stack skycell.1.xx only etc
    • can remake with clean/updates for each field, but maybe MOPS won't care or maybe with conv stacks isn't a big issue?
    • tricky to do in an automated way. giving each field a different refstack processing label is easiest and do on update? will still try to populate other regions though, so need to restrict to particular region with -skycell_id. won't need a sub-label for each 7 to prevent the rerun headache and to only run against proper field since matched by skycell in diffim.
  • will -bothways set cause problems, why was not able to rerun?
  • need MOPS feedback on detections, usefulness, problems

STS observation and tessellation summary

Sample rawExp.comment information for the type of observation

STS.180.ND PP1 center
STS.000.GIQ PP1 dither 1 
STS.180.NDGIQ PP5 center
STS.180.NOM PP3 dither 3
  • 000,090,180,270 -- rotator to keep rough PA on field over season
  • GIQ -- shorter exp_time (20s versus 30s nominal) to not saturate more sources in better seeing conditions
  • ND -- no dither so always with center and typically with 30s, occasional NDGIQ
  • NOM -- nominal IQ?
  • PP# -- 1-7, 7 pointings
  • dither 1-4 -- dither shifts (by ?) and typically with GIQ

Overlay of camera SMFs for PP1-PP7 (colored) on top of the projection skycells defined at http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_Tessellations.

7 projections and their skycell overlap (STS.V4, 0.25"/pixel platescale)

Approximate pattern, E left, N up:

 1 (orange)  2 (red)     
3 (blue)  4 (green)  5 (violet)   
 6 (indigo)  7 (yellow)

To generate SMF sample for position and look at dither pattern:

mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select chipRun.workdir,rawExp.exp_tag,rawExp.exp_name from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and  camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%center%" group by rawExp.exp_id;' | grep -v "workdir" | uniq -f2 | awk '{ print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -p " $1"/"$2"/o%smf" }' | tcsh | awk '{print "imbox "$1}' >& STSsub.1_i_imboxsmf_c.lst

mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select chipRun.workdir,rawExp.exp_tag,rawExp.exp_name from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and  camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%dither 1%" group by rawExp.exp_id;' | grep -v "workdir" | uniq -f2 | awk '{ print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -p " $1"/"$2"/o%smf" }' | tcsh | awk '{print "imbox "$1}' >& STSsub.1_i_imboxsmf_d1.lst

...

dvo -D CATDIR STS.V4 

$KAPA = kapa -noX

region 297.5 17.1 4.2  
style -c black
image
style -c orange 
input STSsub.1_i_imboxsmf.lst

style -c red
input STSsub.2_i_imboxsmf.lst

...

There is significant overlap of projections (and effectively skycells). May cause issue with general input into a refstack skycell with wildly different IQ?

  • some will have direct overlapping pixels into the skycell, for example PP4 can provide chips to make skycell.1.03 that could then end up in a stack for that skycell and vice versa.
  • these could fill in different areas of the skycell with very different PSF functions, but the skycells are also populated by different chips and IQ from rotations as well.
  • will probably want a reasonable input sample, but suspect target PSF will behave oddly for convolved stack
  • using the processed warps, skycell.1.03 looks like
    select exp_name,dateobs,warpRun.warp_id,warpSkyCellMap.skycell_id,warpSkyCellMap.class_id,comment from warpRun join warpSkyCellMap using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where warpRun.data_group like "STS.refstack.2012%" and skycell_id like "%.1.03%" order by exp_name;
    
    | exp_name    | dateobs             | warp_id | skycell_id   | class_id | comment                  |
    +-------------+---------------------+---------+--------------+----------+--------------------------+
    | o5705g0379o | 2011-05-24 12:08:55 |  410461 | skycell.1.03 | XY67     | STS.270.GIQ PP3 dither 1 | 
    | o5705g0379o | 2011-05-24 12:08:55 |  410461 | skycell.1.03 | XY57     | STS.270.GIQ PP3 dither 1 | 
    | o5705g0386o | 2011-05-24 12:13:00 |  410467 | skycell.1.03 | XY57     | STS.270.GIQ PP3 dither 2 | 
    | o5705g0386o | 2011-05-24 12:13:00 |  410467 | skycell.1.03 | XY67     | STS.270.GIQ PP3 dither 2 | 
    | o5705g0387o | 2011-05-24 12:13:35 |  410475 | skycell.1.03 | XY71     | STS.270.GIQ PP4 dither 3 | 
    | o5705g0387o | 2011-05-24 12:13:35 |  410475 | skycell.1.03 | XY72     | STS.270.GIQ PP4 dither 3 | 
    | o5705g0393o | 2011-05-24 12:17:04 |  410478 | skycell.1.03 | XY67     | STS.270.GIQ PP3 dither 3 | 
    | o5705g0393o | 2011-05-24 12:17:04 |  410478 | skycell.1.03 | XY57     | STS.270.GIQ PP3 dither 3 | 
    | o5705g0394o | 2011-05-24 12:17:38 |  410469 | skycell.1.03 | XY72     | STS.270.GIQ PP4 dither 4 | 
    | o5705g0394o | 2011-05-24 12:17:38 |  410469 | skycell.1.03 | XY71     | STS.270.GIQ PP4 dither 4 | 
    | o5705g0395o | 2011-05-24 12:18:14 |  409630 | skycell.1.03 | XY02     | STS.270.GIQ PP1 dither 4 | 
    | o5705g0395o | 2011-05-24 12:18:14 |  409630 | skycell.1.03 | XY03     | STS.270.GIQ PP1 dither 4 | 
    | o5705g0395o | 2011-05-24 12:18:14 |  409630 | skycell.1.03 | XY04     | STS.270.GIQ PP1 dither 4 | 
    
    | o5719g0324o | 2011-06-07 11:27:59 |  410516 | skycell.1.03 | XY10     | STS.180.GIQ PP4 dither 1 | 
    | o5719g0324o | 2011-06-07 11:27:59 |  410516 | skycell.1.03 | XY20     | STS.180.GIQ PP4 dither 1 | 
    | o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY36     | STS.180.GIQ PP1 dither 1 | 
    | o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY37     | STS.180.GIQ PP1 dither 1 | 
    | o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY46     | STS.180.GIQ PP1 dither 1 | 
    | o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY47     | STS.180.GIQ PP1 dither 1 | 
    | o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY26     | STS.180.GIQ PP1 dither 1 | 
    | o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY27     | STS.180.GIQ PP1 dither 1 | 
    | o5719g0330o | 2011-06-07 11:31:21 |  410528 | skycell.1.03 | XY71     | STS.180.GIQ PP3 dither 1 | 
    | o5719g0330o | 2011-06-07 11:31:21 |  410528 | skycell.1.03 | XY72     | STS.180.GIQ PP3 dither 1 | 
    
    | o5767g0396o | 2011-07-25 11:59:08 |  410618 | skycell.1.03 | XY57     | STS.000.ND PP4 center    | 
    | o5767g0396o | 2011-07-25 11:59:08 |  410618 | skycell.1.03 | XY67     | STS.000.ND PP4 center    | 
    | o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY50     | STS.000.ND PP1 center    | 
    | o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY51     | STS.000.ND PP1 center    | 
    | o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY30     | STS.000.ND PP1 center    | 
    | o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY31     | STS.000.ND PP1 center    | 
    | o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY40     | STS.000.ND PP1 center    | 
    | o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY41     | STS.000.ND PP1 center    | 
    | o5767g0402o | 2011-07-25 12:03:25 |  410614 | skycell.1.03 | XY05     | STS.000.ND PP3 center    | 
    | o5767g0402o | 2011-07-25 12:03:25 |  410614 | skycell.1.03 | XY06     | STS.000.ND PP3 center    | 
    
    

Will also be directly overlapping skycells for diffims with at least slightly different refstacks:

skycell.<PP>.xx possible overlap skycell.<PP>.xx
1.03 4.71,4.72; 3.57, 3.67

Refstack

Reference stack using data from 2011 only, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, large elongation.

Chip->Warp processing

Only reprocessing a selected sample to use in the refstack only --

Test sample:

chiptool -definebyquery -set_label STS.mehtest -set_workdir neb://@HOST@.0/meh/STS.mehtest -set_data_group STS.mehtest -set_dist_group NULL -set_tess_id STS.V4 -set_end_stage warp -dbname gpc1 -dateobs_begin 2011-05-24 -dateobs_end 2011-05-25 -filter i% -comment STS% -obs_mode STS -simple -pretend >& teststschip.log

--> 5/24 has 84 exposures, so need more restrictive for test -- do o5705g0373o

chiptool -definebyquery -set_label STS.mehtest -set_workdir neb://@HOST@.0/meh/STS.mehtest -set_data_group STS.mehtest -set_dist_group NULL -set_tess_id STS.V4 -set_end_stage warp -set_reduction STS_DATASET -dbname gpc1 -exp_name o5705g0373o -simple -pretend >& teststschip_o5705g0373o.log

-- test run - STS.mehtest - don't see corner unglow in camera image, dynamic range such that not see strongly?
neb://@HOST@.0/meh/STS.mehtest/o5705g0373o.340508

--> see some patterns in warp background for sample -- significant? see
meh/STS.mehtest/o5705g0373o.340508/o5705g0373o.340508.wrp.408756.skycell.1.04.fits

Determine cuts for exposure sample to reprocess, pick of order 50 exposure sample:

  • time range
    | o5666g0560o | 324290 | STS      | STS      |       30 | 2011-04-15 14:01:18 | i.00000 | STS PP6 test visit 2     | 
    | o5666g0561o | 324291 | STS      | STS      |       30 | 2011-04-15 14:02:10 | i.00000 | STS PP7 test visit 2     | 
    | o5667g0517o | 324909 | RINGS.V3 | STS      |       30 | 2011-04-16 14:39:15 | i.00000 | STS PP8 test visit 1     | 
    | o5667g0518o | 324910 | RINGS.V3 | STS      |       30 | 2011-04-16 14:39:58 | i.00000 | STS PP9 test visit 1     | 
    ..
    | o5681g0558o | 332242 | RINGS.V3 | STS      |       30 | 2011-04-30 15:02:10 | i.00000 | STS PP6 test visit 2     | 
    | o5681g0559o | 332243 | RINGS.V3 | STS      |       30 | 2011-04-30 15:03:32 | i.00000 | STS PP3 test visit 2     | 
    | c5681g0007d | 332244 | NULL     | STS      |       30 | 2011-04-30 15:05:39 | i.00000 | STS PP3 test visit 2     | 
    | o5695g0143o | 335717 | NULL     | STS      |       30 | 2011-05-14 11:06:19 | i.00000 | STS.180.NOM PP4 center   | 
    ..
    | o5695g0144o | 335718 | NULL     | STS      |       30 | 2011-05-14 11:07:04 | i.00000 | STS.180.NOM PP1 center   | 
    | o5695g0225o | 335799 | NULL     | STS      |       30 | 2011-05-14 12:06:25 | i.00000 | STS.180.NOM PP6 dither 5 | 
    | o5695g0226o | 335800 | NULL     | STS      |       30 | 2011-05-14 12:07:09 | i.00000 | STS.180.NOM PP3 dither 5 | 
    | o5705g0373o | 340508 | STS.V4   | STS      |       20 | 2011-05-24 12:05:27 | i.00000 | STS.270.GIQ PP4 dither 1 |  -- talking with Bill, looks like when to start -- 
    | o5705g0374o | 340510 | STS.V4   | STS      |       20 | 2011-05-24 12:06:00 | i.00000 | STS.270.GIQ PP1 dither 1 | 
    
  • camera FWHM, ZPT, background cut
  • dither content check
    SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE obs_mode="STS" and dateobs>"2011-05-23" and dateobs<"2012-01" and  fwhm_major<3.8 and zpt_obs>24.6 and camProcessedExp.bg<100.0 and comment like "%PP1%" group by exp_id;
    ...
    
  • summary
    PP1 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 42, ~half center half dither (16,7,5,6,8)
    PP2 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 47, ~half center half dither
    PP3 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 53, ~half center half dither 
    PP4 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 50, ~half center half dither 
    PP5 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 52, ~half center half dither 
    PP6 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 49, ~half center half dither (14,8,8,10,9)
    PP7 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 49, ~half center half dither (18,9,5,10,7)
    

Run PP1 set first with selected exposures and perl script to make reftest stacks:

mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select rawExp.exp_id from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and  camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%" group by rawExp.exp_id;' | grep -v "exp_id" 

perl sts.v4.refstackexp.20120510_pp1.chip > run_pp1.chip.bat
source run_pp1.chip.bat >& run_pp1.chip.log

Once reftest check out, run remaining PP2-7 exposures:

mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select rawExp.exp_id from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and  camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment not like "%PP1%" group by rawExp.exp_id;' | grep -v "exp_id" 

perl sts.v4.refstackexp.20120510_pp2-7.chip > run_pp2-7.chip.bat
source run_pp2-7.chip.bat >& run_pp2-7.chip.log

--> set priority with lap @200
labeltool -definelabel -set_label STS.refstack.20120510 -set_priority 200 -dbname gpc1
labeltool -updatelabel -label STS.refstack.20120510 -set_priority 190 -dbname gpc1

Stack processing

  • label: STS.refstack.20120510
  • data_group: STS.refstack.20120510
  • workdir: neb://@HOST@.0/gpc1/STS.reftest.20120510
  • reduction: PS1_REFERENCE
        OUTPUT.NOCOMP       BOOL    TRUE
        OUTPUT.LOGFLUX      BOOL    FALSE
        OUTPUT.REPLICATE    BOOL    TRUE
    
    --> compared to other reduction recipes -- cut on input PSF in STACK_DEEP may be useful but can apply in selecting exposures for stacking (not 100% effective however).
    STACK_THREEPI  METADATA
        PSF.INPUT.MAX   F32     12.0
        OUTPUT.NOCOMP   BOOL    FALSE
        OUTPUT.LOGFLUX  BOOL    TRUE
    STACK_DEEP    METADATA
        OUTPUT.NOCOMP   BOOL    TRUE
        OUTPUT.LOGFLUX  BOOL    FALSE
        PSF.INPUT.MAX           F32  7.5
        PSF.INPUT.CLIP.NSIGMA   F32  5.0
    
    

Test stack (reftest) sample like made for MD skycell.055 with different number of input warps and FWHM cut using reprocessed sample (background,ZPT cut)

  • label: STS.reftest.20120510
  • data_group: STS.reftest.20120510.c#f<FWHM cut> where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp
    • c0 additional FWHM cut only
    • c1 additional FWHM cut with elong<1.2
  • skycell.1.03 chosen being on outer edge without large masking fraction or chunks and for comparison in possible overlap tests with other projection (PP4)
    • overlap skycell would be useful to see how much it becomes degraded when added both in overlap and in different spatial regions. for now that would need a corner.
    • in projection 1, the corner 1.01 is like 1.66. look at maskfrac and also Ndetections on the stack
      select warpRun.warp_id,warpSkyfile.skycell_id,warpSkyfile.maskfrac_npix,warpSkyfile.maskfrac_static,warpSkyfile.maskfrac_dynamic,warpSkyfile.maskfrac_advisory,warpSkyfile.good_frac from warpRun join warpSkyfile using (warp_id) where label like "STS.refstack.20120510" and skycell_id like "%1.%" limit 10;
      
      | warp_id | skycell_id   | maskfrac_npix | maskfrac_static | maskfrac_dynamic | maskfrac_advisory | good_frac |
      +---------+--------------+---------------+-----------------+------------------+-------------------+-----------+
      |  409624 | skycell.1.00 |      37519776 |         0.07579 |            8e-06 |          0.006282 |  0.044258 | 
      |  409624 | skycell.1.01 |      37519776 |         0.27474 |          6.3e-05 |          0.094433 |   0.46227 | 
      |  409624 | skycell.1.02 |      37519776 |        0.134148 |          8.4e-05 |          0.129784 |  0.571464 | 
      |  409624 | skycell.1.03 |      37519776 |        0.067499 |          9.8e-05 |          0.144948 |  0.631332 | 
      |  409624 | skycell.1.04 |      37519776 |        0.101891 |         0.000159 |          0.140317 |  0.595121 | 
      |  409624 | skycell.1.05 |      37519776 |        0.140644 |         0.000259 |          0.158277 |  0.565971 | 
      |  409624 | skycell.1.06 |      37519776 |        0.284199 |         0.000173 |          0.115242 |  0.450413 | 
      |  409624 | skycell.1.07 |      37519776 |        0.076937 |          1.3e-05 |          0.010954 |  0.043497 | 
      |  409624 | skycell.1.08 |      37519776 |        0.336794 |            8e-05 |          0.087179 |  0.445569 | 
      |  409624 | skycell.1.09 |      37519776 |        0.370852 |         0.000146 |          0.119304 |  0.519711 | 
      
      ---> want to avoid the corner then and do bottom edge 1.04,1.05 where good_frac ~50%, 1.05 has larger static mask to watch in rotations. 1.03 also should work and seems more stable static and good maskfrac.
      
  • initial test sample:
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c1f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.03  >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell103.log
    
    ...
    
    --> sample unconvolved images -- doesn't seem to be fully filling in regions/gaps, see input details.
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899917.unconv.num.fits -- c1f39 (max inputs 36)
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.num.fits -- c1f38 (max inputs 23)
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.num.fits -- c1f37 (max inputs 12)
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899920.unconv.num.fits -- c0f45 (max inputs 44)
    
    --> input stats for likely FWHM cut, 
    SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.reftest.20120510.c1f38" group by exp_id;
    
    | exp_id | exp_time | dateobs             | zpt_obs | zpt_stdev | fwhm_major | elong | iq_elong | fwhm_iq_maj | bg     | bg_stdev | uq   | lq   | n_stars | comment                  |
    +--------+----------+---------------------+---------+-----------+------------+-------+----------+-------------+--------+----------+------+------+---------+--------------------------+
    | 340540 |       20 | 2011-05-24 12:22:25 | 24.5549 |    0.0259 |    3.69691 | 1.188 | 1.083    | 0.92        | 91.27  | 14.01    | 3.98 | 3.46 | 8648132 | STS.270.GIQ PP1 dither 1 | 
    | 340558 |       20 | 2011-05-24 12:34:54 | 24.5559 |  0.026682 |     3.7342 | 1.148 | 1.073    | 0.86        | 88.49  | 14.04    | 4.04 | 3.45 | 8393568 | STS.270.GIQ PP1 dither 4 | 
    | 340567 |       20 | 2011-05-24 12:39:02 | 24.5563 |  0.025387 |    3.62252 | 1.130 | 1.059    | 0.84        | 88.18  | 13.78    | 3.89 | 3.29 | 8588336 | STS.270.GIQ PP1 dither 1 | 
    | 340573 |       20 | 2011-05-24 12:43:16 | 24.5591 |  0.025946 |    3.64969 | 1.127 | 1.050    | 0.82        | 87.11  | 13.21    | 4.05 | 3.38 | 8380448 | STS.270.GIQ PP1 dither 2 | 
    | 344556 |       20 | 2011-06-07 11:28:32 | 24.4608 |  0.029333 |    3.59708 | 1.165 | 1.079    | 0.88        | 100.50 | 13.30    | 4.04 | 3.39 | 8487768 | STS.180.GIQ PP1 dither 1 | 
    | 344562 |       20 | 2011-06-07 11:32:30 | 24.5164 |  0.024937 |    3.59372 | 1.150 | 1.069    | 0.87        | 97.07  | 13.32    | 3.99 | 3.37 | 8638928 | STS.180.GIQ PP1 dither 2 | 
    | 344569 |       20 | 2011-06-07 11:36:21 | 24.5313 |  0.024183 |    3.69415 | 1.168 | 1.085    | 0.93        | 94.56  | 13.63    | 4.03 | 3.47 | 8641948 | STS.180.GIQ PP1 dither 3 | 
    | 344575 |       20 | 2011-06-07 11:40:18 | 24.5333 |  0.027669 |    3.66644 | 1.154 | 1.070    | 0.91        | 94.17  | 13.25    | 3.99 | 3.39 | 8611680 | STS.180.GIQ PP1 dither 4 | 
    | 344583 |       20 | 2011-06-07 11:44:12 | 24.4983 |  0.040863 |    3.72526 | 1.159 | 1.088    | 0.92        | 96.11  | 13.24    | 4.14 | 3.51 | 8467340 | STS.180.GIQ PP1 dither 1 | 
    | 344589 |       20 | 2011-06-07 11:48:13 | 24.4937 |  0.046644 |    3.72833 | 1.136 | 1.068    | 0.90        | 95.45  | 13.70    | 4.02 | 3.51 | 8327116 | STS.180.GIQ PP1 dither 2 | 
    | 344597 |       20 | 2011-06-07 11:52:11 | 24.4935 |  0.031675 |    3.71614 | 1.141 | 1.066    | 0.87        | 92.70  | 13.54    | 4.13 | 3.51 | 8277828 | STS.180.GIQ PP1 dither 3 | 
    | 344605 |       20 | 2011-06-07 11:56:13 | 24.5409 |  0.025897 |    3.74193 | 1.160 | 1.063    | 0.90        | 89.26  | 13.15    | 4.15 | 3.54 | 8427112 | STS.180.GIQ PP1 dither 4 | 
    | 367982 |       30 | 2011-07-25 11:59:50 | 24.5698 |   0.02507 |    3.79495 | 1.091 | 1.052    | 0.79        | 97.48  | 17.60    | 4.03 | 3.65 | 8991280 | STS.000.ND PP1 center    | 
    | 367996 |       30 | 2011-07-25 12:09:52 | 24.5652 |  0.026531 |     3.6958 | 1.104 | 1.064    | 0.79        | 97.83  | 19.54    | 4.02 | 3.49 | 9311700 | STS.000.ND PP1 center    | 
    | 368010 |       30 | 2011-07-25 12:19:54 | 24.5644 |  0.026506 |    3.74343 | 1.089 | 1.055    | 0.77        | 102.98 | 19.22    | 3.99 | 3.60 | 9108432 | STS.000.ND PP1 center    | 
    | 368833 |       30 | 2011-07-27 07:23:06 | 24.5521 |  0.026382 |    3.75974 | 1.153 | 1.061    | 0.93        | 114.75 | 17.14    | 4.16 | 3.49 | 9435552 | STS.180.ND PP1 center    | 
    | 393751 |       20 | 2011-09-18 08:24:25 | 24.5337 |  0.029066 |    3.54239 | 1.153 | 1.071    | 0.91        | 89.00  | 12.90    | 3.89 | 3.31 | 9090136 | STS.000.GIQ PP1 dither 1 | 
    | 393763 |       20 | 2011-09-18 08:32:30 |  24.533 |   0.02565 |    3.60961 | 1.170 | 1.071    | 0.96        | 88.74  | 12.99    | 3.88 | 3.33 | 9028068 | STS.000.GIQ PP1 dither 3 | 
    | 393778 |       20 | 2011-09-18 08:40:35 | 24.5299 |  0.027023 |    3.55016 | 1.177 | 1.072    | 0.92        | 92.47  | 13.17    | 3.88 | 3.30 | 9102116 | STS.000.GIQ PP1 dither 1 | 
    | 393785 |       20 | 2011-09-18 08:44:33 | 24.5337 |  0.027504 |     3.6808 | 1.135 | 1.065    | 0.92        | 96.03  | 13.46    | 4.03 | 3.48 | 8654104 | STS.000.GIQ PP1 dither 2 | 
    | 393792 |       20 | 2011-09-18 08:48:23 | 24.5315 |  0.026466 |    3.70284 | 1.164 | 1.073    | 0.97        | 94.42  | 13.26    | 4.00 | 3.44 | 8762736 | STS.000.GIQ PP1 dither 3 | 
    | 393799 |       20 | 2011-09-18 08:52:22 | 24.5312 |  0.028034 |    3.71144 | 1.172 | 1.078    | 0.96        | 98.12  | 13.35    | 4.02 | 3.47 | 8695292 | STS.000.GIQ PP1 dither 4 | 
    | 393835 |       20 | 2011-09-18 09:13:19 | 24.5271 |  0.027369 |    3.75097 | 1.157 | 1.076    | 0.99        | 103.99 | 13.49    | 4.07 | 3.47 | 8624396 | STS.000.GIQ PP1 dither 1 | 
    
    --> cuts out a larger fraction of the center position
    
    --> compared to log file for ppStack -- gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.log
          Input 0: 3.137970
          Input 1: 3.397691
          Input 2: 3.309175
          Input 3: 3.310735
          Input 4: 3.114384
          Input 5: 3.043151
          Input 6: 3.346249
          Input 7: 3.141845
          Input 8: 3.471485
          Input 9: 3.238341
          Input 10: 3.433451
          Input 11: 3.225220
          Input 12: 4.423095
          Input 13: 4.086733
          Input 14: 4.282342
          Input 15: 4.067309
          Input 16: 3.030721
          Input 17: 4.047925
          Input 18: 4.245076
          Input 19: 4.328331
          Input 20: 4.284031
          Input 21: 4.074796
          Input 22: 4.197470
          Input FWHMs : 3.662502 +/- 0.504006
              fit ext:   0.965282 sec for 225 of 225 sources
              fit psf:   0.342076 sec for 196 of 225 sources
            apresid: 0.046344 +/- 0.000000 (systematic error) from statistics of 225 psf stars (196 used)
            chisq vs flux fit term 0: 0.000000 +/- 0.000000
            chisq vs flux fit term 1: 0.000000 +/- 0.000000
          try model PS_MODEL_PS1_V1, ap-fit: 0.046344 +/- 0.000000 : sky bias: 0.000000
          Target seeing FWHM: 4.430518
    ...
            fwhm (psf): 4.472250,4.366182 (moments): 0.000000,0.000000
    
    --> even for c1f37 has FWHM>4.2 going into it -- gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.log
    --> however, not as overdriven as feared. --- need to add plot/vector of PSF shape in skycell ---
    

psphot on samples, unconvolved and convolved (compare to ppStack log file):

  • faults because of recent modified psphot config/code to limit peaks and so faults.. need to add -Di PEAKS_NMAX_TOTAL 60000
  • unconvolved stacks still faulting.. so do PASS1 only -D BREAK_POINT PASS1. Convolved ones can be poor, FWHM_MAJ~9-10??
    gethead -auhc *fpa.cmf FWHM_MAJ IQ_FW1
    STS.V4.skycell.1.03.stk.899917.fpa.cmf        4.405078 3.746344
    STS.V4.skycell.1.03.stk.899917.unconv.fpa.cmf 3.921824 3.04026
    STS.V4.skycell.1.03.stk.899918.fpa.cmf        4.359843 3.701267
    STS.V4.skycell.1.03.stk.899918.unconv.fpa.cmf 3.788124 2.973968
    STS.V4.skycell.1.03.stk.899919.fpa.cmf        4.184109 3.576873
    STS.V4.skycell.1.03.stk.899919.unconv.fpa.cmf 3.835106 2.939088
    STS.V4.skycell.1.03.stk.899920.fpa.cmf        4.54595 3.812682
    STS.V4.skycell.1.03.stk.899920.unconv.fpa.cmf 3.862908 3.048509
    --> basically as with MD, +/- pixel, so all are basically similar and going to smaller FWHM cut not improve much, partially because cut doen't always match true FWHM going in.
    
  • previous refstack.20110613
    gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.fits (cleaned)
    gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.cmf
    gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.log
    

Summary of test runs for skycell.1.03:

version data_group (stack_id) filter skycell input warps (rej) skycell FWHM (major) conv/unconv
new STS.refstack.20120510 V4 i 61 (2) 1.38/0.98"
STS.reftest.20120510.c1f39 V4 (899917) i 36 (0) 1.10/0.98"
STS.reftest.20120510.c1f38 V4 (899918) i 34 (0) 1.09/0.95"
STS.reftest.20120510.c1f37 V4 (899919) i 12 (0) 1.05/0.96"
STS.reftest.20120510.c0f45 V4 (899920) i 44 (0) 1.14/0.97"
old STS.refstack.20110613 V3 i 68 (0) ~1.16/0.9" (approx from log since image cleaned)

Look at recent camera/warps to check similar coverage

  • recent observations all 180 (as of 120510)
  • center+dither available
  • sample from rawExp table:
    | 484662 | o6054g0382o | 2012-05-07 13:44:36 | i.00000 |       30 | STS.180.ND PP1 center 
    
    gpc1/STS.nt/2012/05/07/o6054g0382o.484662/
    gpc1/STS.nt/2012/05/07/o6054g0382o.484662/o6054g0382o.484662.wrp.408115.skycell.1.03.fits
    --> bad tracking image, startrails.. try another
    
    gpc1/STS.nt/2012/05/07/o6054g0375o.484656/
    gpc1/STS.nt/2012/05/07/o6054g0375o.484656/o6054g0375o.484656.wrp.408110.skycell.1.03.fits
    --> yes, similar pattern - 
    

Main run -- do by projection PP 1-7:

  • use FWHM cut 3.8 for ~20-40 and see -- if some need fixing, then re-run that skycell, if too much burntool then know for next version
    • should really add the PSF cut to PS1_REFERENCE too to help reject unrepresentative camera FWHM for skycell.
  • run in normal stack, system fairly loaded w/o deepstack running? no, the c3 machines need more work
stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.refstack.20120510 -set_label STS.refstack.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.refstack.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.8 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple  >& stacktool.sts_refstack.20120510.fwhm38_elong12.log
--->  oops forgot to add -select_skycell_id skycell.1.% -- so it is running all projections at once.

Compare to test sample skycell.1.03:

gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.fits
gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.unconv.fits
gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.log
--> run psphot manually and add to table above -- need to set 65000 detections for unconv stack

Look at final stacks: bottom line, more inputs would help, but maybe this is sufficient

  • 4.17 odd bright cell print through
  • 2.02 odd faint chunk print through
  • 2.39 gap faint print through
  • 5.06 missed burntool?
  • 1.09,1.17 large unfilled gap
  • 1.24 long unfilled gap
  • 1.25,1.35 long and large unfilled gap
  • 1.12 large unfilled cell
  • 1.40 unfilled masking corner multi-cell
  • 1.55 rounded edge + corner adds other exp
  • 1.44,5.14 odd bright orphan diffspike-like unmasked
  • 1.11 nice cluster
  • 1.32 faint flux print through from brighter cell

Run additional reftest for selected examples:

  • skycell.1.35 -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.902937.log
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35  >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell135_c2.log
    
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f45 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 4.5 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35  >& stacktool.sts_reftest.20120510.fwhm45_elong12_skycell135_c2.log
    
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35  >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell135_c2.log
    
    ---> the additional input 45 shows lower corner with much worse seeing -- this is problem, each skycell may need to be tuned/limited directly since DB not have proper stats. A reason for using conv stacking 
    
  • skycell.1.40 -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.902942.log
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40  >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell140_c2.log
    
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f45 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 4.5 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40  >& stacktool.sts_reftest.20120510.fwhm45_elong12_skycell140_c2.log
    
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40  >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell140_c2.log
    
    ---> max input not fill in masking much 
    
  • skycell.1.55 -- mix of PP inputs and possible IQ issues, try reducing Ninput greatly. really need to do manually.. -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.log
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f36 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.6 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.55  >& stacktool.sts_reftest.20120510.fwhm36_elong12_skycell155_c2.log
    
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.55  >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell155_c2.log
    
    ---> the fewer inputs makes the IQ worse in upper left, reason to use conv stacks here 
    

Look at Target seeing to see if any odd inputs driving target PSF:

  • Target seeing for PSF
    mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select path_base from stackRun join stackSumSkyfile using (stack_id) where stackRun.label like "STS.refstack.20120510" and  stackRun.skycell_id like "%.1.%";' | grep -v path_base | awk '{print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -l -p "$1"%.log"}' | tcsh | awk '{print " cat "$9}' | tcsh | grep "Target seeing"
          Target seeing FWHM: 3.871291
          Target seeing FWHM: 5.136444
          Target seeing FWHM: 9.224628
          Target seeing FWHM: 5.216270
          Target seeing FWHM: 4.238182
          Target seeing FWHM: 4.433765
          Target seeing FWHM: 8.005375
          Target seeing FWHM: 5.200400
    ...
    --> look at couple with FWHM 8-9 -- #3,7 -- manually match to input file:
    gpc1/STS.refstack.20120510/STS.V4/skycell.1.02/STS.V4.skycell.1.02.stk.902904.log -- 83 Ninput, largest ~5.7 so wacky shapes? even 2.89 FWHM
    gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.log -- 78 Ninput, largest ~4.5 so wacky shapes?
    
  • create log of Input FWHM, Target seeing, and convolved PSF FWHM:
    mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select path_base from stackRun join stackSumSkyfile using (stack_id) where stackRun.label like "STS.refstack.20120510" and  stackRun.skycell_id like "%";' | grep -v path_base | awk '{print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -l -p "$1"%.log"}' | tcsh | awk '{print "echo "$9"; grep  -e \" Input \" -e \"Target seeing\" -e \"fwhm (psf)\" "$9" ;echo \" \""}' | tcsh  > STS.refstack.20120510_input_target_output_fwhm_inputlong.log
    
    

clean-up

Will need to keep warps around for remaking the refstack if necessary but more importantly for MOPS to get stamps without update delay. ~50x7 or 350 exposures on disk for the skycells.

chiptool -updaterun -set_state goto_cleaned -set_label goto_cleaned -data_group STS.refstack.20120510 -dbname gpc1

//warptool -updaterun -set_state goto_cleaned -set_label goto_cleaned -data_group STS.refstack.20120510 -dbname gpc1

Test Diffim Samples

Want to create a sample of diffims with detections for MOPS to look at. Will need to also keep in "full" state so they can get postage stamps quickly. Focus on projection 1 (PP1 exposures, skycell.1.xx).

Basic setup: (warp - convolved stack) -- need examples and compared to unconv stack (why not using unconv for any WSdiff?)

  • what are diffims for other products -- WS, target1, conv vs unconv stack -- part of what needed to do for ppsub testing page still
  • MOPS has no specific requirements for test samples, so try to combine multiple aspects using skycell.1.xx
    • W-S with conv refstack since unconv refstack not an option due to code issues in the filename.
    • 2011 -- 1 or 2 nights -- ones not used in refstack (may be difficult), will want to be manually updated so no chance of being (auto)-cleaned
    • 2012 -- 1 or 2 nights -- will want to be manually updated so no chance of cleaning as well
  • not sure how to queue/run these since normally driven by a survey task. could do last nights for 2012 right off but cleanup window so work with older sample
    • need to setup/change publishing location?
    • turn off cleanup for the STS for now? no, uncertain when would turn back on and want continue cleanup as normal.

Manual Test Set

Want to run quick set locally. Pick center, edge, edge+lots overlap for various filling types - 1.44, 1.55, 1.03?

gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.log
gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.log
gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.log
--> appear mostly well behaved mix.. need to add one more -- 1.06 is wacky
gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.log

Run with convolve TARGET2, TARGET1 and DUAL (can use simiple scripts used for ppSub tests) -- in testppsub_sts120510:

  • need recent night listing with PP1 exp_name -
select exp_id,exp_name,exp_time,dateobs,comment from rawExp where obs_mode like "%STS%" and comment like "%PP1%" and dateobs>"2012-04";
| exp_id | exp_name    | exp_time | dateobs             | comment                  |
+--------+-------------+----------+---------------------+--------------------------+
| 475121 | o6029g0441o |       30 | 2012-04-12 13:40:09 | STS.180.ND PP1 center    | 
| 475128 | o6029g0448o |       30 | 2012-04-12 13:45:11 | STS.180.ND PP1 center    | 
| 475135 | o6029g0455o |       30 | 2012-04-12 13:50:14 | STS.180.ND PP1 center    | 

...

| 483638 | o6049g0349o |       30 | 2012-05-02 13:54:49 | STS.180.ND PP1 center    | 
| 483646 | o6049g0356o |       30 | 2012-05-02 13:59:51 | STS.180.ND PP1 center    | 
| 483651 | o6049g0328o |       30 | 2012-05-02 13:39:38 | STS.180.ND PP1 center    | 
| 483652 | o6049g0363o |       30 | 2012-05-02 14:04:52 | STS.180.ND PP1 center    | 
| 484578 | o6054g0305o |       30 | 2012-05-07 12:48:56 | STS.180.ND PP1 center    | 
| 484590 | o6054g0312o |       30 | 2012-05-07 12:53:58 | STS.180.ND PP1 center    | 
| 484599 | o6054g0319o |       30 | 2012-05-07 12:58:58 | STS.180.ND PP1 center    | 
| 484606 | o6054g0326o |       30 | 2012-05-07 13:04:01 | STS.180.ND PP1 center    | 
| 484613 | o6054g0333o |       30 | 2012-05-07 13:09:03 | STS.180.ND PP1 center    | 
| 484620 | o6054g0340o |       30 | 2012-05-07 13:14:04 | STS.180.ND PP1 center    | 
| 484627 | o6054g0347o |       30 | 2012-05-07 13:19:09 | STS.180.ND PP1 center    | 
| 484634 | o6054g0354o |       30 | 2012-05-07 13:24:13 | STS.180.ND PP1 center    | 
| 484641 | o6054g0361o |       30 | 2012-05-07 13:29:18 | STS.180.ND PP1 center    | 
| 484648 | o6054g0368o |       30 | 2012-05-07 13:34:25 | STS.180.ND PP1 center    | 
| 484656 | o6054g0375o |       30 | 2012-05-07 13:39:31 | STS.180.ND PP1 center    | 
| 484662 | o6054g0382o |       30 | 2012-05-07 13:44:36 | STS.180.ND PP1 center    | 
+--------+-------------+----------+---------------------+--------------------------+


neb://@HOST@.0/gpc1/STS.nt/2012/05/02/o6049g0286o -- cleaned..
neb://@HOST@.0/gpc1/STS.nt/2012/05/07/o6054g0305o -- okay

--> do most recent 2 first - first in list - need script liner to get more of these 
--> will want to know FWHM of input warps too -- will be in ppSub log files 
  • skycell.1.44 sample:
    neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/%unconv%fits
    nebds9 gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits
    --> @4610,5580 region see stars look elongated/trailed versus ones nearby.. looks like multi-cell mask region -- diffims will show if problem
    --> maybe has to do with GIQ? 
    
    ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.44.fits  gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120507 > runppsub_sts144_120507_1.bat
    source runppsub_sts144_120507_1.bat >& runppsub_sts144_120507_1.log &
    
    ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits  gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120507 > runppsub_sts144_120507_2.bat
    source runppsub_sts144_120507_2.bat >& runppsub_sts144_120507_2.log &
    
    --> created batch script  runppsub_sts144_120507_2.bat for example
    ppSub ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.20120507 -inimage neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits -refimage neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits -inmask neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.mask.fits -refmask neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.mask.fits -invariance neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.wt.fits -refvariance neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.wt.fits -insources neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.cmf -refsources neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.cmf -D CONVOLVE.TARGET SINGLE2 -save-inconv -save-refconv -inverse -recipe PPSUB WARPSTACK -recipe PSPHOT DIFF -recipe PPSTATS DIFFSTATS -F PSPHOT.PSF.SAVE PSPHOT.PSF.SKY.SAVE -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF -F PSPHOT.BACKMDL PSPHOT.BACKMDL.MEF -photometry -vvv -debug -trace err 10 -trace ppSub 10 -trace psModules.imcombine 10 -trace psModules.camera 10 -tracedest ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.trace -log ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.log -stats ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.stats -threads 4 -dumpconfig ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.mdc
    
    ppSub ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.20120507 -inimage neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits -refimage neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits -inmask neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.mask.fits -refmask neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.mask.fits -invariance neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.wt.fits -refvariance neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.wt.fits -insources neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.cmf -refsources neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.cmf -D CONVOLVE.TARGET SINGLE1 -save-inconv -save-refconv -inverse -recipe PPSUB WARPSTACK -recipe PSPHOT DIFF -recipe PPSTATS DIFFSTATS -F PSPHOT.PSF.SAVE PSPHOT.PSF.SKY.SAVE -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF -F PSPHOT.BACKMDL PSPHOT.BACKMDL.MEF -photometry -vvv -debug -trace err 10 -trace ppSub 10 -trace psModules.imcombine 10 -trace psModules.camera 10 -tracedest ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.trace -log ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.log -stats ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.stats -threads 4 -dumpconfig ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.mdc
    
    ppSub ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.20120507 -inimage neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits -refimage neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits -inmask neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.mask.fits -refmask neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.mask.fits -invariance neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.wt.fits -refvariance neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.wt.fits -insources neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.cmf -refsources neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.cmf -D CONVOLVE.TARGET SINGLE1 -Db DUAL T -save-inconv -save-refconv -inverse -recipe PPSUB WARPSTACK -recipe PSPHOT DIFF -recipe PPSTATS DIFFSTATS -F PSPHOT.PSF.SAVE PSPHOT.PSF.SKY.SAVE -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF -F PSPHOT.BACKMDL PSPHOT.BACKMDL.MEF -photometry -vvv -debug -trace err 10 -trace ppSub 10 -trace psModules.imcombine 10 -trace psModules.camera 10 -tracedest ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.trace -log ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.log -stats ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.stats -threads 4 -dumpconfig ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.mdc
    
    
    
  • skycell.1.55 sample:
    neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/%unconv%fits
    --> also see odd shape near center but is edge field, but has both PP1 and PP2 inputs - 
    
    ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.55.fits  gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.unconv.fits 20120507 > runppsub_sts155_120507_1.bat
    source runppsub_sts155_120507_1.bat >& runppsub_sts155_120507_1.log &
    
  • skycell.1.03 sample:
    neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/%unconv%fits
    --> looks ok, good overlap mix pp1/4
    
    ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits  gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.unconv.fits  20120507 > runppsub_sts103_120507_1.bat
    source runppsub_sts103_120507_1.bat >& runppsub_sts103_120507_1.log &
    
  • skycell.1.06 sample:
    neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/%unconv%fits
    --> looks ok, crazy overlap mix pp1,2,4 and maybe odd shape in corner
    
    ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.06.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.unconv.fits 20120507 > runppsub_sts106_120507_1.bat
    source runppsub_sts106_120507_1.bat >& runppsub_sts106_120507_1.log &
    

Overall, diffims look poor. MD04 from 20120508 WS look okay, so look at a reftest sample as well

  • look at sample based on FWHM
    SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE obs_mode="STS" and dateobs>"2012-01-01" and comment like "%PP1%" group by exp_id;
    
    | exp_id | exp_name    | exp_time | dateobs             | zpt_obs | zpt_stdev | fwhm_major | elong | iq_elong | bg     | bg_stdev | uq    | lq    | n_stars | comment                  |
    +--------+-------------+----------+---------------------+---------+-----------+------------+-------+----------+--------+----------+-------+-------+---------+--------------------------+
    | 478479 | o6037g0401o |       20 | 2012-04-20 13:46:26 | 24.5799 |  0.023472 |     4.1334 | 1.174 | 1.087    | 76.31  | 12.57    | 4.42  | 3.87  | 8291412 | STS.180.GIQ PP1 dither 1 | 
    | 478485 | o6037g0408o |       20 | 2012-04-20 13:50:22 | 24.5835 |  0.025496 |    4.22466 | 1.130 | 1.065    | 75.50  | 12.68    | 4.55  | 4.00  | 7956904 | STS.180.GIQ PP1 dither 2 | 
    | 478492 | o6037g0415o |       20 | 2012-04-20 13:54:14 | 24.5827 |  0.023232 |    4.16861 | 1.159 | 1.076    | 76.68  | 12.72    | 4.55  | 3.98  | 8128324 | STS.180.GIQ PP1 dither 3 | 
    | 478499 | o6037g0422o |       20 | 2012-04-20 13:58:10 | 24.5839 |  0.025415 |    4.09489 | 1.149 | 1.080    | 78.47  | 12.89    | 4.59  | 3.92  | 8022420 | STS.180.GIQ PP1 dither 4 | 
    | 478506 | o6037g0429o |       20 | 2012-04-20 14:02:08 | 24.5839 |  0.026163 |    4.15877 | 1.147 | 1.075    | 80.47  | 13.00    | 4.65  | 3.97  | 7940228 | STS.180.GIQ PP1 dither 1 | 
    | 478514 | o6037g0436o |       20 | 2012-04-20 14:06:05 | 24.5859 |  0.026272 |    4.32337 | 1.160 | 1.070    | 78.41  | 12.89    | 4.65  | 4.09  | 7879084 | STS.180.GIQ PP1 dither 2 | 
    | 478520 | o6037g0443o |       20 | 2012-04-20 14:09:58 | 24.5857 |  0.027436 |    4.21704 | 1.168 | 1.072    | 77.77  | 12.84    | 4.51  | 3.95  | 8086704 | STS.180.GIQ PP1 dither 3 | 
    | 478527 | o6037g0450o |       20 | 2012-04-20 14:13:50 | 24.5854 |  0.023077 |    4.05132 | 1.167 | 1.071    | 77.00  | 12.81    | 4.30  | 3.84  | 8285424 | STS.180.GIQ PP1 dither 4 | 
    | 478533 | o6037g0457o |       20 | 2012-04-20 14:17:47 | 24.5832 |  0.024982 |    3.96426 | 1.160 | 1.081    | 76.22  | 12.70    | 4.36  | 3.71  | 8430032 | STS.180.GIQ PP1 dither 1 | 
    | 478542 | o6037g0464o |       20 | 2012-04-20 14:21:42 | 24.5834 |  0.022577 |    3.89289 | 1.151 | 1.070    | 76.97  | 12.70    | 4.30  | 3.65  | 8503148 | STS.180.GIQ PP1 dither 2 | 
    | 478548 | o6037g0471o |       20 | 2012-04-20 14:25:40 | 24.5863 |  0.027271 |    3.99574 | 1.135 | 1.071    | 75.40  | 12.73    | 4.42  | 3.84  | 8258836 | STS.180.GIQ PP1 dither 3 | 
    | 478554 | o6037g0478o |       20 | 2012-04-20 14:29:40 | 24.5881 |   0.02494 |    4.14256 | 1.131 | 1.074    | 77.49  | 12.87    | 4.50  | 3.94  | 8057476 | STS.180.GIQ PP1 dither 4 | 
    | 478561 | o6037g0485o |       20 | 2012-04-20 14:34:36 | 24.5874 |  0.025127 |    4.02795 | 1.181 | 1.083    | 76.24  | 12.78    | 4.50  | 3.67  | 8296456 | STS.180.GIQ PP1 dither 1 | 
    | 478569 | o6037g0492o |       20 | 2012-04-20 14:38:35 | 24.5881 |  0.025795 |    4.09032 | 1.244 | 1.081    | 75.42  | 12.66    | 4.31  | 3.88  | 8401392 | STS.180.GIQ PP1 dither 2 | 
    | 478576 | o6037g0499o |       20 | 2012-04-20 14:42:30 | 24.5884 |  0.026093 |    4.01093 | 1.189 | 1.083    | 75.54  | 12.70    | 4.35  | 3.85  | 8364744 | STS.180.GIQ PP1 dither 3 | 
    | 478582 | o6037g0506o |       20 | 2012-04-20 14:46:32 | 24.5848 |  0.023689 |     3.8121 | 1.183 | 1.083    | 76.40  | 12.70    | 4.34  | 3.49  | 8674124 | STS.180.GIQ PP1 dither 4 | 
    | 479810 | o6040g0441o |       30 | 2012-04-23 13:31:03 |  24.608 |  0.031557 |    6.12941 | 1.131 | 1.057    | 147.86 | 17.88    | 6.39  | 5.88  | 5637788 | STS.180.ND PP1 center    | 
    | 479818 | o6040g0448o |       30 | 2012-04-23 13:36:04 | 24.6014 |   0.03195 |    5.88586 | 1.054 | 1.044    | 150.06 | 18.08    | 6.08  | 5.69  | 5697728 | STS.180.ND PP1 center    | 
    | 479822 | o6040g0455o |       30 | 2012-04-23 13:41:07 | 24.6063 |  0.031496 |     5.3039 | 1.078 | 1.048    | 147.06 | 17.63    | 5.56  | 5.12  | 6435688 | STS.180.ND PP1 center    | 
    | 479831 | o6040g0462o |       30 | 2012-04-23 13:46:10 | 24.5993 |  0.037926 |    7.24256 | 1.054 | 1.047    | 153.22 | 18.75    | 7.51  | 7.06  | 4638008 | STS.180.ND PP1 center    | 
    | 479838 | o6040g0469o |       30 | 2012-04-23 13:51:11 | 24.6107 |  0.033072 |    5.79955 | 1.057 | 1.042    | 149.01 | 17.93    | 6.15  | 5.61  | 5894304 | STS.180.ND PP1 center    | 
    | 479845 | o6040g0476o |       30 | 2012-04-23 13:56:11 |  24.611 |  0.034235 |    5.78945 | 1.081 | 1.054    | 143.46 | 17.93    | 6.17  | 5.59  | 5720112 | STS.180.ND PP1 center    | 
    | 479852 | o6040g0483o |       30 | 2012-04-23 14:01:15 | 24.6095 |  0.027292 |    5.23093 | 1.080 | 1.046    | 134.52 | 17.25    | 5.62  | 4.97  | 6420468 | STS.180.ND PP1 center    | 
    | 479859 | o6040g0490o |       30 | 2012-04-23 14:06:17 | 24.6096 |  0.028854 |    5.25701 | 1.113 | 1.056    | 134.46 | 17.07    | 5.68  | 4.99  | 6732892 | STS.180.ND PP1 center    | 
    | 479867 | o6040g0497o |       30 | 2012-04-23 14:11:20 | 24.6074 |   0.02922 |    4.90916 | 1.124 | 1.062    | 132.51 | 16.81    | 5.42  | 4.58  | 7115244 | STS.180.ND PP1 center    | 
    | 479874 | o6040g0504o |       30 | 2012-04-23 14:16:21 | 24.6099 |  0.032976 |    6.24034 | 1.063 | 1.047    | 135.12 | 17.82    | 6.52  | 6.03  | 5343240 | STS.180.ND PP1 center    | 
    | 479880 | o6040g0511o |       30 | 2012-04-23 14:21:21 | 24.6151 |  0.030796 |    5.76475 | 1.078 | 1.046    | 136.42 | 17.50    | 6.09  | 5.56  | 5944992 | STS.180.ND PP1 center    | 
    | 479888 | o6040g0518o |       30 | 2012-04-23 14:26:25 | 24.6136 |  0.033806 |    6.09428 | 1.059 | 1.045    | 134.54 | 17.65    | 6.44  | 5.87  | 5589900 | STS.180.ND PP1 center    | 
    | 482593 | o6045g0420o |       30 | 2012-04-28 13:37:39 | 24.6132 |  0.033623 |    6.26513 | 1.051 | 1.043    | 109.03 | 16.37    | 6.59  | 6.09  | 5474888 | STS.180.ND PP1 center    | 
    | 482600 | o6045g0427o |       30 | 2012-04-28 13:42:41 | 24.6115 |  0.034468 |    6.32739 | 1.136 | 1.066    | 104.49 | 16.41    | 6.54  | 6.13  | 5488984 | STS.180.ND PP1 center    | 
    | 482609 | o6045g0434o |       30 | 2012-04-28 13:47:43 | 24.6127 |  0.030218 |    5.29423 | 1.090 | 1.049    | 109.80 | 16.21    | 5.59  | 5.11  | 6391756 | STS.180.ND PP1 center    | 
    | 482614 | o6045g0441o |       30 | 2012-04-28 13:52:45 | 24.6118 |  0.034794 |    5.23477 | 1.084 | 1.045    | 112.48 | 16.55    | 5.44  | 5.06  | 6164296 | STS.180.ND PP1 center    | 
    | 482620 | o6045g0448o |       30 | 2012-04-28 13:57:46 | 24.6129 |  0.026911 |    5.07853 | 1.095 | 1.051    | 107.76 | 15.90    | 5.23  | 4.89  | 6931872 | STS.180.ND PP1 center    | 
    | 482628 | o6045g0455o |       30 | 2012-04-28 14:02:53 | 24.6097 |  0.029615 |    4.93548 | 1.093 | 1.054    | 111.93 | 16.05    | 5.06  | 4.70  | 7055316 | STS.180.ND PP1 center    | 
    | 482635 | o6045g0462o |       30 | 2012-04-28 14:07:56 | 24.6121 |  0.032459 |    5.05487 | 1.081 | 1.050    | 110.60 | 16.20    | 5.27  | 4.88  | 6650952 | STS.180.ND PP1 center    | 
    | 482642 | o6045g0469o |       30 | 2012-04-28 14:12:59 | 24.6062 |  0.024746 |    4.55978 | 1.093 | 1.054    | 113.28 | 16.00    | 4.80  | 4.41  | 7489016 | STS.180.ND PP1 center    | 
    | 482647 | o6045g0476o |       30 | 2012-04-28 14:18:01 | 24.6072 |  0.027452 |    4.59176 | 1.171 | 1.066    | 107.79 | 15.74    | 4.85  | 4.40  | 7688940 | STS.180.ND PP1 center    | 
    | 482654 | o6045g0483o |       30 | 2012-04-28 14:23:04 | 24.6041 |  0.025282 |    4.21365 | 1.113 | 1.055    | 110.57 | 15.78    | 4.62  | 4.07  | 8007588 | STS.180.ND PP1 center    | 
    | 482661 | o6045g0490o |       30 | 2012-04-28 14:28:11 | 24.6169 |  0.030737 |    5.33823 | 1.077 | 1.051    | 104.92 | 16.05    | 5.60  | 5.22  | 6471096 | STS.180.ND PP1 center    | 
    | 482667 | o6045g0497o |       30 | 2012-04-28 14:33:16 | 24.6211 |  0.034222 |    5.67956 | 1.088 | 1.051    | 104.88 | 16.23    | 6.06  | 5.48  | 5996524 | STS.180.ND PP1 center    | 
    | 483576 | o6049g0286o |       30 | 2012-05-02 13:09:26 | 24.5922 |  0.026936 |    4.77424 | 1.088 | 1.058    | 166.69 | 17.89    | 5.10  | 4.62  | 7102152 | STS.180.ND PP1 center    | 
    | 483583 | o6049g0293o |       30 | 2012-05-02 13:14:27 | 24.5931 |   0.02858 |     5.1493 | 1.114 | 1.057    | 167.40 | 18.27    | 5.46  | 4.93  | 6496992 | STS.180.ND PP1 center    | 
    | 483590 | o6049g0300o |       30 | 2012-05-02 13:19:26 | 24.5953 |  0.036625 |    5.26372 | 1.072 | 1.049    | 164.38 | 18.82    | 5.52  | 5.07  | 5726832 | STS.180.ND PP1 center    | 
    | 483597 | o6049g0307o |       30 | 2012-05-02 13:24:29 | 24.5986 |  0.029531 |    5.12019 | 1.101 | 1.053    | 154.82 | 17.88    | 5.45  | 4.96  | 6618008 | STS.180.ND PP1 center    | 
    | 483604 | o6049g0314o |       30 | 2012-05-02 13:29:35 | 24.5901 |  0.026812 |    4.59395 | 1.158 | 1.074    | 147.96 | 17.25    | 4.95  | 4.23  | 7622296 | STS.180.ND PP1 center    | 
    | 483611 | o6049g0321o |       30 | 2012-05-02 13:34:37 | 24.5992 |  0.025234 |    5.08152 | 1.104 | 1.051    | 148.90 | 17.58    | 5.41  | 4.76  | 6840712 | STS.180.ND PP1 center    | 
    | 483624 | o6049g0335o |       30 | 2012-05-02 13:44:43 | 24.6024 |  0.035925 |    6.44934 | 1.099 | 1.058    | 153.97 | 18.59    | 6.59  | 6.26  | 5262844 | STS.180.ND PP1 center    | 
    | 483630 | o6049g0342o |       30 | 2012-05-02 13:49:48 | 24.5992 |  0.027985 |    4.95281 | 1.129 | 1.060    | 150.63 | 17.73    | 5.17  | 4.63  | 6896656 | STS.180.ND PP1 center    | 
    | 483638 | o6049g0349o |       30 | 2012-05-02 13:54:49 | 24.6021 |   0.02961 |    4.94886 | 1.105 | 1.056    | 149.24 | 17.80    | 5.32  | 4.74  | 6686580 | STS.180.ND PP1 center    | 
    | 483646 | o6049g0356o |       30 | 2012-05-02 13:59:51 | 24.6018 |  0.024099 |    5.00208 | 1.079 | 1.050    | 146.60 | 17.67    | 5.36  | 4.84  | 6704456 | STS.180.ND PP1 center    | 
    | 483651 | o6049g0328o |       30 | 2012-05-02 13:39:38 | 24.6022 |  0.031847 |    5.47493 | 1.090 | 1.056    | 153.10 | 18.14    | 5.76  | 5.32  | 6100872 | STS.180.ND PP1 center    | 
    | 483652 | o6049g0363o |       30 | 2012-05-02 14:04:52 | 24.6066 |  0.031757 |    5.38569 | 1.077 | 1.047    | 143.41 | 17.73    | 5.66  | 5.17  | 6297140 | STS.180.ND PP1 center    | 
    | 484578 | o6054g0305o |       30 | 2012-05-07 12:48:56 | 24.5422 |  0.048277 |    7.94199 | 1.042 | 1.058    | 297.28 | 23.85    | 8.11  | 7.69  | 3526740 | STS.180.ND PP1 center    | 
    | 484590 | o6054g0312o |       30 | 2012-05-07 12:53:58 | 24.5979 |  0.029655 |    5.70616 | 1.104 | 1.054    | 286.37 | 22.42    | 6.10  | 5.41  | 5368872 | STS.180.ND PP1 center    | 
    | 484599 | o6054g0319o |       30 | 2012-05-07 12:58:58 | 24.5956 |  0.026827 |    4.86361 | 1.087 | 1.055    | 284.14 | 21.65    | 5.23  | 4.55  | 6553352 | STS.180.ND PP1 center    | 
    | 484606 | o6054g0326o |       30 | 2012-05-07 13:04:01 |  24.598 |  0.034046 |    6.04375 | 1.065 | 1.049    | 275.47 | 22.27    | 6.37  | 5.74  | 5064844 | STS.180.ND PP1 center    | 
    | 484613 | o6054g0333o |       30 | 2012-05-07 13:09:03 | 24.5886 |  0.038271 |    6.46371 | 1.053 | 1.049    | 272.68 | 22.48    | 6.78  | 6.17  | 4610776 | STS.180.ND PP1 center    | 
    | 484620 | o6054g0340o |       30 | 2012-05-07 13:14:04 | 24.6006 |  0.027721 |    5.30901 | 1.094 | 1.053    | 267.68 | 21.54    | 5.68  | 5.02  | 5955176 | STS.180.ND PP1 center    | 
    | 484627 | o6054g0347o |       30 | 2012-05-07 13:19:09 | 24.6062 |  0.029659 |    5.54425 | 1.109 | 1.059    | 261.91 | 21.38    | 5.92  | 5.23  | 5810048 | STS.180.ND PP1 center    | 
    | 484634 | o6054g0354o |       30 | 2012-05-07 13:24:13 | 24.6036 |   0.02889 |    5.08168 | 1.087 | 1.051    | 261.21 | 21.03    | 5.43  | 4.76  | 6426036 | STS.180.ND PP1 center    | 
    | 484641 | o6054g0361o |       30 | 2012-05-07 13:29:18 |  24.605 |  0.031921 |    5.93175 | 1.064 | 1.046    | 264.68 | 21.74    | 6.29  | 5.65  | 5261756 | STS.180.ND PP1 center    | 
    | 484648 | o6054g0368o |       30 | 2012-05-07 13:34:25 | 24.6058 |   0.03159 |    5.50681 | 1.098 | 1.044    | 259.40 | 21.42    | 5.87  | 5.25  | 5692760 | STS.180.ND PP1 center    | 
    | 484656 | o6054g0375o |       30 | 2012-05-07 13:39:31 | 24.6014 |  0.029152 |    5.17827 | 1.093 | 1.055    | 251.54 | 21.03    | 5.57  | 4.80  | 6133144 | STS.180.ND PP1 center    | 
    | 484662 | o6054g0382o |       30 | 2012-05-07 13:44:36 |  21.388 |  0.747221 |    11.6821 | 1.243 | 1.300    | 279.91 | 30.44    | 49.59 | 10.41 | 1599700 | STS.180.ND PP1 center    | 
    
    --> next one on 20120507 (o6054g0312o) is better FWHM so try that first - looks the same. 
    --> so try 20120420, o6037g0506o -- will need to update and just used the PSS (warp_id=400804, skycell.1.44 bundle) -- FWHM~3.8
    
    gpc1/STS.nt/2012/04/20/o6037g0506o.478582/o6037g0506o.478582.wrp.400804.skycell.1.44.fits
    
    ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/20/o6037g0506o.478582/o6037g0506o.478582.wrp.400804.skycell.1.44.fits  gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120420 > runppsub_sts144_120420_x.bat
    source runppsub_sts144_120420_x.bat >& runppsub_sts144_120420_x.log &
    
    --> better, at least residuals more on just brighter sources and not concentrated cores, but some things that look like diffim detections look odd. Some look like pixel patterns removed in pixel rejection?
    
    

Need to retry diffims above as conv stacks too - seems the same, odd.. --> need to look at the mask images as well -

  • sample reftest diffims
    mysql> select stack_id,data_group,skycell_id from stackRun where data_group like "STS.reftest%c%f%";
    +----------+----------------------------+--------------+
    | stack_id | data_group                 | skycell_id   |
    +----------+----------------------------+--------------+
    |   899920 | STS.reftest.20120510.c0f45 | skycell.1.03 | 
    |   899919 | STS.reftest.20120510.c1f37 | skycell.1.03 | 
    |   899918 | STS.reftest.20120510.c1f38 | skycell.1.03 | 
    |   899917 | STS.reftest.20120510.c1f39 | skycell.1.03 | 
    |   904603 | STS.reftest.20120510.c2f36 | skycell.1.55 | 
    |   904604 | STS.reftest.20120510.c2f37 | skycell.1.55 | 
    |   904605 | STS.reftest.20120510.c2f37 | skycell.1.40 | 
    |   904606 | STS.reftest.20120510.c2f37 | skycell.1.35 | 
    |   904599 | STS.reftest.20120510.c2f39 | skycell.1.35 | 
    |   904601 | STS.reftest.20120510.c2f39 | skycell.1.40 | 
    |   904600 | STS.reftest.20120510.c2f45 | skycell.1.35 | 
    |   904602 | STS.reftest.20120510.c2f45 | skycell.1.40 | 
    
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899917.unconv.fits
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.fits
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.fits
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899920.unconv.fits
    
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904599.unconv.fits
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904600.unconv.fits
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904606.unconv.fits
    
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.904603.unconv.fits
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.904604.unconv.fits
    
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904601.unconv.fits
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904602.unconv.fits
    gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904605.unconv.fits
    
    
  • sample runs of the kind
    ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.fits 20120507_c1f38 > runppsub_sts103_120507_c1f38.bat
    source runppsub_sts103_120507_c1f38.bat >& runppsub_sts103_120507_c1f38.log &
    
    --> ack.. took too long, been cleaned already and no other observations since. would need to update chips too.., use PSS again since quick test and want cleaned again in near future 
    --> copy over locally if want to keep for rerun diffim tests (not now)
    
    --> for conv, edit the .bat, make WSc and drop the .unconv. for now
    cp runppsub_sts103_120507_c1f38.bat runppsub_sts103_120507_c1f38c.bat
    source runppsub_sts103_120507_c1f38c.bat >& runppsub_sts103_120507_c1f38c.log &
    
    --> look similar, try c1f37 - 
    ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.fits  20120507_c1f37 > runppsub_sts103_120507_c1f37.bat
    source runppsub_sts103_120507_c1f37.bat >& runppsub_sts103_120507_c1f37.log &
    
    --> similar as well. are the exp_time different? yes 20120420 is 20s and IQ is better too, so maybe recent ones are just wacky IQ?
    

Diffim sample 1 (faulted)

Sample diffims for projection 1 faulted in publishing. So this is more of an example of some things of what not to do..

Nights to pick for test -- do test run skycell.1.03 again since edge field and can have overlap. Not clear way to get FWHM other than from gpc1 DB, need to know what chips go into that? No, camProcessedExp sufficient.

  • need to know rough FWHM of refstack and nightly exposures to pick sample -- rough FWHM for refstack in log of run, nightly exposures in DB
    • refstack ~3.94 unconv, ~5.5 conv -- this could be a problem, many warps << than this so stack will be deconvolved
      SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE  comment like "STS%PP1%" group by exp_id;
      
    • 120420 - look ok
    • 120507 ~7.52 (7.9 in cam, wrong way even for cam/warp scaling) -- looked poor, don't use as test
  • need to pick one from 2011 not used in refstack
    SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" group by exp_id;
    
    | 393855 | o5822g0346o |       20 | 2011-09-18 09:25:14 | 24.6354 |    3.87493 | 1.166 | 1.08        | 105.39   | 13.63    | 4.11  | 3.61  | 8444848 | STS.000.GIQ PP1 dither 4 | 
    

2012 sample to test before full field updating -- like from above

SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE  comment like "STS%PP1%" group by exp_id;

| 475121 | o6029g0441o |       30 | 2012-04-12 13:40:09 | 24.5321 |    4.06258 | 1.119 | 1.05        | 315.49   | 21.55    | 4.21  | 3.82  | 8078200 | STS.180.ND PP1 center    | 
...
| 476338 | o6031g0408o |       20 | 2012-04-14 13:34:26 | 24.5537 |    4.27532 | 1.180 | 0.98        | 117.51   | 14.15    | 4.77  | 4.05  | 7485604 | STS.180.GIQ PP1 dither 1 | 
...
| 477549 | o6033g0430o |       30 | 2012-04-16 13:31:05 | 24.5586 |    7.96196 | 1.194 | 1.83        | 154.22   | 18.30    | 8.08  | 7.71  | 4586012 | STS.180.ND PP1 center    | 
...
+good+ | 478479 | o6037g0401o |       20 | 2012-04-20 13:46:26 | 24.5799 |     4.1334 | 1.174 | 1.06        | 76.31    | 12.57    | 4.42  | 3.87  | 8291412 | STS.180.GIQ PP1 dither 1 | 
...
| 479810 | o6040g0441o |       30 | 2012-04-23 13:31:03 |  24.608 |    6.12941 | 1.131 | 1.20        | 147.86   | 17.88    | 6.39  | 5.88  | 5637788 | STS.180.ND PP1 center    | 
...
| 482593 | o6045g0420o |       30 | 2012-04-28 13:37:39 | 24.6132 |    6.26513 | 1.051 | 1.07        | 109.03   | 16.37    | 6.59  | 6.09  | 5474888 | STS.180.ND PP1 center    | 
...
| 483576 | o6049g0286o |       30 | 2012-05-02 13:09:26 | 24.5922 |    4.77424 | 1.088 | 0.94        | 166.69   | 17.89    | 5.10  | 4.62  | 7102152 | STS.180.ND PP1 center    | 
...
-avoid-  | 484578 | o6054g0305o |       30 | 2012-05-07 12:48:56 | 24.5422 |    7.94199 | 1.042 | 0.76        | 297.28   | 23.85    | 8.11  | 7.69  | 3526740 | STS.180.ND PP1 center    | 
...
| 487890 | o6062g0361o |       30 | 2012-05-15 12:00:59 | 24.5415 |    4.14248 | 1.177 | 1.04        | 146.86   | 16.61    | 4.58  | 3.86  | 8390052 | STS.180.ND PP1 center    | 
  • o6054g0305o on disk still, so easy to just test try also even if not great
  • try to at least include 4/14 using a GIQ? 4/20 cleaned but know works, so including 4/14 not necessary.

Run test diffims:

  • currently have to use the conv stack, adding c to script/files
    select uri from warpSkyfile where warp_id=409660 and skycell_id like "%1.03";
    
    ./run_ppsub_night_ws.csh gpc1/STS.refstack.20120510/o5822g0346o.393855/o5822g0346o.393855.wrp.409660.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20110918c > runppsub_sts103_110918_1c.bat
    source runppsub_sts103_110918_1c.bat >& runppsub_sts103_110918_1c.log &
    --> good - 
    
select warpSkyfile.uri from warpSkyfile join warpRun using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_name="o6029g0441o" and warpSkyfile.skycell_id like "%1.03";

./run_ppsub_night_ws.csh  gpc1/STS.nt/2012/04/12//o6029g0441o.475121/o6029g0441o.475121.wrp.396474.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120412c > runppsub_sts103_120412_1c.bat
source runppsub_sts103_120412_1c.bat >& runppsub_sts103_120412_1c.log &
--> bilster cores (undersubs?)
select warpSkyfile.uri from warpSkyfile join warpRun using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where warpSkyfile.skycell_id like "%1.03" and exp_name="o6031g0408o";

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/14//o6031g0408o.476338/o6031g0408o.476338.wrp.397423.skycell.1.03.fits   gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120414c > runppsub_sts103_120414_1c.bat
source runppsub_sts103_120414_1c.bat >& runppsub_sts103_120414_1c.log &
--> blister cores (undersub?)
./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/16//o6033g0430o.477549/o6033g0430o.477549.wrp.398532.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120416c > runppsub_sts103_120416_1c.bat
source runppsub_sts103_120416_1c.bat >& runppsub_sts103_120416_1c.log &
--> large conv oversub?
-- remaining set 

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/23//o6040g0441o.479810/o6040g0441o.479810.wrp.401931.skycell.1.03.fits   gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120423c > runppsub_sts103_120423_1c.bat
source runppsub_sts103_120423_1c.bat >& runppsub_sts103_120423_1c.log &
--> larger conv, maybe slight oversub

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/28//o6045g0420o.482593/o6045g0420o.482593.wrp.405569.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120428c > runppsub_sts103_120428_1c.bat
source runppsub_sts103_120428_1c.bat >& runppsub_sts103_120428_1c.log &
--> larger conv, oversub

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/02//o6049g0286o.483576/o6049g0286o.483576.wrp.406354.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120502c > runppsub_sts103_120502_1c.bat
source runppsub_sts103_120502_1c.bat >& runppsub_sts103_120502_1c.log &
--> slightly larger conv, maybe slight oversub

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/15//o6062g0361o.487890/o6062g0361o.487890.wrp.415613.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120515c > runppsub_sts103_120515_1c.bat
source runppsub_sts103_120515_1c.bat >& runppsub_sts103_120515_1c.log &
--> low conv, blister undersubs
  • looks like 20110918, 20120420, 20120423/20120502 are good samples -- use 4/23, seeing some odd conv/pixel rejected artifacts(?) coming out in diffim but not sure if detected and 30s version
  • may also want to try noconv diffims since STS is supposed to be done within certain IQ constraints

Update full field examples for diffim run:

  • set selected nights up for updates - number of exposures ~16(20s)/16(20s)/12(30s) for 20110918, 20120420, 20120423 respectively
    • label = STS.mehdifftest
    • will need to script mysql call based on what to get the chip/warp_id.
    • detrend different since? may not be unique run given reprocessing for refstack oh well.. manually checked and all unique..
    • 20110918 during magicked era, keep eye out for any issues. may have to just reprocess it - try update since just a test anyways and see. possibly have some clean state issues, so keep eye on -- did update for test case anyways so should work
      select exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" group by exp_name;
      
      chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id     
      warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  
      

  • create specific chip->warp update lists
    mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_110918.bat
    
    mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-20%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120420.bat
    
    mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-23%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120423.bat
    
  • warp updates driving ipp017 nuts.. ~10-20% RAM for 5 runs for total ~75%, cpu_wait and swap overload. killing off 1 at time...
      1    ipp017    BUSY   2993.25 0.0.5.fe34  0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400730 --warp_skyfile_id 34217254 --skycell_id skycell.1.22 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0429o.478506/o6037g0429o.478506.wrp.400730.skycell.1.22 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose 
    
      2    ipp017    RESP   3369.78 0.0.6.269  0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400756 --warp_skyfile_id 34219846 --skycell_id skycell.1.21 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0450o.478527/o6037g0450o.478527.wrp.400756.skycell.1.21 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose 
    
    ...
    
  • odd, didn't fault on czarpage for revert?
    select warpRun.warp_id,state,data_state,fault from warpRun join warpSkyfile using (warp_id) where label like "STS.mopsdiff.mehtest" and data_state="update";
    
    warptool -dbname gpc1 -revertwarped -label STS.mopsdiff.mehtest   -fault 4
    --> ok all finished fine now..
    

Setup full projection 1 diffim run:

  • label = STS.mopsdiff.mehtest
  • data_group = STS.mopsdiff.mehtest
  • workdir = meh/STS.mopsdiff.mehtest/
  • dist_group? no, but need to know about publishing - should run on own using diffim info
  • difftool command for normal WS? -definewarpstack
    • need to add inverse since WS has as false, might as well set diff_mode as well -- not part of definewarpstack since implied
    • what about -exposure? would also seem want -- not part of definewarpstack, only definerun since implied
    • also -input_label becomes -warp_label, -template_label becomes stack_label
    • need -pretend to see output -- can't test until updates are done -- but can intermediate test what to expect, see diffs with other fields so need -skycell_id %.1.%
    • wildcards not work so need to do mysql cmdline batch since didn't restrict batch updates.. this is a problem with the large overlap skycells, more restrictions will be needed in future runs
difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -bothways -set_reduction WARPSTACK -simple -pretend

--> 31 exposures, 81 skycells so ~2500 jobs
--> could try one skycell first then full, will see if will remake it -- use pretend again for full and see -- 

difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -bothways -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple 

--> oops, by running 1 skycell, now thinks is done.. now need to use -rerun and guess cleanup the other one - 
difftool -updaterun -set_state goto_cleaned -set_label goto_cleaned -label STS.mopsdiff.mehtest -dbname gpc1

--> also seems to have set data_group to STS.<date> and not used the -set_data_group, why?

difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -set_reduction WARPSTACK -bothways -simple -rerun

-- mistake -- also left -bothways in diffim run and don't want to do that but finished okay.

  • odd fault 4, revert not fixing -- something in psphot
    242183 	skycell.1.29 	STS.mopsdiff.mehtest 	STS.20110918
    

Distribution to MOPS via publishing -

  • setup -- how to associate label to particular datastore location for this run for IPP-MOPS-TEST-2
    | publishClient           | 
    | publishDone             | 
    | publishRun              | 
    
    select * from publishClient;
    
    | client_id | active | product         | stage    | magicked | workdir                         | comment                                           | name            | output_format |
    +-----------+--------+-----------------+----------+----------+---------------------------------+---------------------------------------------------+-----------------+---------------+
    |         1 |      1 | IPP-MOPS        | diff     |        1 | neb://any/gpc1/IPP-MOPS/        | IPP detections for MOPS                           | NULL            |             2 | 
    |         2 |      0 | MONET           | camera   |        0 | neb://any/gpc1/MONET/           | Translated camera-level detections for Dave Monet | NULL            |             1 | 
    |         3 |      0 | Transients      | diff     |        1 | neb://any/gpc1/IPP-Transients/  | IPP detections of possible transients             | NULL            |             1 | 
    |         4 |      0 | IPP-MOPS-UPDATE | diffphot |        1 | neb://any/gpc1/IPP-MOPS-UPDATE/ | Re-photometered IPP detections for MOPS           | NULL            |             1 | 
    |         5 |      1 | IPP-MOPS-TEST   | diff     |        0 | neb://any/gpc1/IPP-MOPS-TEST    | NULL                                              | NULL            |             2 | 
    |        10 |      0 | IPP-MOPS-TEST   | camera   |        0 | neb://any/gpc1/IPP-MOPS-TEST    | NULL                                              | NULL            |             2 | 
    |        11 |      0 | IPP-MOPS-2      | diff     |        1 | neb://any/gpc1/IPP-MOPS-2/      | IPP detections for MOPS / Output format version 2 | IPP-MOPS-2      |             2 | 
    |        12 |      1 | IPP-MOPS-TEST-2 | diff     |        0 | neb://any/gpc1/IPP-MOPS-TEST-2/ | IPP detections for MOPS / Output format version 2 | IPP-MOPS-TEST-2 |             2 | 
    |        13 |      1 | IPP-MOPS-LAP    | diff     |        1 | neb://any/gpc1/IPP-MOPS-LAP     | IPP detections from Large Area Processing         | NULL            |             2 | 
    
    --> want to put to IPP-MOPS-TEST-2 so client_id=12
    
pubtool -h -definerun

-definerun:
  -set_label      set label
  -client_id      search by client_id     -- should this be set client_id or is search include set to use this client_id??
  -label          set and search by label
  -data_group     search by data_group
  -comment        search by comment
  -dateobs_begin  search for exposures by time (>=)
  -dateobs_end    search for exposures by time (<=)
  -filter         search for filter
  -obs_mode       search by observation mode
  -rerun          Re-run publish?
  -pretend        Pretend to define?
  -simple         use simple output format?
  -limit          limit result set
  • simply just
    pubtool -dbname gpc1 -definerun -label STS.mopsdiff.mehtest -simple -pretend
    
    --> but how to direct only to go to client_id=12?
    
  • survey.pro pantasks module in stdscience
    • will need to add label to publishing + add info to stdscience to tell where to put out too -- this is what needed.. similar for distribution --
    • what has used client_id 12? nothing permanent in the ~ipp/stdscience/input file.
    • format to use in stdscience from survey.pro: <tag> <label> <client_id> <comment>
      • what does <tag> do? allows multiple entries per label and is a unique key. The .ds for example, to be able to push out to other locations being magicked (though not implemented now). Still should be different, so call STS.test
survey.add.publish STS.mopsdiff STS.mopsdiff.mehtest 12 NULL

survey.del.publish STS.mopsdiff
--> to remove if needed
  • careful, STS.nightlyscience already defined to go to 5 -- survey.add.publish STS STS.nightlyscience 5 NULL
  • faults! why is it trying to put to LAP? and not just LAP, but MOPS and MOPS-TEST too..
    system failure for: publish_file.pl --pub_id 422412 --camera GPC1 --workdir neb://any/gpc1/IPP-MOPS-LAP --product IPP-MOPS-LAP --stage diff --stage_id 242192 --output_forma
    t 2 --redirect-output --need-magic --dbname gpc1 --verbose
    
  • run manually and see what is happening
    publish_file.pl --pub_id 422376 --camera GPC1 --workdir neb://any/gpc1/IPP-MOPS-TEST-2/ --product IPP-MOPS-TEST-2 --stage diff --stage_id 242198 --output_format 2 --redirect-output --dbname gpc1 --verbose
    
    neb://any/gpc1/IPP-MOPS-TEST-2/
    --> many things in this directory -- and why reports many not found because cleaned?
    

-- MISTAKE -- should be careful manually running pubtool and loading into survey task

  • fault from specific log -- could this astrom issue be because STS fields or improper run?
    neb://any/gpc1/IPP-MOPS-TEST-2//IPP-MOPS-TEST-2.422401.log
    
    astrom value for negative doesn't match at /home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/publish_file.pl line 374
            main::my_die('astrom value for negative doesn\'t match', 422401, 2) called at /home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/publish_file.pl line 316
            main::diff_check('HASH(0x1a4da90)', 'HASH(0x2a26290)', 'negative') called at /home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/publish_file.pl line 243
    Running [/home/panstarrs/ipp/psconfig/ipp-20120404.lin64/bin/pubtool -add -pub_id 422401 -path_base neb://any/gpc1/IPP-MOPS-TEST-2//IPP-MOPS-TEST-2.422401 -dtime_script 2.99998372793198 -hostname ippc10 -fault 2 -dbname gpc1]...
    
    --> note astrom is sqrt($comp->{sigma_ra_2}**2 + $comp->{sigma_dec_2}**2)
    
    --> the diffim this is for 
    neb://ipp007.0/meh/STS.mopsdiff.mehtest/STS.V4/skycell.1.02/STS.V4.skycell.1.02.WS.dif.242223
    meh/STS.mopsdiff.mehtest/STS.V4/skycell.1.02/STS.V4.skycell.1.02.WS.dif.242223.cmf
    meh/STS.mopsdiff.mehtest/STS.V4/skycell.1.02/STS.V4.skycell.1.02.WS.dif.242223.inv.cmf
    
    /data/ipp007.0/nebulous/92/71/2095812973.meh:STS.mopsdiff.mehtest:STS.V4:skycell.1.02:STS.V4.skycell.1.02.WS.dif.242223.cmf
    /data/ipp007.0/nebulous/14/91/2095815037.meh:STS.mopsdiff.mehtest:STS.V4:skycell.1.02:STS.V4.skycell.1.02.WS.dif.242223.inv.cmf
    
    --> and publish_file.pl called from publish.pro - 
    $run = publish_file.pl --pub_id $PUB_ID --camera $CAMERA --workdir $WORKDIR --product $PRODUCT --stage $STAGE --stage_id $STAGE_ID --output_format $OUTPUT_FORMAT --redirect-output        
                                                 
    system failure for: publish_file.pl --pub_id 422401 --camera GPC1 --workdir neb://any/gpc1/IPP-MOPS-TEST-2/ --product IPP-MOPS-TEST-2 --stage diff --stage_id 242223 --output_format 2 --redirect-output --dbname gpc1 --verbose
    
  • attempt to revert - not going to fix
    pubtool -dbname gpc1 -revert -fault 2 -pub_id 422401
    
  • publish cleanup -- drop all the extra runs trying to publish to the other client_id
    pubtool -dbname gpc1 -updaterun -set_state drop -label STS.mopsdiff.mehtest -client_id 1
    pubtool -dbname gpc1 -updaterun -set_state drop -label STS.mopsdiff.mehtest -client_id 5
    pubtool -dbname gpc1 -updaterun -set_state drop -label STS.mopsdiff.mehtest -client_id 13
    --> leaves 27 with client_id=12 are all valid? look at stage_id
    
    select pub_id,diff_id,diffRun.data_group from publishRun join publishDone using (pub_id) join diffRun  where publishRun.label like "STS.mopsdiff.mehtest" and publishRun.state="new" and diff_id=stage_id;
    --> missing 20110918 oddly, they've ended up in another client_id.. 
    
  • right, don't want bothways since it is a WS -- is there a way to skip those made and just publish the .pos. files? doesn't appear to be so, need to clean up and rerun.
    • this true of other WS runs? OSS is also not bothways, MD same.. so need to rerun and set previous pub_id to drop and clean diffRun
      difftool -updaterun -set_state goto_cleaned -set_label goto_cleaned -label STS.mopsdiff.mehtest -dbname gpc1
      pubtool -dbname gpc1 -updaterun -set_state drop -label STS.mopsdiff.mehtest -client_id 12
      
      difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK -simple -rerun -pretend
      
      --> -rerun needed?  yes and says 164 now, why double?? since only 1 way should be the 43 as the number of warps? 16+16+11 exposures
      --> warp_id 258031 has 94 skycells, but reports skycell_count 162
      --> using -skycell_id to test limit, it is 2 per exposure, how.. what are they?
      
    • why is difftool -rerun seem to be producing double the runs now? the sql query for the -definewarpstack
      SELECT
          exp_id,
          warp_id,
          rawExp.filter,
          warpRun.label,
          warpRun.tess_id,
          warpRun.data_group,
          COUNT(skycell_id) as skycell_count
      FROM warpRun
      JOIN warpSkyfile USING(warp_id)
      JOIN fakeRun USING(fake_id)
      JOIN camRun USING(cam_id)
      JOIN chipRun USING(chip_id)
      JOIN rawExp USING(exp_id)
      LEFT JOIN (
          SELECT DISTINCT
              diffRun.*,
              warp_id,
              exp_id
          FROM diffRun
          JOIN diffInputSkyfile USING(diff_id)
          JOIN warpRun
              ON warpRun.warp_id = diffInputSkyfile.warp1
          JOIN fakeRun USING(fake_id)
          JOIN camRun USING(cam_id)
          JOIN chipRun USING(chip_id)
          WHERE warp1 IS NOT NULL
              AND warpRun.state = 'full'
          -- warp where hook 
       AND (warpRun.label = 'STS.mopsdiff.mehtest')
      ) AS diffExp USING(exp_id, warp_id)
      WHERE
          warpRun.state = 'full'
          AND warpSkyfile.fault = 0
          AND warpSkyfile.quality = 0
          -- warp where hook 
       AND (warpSkyfile.skycell_id = 'skycell.1.03') AND (warpRun.label = 'STS.mopsdiff.mehtest') AND ((warpSkyfile.good_frac > 0.20000000 - 0.00000119))
          -- exp where hook 
          -- diff where hook 
      GROUP BY exp_id, warp_id DESC;
      
    • try single skycell again, -pretend outputs lines but nothing actually queued..
      difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple -rerun -pretend
      
      --> retry again using -bothways
      difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple -rerun -bothways -pretend 
      --> nope.. can redo only be done on full runs and not cleaned ones then?
      
    • look at specific diffim and update (since these were cleaned earlier)
      difftool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest  -diff_id 242223
      
      difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple -rerun -pretend
      --> nope..
      
      difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple -rerun
      --> need to go through this in more detail
      
  • suspect that won't redo for 1 skycell because diff_id same for all and need to do all and should be able to do just for diff_id to rerun? would need to update all of them then
    difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK  -simple -rerun -pretend
    
    difftool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest  -diff_id 242223
    
    select * from diffRun where data_group like "STS.2012%";
    --> yes all cleaned
    
    mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select diff_id,state,data_group from diffRun where data_group like "STS.2012%" and state="cleaned";' | grep -v diff_id | awk '{print "difftool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest  -diff_id  "$1}' > ~/updatediff_sts.mopsdiff.mehtest.bat
    
    --> ipp017 RAM overloaded again now in updating diffims -- manually kill off few and revert
    difftool -revertdiffskyfile -fault 4 -label STS.mopsdiff.mehtest -dbname gpc1 
    
    
    difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK  -simple -rerun -pretend
    
    --> needs workdir, label and data_group seem optional unless want to change 
    difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_reduction WARPSTACK  -simple -rerun 
    
    --> nope, still missing something.. queue up a new test..
    
    

Diffim sample 2 (published)

Rerun of sample diffims for projection 1 and published under IPP-MOPS-TEST-2

Need to update chip->warp of (re)selected nights for testing:

  • like before avoid dates used in refstack
    SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" group by exp_id;
    
    
  • try for sample similar/better FWHM and worse
    SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE  comment like "STS%PP1%" group by exp_id;
    
    | 400812 | o5834g0061o |       30 | 2011-09-30 06:48:31 |  24.643 |     4.9952 | 1.117 | 0.90        | 137.86   | 16.97    | 5.30  | 4.83  | 6509204 | STS.090.ND PP1 center    | 
    | 400819 | o5834g0068o |       30 | 2011-09-30 06:53:41 | 24.6431 |      5.927 | 1.198 | 1.38        | 131.72   | 16.92    | 6.25  | 5.55  | 5929444 | STS.090.ND PP1 center    | 
    | 400826 | o5834g0075o |       30 | 2011-09-30 06:58:46 | 24.6413 |    5.41098 | 1.165 | 1.20        | 123.85   | 16.34    | 5.69  | 5.22  | 6411952 | STS.090.ND PP1 center    | 
    | 400833 | o5834g0082o |       30 | 2011-09-30 07:03:51 | 24.6297 |    6.84521 | 1.120 | 1.16        | 121.33   | 17.11    | 7.07  | 6.75  | 4591816 | STS.090.ND PP1 center    | 
    | 400840 | o5834g0089o |       30 | 2011-09-30 07:08:53 | 24.6256 |    6.46738 | 1.069 | 0.84        | 126.92   | 17.59    | 6.74  | 6.23  | 4562012 | STS.090.ND PP1 center    | 
    | 400847 | o5834g0096o |       30 | 2011-09-30 07:13:55 | 24.6386 |    6.42677 | 1.196 | 1.36        | 125.30   | 16.89    | 6.75  | 6.22  | 5325836 | STS.090.ND PP1 center    | 
    | 400854 | o5834g0103o |       30 | 2011-09-30 07:19:00 | 24.6292 |    6.37093 | 1.103 | 0.88        | 123.32   | 17.25    | 6.67  | 6.13  | 4688216 | STS.090.ND PP1 center    | 
    | 400861 | o5834g0110o |       30 | 2011-09-30 07:24:07 | 24.6316 |    6.62746 | 1.180 | 1.04        | 131.42   | 17.69    | 6.82  | 6.42  | 4652948 | STS.090.ND PP1 center    | 
    | 400868 | o5834g0117o |       30 | 2011-09-30 07:29:12 | 24.6365 |    6.15409 | 1.085 | 1.05        | 128.51   | 17.22    | 6.53  | 5.92  | 5107028 | STS.090.ND PP1 center    | 
    | 400875 | o5834g0124o |       30 | 2011-09-30 07:34:21 |  24.641 |     5.2498 | 1.127 | 0.91        | 123.12   | 16.49    | 5.56  | 5.03  | 6108640 | STS.090.ND PP1 center    | 
    | 400883 | o5834g0131o |       30 | 2011-09-30 07:39:28 | 24.6409 |    5.78907 | 1.154 | 1.19        | 127.42   | 16.82    | 6.00  | 5.65  | 5809232 | STS.090.ND PP1 center    | 
    | 400889 | o5834g0138o |       30 | 2011-09-30 07:44:33 | 24.6389 |    6.12522 | 1.064 | 1.12        | 125.19   | 17.02    | 6.48  | 5.96  | 5185056 | STS.090.ND PP1 center    | 
    
    | 475121 | o6029g0441o |       30 | 2012-04-12 13:40:09 | 24.5321 |    4.06258 | 1.119 | 1.05        | 315.49   | 21.55    | 4.21  | 3.82  | 8078200 | STS.180.ND PP1 center    | 
    | 475128 | o6029g0448o |       30 | 2012-04-12 13:45:11 | 24.5325 |    3.91594 | 1.132 | 0.95        | 303.82   | 21.37    | 4.41  | 3.67  | 8132400 | STS.180.ND PP1 center    | 
    | 475135 | o6029g0455o |       30 | 2012-04-12 13:50:14 | 24.5375 |    4.06348 | 1.129 | 0.97        | 307.70   | 21.58    | 4.48  | 3.88  | 7907516 | STS.180.ND PP1 center    | 
    | 475141 | o6029g0462o |       30 | 2012-04-12 13:55:16 | 24.5386 |     4.0157 | 1.135 | 0.97        | 301.07   | 21.32    | 4.55  | 3.78  | 8000044 | STS.180.ND PP1 center    | 
    | 475149 | o6029g0469o |       30 | 2012-04-12 14:00:20 | 24.5421 |    4.29222 | 1.196 | 1.03        | 300.51   | 21.44    | 4.66  | 4.07  | 7669568 | STS.180.ND PP1 center    | 
    | 475156 | o6029g0476o |       30 | 2012-04-12 14:05:25 | 24.5459 |    4.32599 | 1.163 | 1.03        | 304.46   | 21.61    | 4.80  | 4.07  | 7576116 | STS.180.ND PP1 center    | 
    | 475163 | o6029g0483o |       30 | 2012-04-12 14:10:27 | 24.5445 |    3.90778 | 1.119 | 0.92        | 308.26   | 21.57    | 4.16  | 3.71  | 8114964 | STS.180.ND PP1 center    | 
    | 475170 | o6029g0490o |       30 | 2012-04-12 14:15:31 | 24.5486 |    4.25772 | 1.135 | 0.96        | 295.86   | 21.39    | 4.70  | 4.02  | 7605516 | STS.180.ND PP1 center    | 
    | 475177 | o6029g0497o |       30 | 2012-04-12 14:20:35 | 24.5477 |    4.11544 | 1.203 | 1.09        | 378.52   | 23.53    | 4.35  | 3.77  | 7926572 | STS.180.ND PP1 center    | 
    | 475184 | o6029g0504o |       30 | 2012-04-12 14:25:37 | 24.5461 |    3.88865 | 1.175 | 1.00        | 293.10   | 21.07    | 4.35  | 3.62  | 8464032 | STS.180.ND PP1 center    | 
    | 475191 | o6029g0511o |       30 | 2012-04-12 14:30:41 |  24.552 |    4.19097 | 1.172 | 1.10        | 277.41   | 20.75    | 4.38  | 3.89  | 8060959 | STS.180.ND PP1 center    | 
    | 475198 | o6029g0518o |       30 | 2012-04-12 14:35:47 | 24.5526 |    4.09804 | 1.174 | 1.05        | 280.17   | 20.83    | 4.36  | 3.79  | 8122168 | STS.180.ND PP1 center    | 
    
    | 476338 | o6031g0408o |       20 | 2012-04-14 13:34:26 | 24.5537 |    4.27532 | 1.180 | 0.98        | 117.51   | 14.15    | 4.76  | 4.05  | 7485604 | STS.180.GIQ PP1 dither 1 | 
    | 476345 | o6031g0415o |       20 | 2012-04-14 13:38:20 | 24.5533 |    4.17247 | 1.184 | 1.01        | 117.49   | 14.21    | 4.67  | 3.89  | 7636200 | STS.180.GIQ PP1 dither 2 | 
    | 476352 | o6031g0422o |       20 | 2012-04-14 13:42:15 |  24.552 |    4.09387 | 1.144 | 0.94        | 118.19   | 14.27    | 4.45  | 3.90  | 7637616 | STS.180.GIQ PP1 dither 3 | 
    | 476360 | o6031g0429o |       20 | 2012-04-14 13:46:09 |  24.556 |    4.31094 | 1.188 | 1.01        | 118.09   | 14.37    | 4.70  | 4.05  | 7378484 | STS.180.GIQ PP1 dither 4 | 
    | 476366 | o6031g0436o |       20 | 2012-04-14 13:50:05 | 24.5551 |    4.17617 | 1.154 | 0.94        | 119.08   | 14.43    | 4.59  | 3.98  | 7443824 | STS.180.GIQ PP1 dither 1 | 
    | 476373 | o6031g0443o |       20 | 2012-04-14 13:53:56 | 24.5525 |    3.82054 | 1.166 | 0.84        | 121.42   | 14.46    | 4.30  | 3.63  | 7964096 | STS.180.GIQ PP1 dither 2 | 
    | 476380 | o6031g0450o |       20 | 2012-04-14 13:57:48 | 24.5545 |    4.07024 | 1.146 | 0.98        | 123.89   | 14.60    | 4.58  | 3.78  | 7718292 | STS.180.GIQ PP1 dither 3 | 
    | 476389 | o6031g0457o |       20 | 2012-04-14 14:01:41 | 24.5365 |     4.0109 | 1.144 | 0.93        | 133.13   | 15.08    | 4.49  | 3.79  | 7568984 | STS.180.GIQ PP1 dither 4 | 
    | 476397 | o6031g0464o |       20 | 2012-04-14 14:08:15 | 24.5432 |     3.9738 | 1.177 | 0.95        | 131.38   | 14.95    | 4.33  | 3.76  | 7835456 | STS.180.GIQ PP1 dither 1 | 
    | 476404 | o6031g0471o |       20 | 2012-04-14 14:12:12 | 24.5149 |    3.94952 | 1.206 | 0.97        | 138.51   | 15.23    | 4.44  | 3.68  | 7840492 | STS.180.GIQ PP1 dither 2 | 
    | 476411 | o6031g0478o |       20 | 2012-04-14 14:16:12 | 24.5435 |    3.80814 | 1.181 | 0.90        | 126.95   | 14.66    | 4.29  | 3.58  | 8082288 | STS.180.GIQ PP1 dither 3 | 
    | 476418 | o6031g0485o |       20 | 2012-04-14 14:20:07 | 24.5456 |    3.77643 | 1.189 | 0.92        | 128.77   | 14.71    | 4.27  | 3.54  | 8231892 | STS.180.GIQ PP1 dither 4 | 
    | 476425 | o6031g0492o |       20 | 2012-04-14 14:24:14 | 24.5303 |     4.1106 | 1.144 | 0.98        | 139.08   | 15.35    | 4.59  | 3.83  | 7461452 | STS.180.GIQ PP1 dither 1 | 
    | 476432 | o6031g0499o |       20 | 2012-04-14 14:28:10 | 24.5157 |     3.9894 | 1.202 | 0.93        | 143.64   | 15.48    | 4.33  | 3.72  | 7628256 | STS.180.GIQ PP1 dither 2 | 
    | 476437 | o6031g0506o |       20 | 2012-04-14 14:32:10 | 24.4737 |    4.25366 | 1.151 | 0.99        | 157.29   | 15.95    | 4.71  | 3.96  | 7066552 | STS.180.GIQ PP1 dither 3 | 
    | 476445 | o6031g0513o |       20 | 2012-04-14 14:36:14 | 24.5465 |    3.89488 | 1.188 | 0.89        | 139.57   | 15.33    | 4.43  | 3.60  | 7892495 | STS.180.GIQ PP1 dither 4 | 
    | 476453 | o6031g0520o |       20 | 2012-04-14 14:40:10 |  24.555 |    3.64675 | 1.198 | 0.87        | 136.07   | 15.05    | 4.17  | 3.37  | 8450820 | STS.180.GIQ PP1 dither 1 | 
    | 476460 | o6031g0527o |       20 | 2012-04-14 14:44:06 | 24.4858 |    3.62349 | 1.210 | 0.85        | 166.07   | 16.03    | 4.25  | 3.28  | 8069508 | STS.180.GIQ PP1 dither 2 | 
    
    --> only 1 from 9/19 used so use that date? 9/11 better and none used from it - or 9/30 to have sample of worse input (yes)
    --> so 4/14 after 4/12 for close in time observ and still pretty good -- would prefer one not as good.. 
    
  • 20120412:
    mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-12%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiffv2.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiffv2.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiffv2.mehtest_120412.bat
    ...
    
    difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiffv2.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffv2.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffv2.mehtest -set_data_group STS.mopsdiffv2.mehtest  -set_reduction WARPSTACK -simple -pretend
    
    --> sample found, run w/o -pretend and finishes
    

Setup for publishing test:

  • to check but dont't run as would get sent to 5 as well as 12..
    pubtool -dbname gpc1 -definerun -label STS.mopsdiffv2.mehtest -simple -pretend
    
    --> survey tasks ipp-localops.lin64/share/pantasks/modules/survey.pro calls
    pubtool -definerun -label $label -client_id $client_id
    
    pubtool -dbname gpc1 -definerun -label STS.mopsdiffv2.mehtest -client_id 12 -simple -pretend
    --> successful and only put to 12, and just .pos.mops detections similar to MD with diff_mode 2 has for WS so compared to OSS now using SWEETSPOT (noconv) and diff_mode 1 with .pos and .neg.
    

Continue setup next 2012 sample and 2011 sample then

  • now update 9/30/11 and 20120414 nights chips->warps
    mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-30%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiffv2.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiffv2.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiffv2.mehtest_110930.bat
    
    mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-14%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiffv2.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiffv2.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiffv2.mehtest_120414.bat
    
  • pretend test run diffims
    difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiffv2.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffv2.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffv2.mehtest -set_data_group STS.mopsdiffv2.mehtest  -set_reduction WARPSTACK -simple -pretend
    
  • 9/30 only one exposure is going to go through diffs for some reason
    select * from diffRun where data_group like "STS.20110930";
    
    --> diffims make last year for this night, but were WW diffims. try with -rerun 
    
    difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiffv2.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffv2.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffv2.mehtest -set_data_group STS.mopsdiffv2.mehtest  -set_reduction WARPSTACK -simple -rerun -pretend
    
    --> yes, now seems to be there, runs all -- seems rerun works to run new type of diffim but not rerun old one as in diffim test 1 above?
    
    pubtool -dbname gpc1 -definerun -label STS.mopsdiffv2.mehtest -client_id 12 -simple -pretend
    
    --> worked, removed the -pretend and out onto datastore now for MOPS to look at.  
    

Diffim summary:

skycell FWHM input/ref Ndet comment
1.03

MOPS feedback

  • MOPS still looking at

Automated setup for MOPS

Generally already setup for MD in the survey tasks.

  • stdscience pantasks (not implemented yet) -- use name STS.mopsdiff?
    survey.add.publish STS.mopsdiff STS.nightlyscience 12 NULL
    add.label STS.nightlyscience
    
    --> check or remove
    survey.show.publish 
    survey.del.publish STS.mopsdiff
    
    
  • publishing pantasks (already exists)
    add.label STS.nightlyscience
    

Attachments