IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:STS.refstack.20120510

Version 52 (modified by Mark Huber, 14 years ago) ( diff )

--

Summary

  1. Summary
  2. STS observation and tessellation summary
  3. Refstack
    1. Chip->Warp processing
    2. Stack processing
    3. clean-up
  4. Test Diffim samples
    1. manual test set
    2. Diffim sample 1 (faulted)
    3. Diffim sample 2 (published)
  5. MOPS feedback

MOPS would like a refstack for the STS fields in order to get warp-stack diffims. The previous STS.refstack.20110613,20110615 wasn't usable as it was destroyed by magic. STS.20120510 was made using a similar number of inputs from the best exposures from 2011. Not pretty but hopefully sufficient.

Status:

  • 5/8: past refstack from 6/2011 are cleaned, magic destroyed them. no magic problem anymore..
  • 5/10: test samples and selected exposures processed for chip->warp
  • 5/13: test samples for refstack finished, full refstack started
  • 5/15: refstack finished and checked. looking at diffim samples and how to publish detections to MOPS
  • 5/18: test diffim sample 1 using updated warps from past nights in 2011, early 2012
  • 5/20: test diffims finished but fault in publishing, tracing out.
  • 5/27: test sample 1 a mess, restart with test sample 2
  • 5/29: test sample 2 diffim detections published to IPP-MOPS-TEST-2, waiting for feedback if will work

Things to do/fix/test:

  • convolved versus unconvolved stack diffims
  • target PSF limiting
  • skycell overlap in refstack and diffims - want to restrict inputs PP1 into stack skycell.1.xx only etc
    • can remake with clean/updates for each field, but maybe MOPS won't care or maybe with conv stacks isn't a big issue?
    • tricky to do in an automated way. giving each field a different refstack processing label is easiest and do on update? will still try to populate other regions though, so need to restrict to particular region with -skycell_id. won't need a sub-label for each 7 to prevent the rerun headache and to only run against proper field since matched by skycell in diffim.

*

STS observation and tessellation summary

Sample rawExp.comment information for the type of observation 

STS.180.ND PP1 center
STS.000.GIQ PP1 dither 1 
STS.180.NDGIQ PP5 center
STS.180.NOM PP3 dither 3
  • 000,090,180,270 -- rotator to keep rough PA on field over season
  • GIQ -- shorter exp_time (20s versus 30s nominal) to not saturate more sources in better seeing conditions
  • ND -- no dither so always with center and typically with 30s, occasional NDGIQ
  • NOM -- nominal IQ?
  • PP# -- 1-7, 7 pointings
  • dither 1-4 -- dither shifts (by ?) and typically with GIQ

Overlay of camera SMFs for PP1-PP7 (colored) on top of the projection skycells defined at ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_Tessellations.

7 projections and their skycell overlap (STS.V4, 0.25"/pixel platescale)

Approximate pattern, E left, N up: 

 1 (orange)  2 (red)     
3 (blue)  4 (green)  5 (violet)   
 6 (indigo)  7 (yellow)

To generate SMF sample for position and look at dither pattern: 

mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select chipRun.workdir,rawExp.exp_tag,rawExp.exp_name from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and  camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%center%" group by rawExp.exp_id;' | grep -v "workdir" | uniq -f2 | awk '{ print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -p " $1"/"$2"/o%smf" }' | tcsh | awk '{print "imbox "$1}' >& STSsub.1_i_imboxsmf_c.lst

mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select chipRun.workdir,rawExp.exp_tag,rawExp.exp_name from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and  camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%dither 1%" group by rawExp.exp_id;' | grep -v "workdir" | uniq -f2 | awk '{ print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -p " $1"/"$2"/o%smf" }' | tcsh | awk '{print "imbox "$1}' >& STSsub.1_i_imboxsmf_d1.lst

...

dvo -D CATDIR STS.V4 

$KAPA = kapa -noX

region 297.5 17.1 4.2  
style -c black
image
style -c orange 
input STSsub.1_i_imboxsmf.lst

style -c red
input STSsub.2_i_imboxsmf.lst

...

There is significant overlap of projections (and effectively skycells). May cause issue with general input into a refstack skycell with wildly different IQ?

  • some will have direct overlapping pixels into the skycell, for example PP4 can provide chips to make skycell.1.03 that could then end up in a stack for that skycell and vice versa.
  • these could fill in different areas of the skycell with very different PSF functions, but the skycells are also populated by different chips and IQ from rotations as well.
  • will probably want a reasonable input sample, but suspect target PSF will behave oddly for convolved stack
  • using the processed warps, skycell.1.03 looks like 
    select exp_name,dateobs,warpRun.warp_id,warpSkyCellMap.skycell_id,warpSkyCellMap.class_id,comment from warpRun join warpSkyCellMap using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where warpRun.data_group like "STS.refstack.2012%" and skycell_id like "%.1.03%" order by exp_name;

| exp_name    | dateobs             | warp_id | skycell_id   | class_id | comment                  |
+-------------+---------------------+---------+--------------+----------+--------------------------+
| o5705g0379o | 2011-05-24 12:08:55 |  410461 | skycell.1.03 | XY67     | STS.270.GIQ PP3 dither 1 | 
| o5705g0379o | 2011-05-24 12:08:55 |  410461 | skycell.1.03 | XY57     | STS.270.GIQ PP3 dither 1 | 
| o5705g0386o | 2011-05-24 12:13:00 |  410467 | skycell.1.03 | XY57     | STS.270.GIQ PP3 dither 2 | 
| o5705g0386o | 2011-05-24 12:13:00 |  410467 | skycell.1.03 | XY67     | STS.270.GIQ PP3 dither 2 | 
| o5705g0387o | 2011-05-24 12:13:35 |  410475 | skycell.1.03 | XY71     | STS.270.GIQ PP4 dither 3 | 
| o5705g0387o | 2011-05-24 12:13:35 |  410475 | skycell.1.03 | XY72     | STS.270.GIQ PP4 dither 3 | 
| o5705g0393o | 2011-05-24 12:17:04 |  410478 | skycell.1.03 | XY67     | STS.270.GIQ PP3 dither 3 | 
| o5705g0393o | 2011-05-24 12:17:04 |  410478 | skycell.1.03 | XY57     | STS.270.GIQ PP3 dither 3 | 
| o5705g0394o | 2011-05-24 12:17:38 |  410469 | skycell.1.03 | XY72     | STS.270.GIQ PP4 dither 4 | 
| o5705g0394o | 2011-05-24 12:17:38 |  410469 | skycell.1.03 | XY71     | STS.270.GIQ PP4 dither 4 | 
| o5705g0395o | 2011-05-24 12:18:14 |  409630 | skycell.1.03 | XY02     | STS.270.GIQ PP1 dither 4 | 
| o5705g0395o | 2011-05-24 12:18:14 |  409630 | skycell.1.03 | XY03     | STS.270.GIQ PP1 dither 4 | 
| o5705g0395o | 2011-05-24 12:18:14 |  409630 | skycell.1.03 | XY04     | STS.270.GIQ PP1 dither 4 | 

| o5719g0324o | 2011-06-07 11:27:59 |  410516 | skycell.1.03 | XY10     | STS.180.GIQ PP4 dither 1 | 
| o5719g0324o | 2011-06-07 11:27:59 |  410516 | skycell.1.03 | XY20     | STS.180.GIQ PP4 dither 1 | 
| o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY36     | STS.180.GIQ PP1 dither 1 | 
| o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY37     | STS.180.GIQ PP1 dither 1 | 
| o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY46     | STS.180.GIQ PP1 dither 1 | 
| o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY47     | STS.180.GIQ PP1 dither 1 | 
| o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY26     | STS.180.GIQ PP1 dither 1 | 
| o5719g0325o | 2011-06-07 11:28:32 |  409632 | skycell.1.03 | XY27     | STS.180.GIQ PP1 dither 1 | 
| o5719g0330o | 2011-06-07 11:31:21 |  410528 | skycell.1.03 | XY71     | STS.180.GIQ PP3 dither 1 | 
| o5719g0330o | 2011-06-07 11:31:21 |  410528 | skycell.1.03 | XY72     | STS.180.GIQ PP3 dither 1 | 

| o5767g0396o | 2011-07-25 11:59:08 |  410618 | skycell.1.03 | XY57     | STS.000.ND PP4 center    | 
| o5767g0396o | 2011-07-25 11:59:08 |  410618 | skycell.1.03 | XY67     | STS.000.ND PP4 center    | 
| o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY50     | STS.000.ND PP1 center    | 
| o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY51     | STS.000.ND PP1 center    | 
| o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY30     | STS.000.ND PP1 center    | 
| o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY31     | STS.000.ND PP1 center    | 
| o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY40     | STS.000.ND PP1 center    | 
| o5767g0397o | 2011-07-25 11:59:50 |  409645 | skycell.1.03 | XY41     | STS.000.ND PP1 center    | 
| o5767g0402o | 2011-07-25 12:03:25 |  410614 | skycell.1.03 | XY05     | STS.000.ND PP3 center    | 
| o5767g0402o | 2011-07-25 12:03:25 |  410614 | skycell.1.03 | XY06     | STS.000.ND PP3 center    |

Will also be directly overlapping skycells for diffims with at least slightly different refstacks:

skycell.<PP>.xx possible overlap skycell.<PP>.xx
1.03 4.71,4.72; 3.57, 3.67

Refstack

Reference stack using data from 2011 only, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, large elongation.

Chip->Warp processing

Only reprocessing a selected sample to use in the refstack only --

  • label STS.refstack.20120510
  • data_group STS.refstack.20120510
  • dist_group NULL
  • workdir: neb://@HOST@.0/gpc1/STS.refstack.20120510

Test sample: 

chiptool -definebyquery -set_label STS.mehtest -set_workdir neb://@HOST@.0/meh/STS.mehtest -set_data_group STS.mehtest -set_dist_group NULL -set_tess_id STS.V4 -set_end_stage warp -dbname gpc1 -dateobs_begin 2011-05-24 -dateobs_end 2011-05-25 -filter i% -comment STS% -obs_mode STS -simple -pretend >& teststschip.log

--> 5/24 has 84 exposures, so need more restrictive for test -- do o5705g0373o

chiptool -definebyquery -set_label STS.mehtest -set_workdir neb://@HOST@.0/meh/STS.mehtest -set_data_group STS.mehtest -set_dist_group NULL -set_tess_id STS.V4 -set_end_stage warp -set_reduction STS_DATASET -dbname gpc1 -exp_name o5705g0373o -simple -pretend >& teststschip_o5705g0373o.log

-- test run - STS.mehtest - don't see corner unglow in camera image, dynamic range such that not see strongly?
neb://@HOST@.0/meh/STS.mehtest/o5705g0373o.340508

--> see some patterns in warp background for sample -- significant? see
meh/STS.mehtest/o5705g0373o.340508/o5705g0373o.340508.wrp.408756.skycell.1.04.fits

Determine cuts for exposure sample to reprocess, pick of order 50 exposure sample:

  • time range 
    | o5666g0560o | 324290 | STS      | STS      |       30 | 2011-04-15 14:01:18 | i.00000 | STS PP6 test visit 2     | 
| o5666g0561o | 324291 | STS      | STS      |       30 | 2011-04-15 14:02:10 | i.00000 | STS PP7 test visit 2     | 
| o5667g0517o | 324909 | RINGS.V3 | STS      |       30 | 2011-04-16 14:39:15 | i.00000 | STS PP8 test visit 1     | 
| o5667g0518o | 324910 | RINGS.V3 | STS      |       30 | 2011-04-16 14:39:58 | i.00000 | STS PP9 test visit 1     | 
..
| o5681g0558o | 332242 | RINGS.V3 | STS      |       30 | 2011-04-30 15:02:10 | i.00000 | STS PP6 test visit 2     | 
| o5681g0559o | 332243 | RINGS.V3 | STS      |       30 | 2011-04-30 15:03:32 | i.00000 | STS PP3 test visit 2     | 
| c5681g0007d | 332244 | NULL     | STS      |       30 | 2011-04-30 15:05:39 | i.00000 | STS PP3 test visit 2     | 
| o5695g0143o | 335717 | NULL     | STS      |       30 | 2011-05-14 11:06:19 | i.00000 | STS.180.NOM PP4 center   | 
..
| o5695g0144o | 335718 | NULL     | STS      |       30 | 2011-05-14 11:07:04 | i.00000 | STS.180.NOM PP1 center   | 
| o5695g0225o | 335799 | NULL     | STS      |       30 | 2011-05-14 12:06:25 | i.00000 | STS.180.NOM PP6 dither 5 | 
| o5695g0226o | 335800 | NULL     | STS      |       30 | 2011-05-14 12:07:09 | i.00000 | STS.180.NOM PP3 dither 5 | 
| o5705g0373o | 340508 | STS.V4   | STS      |       20 | 2011-05-24 12:05:27 | i.00000 | STS.270.GIQ PP4 dither 1 |  -- talking with Bill, looks like when to start -- 
| o5705g0374o | 340510 | STS.V4   | STS      |       20 | 2011-05-24 12:06:00 | i.00000 | STS.270.GIQ PP1 dither 1 |
  • camera FWHM, ZPT, background cut
  • dither content check 
    SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE obs_mode="STS" and dateobs>"2011-05-23" and dateobs<"2012-01" and  fwhm_major<3.8 and zpt_obs>24.6 and camProcessedExp.bg<100.0 and comment like "%PP1%" group by exp_id;
...
  • summary 
    PP1 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 42, ~half center half dither (16,7,5,6,8)
PP2 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 47, ~half center half dither
PP3 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 53, ~half center half dither 
PP4 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 50, ~half center half dither 
PP5 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 52, ~half center half dither 
PP6 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 49, ~half center half dither (14,8,8,10,9)
PP7 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 49, ~half center half dither (18,9,5,10,7)

Run PP1 set first with selected exposures and perl script to make reftest stacks: 

mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select rawExp.exp_id from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and  camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%" group by rawExp.exp_id;' | grep -v "exp_id" 

perl sts.v4.refstackexp.20120510_pp1.chip > run_pp1.chip.bat
source run_pp1.chip.bat >& run_pp1.chip.log

Once reftest check out, run remaining PP2-7 exposures: 

mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select rawExp.exp_id from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and  camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment not like "%PP1%" group by rawExp.exp_id;' | grep -v "exp_id" 

perl sts.v4.refstackexp.20120510_pp2-7.chip > run_pp2-7.chip.bat
source run_pp2-7.chip.bat >& run_pp2-7.chip.log

--> set priority with lap @200
labeltool -definelabel -set_label STS.refstack.20120510 -set_priority 200 -dbname gpc1
labeltool -updatelabel -label STS.refstack.20120510 -set_priority 190 -dbname gpc1

Stack processing

  • label: STS.refstack.20120510
  • data_group: STS.refstack.20120510
  • workdir: neb://@HOST@.0/gpc1/STS.reftest.20120510
  • reduction: PS1_REFERENCE 
        OUTPUT.NOCOMP       BOOL    TRUE
    OUTPUT.LOGFLUX      BOOL    FALSE
    OUTPUT.REPLICATE    BOOL    TRUE

--> compared to other reduction recipes -- cut on input PSF in STACK_DEEP may be useful but can apply in selecting exposures for stacking (not 100% effective however).
STACK_THREEPI  METADATA
    PSF.INPUT.MAX   F32     12.0
    OUTPUT.NOCOMP   BOOL    FALSE
    OUTPUT.LOGFLUX  BOOL    TRUE
STACK_DEEP    METADATA
    OUTPUT.NOCOMP   BOOL    TRUE
    OUTPUT.LOGFLUX  BOOL    FALSE
    PSF.INPUT.MAX           F32  7.5
    PSF.INPUT.CLIP.NSIGMA   F32  5.0

Test stack (reftest) sample like made for MD skycell.055 with different number of input warps and FWHM cut using reprocessed sample (background,ZPT cut)

  • label: STS.reftest.20120510
  • data_group: STS.reftest.20120510.c#f<FWHM cut> where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp
    • c0 additional FWHM cut only
    • c1 additional FWHM cut with elong<1.2
  • skycell.1.03 chosen being on outer edge without large masking fraction or chunks and for comparison in possible overlap tests with other projection (PP4)
    • overlap skycell would be useful to see how much it becomes degraded when added both in overlap and in different spatial regions. for now that would need a corner.
    • in projection 1, the corner 1.01 is like 1.66. look at maskfrac and also Ndetections on the stack 
      select warpRun.warp_id,warpSkyfile.skycell_id,warpSkyfile.maskfrac_npix,warpSkyfile.maskfrac_static,warpSkyfile.maskfrac_dynamic,warpSkyfile.maskfrac_advisory,warpSkyfile.good_frac from warpRun join warpSkyfile using (warp_id) where label like "STS.refstack.20120510" and skycell_id like "%1.%" limit 10;

| warp_id | skycell_id   | maskfrac_npix | maskfrac_static | maskfrac_dynamic | maskfrac_advisory | good_frac |
+---------+--------------+---------------+-----------------+------------------+-------------------+-----------+
|  409624 | skycell.1.00 |      37519776 |         0.07579 |            8e-06 |          0.006282 |  0.044258 | 
|  409624 | skycell.1.01 |      37519776 |         0.27474 |          6.3e-05 |          0.094433 |   0.46227 | 
|  409624 | skycell.1.02 |      37519776 |        0.134148 |          8.4e-05 |          0.129784 |  0.571464 | 
|  409624 | skycell.1.03 |      37519776 |        0.067499 |          9.8e-05 |          0.144948 |  0.631332 | 
|  409624 | skycell.1.04 |      37519776 |        0.101891 |         0.000159 |          0.140317 |  0.595121 | 
|  409624 | skycell.1.05 |      37519776 |        0.140644 |         0.000259 |          0.158277 |  0.565971 | 
|  409624 | skycell.1.06 |      37519776 |        0.284199 |         0.000173 |          0.115242 |  0.450413 | 
|  409624 | skycell.1.07 |      37519776 |        0.076937 |          1.3e-05 |          0.010954 |  0.043497 | 
|  409624 | skycell.1.08 |      37519776 |        0.336794 |            8e-05 |          0.087179 |  0.445569 | 
|  409624 | skycell.1.09 |      37519776 |        0.370852 |         0.000146 |          0.119304 |  0.519711 | 

---> want to avoid the corner then and do bottom edge 1.04,1.05 where good_frac ~50%, 1.05 has larger static mask to watch in rotations. 1.03 also should work and seems more stable static and good maskfrac.
  • initial test sample: 
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c1f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.03  >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell103.log

...

--> sample unconvolved images -- doesn't seem to be fully filling in regions/gaps, see input details.
gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899917.unconv.num.fits -- c1f39 (max inputs 36)
gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.num.fits -- c1f38 (max inputs 23)
gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.num.fits -- c1f37 (max inputs 12)
gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899920.unconv.num.fits -- c0f45 (max inputs 44)

--> input stats for likely FWHM cut, 
SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.reftest.20120510.c1f38" group by exp_id;

| exp_id | exp_time | dateobs             | zpt_obs | zpt_stdev | fwhm_major | elong | iq_elong | fwhm_iq_maj | bg     | bg_stdev | uq   | lq   | n_stars | comment                  |
+--------+----------+---------------------+---------+-----------+------------+-------+----------+-------------+--------+----------+------+------+---------+--------------------------+
| 340540 |       20 | 2011-05-24 12:22:25 | 24.5549 |    0.0259 |    3.69691 | 1.188 | 1.083    | 0.92        | 91.27  | 14.01    | 3.98 | 3.46 | 8648132 | STS.270.GIQ PP1 dither 1 | 
| 340558 |       20 | 2011-05-24 12:34:54 | 24.5559 |  0.026682 |     3.7342 | 1.148 | 1.073    | 0.86        | 88.49  | 14.04    | 4.04 | 3.45 | 8393568 | STS.270.GIQ PP1 dither 4 | 
| 340567 |       20 | 2011-05-24 12:39:02 | 24.5563 |  0.025387 |    3.62252 | 1.130 | 1.059    | 0.84        | 88.18  | 13.78    | 3.89 | 3.29 | 8588336 | STS.270.GIQ PP1 dither 1 | 
| 340573 |       20 | 2011-05-24 12:43:16 | 24.5591 |  0.025946 |    3.64969 | 1.127 | 1.050    | 0.82        | 87.11  | 13.21    | 4.05 | 3.38 | 8380448 | STS.270.GIQ PP1 dither 2 | 
| 344556 |       20 | 2011-06-07 11:28:32 | 24.4608 |  0.029333 |    3.59708 | 1.165 | 1.079    | 0.88        | 100.50 | 13.30    | 4.04 | 3.39 | 8487768 | STS.180.GIQ PP1 dither 1 | 
| 344562 |       20 | 2011-06-07 11:32:30 | 24.5164 |  0.024937 |    3.59372 | 1.150 | 1.069    | 0.87        | 97.07  | 13.32    | 3.99 | 3.37 | 8638928 | STS.180.GIQ PP1 dither 2 | 
| 344569 |       20 | 2011-06-07 11:36:21 | 24.5313 |  0.024183 |    3.69415 | 1.168 | 1.085    | 0.93        | 94.56  | 13.63    | 4.03 | 3.47 | 8641948 | STS.180.GIQ PP1 dither 3 | 
| 344575 |       20 | 2011-06-07 11:40:18 | 24.5333 |  0.027669 |    3.66644 | 1.154 | 1.070    | 0.91        | 94.17  | 13.25    | 3.99 | 3.39 | 8611680 | STS.180.GIQ PP1 dither 4 | 
| 344583 |       20 | 2011-06-07 11:44:12 | 24.4983 |  0.040863 |    3.72526 | 1.159 | 1.088    | 0.92        | 96.11  | 13.24    | 4.14 | 3.51 | 8467340 | STS.180.GIQ PP1 dither 1 | 
| 344589 |       20 | 2011-06-07 11:48:13 | 24.4937 |  0.046644 |    3.72833 | 1.136 | 1.068    | 0.90        | 95.45  | 13.70    | 4.02 | 3.51 | 8327116 | STS.180.GIQ PP1 dither 2 | 
| 344597 |       20 | 2011-06-07 11:52:11 | 24.4935 |  0.031675 |    3.71614 | 1.141 | 1.066    | 0.87        | 92.70  | 13.54    | 4.13 | 3.51 | 8277828 | STS.180.GIQ PP1 dither 3 | 
| 344605 |       20 | 2011-06-07 11:56:13 | 24.5409 |  0.025897 |    3.74193 | 1.160 | 1.063    | 0.90        | 89.26  | 13.15    | 4.15 | 3.54 | 8427112 | STS.180.GIQ PP1 dither 4 | 
| 367982 |       30 | 2011-07-25 11:59:50 | 24.5698 |   0.02507 |    3.79495 | 1.091 | 1.052    | 0.79        | 97.48  | 17.60    | 4.03 | 3.65 | 8991280 | STS.000.ND PP1 center    | 
| 367996 |       30 | 2011-07-25 12:09:52 | 24.5652 |  0.026531 |     3.6958 | 1.104 | 1.064    | 0.79        | 97.83  | 19.54    | 4.02 | 3.49 | 9311700 | STS.000.ND PP1 center    | 
| 368010 |       30 | 2011-07-25 12:19:54 | 24.5644 |  0.026506 |    3.74343 | 1.089 | 1.055    | 0.77        | 102.98 | 19.22    | 3.99 | 3.60 | 9108432 | STS.000.ND PP1 center    | 
| 368833 |       30 | 2011-07-27 07:23:06 | 24.5521 |  0.026382 |    3.75974 | 1.153 | 1.061    | 0.93        | 114.75 | 17.14    | 4.16 | 3.49 | 9435552 | STS.180.ND PP1 center    | 
| 393751 |       20 | 2011-09-18 08:24:25 | 24.5337 |  0.029066 |    3.54239 | 1.153 | 1.071    | 0.91        | 89.00  | 12.90    | 3.89 | 3.31 | 9090136 | STS.000.GIQ PP1 dither 1 | 
| 393763 |       20 | 2011-09-18 08:32:30 |  24.533 |   0.02565 |    3.60961 | 1.170 | 1.071    | 0.96        | 88.74  | 12.99    | 3.88 | 3.33 | 9028068 | STS.000.GIQ PP1 dither 3 | 
| 393778 |       20 | 2011-09-18 08:40:35 | 24.5299 |  0.027023 |    3.55016 | 1.177 | 1.072    | 0.92        | 92.47  | 13.17    | 3.88 | 3.30 | 9102116 | STS.000.GIQ PP1 dither 1 | 
| 393785 |       20 | 2011-09-18 08:44:33 | 24.5337 |  0.027504 |     3.6808 | 1.135 | 1.065    | 0.92        | 96.03  | 13.46    | 4.03 | 3.48 | 8654104 | STS.000.GIQ PP1 dither 2 | 
| 393792 |       20 | 2011-09-18 08:48:23 | 24.5315 |  0.026466 |    3.70284 | 1.164 | 1.073    | 0.97        | 94.42  | 13.26    | 4.00 | 3.44 | 8762736 | STS.000.GIQ PP1 dither 3 | 
| 393799 |       20 | 2011-09-18 08:52:22 | 24.5312 |  0.028034 |    3.71144 | 1.172 | 1.078    | 0.96        | 98.12  | 13.35    | 4.02 | 3.47 | 8695292 | STS.000.GIQ PP1 dither 4 | 
| 393835 |       20 | 2011-09-18 09:13:19 | 24.5271 |  0.027369 |    3.75097 | 1.157 | 1.076    | 0.99        | 103.99 | 13.49    | 4.07 | 3.47 | 8624396 | STS.000.GIQ PP1 dither 1 | 

--> cuts out a larger fraction of the center position

--> compared to log file for ppStack -- gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.log
      Input 0: 3.137970
      Input 1: 3.397691
      Input 2: 3.309175
      Input 3: 3.310735
      Input 4: 3.114384
      Input 5: 3.043151
      Input 6: 3.346249
      Input 7: 3.141845
      Input 8: 3.471485
      Input 9: 3.238341
      Input 10: 3.433451
      Input 11: 3.225220
      Input 12: 4.423095
      Input 13: 4.086733
      Input 14: 4.282342
      Input 15: 4.067309
      Input 16: 3.030721
      Input 17: 4.047925
      Input 18: 4.245076
      Input 19: 4.328331
      Input 20: 4.284031
      Input 21: 4.074796
      Input 22: 4.197470
      Input FWHMs : 3.662502 +/- 0.504006
          fit ext:   0.965282 sec for 225 of 225 sources
          fit psf:   0.342076 sec for 196 of 225 sources
        apresid: 0.046344 +/- 0.000000 (systematic error) from statistics of 225 psf stars (196 used)
        chisq vs flux fit term 0: 0.000000 +/- 0.000000
        chisq vs flux fit term 1: 0.000000 +/- 0.000000
      try model PS_MODEL_PS1_V1, ap-fit: 0.046344 +/- 0.000000 : sky bias: 0.000000
      Target seeing FWHM: 4.430518
...
        fwhm (psf): 4.472250,4.366182 (moments): 0.000000,0.000000

--> even for c1f37 has FWHM>4.2 going into it -- gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.log
--> however, not as overdriven as feared. --- need to add plot/vector of PSF shape in skycell ---

psphot on samples, unconvolved and convolved (compare to ppStack log file):

  • faults because of recent modified psphot config/code to limit peaks and so faults.. need to add -Di PEAKS_NMAX_TOTAL 60000
  • unconvolved stacks still faulting.. so do PASS1 only -D BREAK_POINT PASS1. Convolved ones can be poor, FWHM_MAJ~9-10?? 
    gethead -auhc *fpa.cmf FWHM_MAJ IQ_FW1
STS.V4.skycell.1.03.stk.899917.fpa.cmf        4.405078 3.746344
STS.V4.skycell.1.03.stk.899917.unconv.fpa.cmf 3.921824 3.04026
STS.V4.skycell.1.03.stk.899918.fpa.cmf        4.359843 3.701267
STS.V4.skycell.1.03.stk.899918.unconv.fpa.cmf 3.788124 2.973968
STS.V4.skycell.1.03.stk.899919.fpa.cmf        4.184109 3.576873
STS.V4.skycell.1.03.stk.899919.unconv.fpa.cmf 3.835106 2.939088
STS.V4.skycell.1.03.stk.899920.fpa.cmf        4.54595 3.812682
STS.V4.skycell.1.03.stk.899920.unconv.fpa.cmf 3.862908 3.048509
--> basically as with MD, +/- pixel, so all are basically similar and going to smaller FWHM cut not improve much, partially because cut doen't always match true FWHM going in.
  • previous refstack.20110613 
    gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.fits (cleaned)
gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.cmf
gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.log

Summary of test runs for skycell.1.03:

version data_group (stack_id) filter skycell input warps (rej) skycell FWHM (major) conv/unconv
new STS.refstack.20120510 V4 i 61 (2) 1.38/0.98"
STS.reftest.20120510.c1f39 V4 (899917) i 36 (0) 1.10/0.98"
STS.reftest.20120510.c1f38 V4 (899918) i 34 (0) 1.09/0.95"
STS.reftest.20120510.c1f37 V4 (899919) i 12 (0) 1.05/0.96"
STS.reftest.20120510.c0f45 V4 (899920) i 44 (0) 1.14/0.97"
old STS.refstack.20110613 V3 i 68 (0) ~1.16/0.9" (approx from log since image cleaned)

Look at recent camera/warps to check similar coverage

  • recent observations all 180 (as of 120510)
  • center+dither available
  • sample from rawExp table: 
    | 484662 | o6054g0382o | 2012-05-07 13:44:36 | i.00000 |       30 | STS.180.ND PP1 center 

gpc1/STS.nt/2012/05/07/o6054g0382o.484662/
gpc1/STS.nt/2012/05/07/o6054g0382o.484662/o6054g0382o.484662.wrp.408115.skycell.1.03.fits
--> bad tracking image, startrails.. try another

gpc1/STS.nt/2012/05/07/o6054g0375o.484656/
gpc1/STS.nt/2012/05/07/o6054g0375o.484656/o6054g0375o.484656.wrp.408110.skycell.1.03.fits
--> yes, similar pattern -

Main run -- do by projection PP 1-7:

  • use FWHM cut 3.8 for ~20-40 and see -- if some need fixing, then re-run that skycell, if too much burntool then know for next version
    • should really add the PSF cut to PS1_REFERENCE too to help reject unrepresentative camera FWHM for skycell.
  • run in normal stack, system fairly loaded w/o deepstack running? no, the c3 machines need more work 
stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.refstack.20120510 -set_label STS.refstack.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.refstack.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.8 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple  >& stacktool.sts_refstack.20120510.fwhm38_elong12.log
--->  oops forgot to add -select_skycell_id skycell.1.% -- so it is running all projections at once.

Compare to test sample skycell.1.03: 

gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.fits
gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.unconv.fits
gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.log
--> run psphot manually and add to table above -- need to set 65000 detections for unconv stack

Look at final stacks: bottom line, more inputs would help, but maybe this is sufficient

  • 4.17 odd bright cell print through
  • 2.02 odd faint chunk print through
  • 2.39 gap faint print through
  • 5.06 missed burntool?
  • 1.09,1.17 large unfilled gap
  • 1.24 long unfilled gap
  • 1.25,1.35 long and large unfilled gap
  • 1.12 large unfilled cell
  • 1.40 unfilled masking corner multi-cell
  • 1.55 rounded edge + corner adds other exp
  • 1.44,5.14 odd bright orphan diffspike-like unmasked
  • 1.11 nice cluster
  • 1.32 faint flux print through from brighter cell

Run additional reftest for selected examples:

  • skycell.1.35 -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.902937.log 
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35  >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell135_c2.log

stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f45 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 4.5 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35  >& stacktool.sts_reftest.20120510.fwhm45_elong12_skycell135_c2.log

stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35  >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell135_c2.log

---> the additional input 45 shows lower corner with much worse seeing -- this is problem, each skycell may need to be tuned/limited directly since DB not have proper stats. A reason for using conv stacking
  • skycell.1.40 -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.902942.log 
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40  >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell140_c2.log

stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f45 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 4.5 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40  >& stacktool.sts_reftest.20120510.fwhm45_elong12_skycell140_c2.log

stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40  >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell140_c2.log

---> max input not fill in masking much
  • skycell.1.55 -- mix of PP inputs and possible IQ issues, try reducing Ninput greatly. really need to do manually.. -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.log 
    stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f36 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.6 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.55  >& stacktool.sts_reftest.20120510.fwhm36_elong12_skycell155_c2.log

stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.55  >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell155_c2.log

---> the fewer inputs makes the IQ worse in upper left, reason to use conv stacks here

Look at Target seeing to see if any odd inputs driving target PSF:

  • Target seeing for PSF 
    mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select path_base from stackRun join stackSumSkyfile using (stack_id) where stackRun.label like "STS.refstack.20120510" and  stackRun.skycell_id like "%.1.%";' | grep -v path_base | awk '{print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -l -p "$1"%.log"}' | tcsh | awk '{print " cat "$9}' | tcsh | grep "Target seeing"
      Target seeing FWHM: 3.871291
      Target seeing FWHM: 5.136444
      Target seeing FWHM: 9.224628
      Target seeing FWHM: 5.216270
      Target seeing FWHM: 4.238182
      Target seeing FWHM: 4.433765
      Target seeing FWHM: 8.005375
      Target seeing FWHM: 5.200400
...
--> look at couple with FWHM 8-9 -- #3,7 -- manually match to input file:
gpc1/STS.refstack.20120510/STS.V4/skycell.1.02/STS.V4.skycell.1.02.stk.902904.log -- 83 Ninput, largest ~5.7 so wacky shapes? even 2.89 FWHM
gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.log -- 78 Ninput, largest ~4.5 so wacky shapes?
  • create log of Input FWHM, Target seeing, and convolved PSF FWHM: 
    mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select path_base from stackRun join stackSumSkyfile using (stack_id) where stackRun.label like "STS.refstack.20120510" and  stackRun.skycell_id like "%";' | grep -v path_base | awk '{print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -l -p "$1"%.log"}' | tcsh | awk '{print "echo "$9"; grep  -e \" Input \" -e \"Target seeing\" -e \"fwhm (psf)\" "$9" ;echo \" \""}' | tcsh  > STS.refstack.20120510_input_target_output_fwhm_inputlong.log

clean-up

Will need to keep warps around for remaking the refstack if necessary but more importantly for MOPS to get stamps without update delay. ~50x7 or 350 exposures on disk for the skycells. 

chiptool -updaterun -set_state goto_cleaned -set_label goto_cleaned -data_group STS.refstack.20120510 -dbname gpc1

//warptool -updaterun -set_state goto_cleaned -set_label goto_cleaned -data_group STS.refstack.20120510 -dbname gpc1

Test Diffim samples

Want to create a sample of diffims with detections for MOPS to look at. Will need to also keep in "full" state so they can get postage stamps quickly. Focus on projection 1.

Basic setup: (warp - convolved stack) -- need examples and compared to unconv stack (why not using this?)

manual test set

Want to run quick set locally. Pick center, edge, edge+lots overlap for various filling types - 1.44, 1.55, 1.03? 

gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.log
gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.log
gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.log
--> appear mostly well behaved mix.. need to add one more -- 1.06 is wacky
gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.log
  • run target2,1 and dual (simiple scripts used for ppSub tests) 
    SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.refstack.20120510" and stackRun.skycell_id like "%1.44" group by exp_id;
  • need night listing with PP1 exp_name - 
select exp_id,exp_name,exp_time,dateobs,comment from rawExp where obs_mode like "%STS%" and comment like "%PP1%" and dateobs>"2012-04";
| exp_id | exp_name    | exp_time | dateobs             | comment                  |
+--------+-------------+----------+---------------------+--------------------------+
| 475121 | o6029g0441o |       30 | 2012-04-12 13:40:09 | STS.180.ND PP1 center    | 
| 475128 | o6029g0448o |       30 | 2012-04-12 13:45:11 | STS.180.ND PP1 center    | 
| 475135 | o6029g0455o |       30 | 2012-04-12 13:50:14 | STS.180.ND PP1 center    | 
| 475141 | o6029g0462o |       30 | 2012-04-12 13:55:16 | STS.180.ND PP1 center    | 
| 475149 | o6029g0469o |       30 | 2012-04-12 14:00:20 | STS.180.ND PP1 center    | 
| 475156 | o6029g0476o |       30 | 2012-04-12 14:05:25 | STS.180.ND PP1 center    | 
| 475163 | o6029g0483o |       30 | 2012-04-12 14:10:27 | STS.180.ND PP1 center    | 
| 475170 | o6029g0490o |       30 | 2012-04-12 14:15:31 | STS.180.ND PP1 center    | 
| 475177 | o6029g0497o |       30 | 2012-04-12 14:20:35 | STS.180.ND PP1 center    | 
| 475184 | o6029g0504o |       30 | 2012-04-12 14:25:37 | STS.180.ND PP1 center    | 
| 475191 | o6029g0511o |       30 | 2012-04-12 14:30:41 | STS.180.ND PP1 center    | 
| 475198 | o6029g0518o |       30 | 2012-04-12 14:35:47 | STS.180.ND PP1 center    | 
| 476338 | o6031g0408o |       20 | 2012-04-14 13:34:26 | STS.180.GIQ PP1 dither 1 | 
| 476345 | o6031g0415o |       20 | 2012-04-14 13:38:20 | STS.180.GIQ PP1 dither 2 | 
| 476352 | o6031g0422o |       20 | 2012-04-14 13:42:15 | STS.180.GIQ PP1 dither 3 | 
| 476360 | o6031g0429o |       20 | 2012-04-14 13:46:09 | STS.180.GIQ PP1 dither 4 | 
| 476366 | o6031g0436o |       20 | 2012-04-14 13:50:05 | STS.180.GIQ PP1 dither 1 | 
| 476373 | o6031g0443o |       20 | 2012-04-14 13:53:56 | STS.180.GIQ PP1 dither 2 | 
| 476380 | o6031g0450o |       20 | 2012-04-14 13:57:48 | STS.180.GIQ PP1 dither 3 | 
| 476389 | o6031g0457o |       20 | 2012-04-14 14:01:41 | STS.180.GIQ PP1 dither 4 | 
| 476397 | o6031g0464o |       20 | 2012-04-14 14:08:15 | STS.180.GIQ PP1 dither 1 | 
| 476404 | o6031g0471o |       20 | 2012-04-14 14:12:12 | STS.180.GIQ PP1 dither 2 | 
| 476411 | o6031g0478o |       20 | 2012-04-14 14:16:12 | STS.180.GIQ PP1 dither 3 | 
| 476418 | o6031g0485o |       20 | 2012-04-14 14:20:07 | STS.180.GIQ PP1 dither 4 | 
| 476425 | o6031g0492o |       20 | 2012-04-14 14:24:14 | STS.180.GIQ PP1 dither 1 | 
| 476432 | o6031g0499o |       20 | 2012-04-14 14:28:10 | STS.180.GIQ PP1 dither 2 | 
| 476437 | o6031g0506o |       20 | 2012-04-14 14:32:10 | STS.180.GIQ PP1 dither 3 | 
| 476445 | o6031g0513o |       20 | 2012-04-14 14:36:14 | STS.180.GIQ PP1 dither 4 | 
| 476453 | o6031g0520o |       20 | 2012-04-14 14:40:10 | STS.180.GIQ PP1 dither 1 | 
| 476460 | o6031g0527o |       20 | 2012-04-14 14:44:06 | STS.180.GIQ PP1 dither 2 | 
| 477549 | o6033g0430o |       30 | 2012-04-16 13:31:05 | STS.180.ND PP1 center    | 
| 477556 | o6033g0437o |       30 | 2012-04-16 13:36:06 | STS.180.ND PP1 center    | 
| 477562 | o6033g0444o |       30 | 2012-04-16 13:41:06 | STS.180.ND PP1 center    | 
| 477569 | o6033g0451o |       30 | 2012-04-16 13:46:09 | STS.180.ND PP1 center    | 
| 477576 | o6033g0458o |       30 | 2012-04-16 13:51:15 | STS.180.ND PP1 center    | 
| 477581 | o6033g0465o |       30 | 2012-04-16 13:56:20 | STS.180.ND PP1 center    | 
| 477588 | o6033g0472o |       30 | 2012-04-16 14:01:24 | STS.180.ND PP1 center    | 
| 477595 | o6033g0479o |       30 | 2012-04-16 14:06:24 | STS.180.ND PP1 center    | 
| 477605 | o6033g0486o |       30 | 2012-04-16 14:11:28 | STS.180.ND PP1 center    | 
| 477612 | o6033g0493o |       30 | 2012-04-16 14:16:30 | STS.180.ND PP1 center    | 
| 477619 | o6033g0500o |       30 | 2012-04-16 14:21:31 | STS.180.ND PP1 center    | 
| 477625 | o6033g0507o |       30 | 2012-04-16 14:26:36 | STS.180.ND PP1 center    | 
| 478479 | o6037g0401o |       20 | 2012-04-20 13:46:26 | STS.180.GIQ PP1 dither 1 | 
| 478485 | o6037g0408o |       20 | 2012-04-20 13:50:22 | STS.180.GIQ PP1 dither 2 | 
| 478492 | o6037g0415o |       20 | 2012-04-20 13:54:14 | STS.180.GIQ PP1 dither 3 | 
| 478499 | o6037g0422o |       20 | 2012-04-20 13:58:10 | STS.180.GIQ PP1 dither 4 | 
| 478506 | o6037g0429o |       20 | 2012-04-20 14:02:08 | STS.180.GIQ PP1 dither 1 | 
| 478514 | o6037g0436o |       20 | 2012-04-20 14:06:05 | STS.180.GIQ PP1 dither 2 | 
| 478520 | o6037g0443o |       20 | 2012-04-20 14:09:58 | STS.180.GIQ PP1 dither 3 | 
| 478527 | o6037g0450o |       20 | 2012-04-20 14:13:50 | STS.180.GIQ PP1 dither 4 | 
| 478533 | o6037g0457o |       20 | 2012-04-20 14:17:47 | STS.180.GIQ PP1 dither 1 | 
| 478542 | o6037g0464o |       20 | 2012-04-20 14:21:42 | STS.180.GIQ PP1 dither 2 | 
| 478548 | o6037g0471o |       20 | 2012-04-20 14:25:40 | STS.180.GIQ PP1 dither 3 | 
| 478554 | o6037g0478o |       20 | 2012-04-20 14:29:40 | STS.180.GIQ PP1 dither 4 | 
| 478561 | o6037g0485o |       20 | 2012-04-20 14:34:36 | STS.180.GIQ PP1 dither 1 | 
| 478569 | o6037g0492o |       20 | 2012-04-20 14:38:35 | STS.180.GIQ PP1 dither 2 | 
| 478576 | o6037g0499o |       20 | 2012-04-20 14:42:30 | STS.180.GIQ PP1 dither 3 | 
| 478582 | o6037g0506o |       20 | 2012-04-20 14:46:32 | STS.180.GIQ PP1 dither 4 | 
| 479810 | o6040g0441o |       30 | 2012-04-23 13:31:03 | STS.180.ND PP1 center    | 
| 479818 | o6040g0448o |       30 | 2012-04-23 13:36:04 | STS.180.ND PP1 center    | 
| 479822 | o6040g0455o |       30 | 2012-04-23 13:41:07 | STS.180.ND PP1 center    | 
| 479831 | o6040g0462o |       30 | 2012-04-23 13:46:10 | STS.180.ND PP1 center    | 
| 479838 | o6040g0469o |       30 | 2012-04-23 13:51:11 | STS.180.ND PP1 center    | 
| 479845 | o6040g0476o |       30 | 2012-04-23 13:56:11 | STS.180.ND PP1 center    | 
| 479852 | o6040g0483o |       30 | 2012-04-23 14:01:15 | STS.180.ND PP1 center    | 
| 479859 | o6040g0490o |       30 | 2012-04-23 14:06:17 | STS.180.ND PP1 center    | 
| 479867 | o6040g0497o |       30 | 2012-04-23 14:11:20 | STS.180.ND PP1 center    | 
| 479874 | o6040g0504o |       30 | 2012-04-23 14:16:21 | STS.180.ND PP1 center    | 
| 479880 | o6040g0511o |       30 | 2012-04-23 14:21:21 | STS.180.ND PP1 center    | 
| 479888 | o6040g0518o |       30 | 2012-04-23 14:26:25 | STS.180.ND PP1 center    | 
| 482593 | o6045g0420o |       30 | 2012-04-28 13:37:39 | STS.180.ND PP1 center    | 
| 482600 | o6045g0427o |       30 | 2012-04-28 13:42:41 | STS.180.ND PP1 center    | 
| 482609 | o6045g0434o |       30 | 2012-04-28 13:47:43 | STS.180.ND PP1 center    | 
| 482614 | o6045g0441o |       30 | 2012-04-28 13:52:45 | STS.180.ND PP1 center    | 
| 482620 | o6045g0448o |       30 | 2012-04-28 13:57:46 | STS.180.ND PP1 center    | 
| 482628 | o6045g0455o |       30 | 2012-04-28 14:02:53 | STS.180.ND PP1 center    | 
| 482635 | o6045g0462o |       30 | 2012-04-28 14:07:56 | STS.180.ND PP1 center    | 
| 482642 | o6045g0469o |       30 | 2012-04-28 14:12:59 | STS.180.ND PP1 center    | 
| 482647 | o6045g0476o |       30 | 2012-04-28 14:18:01 | STS.180.ND PP1 center    | 
| 482654 | o6045g0483o |       30 | 2012-04-28 14:23:04 | STS.180.ND PP1 center    | 
| 482661 | o6045g0490o |       30 | 2012-04-28 14:28:11 | STS.180.ND PP1 center    | 
| 482667 | o6045g0497o |       30 | 2012-04-28 14:33:16 | STS.180.ND PP1 center    | 
| 483576 | o6049g0286o |       30 | 2012-05-02 13:09:26 | STS.180.ND PP1 center    | 
| 483583 | o6049g0293o |       30 | 2012-05-02 13:14:27 | STS.180.ND PP1 center    | 
| 483590 | o6049g0300o |       30 | 2012-05-02 13:19:26 | STS.180.ND PP1 center    | 
| 483597 | o6049g0307o |       30 | 2012-05-02 13:24:29 | STS.180.ND PP1 center    | 
| 483604 | o6049g0314o |       30 | 2012-05-02 13:29:35 | STS.180.ND PP1 center    | 
| 483611 | o6049g0321o |       30 | 2012-05-02 13:34:37 | STS.180.ND PP1 center    | 
| 483624 | o6049g0335o |       30 | 2012-05-02 13:44:43 | STS.180.ND PP1 center    | 
| 483630 | o6049g0342o |       30 | 2012-05-02 13:49:48 | STS.180.ND PP1 center    | 
| 483638 | o6049g0349o |       30 | 2012-05-02 13:54:49 | STS.180.ND PP1 center    | 
| 483646 | o6049g0356o |       30 | 2012-05-02 13:59:51 | STS.180.ND PP1 center    | 
| 483651 | o6049g0328o |       30 | 2012-05-02 13:39:38 | STS.180.ND PP1 center    | 
| 483652 | o6049g0363o |       30 | 2012-05-02 14:04:52 | STS.180.ND PP1 center    | 
| 484578 | o6054g0305o |       30 | 2012-05-07 12:48:56 | STS.180.ND PP1 center    | 
| 484590 | o6054g0312o |       30 | 2012-05-07 12:53:58 | STS.180.ND PP1 center    | 
| 484599 | o6054g0319o |       30 | 2012-05-07 12:58:58 | STS.180.ND PP1 center    | 
| 484606 | o6054g0326o |       30 | 2012-05-07 13:04:01 | STS.180.ND PP1 center    | 
| 484613 | o6054g0333o |       30 | 2012-05-07 13:09:03 | STS.180.ND PP1 center    | 
| 484620 | o6054g0340o |       30 | 2012-05-07 13:14:04 | STS.180.ND PP1 center    | 
| 484627 | o6054g0347o |       30 | 2012-05-07 13:19:09 | STS.180.ND PP1 center    | 
| 484634 | o6054g0354o |       30 | 2012-05-07 13:24:13 | STS.180.ND PP1 center    | 
| 484641 | o6054g0361o |       30 | 2012-05-07 13:29:18 | STS.180.ND PP1 center    | 
| 484648 | o6054g0368o |       30 | 2012-05-07 13:34:25 | STS.180.ND PP1 center    | 
| 484656 | o6054g0375o |       30 | 2012-05-07 13:39:31 | STS.180.ND PP1 center    | 
| 484662 | o6054g0382o |       30 | 2012-05-07 13:44:36 | STS.180.ND PP1 center    | 
+--------+-------------+----------+---------------------+--------------------------+

--> do recent 2 first - first in list - need script liner to get more of these -

neb://@HOST@.0/gpc1/STS.nt/2012/05/02/o6049g0286o -- cleaned.. neb://@HOST@.0/gpc1/STS.nt/2012/05/07/o6054g0305o

--> will want to know FWHM of input warps too

neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/%unconv%fits nebds9 gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits * 1.44 @4610,5580 region see stars look elongated/trailed versus ones nearby.. looks like multi-cell mask region -- diffims will show if problem --> maybe has to do with GIQ?

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.44.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120507 > runppsub_sts144_120507_1.bat source runppsub_sts144_120507_1.bat >& runppsub_sts144_120507_1.log &

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120507 > runppsub_sts144_120507_2.bat source runppsub_sts144_120507_2.bat >& runppsub_sts144_120507_2.log &

neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/%unconv%fits --> also see odd shape near center but is edge field, but has both PP1 and PP2 inputs -

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.55.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.unconv.fits 20120507 > runppsub_sts155_120507_1.bat source runppsub_sts155_120507_1.bat >& runppsub_sts155_120507_1.log &

neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/%unconv%fits --> looks ok, good overlap mix pp1/4

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.unconv.fits 20120507 > runppsub_sts103_120507_1.bat source runppsub_sts103_120507_1.bat >& runppsub_sts103_120507_1.log &

neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/%unconv%fits --> looks ok, crazy overlap mix pp1,2,4 and maybe odd shape in corner

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.06.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.unconv.fits 20120507 > runppsub_sts106_120507_1.bat source runppsub_sts106_120507_1.bat >& runppsub_sts106_120507_1.log &

* diffims look like garbage.. MD04 from 5/8 WS look ok - so look at reftest ones as well.. --> update/pss request an STS field from 5/2? what was fwhm? SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) WHERE obs_mode="STS" and dateobs>"2012-01-01" and comment like "%PP1%" group by exp_id; --> next one on 5/7 is better fwhm so try that first - same --> has ND in comment name -- that mean anything? --> try 4/20 o6037g0506o -- will need to update - warp_id=400804, skycell.1.44 bundle -- FWHM~3.8 --> better, at least messy resid but the things that look like diffim det look odd, some look like things removed in pixel rejection??

gpc1/STS.nt/2012/04/20/o6037g0506o.478582/o6037g0506o.478582.wrp.400804.skycell.1.44.fits

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/20/o6037g0506o.478582/o6037g0506o.478582.wrp.400804.skycell.1.44.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120420 > runppsub_sts144_120420_x.bat source runppsub_sts144_120420_x.bat >& runppsub_sts144_120420_x.log &

--> need to retry diffims above as conv stacks too - seems the same, how is that..

--> need to look at the mask images as well -

--> look at reftest diffims 

mysql> select stack_id,data_group,skycell_id from stackRun where data_group like "STS.reftest%c%f%";
+----------+----------------------------+--------------+
| stack_id | data_group                 | skycell_id   |
+----------+----------------------------+--------------+
|   899920 | STS.reftest.20120510.c0f45 | skycell.1.03 | 
|   899919 | STS.reftest.20120510.c1f37 | skycell.1.03 | 
|   899918 | STS.reftest.20120510.c1f38 | skycell.1.03 | 
|   899917 | STS.reftest.20120510.c1f39 | skycell.1.03 | 
|   904603 | STS.reftest.20120510.c2f36 | skycell.1.55 | 
|   904604 | STS.reftest.20120510.c2f37 | skycell.1.55 | 
|   904605 | STS.reftest.20120510.c2f37 | skycell.1.40 | 
|   904606 | STS.reftest.20120510.c2f37 | skycell.1.35 | 
|   904599 | STS.reftest.20120510.c2f39 | skycell.1.35 | 
|   904601 | STS.reftest.20120510.c2f39 | skycell.1.40 | 
|   904600 | STS.reftest.20120510.c2f45 | skycell.1.35 | 
|   904602 | STS.reftest.20120510.c2f45 | skycell.1.40 | 

gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899917.unconv.fits
gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.fits
gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.fits
gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899920.unconv.fits

gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904599.unconv.fits
gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904600.unconv.fits
gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904606.unconv.fits

gpc1/STS.reftest.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.904603.unconv.fits
gpc1/STS.reftest.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.904604.unconv.fits

gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904601.unconv.fits
gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904602.unconv.fits
gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904605.unconv.fits

./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.fits 20120507_c1f38 > runppsub_sts103_120507_c1f38.bat source runppsub_sts103_120507_c1f38.bat >& runppsub_sts103_120507_c1f38.log &

--> ack.. been cleaned and no others since.. will need to update chips too.. so tedious, use pss - warptool -dbname gpc1 -setskyfiletoupdate -warp_id 408038 -skycell_id skycell.1.03 -set_label --> need to copy over locally -

--> for conv, edit the .bat, make WSc and drop the unconv cp runppsub_sts103_120507_c1f38.bat runppsub_sts103_120507_c1f38c.bat source runppsub_sts103_120507_c1f38c.bat >& runppsub_sts103_120507_c1f38c.log &

--> same, so guess that is good.. try c1f37 - ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.fits 20120507_c1f37 > runppsub_sts103_120507_c1f37.bat source runppsub_sts103_120507_c1f37.bat >& runppsub_sts103_120507_c1f37.log &

--> same.. are the exp_time different? yes 4/20 is 20s.. why would that matter.. IQ is better too, so maybe recent ones are just wacky IQ?

-- need to run psphot on the refstack - staticsky? no that would be a mess and only 1 filter anyways.. --> then can make 

survey.add.publish STS.mopsdiff STS.mopsdiffv2.mehtest 12 NULL
survey.show.publish 
survey.del.publish STS.mopsdiff
add.label STS.mopsdiffv2.mehtest

Diffim sample 1 (faulted)

Sample diffims for projection 1 faulted in publishing. So this is more of an example of what not to do..

Diffim sample 2 (published)

Rerun of sample diffims for projection 1 and published under IPP-MOPS-TEST-2

Need to update chip->warp of (re)selected nights for testing

  • like before avoid dates used in refstack, try sample similar/better FWHM and worse 
    SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" group by exp_id;

SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE  comment like "STS%PP1%" group by exp_id;

| 400812 | o5834g0061o |       30 | 2011-09-30 06:48:31 |  24.643 |     4.9952 | 1.117 | 0.90        | 137.86   | 16.97    | 5.30  | 4.83  | 6509204 | STS.090.ND PP1 center    | 
| 400819 | o5834g0068o |       30 | 2011-09-30 06:53:41 | 24.6431 |      5.927 | 1.198 | 1.38        | 131.72   | 16.92    | 6.25  | 5.55  | 5929444 | STS.090.ND PP1 center    | 
| 400826 | o5834g0075o |       30 | 2011-09-30 06:58:46 | 24.6413 |    5.41098 | 1.165 | 1.20        | 123.85   | 16.34    | 5.69  | 5.22  | 6411952 | STS.090.ND PP1 center    | 
| 400833 | o5834g0082o |       30 | 2011-09-30 07:03:51 | 24.6297 |    6.84521 | 1.120 | 1.16        | 121.33   | 17.11    | 7.07  | 6.75  | 4591816 | STS.090.ND PP1 center    | 
| 400840 | o5834g0089o |       30 | 2011-09-30 07:08:53 | 24.6256 |    6.46738 | 1.069 | 0.84        | 126.92   | 17.59    | 6.74  | 6.23  | 4562012 | STS.090.ND PP1 center    | 
| 400847 | o5834g0096o |       30 | 2011-09-30 07:13:55 | 24.6386 |    6.42677 | 1.196 | 1.36        | 125.30   | 16.89    | 6.75  | 6.22  | 5325836 | STS.090.ND PP1 center    | 
| 400854 | o5834g0103o |       30 | 2011-09-30 07:19:00 | 24.6292 |    6.37093 | 1.103 | 0.88        | 123.32   | 17.25    | 6.67  | 6.13  | 4688216 | STS.090.ND PP1 center    | 
| 400861 | o5834g0110o |       30 | 2011-09-30 07:24:07 | 24.6316 |    6.62746 | 1.180 | 1.04        | 131.42   | 17.69    | 6.82  | 6.42  | 4652948 | STS.090.ND PP1 center    | 
| 400868 | o5834g0117o |       30 | 2011-09-30 07:29:12 | 24.6365 |    6.15409 | 1.085 | 1.05        | 128.51   | 17.22    | 6.53  | 5.92  | 5107028 | STS.090.ND PP1 center    | 
| 400875 | o5834g0124o |       30 | 2011-09-30 07:34:21 |  24.641 |     5.2498 | 1.127 | 0.91        | 123.12   | 16.49    | 5.56  | 5.03  | 6108640 | STS.090.ND PP1 center    | 
| 400883 | o5834g0131o |       30 | 2011-09-30 07:39:28 | 24.6409 |    5.78907 | 1.154 | 1.19        | 127.42   | 16.82    | 6.00  | 5.65  | 5809232 | STS.090.ND PP1 center    | 
| 400889 | o5834g0138o |       30 | 2011-09-30 07:44:33 | 24.6389 |    6.12522 | 1.064 | 1.12        | 125.19   | 17.02    | 6.48  | 5.96  | 5185056 | STS.090.ND PP1 center    | 

| 475121 | o6029g0441o |       30 | 2012-04-12 13:40:09 | 24.5321 |    4.06258 | 1.119 | 1.05        | 315.49   | 21.55    | 4.21  | 3.82  | 8078200 | STS.180.ND PP1 center    | 
| 475128 | o6029g0448o |       30 | 2012-04-12 13:45:11 | 24.5325 |    3.91594 | 1.132 | 0.95        | 303.82   | 21.37    | 4.41  | 3.67  | 8132400 | STS.180.ND PP1 center    | 
| 475135 | o6029g0455o |       30 | 2012-04-12 13:50:14 | 24.5375 |    4.06348 | 1.129 | 0.97        | 307.70   | 21.58    | 4.48  | 3.88  | 7907516 | STS.180.ND PP1 center    | 
| 475141 | o6029g0462o |       30 | 2012-04-12 13:55:16 | 24.5386 |     4.0157 | 1.135 | 0.97        | 301.07   | 21.32    | 4.55  | 3.78  | 8000044 | STS.180.ND PP1 center    | 
| 475149 | o6029g0469o |       30 | 2012-04-12 14:00:20 | 24.5421 |    4.29222 | 1.196 | 1.03        | 300.51   | 21.44    | 4.66  | 4.07  | 7669568 | STS.180.ND PP1 center    | 
| 475156 | o6029g0476o |       30 | 2012-04-12 14:05:25 | 24.5459 |    4.32599 | 1.163 | 1.03        | 304.46   | 21.61    | 4.80  | 4.07  | 7576116 | STS.180.ND PP1 center    | 
| 475163 | o6029g0483o |       30 | 2012-04-12 14:10:27 | 24.5445 |    3.90778 | 1.119 | 0.92        | 308.26   | 21.57    | 4.16  | 3.71  | 8114964 | STS.180.ND PP1 center    | 
| 475170 | o6029g0490o |       30 | 2012-04-12 14:15:31 | 24.5486 |    4.25772 | 1.135 | 0.96        | 295.86   | 21.39    | 4.70  | 4.02  | 7605516 | STS.180.ND PP1 center    | 
| 475177 | o6029g0497o |       30 | 2012-04-12 14:20:35 | 24.5477 |    4.11544 | 1.203 | 1.09        | 378.52   | 23.53    | 4.35  | 3.77  | 7926572 | STS.180.ND PP1 center    | 
| 475184 | o6029g0504o |       30 | 2012-04-12 14:25:37 | 24.5461 |    3.88865 | 1.175 | 1.00        | 293.10   | 21.07    | 4.35  | 3.62  | 8464032 | STS.180.ND PP1 center    | 
| 475191 | o6029g0511o |       30 | 2012-04-12 14:30:41 |  24.552 |    4.19097 | 1.172 | 1.10        | 277.41   | 20.75    | 4.38  | 3.89  | 8060959 | STS.180.ND PP1 center    | 
| 475198 | o6029g0518o |       30 | 2012-04-12 14:35:47 | 24.5526 |    4.09804 | 1.174 | 1.05        | 280.17   | 20.83    | 4.36  | 3.79  | 8122168 | STS.180.ND PP1 center    | 

| 476338 | o6031g0408o |       20 | 2012-04-14 13:34:26 | 24.5537 |    4.27532 | 1.180 | 0.98        | 117.51   | 14.15    | 4.76  | 4.05  | 7485604 | STS.180.GIQ PP1 dither 1 | 
| 476345 | o6031g0415o |       20 | 2012-04-14 13:38:20 | 24.5533 |    4.17247 | 1.184 | 1.01        | 117.49   | 14.21    | 4.67  | 3.89  | 7636200 | STS.180.GIQ PP1 dither 2 | 
| 476352 | o6031g0422o |       20 | 2012-04-14 13:42:15 |  24.552 |    4.09387 | 1.144 | 0.94        | 118.19   | 14.27    | 4.45  | 3.90  | 7637616 | STS.180.GIQ PP1 dither 3 | 
| 476360 | o6031g0429o |       20 | 2012-04-14 13:46:09 |  24.556 |    4.31094 | 1.188 | 1.01        | 118.09   | 14.37    | 4.70  | 4.05  | 7378484 | STS.180.GIQ PP1 dither 4 | 
| 476366 | o6031g0436o |       20 | 2012-04-14 13:50:05 | 24.5551 |    4.17617 | 1.154 | 0.94        | 119.08   | 14.43    | 4.59  | 3.98  | 7443824 | STS.180.GIQ PP1 dither 1 | 
| 476373 | o6031g0443o |       20 | 2012-04-14 13:53:56 | 24.5525 |    3.82054 | 1.166 | 0.84        | 121.42   | 14.46    | 4.30  | 3.63  | 7964096 | STS.180.GIQ PP1 dither 2 | 
| 476380 | o6031g0450o |       20 | 2012-04-14 13:57:48 | 24.5545 |    4.07024 | 1.146 | 0.98        | 123.89   | 14.60    | 4.58  | 3.78  | 7718292 | STS.180.GIQ PP1 dither 3 | 
| 476389 | o6031g0457o |       20 | 2012-04-14 14:01:41 | 24.5365 |     4.0109 | 1.144 | 0.93        | 133.13   | 15.08    | 4.49  | 3.79  | 7568984 | STS.180.GIQ PP1 dither 4 | 
| 476397 | o6031g0464o |       20 | 2012-04-14 14:08:15 | 24.5432 |     3.9738 | 1.177 | 0.95        | 131.38   | 14.95    | 4.33  | 3.76  | 7835456 | STS.180.GIQ PP1 dither 1 | 
| 476404 | o6031g0471o |       20 | 2012-04-14 14:12:12 | 24.5149 |    3.94952 | 1.206 | 0.97        | 138.51   | 15.23    | 4.44  | 3.68  | 7840492 | STS.180.GIQ PP1 dither 2 | 
| 476411 | o6031g0478o |       20 | 2012-04-14 14:16:12 | 24.5435 |    3.80814 | 1.181 | 0.90        | 126.95   | 14.66    | 4.29  | 3.58  | 8082288 | STS.180.GIQ PP1 dither 3 | 
| 476418 | o6031g0485o |       20 | 2012-04-14 14:20:07 | 24.5456 |    3.77643 | 1.189 | 0.92        | 128.77   | 14.71    | 4.27  | 3.54  | 8231892 | STS.180.GIQ PP1 dither 4 | 
| 476425 | o6031g0492o |       20 | 2012-04-14 14:24:14 | 24.5303 |     4.1106 | 1.144 | 0.98        | 139.08   | 15.35    | 4.59  | 3.83  | 7461452 | STS.180.GIQ PP1 dither 1 | 
| 476432 | o6031g0499o |       20 | 2012-04-14 14:28:10 | 24.5157 |     3.9894 | 1.202 | 0.93        | 143.64   | 15.48    | 4.33  | 3.72  | 7628256 | STS.180.GIQ PP1 dither 2 | 
| 476437 | o6031g0506o |       20 | 2012-04-14 14:32:10 | 24.4737 |    4.25366 | 1.151 | 0.99        | 157.29   | 15.95    | 4.71  | 3.96  | 7066552 | STS.180.GIQ PP1 dither 3 | 
| 476445 | o6031g0513o |       20 | 2012-04-14 14:36:14 | 24.5465 |    3.89488 | 1.188 | 0.89        | 139.57   | 15.33    | 4.43  | 3.60  | 7892495 | STS.180.GIQ PP1 dither 4 | 
| 476453 | o6031g0520o |       20 | 2012-04-14 14:40:10 |  24.555 |    3.64675 | 1.198 | 0.87        | 136.07   | 15.05    | 4.17  | 3.37  | 8450820 | STS.180.GIQ PP1 dither 1 | 
| 476460 | o6031g0527o |       20 | 2012-04-14 14:44:06 | 24.4858 |    3.62349 | 1.210 | 0.85        | 166.07   | 16.03    | 4.25  | 3.28  | 8069508 | STS.180.GIQ PP1 dither 2 |
  • 20110930: 
    mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-30%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiffv2.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiffv2.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiffv2.mehtest_110930.bat
...

difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiffv2.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffv2.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffv2.mehtest -set_data_group STS.mopsdiffv2.mehtest  -set_reduction WARPSTACK -simple -pretend

Setup for publishing:

  • check but DONT run -- would get sent to 5 as well as 12.. 
    pubtool -dbname gpc1 -definerun -label STS.mopsdiffv2.mehtest -simple -pretend

--> may need to add the date to the survey task list.. either that or manually load it with pubtool, need to figure out how to only do 12 
ns.del.date, ns.add.date
ns.set.date 2010-09-11 TOWARP -- one for pub?
--> seems like a bad idea.. so how about pubtool - basically calls from 
ipp-localops.lin64/share/pantasks/modules/survey.pro 
pubtool -definerun -label $label -client_id $client_id

pubtool -dbname gpc1 -definerun -label STS.mopsdiffv2.mehtest -client_id 12 -simple -pretend
--> seems to work and only put to 12 -- what happened before then??
--> and works but just pos detections - thats what MD with diff_mode 2 has for WS so.. OSS now using SWEETSPOT (noconv) and diff_mode 1

-- works so setup next 2012 sample and 2011 sample then -- try pick 2011 w/o many in stack - 

SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" group by exp_id;
--> only 1 from 9/19 used so use that date? 9/11 better and none used from it - or 9/30 to have sample of worse input (yes)
--> so 4/14 after 4/12 for close in time observ and still pretty good -- would prefer one not as good.. 

mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-30%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiffv2.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiffv2.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiffv2.mehtest_110930.bat

mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-14%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiffv2.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiffv2.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiffv2.mehtest_120414.bat

difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiffv2.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffv2.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffv2.mehtest -set_data_group STS.mopsdiffv2.mehtest  -set_reduction WARPSTACK -simple -pretend
  • 9/30 only 1 is going to go through diffs for some reason. because diffs make last year.. but were WW diffims 
    select * from diffRun where data_group like "STS.20110930";

--> try with -rerun -- yes seems to be there, runs then all -- seems rerun works to run new type of diffim but not rerun old one?

difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiffv2.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffv2.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffv2.mehtest -set_data_group STS.mopsdiffv2.mehtest  -set_reduction WARPSTACK -simple -rerun -pretend

pubtool -dbname gpc1 -definerun -label STS.mopsdiffv2.mehtest -client_id 12 -simple -pretend

--> seems to have worked, out onto datastore now for MOPS to look at.
skycell FWHM input/ref Ndet comment
1.03

MOPS feedback

Attachments (1)

Download all attachments as: .zip

Note: See TracWiki for help on using the wiki.