= Summary = [[PageOutline]] MOPS would like a refstack for the STS fields in order to get warp-stack diffims. The previous STS.refstack.20110613,20110615 wasn't usable as it was destroyed by magic. STS.20120510 was made using a similar number of inputs from the best exposures from 2011. Not pretty but hopefully sufficient. Status: * 5/8: past refstack from 6/2011 are cleaned, magic destroyed them. no magic problem anymore.. * 5/10: test samples and selected exposures processed for chip->warp * 5/13: test samples for refstack finished, full refstack started * 5/15: refstack finished and checked. looking at diffim samples and how to publish detections to MOPS * 5/18: test diffim sample 1 using updated warps from past nights in 2011, early 2012 * 5/20: test diffims finished but fault in publishing, tracing out. * 5/27: test sample 1 a mess, restart with test sample 2 * 5/29: test sample 2 diffim detections published to IPP-MOPS-TEST-2, waiting for feedback if will work Things to do/fix/test: * convolved versus unconvolved stack diffims * target PSF limiting * skycell overlap in refstack and diffims - want to restrict inputs PP1 into stack skycell.1.xx only etc * can remake with clean/updates for each field, but maybe MOPS won't care or maybe with conv stacks isn't a big issue? * tricky to do in an automated way. giving each field a different refstack processing label is easiest and do on update? will still try to populate other regions though, so need to restrict to particular region with -skycell_id. won't need a sub-label for each 7 to prevent the rerun headache and to only run against proper field since matched by skycell in diffim. * = STS observation and tessellation summary = Sample rawExp.comment information for the type of observation {{{ STS.180.ND PP1 center STS.000.GIQ PP1 dither 1 STS.180.NDGIQ PP5 center STS.180.NOM PP3 dither 3 }}} * 000,090,180,270 -- rotator to keep rough PA on field over season * GIQ -- shorter exp_time (20s versus 30s nominal) to not saturate more sources in better seeing conditions * ND -- no dither so always with center and typically with 30s, occasional NDGIQ * NOM -- nominal IQ? * PP# -- 1-7, 7 pointings * dither 1-4 -- dither shifts (by ?) and typically with GIQ Overlay of camera SMFs for PP1-PP7 (colored) on top of the projection skycells defined at http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_Tessellations. || [[Image(STS_i_smf_fieldcolor.png,300px)]] || 7 projections and their skycell overlap (STS.V4, 0.25"/pixel platescale) || Approximate pattern, E left, N up: {{{ 1 (orange) 2 (red) 3 (blue) 4 (green) 5 (violet) 6 (indigo) 7 (yellow) }}} To generate SMF sample for position and look at dither pattern: {{{ mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select chipRun.workdir,rawExp.exp_tag,rawExp.exp_name from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%center%" group by rawExp.exp_id;' | grep -v "workdir" | uniq -f2 | awk '{ print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -p " $1"/"$2"/o%smf" }' | tcsh | awk '{print "imbox "$1}' >& STSsub.1_i_imboxsmf_c.lst mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select chipRun.workdir,rawExp.exp_tag,rawExp.exp_name from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%dither 1%" group by rawExp.exp_id;' | grep -v "workdir" | uniq -f2 | awk '{ print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -p " $1"/"$2"/o%smf" }' | tcsh | awk '{print "imbox "$1}' >& STSsub.1_i_imboxsmf_d1.lst ... dvo -D CATDIR STS.V4 $KAPA = kapa -noX region 297.5 17.1 4.2 style -c black image style -c orange input STSsub.1_i_imboxsmf.lst style -c red input STSsub.2_i_imboxsmf.lst ... }}} There is significant overlap of projections (and effectively skycells). May cause issue with general input into a refstack skycell with wildly different IQ? * some will have direct overlapping pixels into the skycell, for example PP4 can provide chips to make skycell.1.03 that could then end up in a stack for that skycell and vice versa. * these could fill in different areas of the skycell with very different PSF functions, but the skycells are also populated by different chips and IQ from rotations as well. * will probably want a reasonable input sample, but suspect target PSF will behave oddly for convolved stack * using the processed warps, skycell.1.03 looks like {{{ select exp_name,dateobs,warpRun.warp_id,warpSkyCellMap.skycell_id,warpSkyCellMap.class_id,comment from warpRun join warpSkyCellMap using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where warpRun.data_group like "STS.refstack.2012%" and skycell_id like "%.1.03%" order by exp_name; | exp_name | dateobs | warp_id | skycell_id | class_id | comment | +-------------+---------------------+---------+--------------+----------+--------------------------+ | o5705g0379o | 2011-05-24 12:08:55 | 410461 | skycell.1.03 | XY67 | STS.270.GIQ PP3 dither 1 | | o5705g0379o | 2011-05-24 12:08:55 | 410461 | skycell.1.03 | XY57 | STS.270.GIQ PP3 dither 1 | | o5705g0386o | 2011-05-24 12:13:00 | 410467 | skycell.1.03 | XY57 | STS.270.GIQ PP3 dither 2 | | o5705g0386o | 2011-05-24 12:13:00 | 410467 | skycell.1.03 | XY67 | STS.270.GIQ PP3 dither 2 | | o5705g0387o | 2011-05-24 12:13:35 | 410475 | skycell.1.03 | XY71 | STS.270.GIQ PP4 dither 3 | | o5705g0387o | 2011-05-24 12:13:35 | 410475 | skycell.1.03 | XY72 | STS.270.GIQ PP4 dither 3 | | o5705g0393o | 2011-05-24 12:17:04 | 410478 | skycell.1.03 | XY67 | STS.270.GIQ PP3 dither 3 | | o5705g0393o | 2011-05-24 12:17:04 | 410478 | skycell.1.03 | XY57 | STS.270.GIQ PP3 dither 3 | | o5705g0394o | 2011-05-24 12:17:38 | 410469 | skycell.1.03 | XY72 | STS.270.GIQ PP4 dither 4 | | o5705g0394o | 2011-05-24 12:17:38 | 410469 | skycell.1.03 | XY71 | STS.270.GIQ PP4 dither 4 | | o5705g0395o | 2011-05-24 12:18:14 | 409630 | skycell.1.03 | XY02 | STS.270.GIQ PP1 dither 4 | | o5705g0395o | 2011-05-24 12:18:14 | 409630 | skycell.1.03 | XY03 | STS.270.GIQ PP1 dither 4 | | o5705g0395o | 2011-05-24 12:18:14 | 409630 | skycell.1.03 | XY04 | STS.270.GIQ PP1 dither 4 | | o5719g0324o | 2011-06-07 11:27:59 | 410516 | skycell.1.03 | XY10 | STS.180.GIQ PP4 dither 1 | | o5719g0324o | 2011-06-07 11:27:59 | 410516 | skycell.1.03 | XY20 | STS.180.GIQ PP4 dither 1 | | o5719g0325o | 2011-06-07 11:28:32 | 409632 | skycell.1.03 | XY36 | STS.180.GIQ PP1 dither 1 | | o5719g0325o | 2011-06-07 11:28:32 | 409632 | skycell.1.03 | XY37 | STS.180.GIQ PP1 dither 1 | | o5719g0325o | 2011-06-07 11:28:32 | 409632 | skycell.1.03 | XY46 | STS.180.GIQ PP1 dither 1 | | o5719g0325o | 2011-06-07 11:28:32 | 409632 | skycell.1.03 | XY47 | STS.180.GIQ PP1 dither 1 | | o5719g0325o | 2011-06-07 11:28:32 | 409632 | skycell.1.03 | XY26 | STS.180.GIQ PP1 dither 1 | | o5719g0325o | 2011-06-07 11:28:32 | 409632 | skycell.1.03 | XY27 | STS.180.GIQ PP1 dither 1 | | o5719g0330o | 2011-06-07 11:31:21 | 410528 | skycell.1.03 | XY71 | STS.180.GIQ PP3 dither 1 | | o5719g0330o | 2011-06-07 11:31:21 | 410528 | skycell.1.03 | XY72 | STS.180.GIQ PP3 dither 1 | | o5767g0396o | 2011-07-25 11:59:08 | 410618 | skycell.1.03 | XY57 | STS.000.ND PP4 center | | o5767g0396o | 2011-07-25 11:59:08 | 410618 | skycell.1.03 | XY67 | STS.000.ND PP4 center | | o5767g0397o | 2011-07-25 11:59:50 | 409645 | skycell.1.03 | XY50 | STS.000.ND PP1 center | | o5767g0397o | 2011-07-25 11:59:50 | 409645 | skycell.1.03 | XY51 | STS.000.ND PP1 center | | o5767g0397o | 2011-07-25 11:59:50 | 409645 | skycell.1.03 | XY30 | STS.000.ND PP1 center | | o5767g0397o | 2011-07-25 11:59:50 | 409645 | skycell.1.03 | XY31 | STS.000.ND PP1 center | | o5767g0397o | 2011-07-25 11:59:50 | 409645 | skycell.1.03 | XY40 | STS.000.ND PP1 center | | o5767g0397o | 2011-07-25 11:59:50 | 409645 | skycell.1.03 | XY41 | STS.000.ND PP1 center | | o5767g0402o | 2011-07-25 12:03:25 | 410614 | skycell.1.03 | XY05 | STS.000.ND PP3 center | | o5767g0402o | 2011-07-25 12:03:25 | 410614 | skycell.1.03 | XY06 | STS.000.ND PP3 center | }}} Will also be directly overlapping skycells for diffims with at least slightly different refstacks: || skycell..xx || possible overlap skycell..xx || || || || || || 1.03 || 4.71,4.72; 3.57, 3.67 || || || || || ---- = Refstack = Reference stack using data from 2011 only, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, large elongation. == Chip->Warp processing == Only reprocessing a selected sample to use in the refstack only -- * label STS.refstack.20120510 * data_group STS.refstack.20120510 * dist_group NULL * workdir: neb://@HOST@.0/gpc1/STS.refstack.20120510 Test sample: {{{ chiptool -definebyquery -set_label STS.mehtest -set_workdir neb://@HOST@.0/meh/STS.mehtest -set_data_group STS.mehtest -set_dist_group NULL -set_tess_id STS.V4 -set_end_stage warp -dbname gpc1 -dateobs_begin 2011-05-24 -dateobs_end 2011-05-25 -filter i% -comment STS% -obs_mode STS -simple -pretend >& teststschip.log --> 5/24 has 84 exposures, so need more restrictive for test -- do o5705g0373o chiptool -definebyquery -set_label STS.mehtest -set_workdir neb://@HOST@.0/meh/STS.mehtest -set_data_group STS.mehtest -set_dist_group NULL -set_tess_id STS.V4 -set_end_stage warp -set_reduction STS_DATASET -dbname gpc1 -exp_name o5705g0373o -simple -pretend >& teststschip_o5705g0373o.log -- test run - STS.mehtest - don't see corner unglow in camera image, dynamic range such that not see strongly? neb://@HOST@.0/meh/STS.mehtest/o5705g0373o.340508 --> see some patterns in warp background for sample -- significant? see meh/STS.mehtest/o5705g0373o.340508/o5705g0373o.340508.wrp.408756.skycell.1.04.fits }}} Determine cuts for exposure sample to reprocess, pick of order 50 exposure sample: * time range {{{ | o5666g0560o | 324290 | STS | STS | 30 | 2011-04-15 14:01:18 | i.00000 | STS PP6 test visit 2 | | o5666g0561o | 324291 | STS | STS | 30 | 2011-04-15 14:02:10 | i.00000 | STS PP7 test visit 2 | | o5667g0517o | 324909 | RINGS.V3 | STS | 30 | 2011-04-16 14:39:15 | i.00000 | STS PP8 test visit 1 | | o5667g0518o | 324910 | RINGS.V3 | STS | 30 | 2011-04-16 14:39:58 | i.00000 | STS PP9 test visit 1 | .. | o5681g0558o | 332242 | RINGS.V3 | STS | 30 | 2011-04-30 15:02:10 | i.00000 | STS PP6 test visit 2 | | o5681g0559o | 332243 | RINGS.V3 | STS | 30 | 2011-04-30 15:03:32 | i.00000 | STS PP3 test visit 2 | | c5681g0007d | 332244 | NULL | STS | 30 | 2011-04-30 15:05:39 | i.00000 | STS PP3 test visit 2 | | o5695g0143o | 335717 | NULL | STS | 30 | 2011-05-14 11:06:19 | i.00000 | STS.180.NOM PP4 center | .. | o5695g0144o | 335718 | NULL | STS | 30 | 2011-05-14 11:07:04 | i.00000 | STS.180.NOM PP1 center | | o5695g0225o | 335799 | NULL | STS | 30 | 2011-05-14 12:06:25 | i.00000 | STS.180.NOM PP6 dither 5 | | o5695g0226o | 335800 | NULL | STS | 30 | 2011-05-14 12:07:09 | i.00000 | STS.180.NOM PP3 dither 5 | | o5705g0373o | 340508 | STS.V4 | STS | 20 | 2011-05-24 12:05:27 | i.00000 | STS.270.GIQ PP4 dither 1 | -- talking with Bill, looks like when to start -- | o5705g0374o | 340510 | STS.V4 | STS | 20 | 2011-05-24 12:06:00 | i.00000 | STS.270.GIQ PP1 dither 1 | }}} * camera FWHM, ZPT, background cut * dither content check {{{ SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) WHERE obs_mode="STS" and dateobs>"2011-05-23" and dateobs<"2012-01" and fwhm_major<3.8 and zpt_obs>24.6 and camProcessedExp.bg<100.0 and comment like "%PP1%" group by exp_id; ... }}} * summary {{{ PP1 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 42, ~half center half dither (16,7,5,6,8) PP2 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 47, ~half center half dither PP3 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 53, ~half center half dither PP4 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 50, ~half center half dither PP5 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 52, ~half center half dither PP6 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 49, ~half center half dither (14,8,8,10,9) PP7 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 49, ~half center half dither (18,9,5,10,7) }}} Run PP1 set first with selected exposures and perl script to make reftest stacks: {{{ mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select rawExp.exp_id from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%" group by rawExp.exp_id;' | grep -v "exp_id" perl sts.v4.refstackexp.20120510_pp1.chip > run_pp1.chip.bat source run_pp1.chip.bat >& run_pp1.chip.log }}} Once reftest check out, run remaining PP2-7 exposures: {{{ mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select rawExp.exp_id from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment not like "%PP1%" group by rawExp.exp_id;' | grep -v "exp_id" perl sts.v4.refstackexp.20120510_pp2-7.chip > run_pp2-7.chip.bat source run_pp2-7.chip.bat >& run_pp2-7.chip.log --> set priority with lap @200 labeltool -definelabel -set_label STS.refstack.20120510 -set_priority 200 -dbname gpc1 labeltool -updatelabel -label STS.refstack.20120510 -set_priority 190 -dbname gpc1 }}} == Stack processing == * label: STS.refstack.20120510 * data_group: STS.refstack.20120510 * workdir: neb://@HOST@.0/gpc1/STS.reftest.20120510 * reduction: PS1_REFERENCE {{{ OUTPUT.NOCOMP BOOL TRUE OUTPUT.LOGFLUX BOOL FALSE OUTPUT.REPLICATE BOOL TRUE --> compared to other reduction recipes -- cut on input PSF in STACK_DEEP may be useful but can apply in selecting exposures for stacking (not 100% effective however). STACK_THREEPI METADATA PSF.INPUT.MAX F32 12.0 OUTPUT.NOCOMP BOOL FALSE OUTPUT.LOGFLUX BOOL TRUE STACK_DEEP METADATA OUTPUT.NOCOMP BOOL TRUE OUTPUT.LOGFLUX BOOL FALSE PSF.INPUT.MAX F32 7.5 PSF.INPUT.CLIP.NSIGMA F32 5.0 }}} Test stack (reftest) sample like made for MD skycell.055 with different number of input warps and FWHM cut using reprocessed sample (background,ZPT cut) * label: STS.reftest.20120510 * data_group: STS.reftest.20120510.c#f where #=comparison run, =cut want to avoid the corner then and do bottom edge 1.04,1.05 where good_frac ~50%, 1.05 has larger static mask to watch in rotations. 1.03 also should work and seems more stable static and good maskfrac. }}} * initial test sample: {{{ stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c1f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.03 >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell103.log ... --> sample unconvolved images -- doesn't seem to be fully filling in regions/gaps, see input details. gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899917.unconv.num.fits -- c1f39 (max inputs 36) gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.num.fits -- c1f38 (max inputs 23) gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.num.fits -- c1f37 (max inputs 12) gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899920.unconv.num.fits -- c0f45 (max inputs 44) --> input stats for likely FWHM cut, SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.reftest.20120510.c1f38" group by exp_id; | exp_id | exp_time | dateobs | zpt_obs | zpt_stdev | fwhm_major | elong | iq_elong | fwhm_iq_maj | bg | bg_stdev | uq | lq | n_stars | comment | +--------+----------+---------------------+---------+-----------+------------+-------+----------+-------------+--------+----------+------+------+---------+--------------------------+ | 340540 | 20 | 2011-05-24 12:22:25 | 24.5549 | 0.0259 | 3.69691 | 1.188 | 1.083 | 0.92 | 91.27 | 14.01 | 3.98 | 3.46 | 8648132 | STS.270.GIQ PP1 dither 1 | | 340558 | 20 | 2011-05-24 12:34:54 | 24.5559 | 0.026682 | 3.7342 | 1.148 | 1.073 | 0.86 | 88.49 | 14.04 | 4.04 | 3.45 | 8393568 | STS.270.GIQ PP1 dither 4 | | 340567 | 20 | 2011-05-24 12:39:02 | 24.5563 | 0.025387 | 3.62252 | 1.130 | 1.059 | 0.84 | 88.18 | 13.78 | 3.89 | 3.29 | 8588336 | STS.270.GIQ PP1 dither 1 | | 340573 | 20 | 2011-05-24 12:43:16 | 24.5591 | 0.025946 | 3.64969 | 1.127 | 1.050 | 0.82 | 87.11 | 13.21 | 4.05 | 3.38 | 8380448 | STS.270.GIQ PP1 dither 2 | | 344556 | 20 | 2011-06-07 11:28:32 | 24.4608 | 0.029333 | 3.59708 | 1.165 | 1.079 | 0.88 | 100.50 | 13.30 | 4.04 | 3.39 | 8487768 | STS.180.GIQ PP1 dither 1 | | 344562 | 20 | 2011-06-07 11:32:30 | 24.5164 | 0.024937 | 3.59372 | 1.150 | 1.069 | 0.87 | 97.07 | 13.32 | 3.99 | 3.37 | 8638928 | STS.180.GIQ PP1 dither 2 | | 344569 | 20 | 2011-06-07 11:36:21 | 24.5313 | 0.024183 | 3.69415 | 1.168 | 1.085 | 0.93 | 94.56 | 13.63 | 4.03 | 3.47 | 8641948 | STS.180.GIQ PP1 dither 3 | | 344575 | 20 | 2011-06-07 11:40:18 | 24.5333 | 0.027669 | 3.66644 | 1.154 | 1.070 | 0.91 | 94.17 | 13.25 | 3.99 | 3.39 | 8611680 | STS.180.GIQ PP1 dither 4 | | 344583 | 20 | 2011-06-07 11:44:12 | 24.4983 | 0.040863 | 3.72526 | 1.159 | 1.088 | 0.92 | 96.11 | 13.24 | 4.14 | 3.51 | 8467340 | STS.180.GIQ PP1 dither 1 | | 344589 | 20 | 2011-06-07 11:48:13 | 24.4937 | 0.046644 | 3.72833 | 1.136 | 1.068 | 0.90 | 95.45 | 13.70 | 4.02 | 3.51 | 8327116 | STS.180.GIQ PP1 dither 2 | | 344597 | 20 | 2011-06-07 11:52:11 | 24.4935 | 0.031675 | 3.71614 | 1.141 | 1.066 | 0.87 | 92.70 | 13.54 | 4.13 | 3.51 | 8277828 | STS.180.GIQ PP1 dither 3 | | 344605 | 20 | 2011-06-07 11:56:13 | 24.5409 | 0.025897 | 3.74193 | 1.160 | 1.063 | 0.90 | 89.26 | 13.15 | 4.15 | 3.54 | 8427112 | STS.180.GIQ PP1 dither 4 | | 367982 | 30 | 2011-07-25 11:59:50 | 24.5698 | 0.02507 | 3.79495 | 1.091 | 1.052 | 0.79 | 97.48 | 17.60 | 4.03 | 3.65 | 8991280 | STS.000.ND PP1 center | | 367996 | 30 | 2011-07-25 12:09:52 | 24.5652 | 0.026531 | 3.6958 | 1.104 | 1.064 | 0.79 | 97.83 | 19.54 | 4.02 | 3.49 | 9311700 | STS.000.ND PP1 center | | 368010 | 30 | 2011-07-25 12:19:54 | 24.5644 | 0.026506 | 3.74343 | 1.089 | 1.055 | 0.77 | 102.98 | 19.22 | 3.99 | 3.60 | 9108432 | STS.000.ND PP1 center | | 368833 | 30 | 2011-07-27 07:23:06 | 24.5521 | 0.026382 | 3.75974 | 1.153 | 1.061 | 0.93 | 114.75 | 17.14 | 4.16 | 3.49 | 9435552 | STS.180.ND PP1 center | | 393751 | 20 | 2011-09-18 08:24:25 | 24.5337 | 0.029066 | 3.54239 | 1.153 | 1.071 | 0.91 | 89.00 | 12.90 | 3.89 | 3.31 | 9090136 | STS.000.GIQ PP1 dither 1 | | 393763 | 20 | 2011-09-18 08:32:30 | 24.533 | 0.02565 | 3.60961 | 1.170 | 1.071 | 0.96 | 88.74 | 12.99 | 3.88 | 3.33 | 9028068 | STS.000.GIQ PP1 dither 3 | | 393778 | 20 | 2011-09-18 08:40:35 | 24.5299 | 0.027023 | 3.55016 | 1.177 | 1.072 | 0.92 | 92.47 | 13.17 | 3.88 | 3.30 | 9102116 | STS.000.GIQ PP1 dither 1 | | 393785 | 20 | 2011-09-18 08:44:33 | 24.5337 | 0.027504 | 3.6808 | 1.135 | 1.065 | 0.92 | 96.03 | 13.46 | 4.03 | 3.48 | 8654104 | STS.000.GIQ PP1 dither 2 | | 393792 | 20 | 2011-09-18 08:48:23 | 24.5315 | 0.026466 | 3.70284 | 1.164 | 1.073 | 0.97 | 94.42 | 13.26 | 4.00 | 3.44 | 8762736 | STS.000.GIQ PP1 dither 3 | | 393799 | 20 | 2011-09-18 08:52:22 | 24.5312 | 0.028034 | 3.71144 | 1.172 | 1.078 | 0.96 | 98.12 | 13.35 | 4.02 | 3.47 | 8695292 | STS.000.GIQ PP1 dither 4 | | 393835 | 20 | 2011-09-18 09:13:19 | 24.5271 | 0.027369 | 3.75097 | 1.157 | 1.076 | 0.99 | 103.99 | 13.49 | 4.07 | 3.47 | 8624396 | STS.000.GIQ PP1 dither 1 | --> cuts out a larger fraction of the center position --> compared to log file for ppStack -- gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.log Input 0: 3.137970 Input 1: 3.397691 Input 2: 3.309175 Input 3: 3.310735 Input 4: 3.114384 Input 5: 3.043151 Input 6: 3.346249 Input 7: 3.141845 Input 8: 3.471485 Input 9: 3.238341 Input 10: 3.433451 Input 11: 3.225220 Input 12: 4.423095 Input 13: 4.086733 Input 14: 4.282342 Input 15: 4.067309 Input 16: 3.030721 Input 17: 4.047925 Input 18: 4.245076 Input 19: 4.328331 Input 20: 4.284031 Input 21: 4.074796 Input 22: 4.197470 Input FWHMs : 3.662502 +/- 0.504006 fit ext: 0.965282 sec for 225 of 225 sources fit psf: 0.342076 sec for 196 of 225 sources apresid: 0.046344 +/- 0.000000 (systematic error) from statistics of 225 psf stars (196 used) chisq vs flux fit term 0: 0.000000 +/- 0.000000 chisq vs flux fit term 1: 0.000000 +/- 0.000000 try model PS_MODEL_PS1_V1, ap-fit: 0.046344 +/- 0.000000 : sky bias: 0.000000 Target seeing FWHM: 4.430518 ... fwhm (psf): 4.472250,4.366182 (moments): 0.000000,0.000000 --> even for c1f37 has FWHM>4.2 going into it -- gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.log --> however, not as overdriven as feared. --- need to add plot/vector of PSF shape in skycell --- }}} psphot on samples, unconvolved and convolved (compare to ppStack log file): * faults because of recent modified psphot config/code to limit peaks and so faults.. need to add -Di PEAKS_NMAX_TOTAL 60000 * unconvolved stacks still faulting.. so do PASS1 only -D BREAK_POINT PASS1. Convolved ones can be poor, FWHM_MAJ~9-10?? {{{ gethead -auhc *fpa.cmf FWHM_MAJ IQ_FW1 STS.V4.skycell.1.03.stk.899917.fpa.cmf 4.405078 3.746344 STS.V4.skycell.1.03.stk.899917.unconv.fpa.cmf 3.921824 3.04026 STS.V4.skycell.1.03.stk.899918.fpa.cmf 4.359843 3.701267 STS.V4.skycell.1.03.stk.899918.unconv.fpa.cmf 3.788124 2.973968 STS.V4.skycell.1.03.stk.899919.fpa.cmf 4.184109 3.576873 STS.V4.skycell.1.03.stk.899919.unconv.fpa.cmf 3.835106 2.939088 STS.V4.skycell.1.03.stk.899920.fpa.cmf 4.54595 3.812682 STS.V4.skycell.1.03.stk.899920.unconv.fpa.cmf 3.862908 3.048509 --> basically as with MD, +/- pixel, so all are basically similar and going to smaller FWHM cut not improve much, partially because cut doen't always match true FWHM going in. }}} * previous refstack.20110613 {{{ gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.fits (cleaned) gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.cmf gpc1/STS.refstack.20110613/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.310672.log }}} Summary of test runs for skycell.1.03: || || || || || || '''version data_group''' (stack_id) || '''filter''' || '''skycell input warps (rej)''' || '''skycell FWHM (major) conv/unconv''' || || || || || || || new ''STS.refstack.20120510'' V4 || i || 61 (2) || 1.38/0.98" || || || || || || || STS.reftest.20120510.c1f39 V4 (899917) || i || 36 (0) || 1.10/0.98" || || STS.reftest.20120510.c1f38 V4 (899918) || i || 34 (0) || 1.09/0.95" || || STS.reftest.20120510.c1f37 V4 (899919) || i || 12 (0) || 1.05/0.96" || || STS.reftest.20120510.c0f45 V4 (899920) || i || 44 (0) || 1.14/0.97" || || || || || || || old STS.refstack.20110613 V3 || i || 68 (0) || ~1.16/0.9" (approx from log since image cleaned) || || || || || || Look at recent camera/warps to check similar coverage * recent observations all 180 (as of 120510) * center+dither available * sample from rawExp table: {{{ | 484662 | o6054g0382o | 2012-05-07 13:44:36 | i.00000 | 30 | STS.180.ND PP1 center gpc1/STS.nt/2012/05/07/o6054g0382o.484662/ gpc1/STS.nt/2012/05/07/o6054g0382o.484662/o6054g0382o.484662.wrp.408115.skycell.1.03.fits --> bad tracking image, startrails.. try another gpc1/STS.nt/2012/05/07/o6054g0375o.484656/ gpc1/STS.nt/2012/05/07/o6054g0375o.484656/o6054g0375o.484656.wrp.408110.skycell.1.03.fits --> yes, similar pattern - }}} Main run -- do by projection PP 1-7: * use FWHM cut 3.8 for ~20-40 and see -- if some need fixing, then re-run that skycell, if too much burntool then know for next version * should really add the PSF cut to PS1_REFERENCE too to help reject unrepresentative camera FWHM for skycell. * run in normal stack, system fairly loaded w/o deepstack running? no, the c3 machines need more work {{{ stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.refstack.20120510 -set_label STS.refstack.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.refstack.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.8 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple >& stacktool.sts_refstack.20120510.fwhm38_elong12.log ---> oops forgot to add -select_skycell_id skycell.1.% -- so it is running all projections at once. }}} Compare to test sample skycell.1.03: {{{ gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.unconv.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.log --> run psphot manually and add to table above -- need to set 65000 detections for unconv stack }}} Look at final stacks: bottom line, more inputs would help, but maybe this is sufficient * 4.17 odd bright cell print through * 2.02 odd faint chunk print through * 2.39 gap faint print through * 5.06 missed burntool? * 1.09,1.17 large unfilled gap * 1.24 long unfilled gap * 1.25,1.35 long and large unfilled gap * 1.12 large unfilled cell * 1.40 unfilled masking corner multi-cell * 1.55 rounded edge + corner adds other exp * 1.44,5.14 odd bright orphan diffspike-like unmasked * 1.11 nice cluster * 1.32 faint flux print through from brighter cell Run additional reftest for selected examples: * skycell.1.35 -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.902937.log {{{ stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35 >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell135_c2.log stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f45 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 4.5 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35 >& stacktool.sts_reftest.20120510.fwhm45_elong12_skycell135_c2.log stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.35 >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell135_c2.log ---> the additional input 45 shows lower corner with much worse seeing -- this is problem, each skycell may need to be tuned/limited directly since DB not have proper stats. A reason for using conv stacking }}} * skycell.1.40 -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.902942.log {{{ stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f39 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.9 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40 >& stacktool.sts_reftest.20120510.fwhm39_elong12_skycell140_c2.log stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f45 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 4.5 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40 >& stacktool.sts_reftest.20120510.fwhm45_elong12_skycell140_c2.log stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.40 >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell140_c2.log ---> max input not fill in masking much }}} * skycell.1.55 -- mix of PP inputs and possible IQ issues, try reducing Ninput greatly. really need to do manually.. -- gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.log {{{ stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f36 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.6 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.55 >& stacktool.sts_reftest.20120510.fwhm36_elong12_skycell155_c2.log stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group STS.reftest.20120510.c2f37 -set_label STS.reftest.20120510 -set_workdir neb://@HOST@.0/gpc1/STS.reftest.20120510 -select_data_group STS.refstack.20120510 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 3.7 -select_elong_max 1.2 -set_reduction PS1_REFERENCE -simple -select_skycell_id skycell.1.55 >& stacktool.sts_reftest.20120510.fwhm37_elong12_skycell155_c2.log ---> the fewer inputs makes the IQ worse in upper left, reason to use conv stacks here }}} Look at Target seeing to see if any odd inputs driving target PSF: * Target seeing for PSF {{{ mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select path_base from stackRun join stackSumSkyfile using (stack_id) where stackRun.label like "STS.refstack.20120510" and stackRun.skycell_id like "%.1.%";' | grep -v path_base | awk '{print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -l -p "$1"%.log"}' | tcsh | awk '{print " cat "$9}' | tcsh | grep "Target seeing" Target seeing FWHM: 3.871291 Target seeing FWHM: 5.136444 Target seeing FWHM: 9.224628 Target seeing FWHM: 5.216270 Target seeing FWHM: 4.238182 Target seeing FWHM: 4.433765 Target seeing FWHM: 8.005375 Target seeing FWHM: 5.200400 ... --> look at couple with FWHM 8-9 -- #3,7 -- manually match to input file: gpc1/STS.refstack.20120510/STS.V4/skycell.1.02/STS.V4.skycell.1.02.stk.902904.log -- 83 Ninput, largest ~5.7 so wacky shapes? even 2.89 FWHM gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.log -- 78 Ninput, largest ~4.5 so wacky shapes? }}} * create log of Input FWHM, Target seeing, and convolved PSF FWHM: {{{ mysql -hippdb03 -uippuser -pxxxx gpc1 -B -e 'select path_base from stackRun join stackSumSkyfile using (stack_id) where stackRun.label like "STS.refstack.20120510" and stackRun.skycell_id like "%";' | grep -v path_base | awk '{print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -l -p "$1"%.log"}' | tcsh | awk '{print "echo "$9"; grep -e \" Input \" -e \"Target seeing\" -e \"fwhm (psf)\" "$9" ;echo \" \""}' | tcsh > STS.refstack.20120510_input_target_output_fwhm_inputlong.log }}} == clean-up == Will need to keep warps around for remaking the refstack if necessary but more importantly for MOPS to get stamps without update delay. ~50x7 or 350 exposures on disk for the skycells. {{{ chiptool -updaterun -set_state goto_cleaned -set_label goto_cleaned -data_group STS.refstack.20120510 -dbname gpc1 //warptool -updaterun -set_state goto_cleaned -set_label goto_cleaned -data_group STS.refstack.20120510 -dbname gpc1 }}} ---- = Test Diffim Samples = Want to create a sample of diffims with detections for MOPS to look at. Will need to also keep in "full" state so they can get postage stamps quickly. Focus on projection 1 (PP1 exposures, skycell.1.xx). Basic setup: (warp - convolved stack) -- need examples and compared to unconv stack (why not using unconv for any WSdiff?) * what are diffims for other products -- WS, target1, conv vs unconv stack -- part of what needed to do for ppsub testing page still * MOPS has no specific requirements for test samples, so try to combine multiple aspects using skycell.1.xx * W-S with conv refstack since unconv refstack not an option due to code issues in the filename. * 2011 -- 1 or 2 nights -- ones not used in refstack (may be difficult), will want to be manually updated so no chance of being (auto)-cleaned * 2012 -- 1 or 2 nights -- will want to be manually updated so no chance of cleaning as well * not sure how to queue/run these since normally driven by a survey task. could do last nights for 2012 right off but cleanup window so work with older sample * need to setup/change publishing location? * turn off cleanup for the STS for now? no, uncertain when would turn back on and want continue cleanup as normal. == Manual Test Set == Want to run quick set locally. Pick center, edge, edge+lots overlap for various filling types - 1.44, 1.55, 1.03? {{{ gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.log gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.log gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.log --> appear mostly well behaved mix.. need to add one more -- 1.06 is wacky gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.log }}} Run with convolve target2,1 and dual (can use simiple scripts used for ppSub tests) -- in testppsub_sts120510: * need recent night listing with PP1 exp_name - {{{ select exp_id,exp_name,exp_time,dateobs,comment from rawExp where obs_mode like "%STS%" and comment like "%PP1%" and dateobs>"2012-04"; | exp_id | exp_name | exp_time | dateobs | comment | +--------+-------------+----------+---------------------+--------------------------+ | 475121 | o6029g0441o | 30 | 2012-04-12 13:40:09 | STS.180.ND PP1 center | | 475128 | o6029g0448o | 30 | 2012-04-12 13:45:11 | STS.180.ND PP1 center | | 475135 | o6029g0455o | 30 | 2012-04-12 13:50:14 | STS.180.ND PP1 center | ... | 483638 | o6049g0349o | 30 | 2012-05-02 13:54:49 | STS.180.ND PP1 center | | 483646 | o6049g0356o | 30 | 2012-05-02 13:59:51 | STS.180.ND PP1 center | | 483651 | o6049g0328o | 30 | 2012-05-02 13:39:38 | STS.180.ND PP1 center | | 483652 | o6049g0363o | 30 | 2012-05-02 14:04:52 | STS.180.ND PP1 center | | 484578 | o6054g0305o | 30 | 2012-05-07 12:48:56 | STS.180.ND PP1 center | | 484590 | o6054g0312o | 30 | 2012-05-07 12:53:58 | STS.180.ND PP1 center | | 484599 | o6054g0319o | 30 | 2012-05-07 12:58:58 | STS.180.ND PP1 center | | 484606 | o6054g0326o | 30 | 2012-05-07 13:04:01 | STS.180.ND PP1 center | | 484613 | o6054g0333o | 30 | 2012-05-07 13:09:03 | STS.180.ND PP1 center | | 484620 | o6054g0340o | 30 | 2012-05-07 13:14:04 | STS.180.ND PP1 center | | 484627 | o6054g0347o | 30 | 2012-05-07 13:19:09 | STS.180.ND PP1 center | | 484634 | o6054g0354o | 30 | 2012-05-07 13:24:13 | STS.180.ND PP1 center | | 484641 | o6054g0361o | 30 | 2012-05-07 13:29:18 | STS.180.ND PP1 center | | 484648 | o6054g0368o | 30 | 2012-05-07 13:34:25 | STS.180.ND PP1 center | | 484656 | o6054g0375o | 30 | 2012-05-07 13:39:31 | STS.180.ND PP1 center | | 484662 | o6054g0382o | 30 | 2012-05-07 13:44:36 | STS.180.ND PP1 center | +--------+-------------+----------+---------------------+--------------------------+ neb://@HOST@.0/gpc1/STS.nt/2012/05/02/o6049g0286o -- cleaned.. neb://@HOST@.0/gpc1/STS.nt/2012/05/07/o6054g0305o -- okay --> do most recent 2 first - first in list - need script liner to get more of these --> will want to know FWHM of input warps too -- will be in ppSub log files }}} * skycell.1.44 sample: {{{ neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/%unconv%fits nebds9 gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits --> @4610,5580 region see stars look elongated/trailed versus ones nearby.. looks like multi-cell mask region -- diffims will show if problem --> maybe has to do with GIQ? ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.44.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120507 > runppsub_sts144_120507_1.bat source runppsub_sts144_120507_1.bat >& runppsub_sts144_120507_1.log & ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120507 > runppsub_sts144_120507_2.bat source runppsub_sts144_120507_2.bat >& runppsub_sts144_120507_2.log & --> created batch script runppsub_sts144_120507_2.bat for example ppSub ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.20120507 -inimage neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits -refimage neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits -inmask neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.mask.fits -refmask neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.mask.fits -invariance neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.wt.fits -refvariance neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.wt.fits -insources neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.cmf -refsources neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.cmf -D CONVOLVE.TARGET SINGLE2 -save-inconv -save-refconv -inverse -recipe PPSUB WARPSTACK -recipe PSPHOT DIFF -recipe PPSTATS DIFFSTATS -F PSPHOT.PSF.SAVE PSPHOT.PSF.SKY.SAVE -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF -F PSPHOT.BACKMDL PSPHOT.BACKMDL.MEF -photometry -vvv -debug -trace err 10 -trace ppSub 10 -trace psModules.imcombine 10 -trace psModules.camera 10 -tracedest ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.trace -log ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.log -stats ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.stats -threads 4 -dumpconfig ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_2.mdc ppSub ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.20120507 -inimage neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits -refimage neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits -inmask neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.mask.fits -refmask neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.mask.fits -invariance neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.wt.fits -refvariance neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.wt.fits -insources neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.cmf -refsources neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.cmf -D CONVOLVE.TARGET SINGLE1 -save-inconv -save-refconv -inverse -recipe PPSUB WARPSTACK -recipe PSPHOT DIFF -recipe PPSTATS DIFFSTATS -F PSPHOT.PSF.SAVE PSPHOT.PSF.SKY.SAVE -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF -F PSPHOT.BACKMDL PSPHOT.BACKMDL.MEF -photometry -vvv -debug -trace err 10 -trace ppSub 10 -trace psModules.imcombine 10 -trace psModules.camera 10 -tracedest ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.trace -log ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.log -stats ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.stats -threads 4 -dumpconfig ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_1.mdc ppSub ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.20120507 -inimage neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.fits -refimage neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits -inmask neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.mask.fits -refmask neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.mask.fits -invariance neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.wt.fits -refvariance neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.wt.fits -insources neb://@HOST@.0//gpc1/STS.nt/2012/05/07/o6054g0312o.484590/o6054g0312o.484590.wrp.408044.skycell.1.44.cmf -refsources neb://@HOST@.0//gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.cmf -D CONVOLVE.TARGET SINGLE1 -Db DUAL T -save-inconv -save-refconv -inverse -recipe PPSUB WARPSTACK -recipe PSPHOT DIFF -recipe PPSTATS DIFFSTATS -F PSPHOT.PSF.SAVE PSPHOT.PSF.SKY.SAVE -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF -F PSPHOT.BACKMDL PSPHOT.BACKMDL.MEF -photometry -vvv -debug -trace err 10 -trace ppSub 10 -trace psModules.imcombine 10 -trace psModules.camera 10 -tracedest ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.trace -log ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.log -stats ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.stats -threads 4 -dumpconfig ./20120507/o6054g0312o.484590.wrp.408044.skycell.1.44.WS_d.mdc }}} * skycell.1.55 sample: {{{ neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/%unconv%fits --> also see odd shape near center but is edge field, but has both PP1 and PP2 inputs - ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.55.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.902957.unconv.fits 20120507 > runppsub_sts155_120507_1.bat source runppsub_sts155_120507_1.bat >& runppsub_sts155_120507_1.log & }}} * skycell.1.03 sample: {{{ neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/%unconv%fits --> looks ok, good overlap mix pp1/4 ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.902905.unconv.fits 20120507 > runppsub_sts103_120507_1.bat source runppsub_sts103_120507_1.bat >& runppsub_sts103_120507_1.log & }}} * skycell.1.06 sample: {{{ neb-ls gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/%unconv%fits --> looks ok, crazy overlap mix pp1,2,4 and maybe odd shape in corner ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.06.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.unconv.fits 20120507 > runppsub_sts106_120507_1.bat source runppsub_sts106_120507_1.bat >& runppsub_sts106_120507_1.log & }}} Overall, diffims look poor. MD04 from 20120508 WS look okay, so look at a reftest sample as well {{{ --> update/pss request an STS field from 5/2? what was fwhm? SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) WHERE obs_mode="STS" and dateobs>"2012-01-01" and comment like "%PP1%" group by exp_id; --> next one on 5/7 is better fwhm so try that first - same --> has ND in comment name -- that mean anything? --> try 4/20 o6037g0506o -- will need to update - warp_id=400804, skycell.1.44 bundle -- FWHM~3.8 --> better, at least messy resid but the things that look like diffim det look odd, some look like things removed in pixel rejection?? gpc1/STS.nt/2012/04/20/o6037g0506o.478582/o6037g0506o.478582.wrp.400804.skycell.1.44.fits ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/20/o6037g0506o.478582/o6037g0506o.478582.wrp.400804.skycell.1.44.fits gpc1/STS.refstack.20120510/STS.V4/skycell.1.44/STS.V4.skycell.1.44.stk.902946.unconv.fits 20120420 > runppsub_sts144_120420_x.bat source runppsub_sts144_120420_x.bat >& runppsub_sts144_120420_x.log & }}} Need to retry diffims above as conv stacks too - seems the same, how is that.. --> need to look at the mask images as well - * sample reftest diffims {{{ mysql> select stack_id,data_group,skycell_id from stackRun where data_group like "STS.reftest%c%f%"; +----------+----------------------------+--------------+ | stack_id | data_group | skycell_id | +----------+----------------------------+--------------+ | 899920 | STS.reftest.20120510.c0f45 | skycell.1.03 | | 899919 | STS.reftest.20120510.c1f37 | skycell.1.03 | | 899918 | STS.reftest.20120510.c1f38 | skycell.1.03 | | 899917 | STS.reftest.20120510.c1f39 | skycell.1.03 | | 904603 | STS.reftest.20120510.c2f36 | skycell.1.55 | | 904604 | STS.reftest.20120510.c2f37 | skycell.1.55 | | 904605 | STS.reftest.20120510.c2f37 | skycell.1.40 | | 904606 | STS.reftest.20120510.c2f37 | skycell.1.35 | | 904599 | STS.reftest.20120510.c2f39 | skycell.1.35 | | 904601 | STS.reftest.20120510.c2f39 | skycell.1.40 | | 904600 | STS.reftest.20120510.c2f45 | skycell.1.35 | | 904602 | STS.reftest.20120510.c2f45 | skycell.1.40 | gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899917.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899920.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904599.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904600.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.35/STS.V4.skycell.1.35.stk.904606.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.904603.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.55/STS.V4.skycell.1.55.stk.904604.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904601.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904602.unconv.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.40/STS.V4.skycell.1.40.stk.904605.unconv.fits }}} {{{ ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.fits 20120507_c1f38 > runppsub_sts103_120507_c1f38.bat source runppsub_sts103_120507_c1f38.bat >& runppsub_sts103_120507_c1f38.log & --> ack.. been cleaned and no others since.. will need to update chips too.. so tedious, use pss - //warptool -dbname gpc1 -setskyfiletoupdate -warp_id 408038 -skycell_id skycell.1.03 -set_label --> need to copy over locally - --> for conv, edit the .bat, make WSc and drop the unconv cp runppsub_sts103_120507_c1f38.bat runppsub_sts103_120507_c1f38c.bat source runppsub_sts103_120507_c1f38c.bat >& runppsub_sts103_120507_c1f38c.log & --> same, so guess that is good.. try c1f37 - ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/07/o6054g0305o.484578/o6054g0305o.484578.wrp.408038.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.fits 20120507_c1f37 > runppsub_sts103_120507_c1f37.bat source runppsub_sts103_120507_c1f37.bat >& runppsub_sts103_120507_c1f37.log & --> same.. are the exp_time different? yes 4/20 is 20s.. why would that matter.. IQ is better too, so maybe recent ones are just wacky IQ? }}} * run psphot on the refstack or staticsky? staticsky would be overkill and 1 filter anyways. {{{ survey.add.publish STS.mopsdiff STS.mopsdiffv2.mehtest 12 NULL survey.show.publish survey.del.publish STS.mopsdiff add.label STS.mopsdiffv2.mehtest }}} == Diffim sample 1 (faulted) == Sample diffims for projection 1 faulted in publishing. So this is more of an example of some things of what not to do.. --> dont know how to tell fwhm other than from db - do 1.03 since edge field and wacky overlap, but need to know what chips go into that? no, camProcessedExp * need to know rough FWHM of refstack and nightly exposures to pick sample * need to pick one from 2011 not used in refstack SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) WHERE comment like "STS%PP1%" group by exp_id; * refstack ~3.94 * 120420 - look ok * 120507 ~7.52 (7.9 in cam, wrong way even for cam/warp scaling) -- looked poor, don't use as test select exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.refstack.20120510" and stackRun.skycell_id like "%1.03" and comment like "%PP1%" group by exp_id; SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" group by exp_id; //SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" and not in ( select * FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.refstack.20120510" and stackRun.skycell_id like "%1.03" and comment like "%PP1%" ) group by exp_id; | 393855 | o5822g0346o | 20 | 2011-09-18 09:25:14 | 24.6354 | 3.87493 | 1.166 | 1.08 | 105.39 | 13.63 | 4.11 | 3.61 | 8444848 | STS.000.GIQ PP1 dither 4 | --> 2012 sample to test before updating SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) WHERE comment like "STS%PP1%" group by exp_id; | 475121 | o6029g0441o | 30 | 2012-04-12 13:40:09 | 24.5321 | 4.06258 | 1.119 | 1.05 | 315.49 | 21.55 | 4.21 | 3.82 | 8078200 | STS.180.ND PP1 center | | 476338 | o6031g0408o | 20 | 2012-04-14 13:34:26 | 24.5537 | 4.27532 | 1.180 | 0.98 | 117.51 | 14.15 | 4.77 | 4.05 | 7485604 | STS.180.GIQ PP1 dither 1 | | 477549 | o6033g0430o | 30 | 2012-04-16 13:31:05 | 24.5586 | 7.96196 | 1.194 | 1.83 | 154.22 | 18.30 | 8.08 | 7.71 | 4586012 | STS.180.ND PP1 center | ++ | 478479 | o6037g0401o | 20 | 2012-04-20 13:46:26 | 24.5799 | 4.1334 | 1.174 | 1.06 | 76.31 | 12.57 | 4.42 | 3.87 | 8291412 | STS.180.GIQ PP1 dither 1 | | 479810 | o6040g0441o | 30 | 2012-04-23 13:31:03 | 24.608 | 6.12941 | 1.131 | 1.20 | 147.86 | 17.88 | 6.39 | 5.88 | 5637788 | STS.180.ND PP1 center | | 482593 | o6045g0420o | 30 | 2012-04-28 13:37:39 | 24.6132 | 6.26513 | 1.051 | 1.07 | 109.03 | 16.37 | 6.59 | 6.09 | 5474888 | STS.180.ND PP1 center | | 483576 | o6049g0286o | 30 | 2012-05-02 13:09:26 | 24.5922 | 4.77424 | 1.088 | 0.94 | 166.69 | 17.89 | 5.10 | 4.62 | 7102152 | STS.180.ND PP1 center | /// | 484578 | o6054g0305o | 30 | 2012-05-07 12:48:56 | 24.5422 | 7.94199 | 1.042 | 0.76 | 297.28 | 23.85 | 8.11 | 7.69 | 3526740 | STS.180.ND PP1 center | | 487890 | o6062g0361o | 30 | 2012-05-15 12:00:59 | 24.5415 | 4.14248 | 1.177 | 1.04 | 146.86 | 16.61 | 4.58 | 3.86 | 8390052 | STS.180.ND PP1 center | --> on disk still so try also --> need warp_ids? try exposure name - in any case need to see state in - yes exp_name works - select * from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_name="o6062g0361o"; --> try to at least include 4/14 sing a GIQ? 4/20 cleaned but know work so 4/14 not necessary -- --> have to use the conv stack.. select uri from warpSkyfile where warp_id=409660 and skycell_id like "%1.03"; ./run_ppsub_night_ws.csh gpc1/STS.refstack.20120510/o5822g0346o.393855/o5822g0346o.393855.wrp.409660.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20110918c > runppsub_sts103_110918_1c.bat source runppsub_sts103_110918_1c.bat >& runppsub_sts103_110918_1c.log & --> good - select warpSkyfile.uri from warpSkyfile join warpRun using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_name="o6029g0441o" and warpSkyfile.skycell_id like "%1.03"; ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/12//o6029g0441o.475121/o6029g0441o.475121.wrp.396474.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120412c > runppsub_sts103_120412_1c.bat source runppsub_sts103_120412_1c.bat >& runppsub_sts103_120412_1c.log & --> bilster cores (undersubs) select warpSkyfile.uri from warpSkyfile join warpRun using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where warpSkyfile.skycell_id like "%1.03" and exp_name="o6031g0408o"; ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/14//o6031g0408o.476338/o6031g0408o.476338.wrp.397423.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120414c > runppsub_sts103_120414_1c.bat source runppsub_sts103_120414_1c.bat >& runppsub_sts103_120414_1c.log & --> blister cores (undersub) ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/16//o6033g0430o.477549/o6033g0430o.477549.wrp.398532.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120416c > runppsub_sts103_120416_1c.bat source runppsub_sts103_120416_1c.bat >& runppsub_sts103_120416_1c.log & --> large conv, oversub ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/23//o6040g0441o.479810/o6040g0441o.479810.wrp.401931.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120423c > runppsub_sts103_120423_1c.bat source runppsub_sts103_120423_1c.bat >& runppsub_sts103_120423_1c.log & --> larger conv, maybe slight oversub ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/28//o6045g0420o.482593/o6045g0420o.482593.wrp.405569.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120428c > runppsub_sts103_120428_1c.bat source runppsub_sts103_120428_1c.bat >& runppsub_sts103_120428_1c.log & --> larger conv, oversub ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/02//o6049g0286o.483576/o6049g0286o.483576.wrp.406354.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120502c > runppsub_sts103_120502_1c.bat source runppsub_sts103_120502_1c.bat >& runppsub_sts103_120502_1c.log & --> slightly larger conv, maybe slight oversub ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/15//o6062g0361o.487890/o6062g0361o.487890.wrp.415613.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120515c > runppsub_sts103_120515_1c.bat source runppsub_sts103_120515_1c.bat >& runppsub_sts103_120515_1c.log & --> low conv, blister undersubs --> looks like 110918, 120420, 120423/120502 -- use 4/23, seeing some odd conv/pixrej artifacts coming out in diffim but not sure if detected and 30s version --> may also want to try unconv since STS is supposed to be done w/in certain IQ? -- set select nights up for updates - number exposures ~ 16(20s)/16(20s)/12(30s) for 110918, 120420, 120423 --> label STS.mehdifftest --> will need to script mysql call based on what to get the chip/warp_id? detrend different since? may not be unique run given reprocessing for refstack oh well.. need PP1 as well --> manually checked and all unique.. select exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" group by exp_name; chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id warptool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id --> will have some clean state issues possibly.. --> note dateobs is 2 groups.. *** 110918 may have magic issues.. may have to just reprocess it - try update since just a test anyways and see -- did update for test case anyways so should work mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_110918.bat mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-20%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120420.bat mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-23%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120423.bat --> warp updates driving ipp017 nuts.. ~10-20% ram for 5 runs for total ~75%, cpu_wait and swap overload. killing 1 at time 1 ipp017 BUSY 2993.25 0.0.5.fe34 0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400730 --warp_skyfile_id 34217254 --skycell_id skycell.1.22 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0429o.478506/o6037g0429o.478506.wrp.400730.skycell.1.22 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose 2 ipp017 RESP 3369.78 0.0.6.269 0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400756 --warp_skyfile_id 34219846 --skycell_id skycell.1.21 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0450o.478527/o6037g0450o.478527.wrp.400756.skycell.1.21 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose --> hm, didnt fault on czarpage for revert? fault=4 select warpRun.warp_id,state,data_state,fault from warpRun join warpSkyfile using (warp_id) where label like "STS.mopsdiff.mehtest" and data_state="update"; warptool -dbname gpc1 -revertwarped -label STS.mopsdiff.mehtest -fault 4 --> ok all finished fine now.. -- setup diff run label STS.mopsdiff.mehtest data_group STS.mopsdiff.mehtest workdir meh/STS.mopsdiff.mehtest/ distribution ? need to know about publishing - does it need normal dist? should run on own using diffim info --> difftool command for normal WS? --> need to add inverse since WS has as false, might as well set diff_mode as well -- not part of definewarpstack since implied --> what about -exposure? would also seem want -- not part of definewarpstack, only definerun since implied --> also -input_label becomes -warp_label, -template_label becomes stack_label --> need -pretend to see output -- cant test until updates are done -- but can intermediate test what to expect, see diffs with other fields so need -skycell_id %.1.% *** wildcards not work so need to do mysql cmdline batch since didnt restrict batch updates.. or who cares, just run them.. this is a problem with the large overlap skycells, and more restrictions will be needed in future runs difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -bothways -set_reduction WARPSTACK -simple -pretend --> 31 exposures, 81 skycells so ~2500 jobs --> could try one skycell first then full, will see if will remake it -- use pretend again for full and see -- difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -bothways -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple *** oops, by running 1 skycell, now thinks is down.. forgot.. now need to use -rerun and guess cleanup the other one - difftool -updaterun -set_state goto_cleaned -set_label goto_cleaned -label STS.mopsdiff.mehtest -dbname gpc1 --> huh.. didnt set data_group right, used the base one STS.date.. wonder why.. difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -set_reduction WARPSTACK -bothways -simple -rerun --> odd fault 4, won't fix -- something in psphot 242183 skycell.1.29 STS.mopsdiff.mehtest STS.20110918 -- distribution to MOPS publishing - --> setup ? dont know how to associate label to particular datastore location for this.. -defineclient -stage define stage (required) -product define product (required) -workdir define workdir (required) -output_format define output format -comment define comment -name define name -unmagicked allow unmagicked data? -updateclient: -client_id search by client_id -stage search by stage -product search by product -comment search by comment (LIKE) -active set to active state -inactive set to inactive state | publishClient | | publishDone | | publishRun | mysql> select * from publishClient; +-----------+--------+-----------------+----------+----------+---------------------------------+---------------------------------------------------+-----------------+---------------+ | client_id | active | product | stage | magicked | workdir | comment | name | output_format | +-----------+--------+-----------------+----------+----------+---------------------------------+---------------------------------------------------+-----------------+---------------+ | 1 | 1 | IPP-MOPS | diff | 1 | neb://any/gpc1/IPP-MOPS/ | IPP detections for MOPS | NULL | 2 | | 2 | 0 | MONET | camera | 0 | neb://any/gpc1/MONET/ | Translated camera-level detections for Dave Monet | NULL | 1 | | 3 | 0 | Transients | diff | 1 | neb://any/gpc1/IPP-Transients/ | IPP detections of possible transients | NULL | 1 | | 4 | 0 | IPP-MOPS-UPDATE | diffphot | 1 | neb://any/gpc1/IPP-MOPS-UPDATE/ | Re-photometered IPP detections for MOPS | NULL | 1 | | 5 | 1 | IPP-MOPS-TEST | diff | 0 | neb://any/gpc1/IPP-MOPS-TEST | NULL | NULL | 2 | | 10 | 0 | IPP-MOPS-TEST | camera | 0 | neb://any/gpc1/IPP-MOPS-TEST | NULL | NULL | 2 | | 11 | 0 | IPP-MOPS-2 | diff | 1 | neb://any/gpc1/IPP-MOPS-2/ | IPP detections for MOPS / Output format version 2 | IPP-MOPS-2 | 2 | | 12 | 1 | IPP-MOPS-TEST-2 | diff | 0 | neb://any/gpc1/IPP-MOPS-TEST-2/ | IPP detections for MOPS / Output format version 2 | IPP-MOPS-TEST-2 | 2 | | 13 | 1 | IPP-MOPS-LAP | diff | 1 | neb://any/gpc1/IPP-MOPS-LAP | IPP detections from Large Area Processing | NULL | 2 | -definerun: -set_label set label -client_id search by client_id -- should this be set client_id?? -label set and search by label -data_group search by data_group -comment search by comment -dateobs_begin search for exposures by time (>=) -dateobs_end search for exposures by time (<=) -filter search for filter -obs_mode search by observation mode -rerun Re-run publish? -pretend Pretend to define? -simple use simple output format? -limit limit result set --> should simply be pubtool -dbname gpc1 -definerun -label STS.mopsdiff.mehtest -simple -pretend --> will need to add label to publishing + add info to stdscience to tell where to put out too -- this is what needed.. similar for distribution -- --> so what has used client_id 12? nothing --> should add comment line for each entry field.. -- see survey.pro --