= Summary STS.refstack.20120510 = [[PageOutline]] MOPS would like a refstack for the STS fields to do warp-stack diffims. The previous STS.refstack.20110613,20110615 wasn't usable as it was destroyed by magic. STS.20120510 was made using a similar number of inputs from the best exposures from 2011. Not pretty but hopefully sufficient. Status: * 5/8: past refstack from 6/2011 are cleaned, magic destroyed them. no magic problem anymore.. * 5/10: test samples and selected exposures processed for chip->warp * 5/13: test samples for refstack finished, full refstack started * 5/15: refstack finished and checked. looking at diffim samples and how to publish detections to MOPS * 5/18: test diffim sample 1 using updated warps from past nights in 2011, early 2012 * 5/20: test diffims finished but fault in publishing, tracing out. * 5/27: test sample 1 a mess, restart with test sample 2 * 5/29: test sample 2 diffim detections published to IPP-MOPS-TEST-2 Details to add: * FWHM of test stacks (STS.reftest.20120510.c#f#) like done with MD refstacks * samole unconv vs convolved stacks and issues found * test diffims faults, details and issues with recent STS data = STS observation and tessellation summary = Sample rawExp.comment information for the type of observation {{{ STS.180.ND PP1 center STS.000.GIQ PP1 dither 1 STS.180.NDGIQ PP5 center STS.180.NOM PP3 dither 3 }}} * 000,090,180,270 -- rotator to keep rough PA on field over season * GIQ -- shorter exp_time (20s versus 30s nominal) to not saturate more sources in better seeing conditions * ND -- no dither so always with center and typically with 30s, occasional NDGIQ * NOM -- nominal IQ? * PP# -- 1-7, 7 pointings * dither 1-4 -- dither shifts (by ?) and typically with GIQ Overlay of camera SMFs for PP1-PP7 (colored) on top of the projection skycells defined at http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_Tessellations. || [[Image(STS_i_smf_fieldcolor.png,300px)]] || 7 projections and their skycell overlap (STS.V4, 0.25"/pixel platescale) || Approximate pattern, E left, N up: {{{ 1 (orange) 2 (red) 3 (blue) 4 (green) 5 (violet) 6 (indigo) 7 (yellow) }}} To generate SMF sample for position and look at dither pattern: {{{ mysql -hippdb03 -uippuser -pippuser gpc1 -B -e 'select chipRun.workdir,rawExp.exp_tag,rawExp.exp_name from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%center%" group by rawExp.exp_id;' | grep -v "workdir" | uniq -f2 | awk '{ print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -p " $1"/"$2"/o%smf" }' | tcsh | awk '{print "imbox "$1}' >& STSsub.1_i_imboxsmf_c.lst mysql -hippdb03 -uippuser -pippuser gpc1 -B -e 'select chipRun.workdir,rawExp.exp_tag,rawExp.exp_name from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%dither 1%" group by rawExp.exp_id;' | grep -v "workdir" | uniq -f2 | awk '{ print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -p " $1"/"$2"/o%smf" }' | tcsh | awk '{print "imbox "$1}' >& STSsub.1_i_imboxsmf_d1.lst ... $KAPA = kapa -noX region 297.5 17.1 4.2 style -c black image style -c orange input STSsub.1_i_imboxsmf.lst style -c red input STSsub.2_i_imboxsmf.lst ... }}} There is significant overlap of projections and skycells, may cause issue with general input into a refstack skycell with wildly different IQ. '''add a sample cross-match table for example tests''' || skycell A.xx || skycell B.xx || || || || || 1.03 || 4. || || || || ---- = Refstack = Reference stack using data from 2011 only, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, large elongation. == Chip->Warp processing == Only reprocessing a selected sample to use in the refstack only -- * label STS.refstack.20120510 * data_group STS.refstack.20120510 * dist_group NULL * workdir: neb://@HOST@.0/gpc1/STS.refstack.20120510 Test sample: {{{ chiptool -definebyquery -set_label STS.mehtest -set_workdir neb://@HOST@.0/meh/STS.mehtest -set_data_group STS.mehtest -set_dist_group NULL -set_tess_id STS.V4 -set_end_stage warp -dbname gpc1 -dateobs_begin 2011-05-24 -dateobs_end 2011-05-25 -filter i% -comment STS% -obs_mode STS -simple -pretend >& teststschip.log --> 5/24 has 84 exposures, yep so need more restrictive for test - do o5705g0373o chiptool -definebyquery -set_label STS.mehtest -set_workdir neb://@HOST@.0/meh/STS.mehtest -set_data_group STS.mehtest -set_dist_group NULL -set_tess_id STS.V4 -set_end_stage warp -set_reduction STS_DATASET -dbname gpc1 -exp_name o5705g0373o -simple -pretend >& teststschip_o5705g0373o.log -- test run - STS.mehtest - dont see corner unglow in camera image, dynamic range such that not see strongly? neb://@HOST@.0/meh/STS.mehtest/o5705g0373o.340508 --> see some patterns in warp background for sample -- significant? see meh/STS.mehtest/o5705g0373o.340508/o5705g0373o.340508.wrp.408756.skycell.1.04.fits }}} Determine cuts for exposure sample to reprocess, pick of order 50 exposure sample: * time-start {{{ | o5666g0560o | 324290 | STS | STS | 30 | 2011-04-15 14:01:18 | i.00000 | STS PP6 test visit 2 | | o5666g0561o | 324291 | STS | STS | 30 | 2011-04-15 14:02:10 | i.00000 | STS PP7 test visit 2 | | o5667g0517o | 324909 | RINGS.V3 | STS | 30 | 2011-04-16 14:39:15 | i.00000 | STS PP8 test visit 1 | | o5667g0518o | 324910 | RINGS.V3 | STS | 30 | 2011-04-16 14:39:58 | i.00000 | STS PP9 test visit 1 | .. | o5681g0558o | 332242 | RINGS.V3 | STS | 30 | 2011-04-30 15:02:10 | i.00000 | STS PP6 test visit 2 | | o5681g0559o | 332243 | RINGS.V3 | STS | 30 | 2011-04-30 15:03:32 | i.00000 | STS PP3 test visit 2 | | c5681g0007d | 332244 | NULL | STS | 30 | 2011-04-30 15:05:39 | i.00000 | STS PP3 test visit 2 | | o5695g0143o | 335717 | NULL | STS | 30 | 2011-05-14 11:06:19 | i.00000 | STS.180.NOM PP4 center | .. | o5695g0144o | 335718 | NULL | STS | 30 | 2011-05-14 11:07:04 | i.00000 | STS.180.NOM PP1 center | | o5695g0225o | 335799 | NULL | STS | 30 | 2011-05-14 12:06:25 | i.00000 | STS.180.NOM PP6 dither 5 | | o5695g0226o | 335800 | NULL | STS | 30 | 2011-05-14 12:07:09 | i.00000 | STS.180.NOM PP3 dither 5 | | o5705g0373o | 340508 | STS.V4 | STS | 20 | 2011-05-24 12:05:27 | i.00000 | STS.270.GIQ PP4 dither 1 | -- talking with Bill, looks like when to start -- | o5705g0374o | 340510 | STS.V4 | STS | 20 | 2011-05-24 12:06:00 | i.00000 | STS.270.GIQ PP1 dither 1 | }}} * camera FWHM, ZPT, background cut * dither content check {{{ SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) WHERE obs_mode="STS" and dateobs>"2011-05-23" and dateobs<"2012-01" and fwhm_major<3.8 and zpt_obs>24.6 and camProcessedExp.bg<100.0 and comment like "%PP1%" group by exp_id; ... }}} * summary {{{ PP1 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 42, ~half center half dither (16,7,5,6,8) PP2 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 47, ~half center half dither PP3 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 53, ~half center half dither PP4 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 50, ~half center half dither PP5 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 52, ~half center half dither PP6 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 49, ~half center half dither (14,8,8,10,9) PP7 fwhm_major<3.9 and zpt_obs>24.5 and camProcessedExp.bg<120.0 and camProcessedExp.bg_stdev<20.0 --> 49, ~half center half dither (18,9,5,10,7) }}} Run PP1 set first with selected exposures and perl script to make reftest stacks: {{{ mysql -hippdb03 -uippuser -pippuser gpc1 -B -e 'select rawExp.exp_id from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment like "%PP1%" group by rawExp.exp_id;' | grep -v "exp_id" perl sts.v4.refstackexp.20120510_pp1.chip > run_pp1.chip.bat source run_pp1.chip.bat >& run_pp1.chip.log }}} Once reftest check out, run remaining PP2-7 exposures: {{{ mysql -hippdb03 -uippuser -pippuser gpc1 -B -e 'select rawExp.exp_id from chipRun join rawExp using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) where rawExp.obs_mode="STS" and rawExp.dateobs>"2011-05-23" and rawExp.dateobs<"2012-01" and camProcessedExp.fwhm_major<3.9 and camProcessedExp.zpt_obs>24.5 and camProcessedExp.bg<120.0 and rawExp.comment not like "%PP1%" group by rawExp.exp_id;' | grep -v "exp_id" perl sts.v4.refstackexp.20120510_pp2-7.chip > run_pp2-7.chip.bat source run_pp2-7.chip.bat >& run_pp2-7.chip.log --> set priority with lap @200 labeltool -definelabel -set_label STS.refstack.20120510 -set_priority 200 -dbname gpc1 labeltool -updatelabel -label STS.refstack.20120510 -set_priority 190 -dbname gpc1 }}} == Stack processing == * label: STS.refstack.20120510 * data_group: STS.refstack.20120510 * workdir: * reduction: PS1_REFERENCE {{{ OUTPUT.NOCOMP BOOL TRUE OUTPUT.LOGFLUX BOOL FALSE OUTPUT.REPLICATE BOOL TRUE }}} Test stack (reftest) sample also made again for skycell.055 with different number of input warps. * label: STS.reftest.201204xx (xx=17 for g, 18 for r, 19 for i, 20 for z, 21 for y) * data_group: * STS.reftest.201204xx.c#f where #=comparison run, =cut where all means 2009+2010+2011 data possible, =cut where 2011 means 2011 data only, =cut & stacktool.sts_reftest.20120510.fwhm39_elong12_skycell103.log }}} gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899917.unconv.num.fits -- c1f39 (max 36) gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.unconv.num.fits -- c1f38 (max 23) gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899919.unconv.num.fits -- c1f37 (max 12) gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899920.unconv.num.fits -- c0f45 (max 44) SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.reftest.20120510.c1f38" group by exp_id; -- psphot on samples - run_psphot_single.csh on unconv+conv -- *** faults because of modified psphot config/code to limit things -- too many peaks and faults.. --> config setting? tag must be different than trunk at 40000 rather than 50000 -- -Di PEAKS_NMAX_TOTAL 60000 --> unconv images still faulting.. so do pass1 only.. conv ones are poor as well, FWHM_MAJ~9-10.. {{{ gethead -auhc *fpa.cmf FWHM_MAJ IQ_FW1 STS.V4.skycell.1.03.stk.899917.fpa.cmf 4.405078 3.746344 STS.V4.skycell.1.03.stk.899917.unconv.fpa.cmf 3.921824 3.04026 STS.V4.skycell.1.03.stk.899918.fpa.cmf 4.359843 3.701267 STS.V4.skycell.1.03.stk.899918.unconv.fpa.cmf 3.788124 2.973968 STS.V4.skycell.1.03.stk.899919.fpa.cmf 4.184109 3.576873 STS.V4.skycell.1.03.stk.899919.unconv.fpa.cmf 3.835106 2.939088 STS.V4.skycell.1.03.stk.899920.fpa.cmf 4.54595 3.812682 STS.V4.skycell.1.03.stk.899920.unconv.fpa.cmf 3.862908 3.048509 }}} {{{ -- look at stacks - bottom line, more inputs would help, but maybe this is sufficient --> 4.17 odd bright cell print through --> 2.02 odd faint chunk print through --> 2.39 gap faint print through --> 5.06 missed burntool? --> 1.09,1.17 large unfilled gap --> 1.24 long unfilled gap --> 1.25,1.35 long and large unfilled gap --> 1.12 large unfilled cell --> 1.40 unfilled masking corner multi-cell --> 1.55 rounded edge + corner adds other exp --> 1.44,5.14 odd bright orphan diffspike-like unmasked --> 1.11 nice cluster --> 1.32 faint flux print through from brighter cell }}} || || || || || || '''version data_group''' (stack_id) || '''filter''' || '''center skycell input warps (rej)''' || '''center skycell FWHM (major)''' || || || || || || || new ''STS.refstack.20120510'' V4 || i || 96 (2) || 1.27" || || STS.reftest.201205.c1f49 V4 () || i || 34 (1) || 1.20" || || old STS.refstack.2011 V3 || i || 78 (4) || 1.45" || -- look at Target seeing to see if any odd inputs driving mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select path_base from stackRun join stackSumSkyfile using (stack_id) where stackRun.label like "STS.refstack.20120510" and stackRun.skycell_id like "%.1.%";' | grep -v path_base | awk '{print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -l -p "$1"%.log"}' | tcsh | awk '{print " cat "$9}' | tcsh | grep "Target seeing" --> couple with 8-9 -- #3,7 = gpc1/STS.refstack.20120510/STS.V4/skycell.1.02/STS.V4.skycell.1.02.stk.902904.log -- 83 Nin, largest ~5.7 so wacky shapes then, even 2.89 gpc1/STS.refstack.20120510/STS.V4/skycell.1.06/STS.V4.skycell.1.06.stk.902908.log -- 78 Nun, largest ~4.5 so wacky shapes.. mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select path_base from stackRun join stackSumSkyfile using (stack_id) where stackRun.label like "STS.refstack.20120510" and stackRun.skycell_id like "%";' | grep -v path_base | awk '{print "/home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/neb-ls -l -p "$1"%.log"}' | tcsh | awk '{print "echo "$9"; grep -e \" Input \" -e \"Target seeing\" -e \"fwhm (psf)\" "$9" ;echo \" \""}' | tcsh > STS.refstack.20120510_input_target_output_fwhm_inputlong.log ---- = Test Diffim samples = Want to create a sample of diffims with detections for MOPS to look at. Will need to also keep in "full" state so they can get postage stamps quickly. Focus on projection 1. {{{ survey.add.publish STS.mopsdiff STS.mopsdiffv2.mehtest 12 NULL survey.show.publish survey.del.publish STS.mopsdiff add.label STS.mopsdiffv2.mehtest }}} == Diffim sample 1 == Sample diffims for projection 1 faulted in publishing == Diffim sample 2 == Rerun of sample diffims for projection 1 and published under IPP-MOPS-TEST-2 {{{ SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" group by exp_id; }}} {{{ mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-30%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiffv2.mehtest -chip_id "$4"; warptool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiffv2.mehtest -warp_id "$5}' > ~/updatechipwarp_sts.mopsdiffv2.mehtest_110930.bat ... difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiffv2.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffv2.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffv2.mehtest -set_data_group STS.mopsdiffv2.mehtest -set_reduction WARPSTACK -simple -pretend }}} {{{ select * from diffRun where data_group like "STS.20110930"; difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiffv2.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffv2.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffv2.mehtest -set_data_group STS.mopsdiffv2.mehtest -set_reduction WARPSTACK -simple -rerun -pretend pubtool -dbname gpc1 -definerun -label STS.mopsdiffv2.mehtest -client_id 12 -simple -pretend }}} || || || skycell || FWHM input/ref || Ndet || comment || || || || 1.03 ||