IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 54 and Version 55 of STS.refstack.20120510


Ignore:
Timestamp:
Jun 15, 2012, 8:11:01 PM (14 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • STS.refstack.20120510

    v54 v55  
    517517----
    518518
    519 = Test Diffim samples =
     519= Test Diffim Samples =
    520520Want to create a sample of diffims with detections for MOPS to look at. Will need to also keep in "full" state so they can get postage stamps quickly. Focus on projection 1.
    521521
    522522Basic setup: (warp - convolved stack) -- need examples and compared to unconv stack (why not using this?)
    523 
    524 
    525 == manual test set ==
     523 * what are diffims for others -- WS, target1, conv vs unconv stack -- part of what needed to do for ppsub testing page still
     524 * MOPS has no specific requirements for test samples, so try to combine multiple aspects using skycell.1.xx
     525  * W-S w/ conv refstack since unconv refstack not an option due to code issues.
     526  * 2011 -- 1 or 2 nights -- ones not used in refstack (may be difficult), will need to be updated so no chance of cleaning
     527  * 2012 -- 1 or 2 nights -- will need to be updated so no chance of cleaning
     528 * not sure how to queue/run these in any case since driven by a survey task. could do last nights for 2012 right off but cleanup window so do older sample
     529  * need to setup/change publishing location
     530  * turn off cleanup for the STS for now? no, uncertain when will turn on so continue cleanup as normal.
     531
     532== Manual Test Set ==
    526533Want to run quick set locally. Pick center, edge, edge+lots overlap for various filling types - 1.44, 1.55, 1.03?
    527534{{{
    … …  
    788795== Diffim sample 1 (faulted) ==
    789796
    790 Sample diffims for projection 1 faulted in publishing. So this is more of an example of what not to do..
     797Sample diffims for projection 1 faulted in publishing. So this is more of an example of some things of what not to do..
     798
     799--> dont know how to tell fwhm other than from db - do 1.03 since edge field and wacky overlap, but need to know what chips go into that? no, camProcessedExp
     800 * need to know rough FWHM of refstack and nightly exposures to pick sample
     801 * need to pick one from 2011 not used in refstack
     802
     803SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE  comment like "STS%PP1%" group by exp_id;
     804
     805 * refstack ~3.94
     806 * 120420 - look ok
     807 * 120507 ~7.52 (7.9 in cam, wrong way even for cam/warp scaling) -- looked poor, don't use as test
     808
     809
     810select exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.refstack.20120510" and stackRun.skycell_id like "%1.03" and comment like "%PP1%" group by exp_id;
     811
     812SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" group by exp_id;
     813
     814
     815//SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) WHERE warpRun.data_group like "STS.refstack.20120510" and comment like "%PP1%" and not in ( select * FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join stackInputSkyfile using (warp_id) join stackRun using (stack_id) WHERE stackRun.data_group like "STS.refstack.20120510" and stackRun.skycell_id like "%1.03" and comment like "%PP1%" ) group by exp_id;
     816
     817
     818 
     819
     820| 393855 | o5822g0346o |       20 | 2011-09-18 09:25:14 | 24.6354 |    3.87493 | 1.166 | 1.08        | 105.39   | 13.63    | 4.11  | 3.61  | 8444848 | STS.000.GIQ PP1 dither 4 |
     821
     822
     823--> 2012 sample to test before updating
     824SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE  comment like "STS%PP1%" group by exp_id;
     825
     826| 475121 | o6029g0441o |       30 | 2012-04-12 13:40:09 | 24.5321 |    4.06258 | 1.119 | 1.05        | 315.49   | 21.55    | 4.21  | 3.82  | 8078200 | STS.180.ND PP1 center    |
     827
     828| 476338 | o6031g0408o |       20 | 2012-04-14 13:34:26 | 24.5537 |    4.27532 | 1.180 | 0.98        | 117.51   | 14.15    | 4.77  | 4.05  | 7485604 | STS.180.GIQ PP1 dither 1 |
     829
     830| 477549 | o6033g0430o |       30 | 2012-04-16 13:31:05 | 24.5586 |    7.96196 | 1.194 | 1.83        | 154.22   | 18.30    | 8.08  | 7.71  | 4586012 | STS.180.ND PP1 center    |
     831
     832++ | 478479 | o6037g0401o |       20 | 2012-04-20 13:46:26 | 24.5799 |     4.1334 | 1.174 | 1.06        | 76.31    | 12.57    | 4.42  | 3.87  | 8291412 | STS.180.GIQ PP1 dither 1 |
     833
     834| 479810 | o6040g0441o |       30 | 2012-04-23 13:31:03 |  24.608 |    6.12941 | 1.131 | 1.20        | 147.86   | 17.88    | 6.39  | 5.88  | 5637788 | STS.180.ND PP1 center    |
     835
     836| 482593 | o6045g0420o |       30 | 2012-04-28 13:37:39 | 24.6132 |    6.26513 | 1.051 | 1.07        | 109.03   | 16.37    | 6.59  | 6.09  | 5474888 | STS.180.ND PP1 center    |
     837
     838| 483576 | o6049g0286o |       30 | 2012-05-02 13:09:26 | 24.5922 |    4.77424 | 1.088 | 0.94        | 166.69   | 17.89    | 5.10  | 4.62  | 7102152 | STS.180.ND PP1 center    |
     839
     840/// | 484578 | o6054g0305o |       30 | 2012-05-07 12:48:56 | 24.5422 |    7.94199 | 1.042 | 0.76        | 297.28   | 23.85    | 8.11  | 7.69  | 3526740 | STS.180.ND PP1 center    |
     841
     842| 487890 | o6062g0361o |       30 | 2012-05-15 12:00:59 | 24.5415 |    4.14248 | 1.177 | 1.04        | 146.86   | 16.61    | 4.58  | 3.86  | 8390052 | STS.180.ND PP1 center    |
     843--> on disk still so try also
     844--> need warp_ids? try exposure name - in any case need to see state in - yes exp_name works -
     845select * from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_name="o6062g0361o";
     846
     847--> try to at least include 4/14 sing a GIQ? 4/20 cleaned but know work so 4/14 not necessary --
     848--> have to use the conv stack..
     849
     850
     851select uri from warpSkyfile where warp_id=409660 and skycell_id like "%1.03";
     852./run_ppsub_night_ws.csh gpc1/STS.refstack.20120510/o5822g0346o.393855/o5822g0346o.393855.wrp.409660.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20110918c > runppsub_sts103_110918_1c.bat
     853source runppsub_sts103_110918_1c.bat >& runppsub_sts103_110918_1c.log &
     854--> good -
     855
     856select warpSkyfile.uri from warpSkyfile join warpRun using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_name="o6029g0441o" and warpSkyfile.skycell_id like "%1.03";
     857./run_ppsub_night_ws.csh  gpc1/STS.nt/2012/04/12//o6029g0441o.475121/o6029g0441o.475121.wrp.396474.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120412c > runppsub_sts103_120412_1c.bat
     858source runppsub_sts103_120412_1c.bat >& runppsub_sts103_120412_1c.log &
     859--> bilster cores (undersubs)
     860
     861select warpSkyfile.uri from warpSkyfile join warpRun using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where warpSkyfile.skycell_id like "%1.03" and exp_name="o6031g0408o";
     862./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/14//o6031g0408o.476338/o6031g0408o.476338.wrp.397423.skycell.1.03.fits   gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120414c > runppsub_sts103_120414_1c.bat
     863source runppsub_sts103_120414_1c.bat >& runppsub_sts103_120414_1c.log &
     864--> blister cores (undersub)
     865
     866./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/16//o6033g0430o.477549/o6033g0430o.477549.wrp.398532.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120416c > runppsub_sts103_120416_1c.bat
     867source runppsub_sts103_120416_1c.bat >& runppsub_sts103_120416_1c.log &
     868--> large conv, oversub
     869
     870./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/23//o6040g0441o.479810/o6040g0441o.479810.wrp.401931.skycell.1.03.fits   gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120423c > runppsub_sts103_120423_1c.bat
     871source runppsub_sts103_120423_1c.bat >& runppsub_sts103_120423_1c.log &
     872--> larger conv, maybe slight oversub
     873
     874./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/28//o6045g0420o.482593/o6045g0420o.482593.wrp.405569.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120428c > runppsub_sts103_120428_1c.bat
     875source runppsub_sts103_120428_1c.bat >& runppsub_sts103_120428_1c.log &
     876--> larger conv, oversub
     877
     878./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/02//o6049g0286o.483576/o6049g0286o.483576.wrp.406354.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120502c > runppsub_sts103_120502_1c.bat
     879source runppsub_sts103_120502_1c.bat >& runppsub_sts103_120502_1c.log &
     880--> slightly larger conv, maybe slight oversub
     881
     882./run_ppsub_night_ws.csh gpc1/STS.nt/2012/05/15//o6062g0361o.487890/o6062g0361o.487890.wrp.415613.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120515c > runppsub_sts103_120515_1c.bat
     883source runppsub_sts103_120515_1c.bat >& runppsub_sts103_120515_1c.log &
     884--> low conv, blister undersubs
     885
     886--> looks like 110918, 120420, 120423/120502 -- use 4/23, seeing some odd conv/pixrej artifacts coming out in diffim but not sure if detected and 30s version
     887
     888
     889
     890--> may also want to try unconv since STS is supposed to be done w/in certain IQ?
     891
     892
     893-- set select nights up for updates - number exposures ~ 16(20s)/16(20s)/12(30s) for 110918, 120420, 120423
     894--> label STS.mehdifftest
     895--> will need to script mysql call based on what to get the chip/warp_id? detrend different since? may not be unique run given reprocessing for refstack oh well.. need PP1 as well
     896--> manually checked and all unique..
     897
     898select exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" group by exp_name;
     899
     900chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id     
     901warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id 
     902
     903--> will have some clean state issues possibly..
     904--> note dateobs is 2 groups..
     905*** 110918 may have magic issues.. may have to just reprocess it - try update since just a test anyways and see -- did update for test case anyways so should work
     906
     907mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_110918.bat
     908
     909mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-20%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120420.bat
     910
     911mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-23%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120423.bat
     912
     913--> warp updates driving ipp017 nuts.. ~10-20% ram for 5 runs for total ~75%, cpu_wait and swap overload. killing 1 at time
     914
     915  1    ipp017    BUSY   2993.25 0.0.5.fe34  0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400730 --warp_skyfile_id 34217254 --skycell_id skycell.1.22 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0429o.478506/o6037g0429o.478506.wrp.400730.skycell.1.22 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose
     916
     917  2    ipp017    RESP   3369.78 0.0.6.269  0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400756 --warp_skyfile_id 34219846 --skycell_id skycell.1.21 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0450o.478527/o6037g0450o.478527.wrp.400756.skycell.1.21 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose
     918
     919--> hm, didnt fault on czarpage for revert? fault=4
     920
     921select warpRun.warp_id,state,data_state,fault from warpRun join warpSkyfile using (warp_id) where label like "STS.mopsdiff.mehtest" and data_state="update";
     922
     923warptool -dbname gpc1 -revertwarped -label STS.mopsdiff.mehtest   -fault 4
     924--> ok all finished fine now..
     925
     926
     927
     928-- setup diff run
     929label STS.mopsdiff.mehtest
     930data_group STS.mopsdiff.mehtest
     931workdir meh/STS.mopsdiff.mehtest/
     932distribution ? need to know about publishing - does it need normal dist? should run on own using diffim info
     933--> difftool command for normal WS?
     934--> need to add inverse since WS has as false, might as well set diff_mode as well -- not part of definewarpstack since implied
     935--> what about -exposure? would also seem want -- not part of definewarpstack, only definerun since implied
     936--> also -input_label becomes -warp_label, -template_label becomes stack_label
     937--> need -pretend to see output -- cant test until updates are done -- but can intermediate test what to expect, see diffs with other fields so need -skycell_id %.1.%
     938*** wildcards not work so need to do mysql cmdline batch since didnt restrict batch updates.. or who cares, just run them.. this is a problem with the large overlap skycells, and more restrictions will be needed in future runs
     939
     940difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -bothways -set_reduction WARPSTACK -simple -pretend
     941
     942--> 31 exposures, 81 skycells so ~2500 jobs
     943--> could try one skycell first then full, will see if will remake it -- use pretend again for full and see --
     944
     945difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -bothways -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple
     946
     947*** oops, by running 1 skycell, now thinks is down.. forgot.. now need to use -rerun and guess cleanup the other one -
     948difftool -updaterun -set_state goto_cleaned -set_label goto_cleaned -label STS.mopsdiff.mehtest -dbname gpc1
     949
     950--> huh.. didnt set data_group right, used the base one STS.date.. wonder why..
     951
     952difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -set_reduction WARPSTACK -bothways -simple -rerun
     953
     954--> odd fault 4, won't fix -- something in psphot
     955242183  skycell.1.29    STS.mopsdiff.mehtest    STS.20110918
     956
     957
     958
     959-- distribution to MOPS publishing -
     960--> setup ? dont know how to associate label to particular datastore location for this..
     961-defineclient
     962  -stage          define stage (required)
     963  -product        define product (required)
     964  -workdir        define workdir (required)
     965  -output_format  define output format
     966  -comment        define comment
     967  -name           define name
     968  -unmagicked     allow unmagicked data?
     969-updateclient:
     970  -client_id  search by client_id
     971  -stage      search by stage
     972  -product    search by product
     973  -comment    search by comment (LIKE)
     974  -active     set to active state
     975  -inactive   set to inactive state
     976
     977| publishClient           |
     978| publishDone             |
     979| publishRun              |
     980
     981mysql> select * from publishClient;
     982+-----------+--------+-----------------+----------+----------+---------------------------------+---------------------------------------------------+-----------------+---------------+
     983| client_id | active | product         | stage    | magicked | workdir                         | comment                                           | name            | output_format |
     984+-----------+--------+-----------------+----------+----------+---------------------------------+---------------------------------------------------+-----------------+---------------+
     985|         1 |      1 | IPP-MOPS        | diff     |        1 | neb://any/gpc1/IPP-MOPS/        | IPP detections for MOPS                           | NULL            |             2 |
     986|         2 |      0 | MONET           | camera   |        0 | neb://any/gpc1/MONET/           | Translated camera-level detections for Dave Monet | NULL            |             1 |
     987|         3 |      0 | Transients      | diff     |        1 | neb://any/gpc1/IPP-Transients/  | IPP detections of possible transients             | NULL            |             1 |
     988|         4 |      0 | IPP-MOPS-UPDATE | diffphot |        1 | neb://any/gpc1/IPP-MOPS-UPDATE/ | Re-photometered IPP detections for MOPS           | NULL            |             1 |
     989|         5 |      1 | IPP-MOPS-TEST   | diff     |        0 | neb://any/gpc1/IPP-MOPS-TEST    | NULL                                              | NULL            |             2 |
     990|        10 |      0 | IPP-MOPS-TEST   | camera   |        0 | neb://any/gpc1/IPP-MOPS-TEST    | NULL                                              | NULL            |             2 |
     991|        11 |      0 | IPP-MOPS-2      | diff     |        1 | neb://any/gpc1/IPP-MOPS-2/      | IPP detections for MOPS / Output format version 2 | IPP-MOPS-2      |             2 |
     992|        12 |      1 | IPP-MOPS-TEST-2 | diff     |        0 | neb://any/gpc1/IPP-MOPS-TEST-2/ | IPP detections for MOPS / Output format version 2 | IPP-MOPS-TEST-2 |             2 |
     993|        13 |      1 | IPP-MOPS-LAP    | diff     |        1 | neb://any/gpc1/IPP-MOPS-LAP     | IPP detections from Large Area Processing         | NULL            |             2 |
     994
     995-definerun:
     996  -set_label      set label
     997  -client_id      search by client_id     -- should this be set client_id??
     998  -label          set and search by label
     999  -data_group     search by data_group
     1000  -comment        search by comment
     1001  -dateobs_begin  search for exposures by time (>=)
     1002  -dateobs_end    search for exposures by time (<=)
     1003  -filter         search for filter
     1004  -obs_mode       search by observation mode
     1005  -rerun          Re-run publish?
     1006  -pretend        Pretend to define?
     1007  -simple         use simple output format?
     1008  -limit          limit result set
     1009
     1010--> should simply be
     1011
     1012pubtool -dbname gpc1 -definerun -label STS.mopsdiff.mehtest -simple -pretend
     1013
     1014
     1015--> will need to add label to publishing + add info to stdscience to tell where to put out too -- this is what needed.. similar for distribution --
     1016--> so what has used client_id 12? nothing
     1017--> should add comment line for each entry field.. -- see survey.pro --  <tag>  <label>  <client_id>  <comment>
     1018--> so what does <tag> do? allows multiple entries per label and is a unique key, ie for the .ds for example to be able to push out to other locations being magicked, so not important now? still needs to be different so call STS.test
     1019
     1020survey.add.publish STS.mopsdiff STS.mopsdiff.mehtest 12 NULL
     1021
     1022survey.del.publish STS.mopsdiff
     1023
     1024--> careful STS.nightlyscience already defn to go to 5 --  survey.add.publish STS  STS.nightlyscience 5 NULL
     1025--> faults! why is it trying to put to LAP?? and not just LAP, MOPS and MOPS-TEST too..
     1026system failure for: publish_file.pl --pub_id 422412 --camera GPC1 --workdir neb://any/gpc1/IPP-MOPS-LAP --product IPP-MOPS-LAP --stage diff --stage_id 242192 --output_forma
     1027t 2 --redirect-output --need-magic --dbname gpc1 --verbose
     1028
     1029--> run manually
     1030publish_file.pl --pub_id 422376 --camera GPC1 --workdir neb://any/gpc1/IPP-MOPS-TEST-2/ --product IPP-MOPS-TEST-2 --stage diff --stage_id 242198 --output_format 2 --redirect-output --dbname gpc1 --verbose
     1031
     1032neb://any/gpc1/IPP-MOPS-TEST-2/
     1033--> loads of things in this dir - and why reports many not found because cleaned?
     1034
     1035*** suspect shouldnt run pubtool and use survey task, pubtool generically loaded for all then..
     1036
     1037
     1038--> actual fault from log -- could this astrom issue be because of STS?
     1039neb://any/gpc1/IPP-MOPS-TEST-2//IPP-MOPS-TEST-2.422401.log
     1040
     1041astrom value for negative doesn't match at /home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/publish_file.pl line 374
     1042        main::my_die('astrom value for negative doesn\'t match', 422401, 2) called at /home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/publish_file.pl line 316
     1043        main::diff_check('HASH(0x1a4da90)', 'HASH(0x2a26290)', 'negative') called at /home/panstarrs/ipp/psconfig//ipp-20120404.lin64/bin/publish_file.pl line 243
     1044Running [/home/panstarrs/ipp/psconfig/ipp-20120404.lin64/bin/pubtool -add -pub_id 422401 -path_base neb://any/gpc1/IPP-MOPS-TEST-2//IPP-MOPS-TEST-2.422401 -dtime_script 2.99998372793198 -hostname ippc10 -fault 2 -dbname gpc1]...
     1045
     1046astrom => sqrt($comp->{sigma_ra_2}**2 + $comp->{sigma_dec_2}**2)
     1047
     1048
     1049neb://ipp007.0/meh/STS.mopsdiff.mehtest/STS.V4/skycell.1.02/STS.V4.skycell.1.02.WS.dif.242223
     1050meh/STS.mopsdiff.mehtest/STS.V4/skycell.1.02/STS.V4.skycell.1.02.WS.dif.242223.cmf
     1051meh/STS.mopsdiff.mehtest/STS.V4/skycell.1.02/STS.V4.skycell.1.02.WS.dif.242223.inv.cmf
     1052
     1053/data/ipp007.0/nebulous/92/71/2095812973.meh:STS.mopsdiff.mehtest:STS.V4:skycell.1.02:STS.V4.skycell.1.02.WS.dif.242223.cmf
     1054/data/ipp007.0/nebulous/14/91/2095815037.meh:STS.mopsdiff.mehtest:STS.V4:skycell.1.02:STS.V4.skycell.1.02.WS.dif.242223.inv.cmf
     1055
     1056
     1057neb://ipp008.0/meh/STS.mopsdiff.mehtest/STS.V4/skycell.1.03/STS.V4.skycell.1.03.WS.dif.242223
     1058
     1059
     1060--> publish_file.pl called from publish.pro -
     1061$run = publish_file.pl --pub_id $PUB_ID --camera $CAMERA --workdir $WORKDIR --product $PRODUCT --stage $STAGE --stage_id $STAGE_ID --output_format $OUTPUT_FORMAT --redirect-output       
     1062                                             
     1063system failure for: publish_file.pl --pub_id 422401 --camera GPC1 --workdir neb://any/gpc1/IPP-MOPS-TEST-2/ --product IPP-MOPS-TEST-2 --stage diff --stage_id 242223 --output_format 2 --redirect-output --dbname gpc1 --verbose
     1064
     1065
     1066--> attempt revert -
     1067pubtool -dbname gpc1 -revert -fault 2 -pub_id 422401
     1068
     1069
     1070--> does it still need to be added to distribution so that it ends up on the datastore?? no looks like in the log that pubtool is run
     1071
     1072--> publish cleanup/drop -
     1073pubtool -dbname gpc1 -updaterun -set_state drop -label STS.mopsdiff.mehtest -client_id 1
     1074pubtool -dbname gpc1 -updaterun -set_state drop -label STS.mopsdiff.mehtest -client_id 5
     1075pubtool -dbname gpc1 -updaterun -set_state drop -label STS.mopsdiff.mehtest -client_id 13
     1076--> leaves 27 with client_id=12 are all valid? look at stage_id
     1077select pub_id,diff_id,diffRun.data_group from publishRun join publishDone using (pub_id) join diffRun  where publishRun.label like "STS.mopsdiff.mehtest" and publishRun.state="new" and diff_id=stage_id;
     1078--> missing 110918 oddly, they've ended up in another client_id.. ugh..
     1079
     1080
     1081*** dont want bothways since it is a WS? is there a way to skip those? no, publish runs directly off diffRun settings, need to clean and rerun..
     1082--> this true of other WS runs? OSS is also not bothways, MD same.. so need to rerun and set previous pub to drop and clean diffRun
     1083
     1084difftool -updaterun -set_state goto_cleaned -set_label goto_cleaned -label STS.mopsdiff.mehtest -dbname gpc1
     1085pubtool -dbname gpc1 -updaterun -set_state drop -label STS.mopsdiff.mehtest -client_id 12
     1086
     1087difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK -simple -rerun -pretend
     1088
     1089--> data_group not set right.. set to date
     1090--> -rerun needed? use -pretend to see -- yes need -rerun - and says 164 now, why double?? since only 1 way should be the 43 as the number of warps?
     1091--> 16+16+11 exposures
     1092--> warp_id 258031 has 94 skycells
     1093--> but reports skycell_count 162
     1094--> using -skycell_id to limit it is 2 per exposure, how.. what are they?
     1095SELECT
     1096    exp_id,
     1097    warp_id,
     1098    rawExp.filter,
     1099    warpRun.label,
     1100    warpRun.tess_id,
     1101    warpRun.data_group,
     1102    COUNT(skycell_id) as skycell_count
     1103FROM warpRun
     1104JOIN warpSkyfile USING(warp_id)
     1105JOIN fakeRun USING(fake_id)
     1106JOIN camRun USING(cam_id)
     1107JOIN chipRun USING(chip_id)
     1108JOIN rawExp USING(exp_id)
     1109LEFT JOIN (
     1110    SELECT DISTINCT
     1111        diffRun.*,
     1112        warp_id,
     1113        exp_id
     1114    FROM diffRun
     1115    JOIN diffInputSkyfile USING(diff_id)
     1116    JOIN warpRun
     1117        ON warpRun.warp_id = diffInputSkyfile.warp1
     1118    JOIN fakeRun USING(fake_id)
     1119    JOIN camRun USING(cam_id)
     1120    JOIN chipRun USING(chip_id)
     1121    WHERE warp1 IS NOT NULL
     1122        AND warpRun.state = 'full'
     1123    -- warp where hook
     1124 AND (warpRun.label = 'STS.mopsdiff.mehtest')
     1125) AS diffExp USING(exp_id, warp_id)
     1126WHERE
     1127    warpRun.state = 'full'
     1128    AND warpSkyfile.fault = 0
     1129    AND warpSkyfile.quality = 0
     1130    -- warp where hook
     1131 AND (warpSkyfile.skycell_id = 'skycell.1.03') AND (warpRun.label = 'STS.mopsdiff.mehtest') AND ((warpSkyfile.good_frac > 0.20000000 - 0.00000119))
     1132    -- exp where hook
     1133    -- diff where hook
     1134GROUP BY exp_id, warp_id DESC;
     1135
     1136--> try single skycell again, -pretend outputs lines but nothing actually queued..
     1137difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple -rerun -pretend
     1138--> retry again using -bothways
     1139difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple -rerun -bothways -pretend
     1140--> nope.. can redo only be done on full runs and not cleaned ones then?
     1141
     1142difftool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest  -diff_id 242223
     1143
     1144difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple -rerun -pretend
     1145--> nope..
     1146difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple -rerun
     1147--> need to go through this explicitly..
     1148
     1149*** suspect that wont redo for 1 skycell because diff_id same for all and need to do all and should be able to do just for diff_id to rerun? would need to update all of them then
     1150
     1151difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK  -simple -rerun -pretend
     1152
     1153difftool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest  -diff_id 242223
     1154select * from diffRun where data_group like "STS.2012%";
     1155
     1156mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select diff_id,state,data_group from diffRun where data_group like "STS.2012%" and state="cleaned";' | grep -v diff_id | awk '{print "difftool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest  -diff_id  "$1}' > ~/updatediff_sts.mopsdiff.mehtest.bat
     1157
     1158--> ipp017 ram overuse
     1159difftool -revertdiffskyfile -fault 4 -label STS.mopsdiff.mehtest -dbname gpc1
     1160
     1161
     1162difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiffredo.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_data_group STS.mopsdiffredo.mehtest -set_reduction WARPSTACK  -simple -rerun -pretend
     1163
     1164--> needs workdir, label and data_group optional
     1165difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_reduction WARPSTACK  -simple -rerun -pretend
    7911166
    7921167