IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 63 and Version 64 of STS.refstack.20120510


Ignore:
Timestamp:
Jun 21, 2012, 11:25:37 AM (14 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • STS.refstack.20120510

    v63 v64  
    783783Sample diffims for projection 1 faulted in publishing. So this is more of an example of some things of what not to do..
    784784
    785 -->   --
    786785Nights to pick for test -- do test run skycell.1.03 again since edge field and can have overlap.  Not clear way to get FWHM other than from gpc1 DB, need to know what chips go into that? No, camProcessedExp sufficient.
    787786 * need to know rough FWHM of refstack and nightly exposures to pick sample -- rough FWHM for refstack in log of run, nightly exposures in DB
    … …  
    792791  * 120420 - look ok
    793792  * 120507 ~7.52 (7.9 in cam, wrong way even for cam/warp scaling) -- looked poor, don't use as test
     793
    794794 * need to pick one from 2011 not used in refstack
    795795{{{
    … …  
    799799}}}
    800800
    801 2012 sample to test before updating
     8012012 sample to test before full field updating -- like from above
    802802{{{
    803803SELECT exp_id,exp_name,exp_time,dateobs,zpt_obs,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id)  WHERE  comment like "STS%PP1%" group by exp_id;
    … …  
    809809| 477549 | o6033g0430o |       30 | 2012-04-16 13:31:05 | 24.5586 |    7.96196 | 1.194 | 1.83        | 154.22   | 18.30    | 8.08  | 7.71  | 4586012 | STS.180.ND PP1 center    |
    810810...
    811 ++++ | 478479 | o6037g0401o |       20 | 2012-04-20 13:46:26 | 24.5799 |     4.1334 | 1.174 | 1.06        | 76.31    | 12.57    | 4.42  | 3.87  | 8291412 | STS.180.GIQ PP1 dither 1 |
     811+good+ | 478479 | o6037g0401o |       20 | 2012-04-20 13:46:26 | 24.5799 |     4.1334 | 1.174 | 1.06        | 76.31    | 12.57    | 4.42  | 3.87  | 8291412 | STS.180.GIQ PP1 dither 1 |
    812812...
    813813| 479810 | o6040g0441o |       30 | 2012-04-23 13:31:03 |  24.608 |    6.12941 | 1.131 | 1.20        | 147.86   | 17.88    | 6.39  | 5.88  | 5637788 | STS.180.ND PP1 center    |
    … …  
    817817| 483576 | o6049g0286o |       30 | 2012-05-02 13:09:26 | 24.5922 |    4.77424 | 1.088 | 0.94        | 166.69   | 17.89    | 5.10  | 4.62  | 7102152 | STS.180.ND PP1 center    |
    818818...
    819 xxxx  | 484578 | o6054g0305o |       30 | 2012-05-07 12:48:56 | 24.5422 |    7.94199 | 1.042 | 0.76        | 297.28   | 23.85    | 8.11  | 7.69  | 3526740 | STS.180.ND PP1 center    |
     819-avoid-  | 484578 | o6054g0305o |       30 | 2012-05-07 12:48:56 | 24.5422 |    7.94199 | 1.042 | 0.76        | 297.28   | 23.85    | 8.11  | 7.69  | 3526740 | STS.180.ND PP1 center    |
    820820...
    821821| 487890 | o6062g0361o |       30 | 2012-05-15 12:00:59 | 24.5415 |    4.14248 | 1.177 | 1.04        | 146.86   | 16.61    | 4.58  | 3.86  | 8390052 | STS.180.ND PP1 center    |
    822822}}}
    823823
    824 --> on disk still so try also
    825 --> need warp_ids? try exposure name - in any case need to see state in - yes exp_name works -
    826 
    827 select * from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_name="o6062g0361o";
    828 
    829 --> try to at least include 4/14 sing a GIQ? 4/20 cleaned but know work so 4/14 not necessary --
    830 --> have to use the conv stack..
    831 
    832 
     824 * o6054g0305o on disk still so easy to just test try also
     825 * try to at least include 4/14 using a GIQ? 4/20 cleaned but know work, so including 4/14 not necessary.
     826 * have to use the conv stack
     827
     828Run test diffims:
    833829{{{
    834830select uri from warpSkyfile where warp_id=409660 and skycell_id like "%1.03";
    … …  
    844840./run_ppsub_night_ws.csh  gpc1/STS.nt/2012/04/12//o6029g0441o.475121/o6029g0441o.475121.wrp.396474.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120412c > runppsub_sts103_120412_1c.bat
    845841source runppsub_sts103_120412_1c.bat >& runppsub_sts103_120412_1c.log &
    846 --> bilster cores (undersubs)
     842--> bilster cores (undersubs?)
    847843}}}
    848844
    … …  
    852848./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/14//o6031g0408o.476338/o6031g0408o.476338.wrp.397423.skycell.1.03.fits   gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120414c > runppsub_sts103_120414_1c.bat
    853849source runppsub_sts103_120414_1c.bat >& runppsub_sts103_120414_1c.log &
    854 --> blister cores (undersub)
     850--> blister cores (undersub?)
    855851}}}
    856852
    … …  
    858854./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/16//o6033g0430o.477549/o6033g0430o.477549.wrp.398532.skycell.1.03.fits  gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120416c > runppsub_sts103_120416_1c.bat
    859855source runppsub_sts103_120416_1c.bat >& runppsub_sts103_120416_1c.log &
    860 --> large conv, oversub
    861 }}}
    862 
     856--> large conv oversub?
     857}}}
     858
     859 * remaining set
     860{{{
    863861./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/23//o6040g0441o.479810/o6040g0441o.479810.wrp.401931.skycell.1.03.fits   gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120423c > runppsub_sts103_120423_1c.bat
    864862source runppsub_sts103_120423_1c.bat >& runppsub_sts103_120423_1c.log &
    … …  
    878876}}}
    879877
    880 --> looks like 110918, 120420, 120423/120502 -- use 4/23, seeing some odd conv/pixrej artifacts coming out in diffim but not sure if detected and 30s version
    881 --> may also want to try unconv since STS is supposed to be done w/in certain IQ?
    882 
    883 
    884  * set select nights up for updates - number exposures ~ 16(20s)/16(20s)/12(30s) for 20110918, 20120420, 20120423
     878 * looks like 20110918, 20120420, 20120423/20120502 are good samples -- use 4/23, seeing some odd conv/pixel rejected artifacts(?) coming out in diffim but not sure if detected and 30s version
     879 * may also want to try noconv since STS is supposed to be done within certain IQ constraints
     880
     881
     882Update full field for diffim run:
     883 * set selected nights up for updates - number of exposures ~16(20s)/16(20s)/12(30s) for 20110918, 20120420, 20120423 respectively
    885884  * label STS.mehdifftest
    886 --> will need to script mysql call based on what to get the chip/warp_id? detrend different since? may not be unique run given reprocessing for refstack oh well.. need PP1 as well
    887 --> manually checked and all unique..
     885  * will need to script mysql call based on what to get the chip/warp_id? detrend different since? may not be unique run given reprocessing for refstack oh well.. need PP1 as well
     886  * manually checked and all unique..
    888887{{{
    889888select exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" group by exp_name;
    … …  
    893892}}}
    894893
    895 --> will have some clean state issues possibly..
    896 --> note dateobs is 2 groups..
     894 * will have some clean state issues possibly..
     895 * note dateobs is 2 groups..
    897896
    898897 * 20110918 during magicked era, keep eye out for any issues. may have to just reprocess it - try update since just a test anyways and see -- did update for test case anyways so should work
    899 
    900 mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_110918.bat
    901 
    902 mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-20%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120420.bat
    903 
    904 mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-23%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120423.bat
    905 
    906 --> warp updates driving ipp017 nuts.. ~10-20% ram for 5 runs for total ~75%, cpu_wait and swap overload. killing 1 at time
    907 
     898 * create chip->warp update lists
     899{{{
     900mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_110918.bat
     901
     902mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-20%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120420.bat
     903
     904mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2012-04-23%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1  -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id  "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_120423.bat
     905}}}
     906
     907 *  warp updates driving ipp017 nuts.. ~10-20% RAM for 5 runs for total ~75%, cpu_wait and swap overload. killing 1 at time...
     908{{{
    908909  1    ipp017    BUSY   2993.25 0.0.5.fe34  0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400730 --warp_skyfile_id 34217254 --skycell_id skycell.1.22 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0429o.478506/o6037g0429o.478506.wrp.400730.skycell.1.22 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose
    909910
    910911  2    ipp017    RESP   3369.78 0.0.6.269  0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400756 --warp_skyfile_id 34219846 --skycell_id skycell.1.21 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0450o.478527/o6037g0450o.478527.wrp.400756.skycell.1.21 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose
    911912
    912 --> hm, didnt fault on czarpage for revert? fault=4
    913 
     913...
     914}}}
     915
     916 * hm, didn't fault on czarpage for revert? fault=4
     917{{{
    914918select warpRun.warp_id,state,data_state,fault from warpRun join warpSkyfile using (warp_id) where label like "STS.mopsdiff.mehtest" and data_state="update";
    915919
    916920warptool -dbname gpc1 -revertwarped -label STS.mopsdiff.mehtest   -fault 4
    917921--> ok all finished fine now..
    918 
     922}}}
    919923
    920924
    921925Setup full projection 1 diffim run:
    922  
    923 label STS.mopsdiff.mehtest
    924 data_group STS.mopsdiff.mehtest
    925 workdir meh/STS.mopsdiff.mehtest/
    926 distribution ? need to know about publishing - does it need normal dist? should run on own using diffim info
     926 * label = STS.mopsdiff.mehtest
     927 * data_group = STS.mopsdiff.mehtest
     928 * workdir = meh/STS.mopsdiff.mehtest/
     929 * distribution? no, need to know about publishing - does it need normal distribution? should run on own using diffim info
     930{{{
    927931--> difftool command for normal WS?
    928932--> need to add inverse since WS has as false, might as well set diff_mode as well -- not part of definewarpstack since implied
    … …  
    931935--> need -pretend to see output -- cant test until updates are done -- but can intermediate test what to expect, see diffs with other fields so need -skycell_id %.1.%
    932936*** wildcards not work so need to do mysql cmdline batch since didnt restrict batch updates.. or who cares, just run them.. this is a problem with the large overlap skycells, and more restrictions will be needed in future runs
    933 
     937}}}
     938
     939{{{
    934940difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -bothways -set_reduction WARPSTACK -simple -pretend
    935941
    … …  
    945951
    946952difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -set_reduction WARPSTACK -bothways -simple -rerun
     953}}}
    947954
    948955'''trouble -- left -bothways in run and don't want to do that'''
    949956
    950 --> odd fault 4, won't fix -- something in psphot
     957 * odd fault 4, won't fix -- something in psphot
     958{{{
    951959242183  skycell.1.29    STS.mopsdiff.mehtest    STS.20110918
    952 
    953 
    954 
    955 -- distribution to MOPS publishing -
    956 --> setup ? dont know how to associate label to particular datastore location for this..
    957 -defineclient
    958   -stage          define stage (required)
    959   -product        define product (required)
    960   -workdir        define workdir (required)
    961   -output_format  define output format
    962   -comment        define comment
    963   -name           define name
    964   -unmagicked     allow unmagicked data?
    965 -updateclient:
    966   -client_id  search by client_id
    967   -stage      search by stage
    968   -product    search by product
    969   -comment    search by comment (LIKE)
    970   -active     set to active state
    971   -inactive   set to inactive state
     960}}}
     961
     962
     963Distribution to MOPS via publishing -
     964 * setup? how to associate label to particular datastore location for this run for IPP-MOPS-TEST-2
     965{{{
    972966
    973967| publishClient           |
    974968| publishDone             |
    975969| publishRun              |
    976 
    977 {{{
    978 mysql> select * from publishClient;
    979 +-----------+--------+-----------------+----------+----------+---------------------------------+---------------------------------------------------+-----------------+---------------+
     970}}}
     971
     972select * from publishClient;
     973
    980974| client_id | active | product         | stage    | magicked | workdir                         | comment                                           | name            | output_format |
    981975+-----------+--------+-----------------+----------+----------+---------------------------------+---------------------------------------------------+-----------------+---------------+
    … …  
    989983|        12 |      1 | IPP-MOPS-TEST-2 | diff     |        0 | neb://any/gpc1/IPP-MOPS-TEST-2/ | IPP detections for MOPS / Output format version 2 | IPP-MOPS-TEST-2 |             2 |
    990984|        13 |      1 | IPP-MOPS-LAP    | diff     |        1 | neb://any/gpc1/IPP-MOPS-LAP     | IPP detections from Large Area Processing         | NULL            |             2 |
     985
     986--> want to put to IPP-MOPS-TEST-2 so client_id=12
    991987}}}
    992988
    … …  
    11561152select * from diffRun where data_group like "STS.2012%";
    11571153
    1158 mysql -hippdb01 -uippuser -pippuser gpc1 -B -e 'select diff_id,state,data_group from diffRun where data_group like "STS.2012%" and state="cleaned";' | grep -v diff_id | awk '{print "difftool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest  -diff_id  "$1}' > ~/updatediff_sts.mopsdiff.mehtest.bat
     1154mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select diff_id,state,data_group from diffRun where data_group like "STS.2012%" and state="cleaned";' | grep -v diff_id | awk '{print "difftool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest  -diff_id  "$1}' > ~/updatediff_sts.mopsdiff.mehtest.bat
    11591155
    11601156--> ipp017 ram overuse
    … …  
    11661162--> needs workdir, label and data_group optional
    11671163difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_workdir neb://@HOST@.0/meh/STS.mopsdiffredo.mehtest -set_reduction WARPSTACK  -simple -rerun -pretend
    1168 
    11691164
    11701165== Diffim sample 2 (published) ==