IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 15 and Version 16 of PS1_IPP_Czarlog_20190318


Ignore:
Timestamp:
Mar 25, 2019, 10:37:25 AM (7 years ago)
Author:
fairlamb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • PS1_IPP_Czarlog_20190318

    v15 v16  
    801801'''Maybe Rob job not all good'''
    802802
    803 Maybe Rob job not all good
    804 
     803Have a look at theses exposures:
     804{{{
    8058051       MOPS    918892  weryk_5C95386D  2019-03-22 20:19:52     25      217487  00:10:09        2019-03-22 20:30:01
    8068061       MOPS    918892  weryk_5C95386D  2019-03-22 20:19:52     25      217488  00:10:09        2019-03-22 20:30:01
    8078071       MOPS    918892  weryk_5C95386D  2019-03-22 20:19:52     25      217490  00:10:09        2019-03-22 20:30:01
    8088082       MOPS    918892  weryk_5C95386D  2019-03-22 20:19:52     25      217508  00:10:09        2019-03-22 20:30:01
     809}}}
    809810Check 'em oot:
    810811{{{
    … …  
    831832}}}
    832833
    833 warptool -dbname gpc1 -listrun -simple -warp_id 1196545
    834 warptool -dbname gpc1 -listrun -simple -warp_id 1195189
    835 warptool -dbname gpc1 -listrun -simple -warp_id 1195185
    836 warptool -dbname gpc1 -listrun -simple -warp_id 1289563
    837 warptool -dbname gpc1 -listrun -simple -warp_id 1289536
    838 warptool -dbname gpc1 -listrun -simple -warp_id 1289497
    839 warptool -dbname gpc1 -listrun -simple -warp_id 1289466
    840 warptool -dbname gpc1 -listrun -simple -warp_id 1979742
    841 warptool -dbname gpc1 -listrun -simple -warp_id 1979741
    842 warptool -dbname gpc1 -listrun -simple -warp_id 1979739
    843 warptool -dbname gpc1 -listrun -simple -warp_id 1979740
    844 
    845 warp    chip    exp
    846 1196545 1247225 536246 o6222g0265o
    847 1195189 1245021 661797 o6568g0237o
    848 1195185 1245018 661777 o6568g0218o
    849 1289563 1347972 841210 o7011g0400o
    850 1289536 1347954 841190 o7011g0382o
    851 1289497 1347936 841174 o7011g0364o
    852 1289466 1347918 841156 o7011g0346o
    853 1979742 2036560 1354013 o8120g0081o
    854 1979741 2036559 1354012 o8120g0080o
    855 1979739 2036558 1354011 o8120g0079o
    856 1979740 2036557 1354010 o8120g0078o
    857 
    858 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id
    859 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id
    860 
     834Use the above to update the runs, aka:
     835{{{
    861836chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1247225
    862 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1245021
    863 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1245018
    864 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1347972
    865 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1347954
    866 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1347936
    867 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1347918
    868 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  2036560
    869 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  2036559
    870 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  2036558
    871 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  2036557
    872 
    873837warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1196545
    874 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1195189
    875 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1195185
    876 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1289563
    877 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1289536
    878 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1289497
    879 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1289466
    880 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1979742
    881 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1979741
    882 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1979739
    883 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1979740
    884 
    885 
    886 
    887 More work from Weryk
    888 
    889 1       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      306270  00:13:08        2019-03-23 00:30:01
    890 1       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      306271  00:13:08        2019-03-23 00:30:01
    891 1       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      306274  00:13:08        2019-03-23 00:30:01
    892 1       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      306275  00:13:08        2019-03-23 00:30:01
    893 1       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      308291  00:13:08        2019-03-23 00:30:01
    894 1       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      308294  00:13:08        2019-03-23 00:30:01
    895 1       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      308298  00:13:08        2019-03-23 00:30:01
    896 1       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      308994  00:13:08        2019-03-23 00:30:01
    897 2       MOPS    918909  weryk_5C95761A  2019-03-23 00:17:15     25      606306  00:12:46        2019-03-23 00:30:01
    898 
    899 Check these out... they might be good since I've just updated the run on some other jobs
    900 
    901 
    902 
    903 
    904 '''We need to prepare for nightly processing again'''
    905 
     838}}}
    906839
    907840
    … …  
    915848''' ganglia demon failed across the cluster '''
    916849
    917 JRF: Had to restart it on a whole load of machines... Add the command to fix it here (note that the ubuntu ones look a little strange)
    918 
     850JRF: Had to restart it on a whole load of machines...
     851{{{
     852sudo /etc/init.d/gmond status
     853sudo /etc/init.d/gmond restart
     854}}}
     855on the newer ubuntu machines ipp127+:
     856{{{
     857sudo service ganglia-monitor status
     858sudo service ganglia-monitor restart
     859}}}
     860Though the above didn't always work on the ubuntu machines
    919861
    920862''ps_ud_MOPS error'''
    … …  
    923865Tried to revert, it failed. Then tried to update, it also failed.
    924866
    925 
    926867So I'm just going to rerun the exposure:
     868{{{
    927869chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190323 -set_reduction LAP_SCIENCE -exp_name 627000
    928 
     870}}}
    929871Check it's done
     872{{{
    930873SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name="o6473g0463o";
    931 
    932 Cleanup the old stuff:
     874}}}
     875Then clean up the old stuff
    933876
    934877
    935878''' pstamps '''
    936879
     880{{{
    93788110960392 new warp 1999761 skycell.0751.033 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918963 ps_ud_MOPS F 0 0 MOPS 918963 /data/ipp113.0/pstamp/work/2019/03/24/918963/checkdep.10960392.log 500
    93888210960394 new warp 1999761 skycell.0751.034 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918963 ps_ud_MOPS F 0 0 MOPS 918963 /data/ipp113.0/pstamp/work/2019/03/24/918963/checkdep.10960394.log 500
    … …  
    951895| o8185g0309o | 1379681 | 2057059 | ps_ud_MOPS | update     | 1999761 | ps_ud_MOPS      | update     |
    952896+-------------+---------+---------+------------+------------+---------+-----------------+------------+
     897}}}
    953898
    954899I'm just gonna try and update them, see what happens.
     900{{{
    955901chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1765399
    956902chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1929751
    … …  
    961907warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1875836
    962908warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1999761
     909}}}
     910
     911It worked!
    963912
    964913=== Sunday : 2019.03.24 ===
    … …  
    966915There are a bunch of chips and warps stuck in the full state...
    967916
     917{{{
    968918SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipProcessedImfile.fault != 0 AND chipRun.label like 'update.LAP.PV3' AND chipRun.state LIKE "full" LIMIT 80 OFFSET 0;
    969919
    … …  
    980930| o6782g0200o | 732898 | 1047451 | full  | update.LAP.PV3 |     2 | GPC1   |
    981931| o6782g0200o | 732898 | 1047451 | full  | update.LAP.PV3 |     2 | GPC1   |
    982 | o5297g0292o | 156364 | 1047462 | full  | update.LAP.PV3 |     2 | GPC1   |
    983 | o5297g0292o | 156364 | 1047462 | full  | update.LAP.PV3 |     2 | GPC1   |
     932| o5297g0292o | 156364 | 1047462 | full  | update.LAP.PV3 |     2 | GPC1   | 
    984933| o5306g0280o | 159686 | 1047465 | full  | update.LAP.PV3 |     2 | GPC1   |
    985934| o5653g0302o | 320134 | 1047468 | full  | update.LAP.PV3 |     2 | GPC1   |
    986935| o6001g1268o | 465334 | 1047470 | full  | update.LAP.PV3 |     2 | GPC1   |
    987 | o6001g1268o | 465334 | 1047470 | full  | update.LAP.PV3 |     2 | GPC1   |
    988 | o6001g1269o | 465335 | 1047471 | full  | update.LAP.PV3 |     2 | GPC1   |
    989936| o6001g1269o | 465335 | 1047471 | full  | update.LAP.PV3 |     2 | GPC1   |
    990937| o6001g1269o | 465335 | 1047471 | full  | update.LAP.PV3 |     2 | GPC1   |
    … …  
    993940| o6013g0292o | 469066 | 1047474 | full  | update.LAP.PV3 |     2 | GPC1   |
    994941| o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
    995 | o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
    996 | o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
    997 | o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
    998 | o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
    999942+-------------+--------+---------+-------+----------------+-------+--------+
    1000 
     943}}}
     944
     945Attempt to clean...
     946{{{
    1001947chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047387
    1002 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047394
    1003 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047433
    1004 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047440
    1005 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047444
    1006 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047445
    1007 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047446
    1008 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047451
    1009 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047462
    1010 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047465
    1011 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047468
    1012 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047470
    1013 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047471
    1014 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047472
    1015 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047473
    1016 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047474
     948.
    1017949chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047475
    1018 
     950}}}
     951then update...
     952{{{
    1019953chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047387
    1020 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047394
    1021 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047433
    1022 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047440
    1023 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047444
    1024 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047445
    1025 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047446
    1026 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047451
    1027 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047462
    1028 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047465
    1029 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047468
    1030 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047470
    1031 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047471
    1032 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047472
    1033 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047473
    1034 chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047474
     954.
    1035955chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047475
    1036 
     956}}}
    1037957HOWEVER, this failed and I'm not sure why...
    1038 So there are now some fault 2 entries with goto_cleaned...
     958So there are now some fault 2 entries with goto_cleaned... query this to get them:
     959{{{
     960SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipRun.chip_id IN (1047387,1047394,1047433,1047440,1047444,1047445,1047446,1047451,1047462,1047465,1047468,1047470,1047471,1047472,1047473,1047474,1047475);
     961}}}
     962
    1039963FIX ON MONDAY!
    1040964
    1041 {{{
    1042 SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipRun.chip_id IN (1047387,1047394,1047433,1047440,1047444,1047445,1047446,1047451,1047462,1047465,1047468,1047470,1047471,1047472,1047473,1047474,1047475);
    1043 }}}
    1044 
    1045 Remaining chip issues:
     965'''Remaining chip issues:'''
    1046966
    1047967All 4 of them:
    … …  
    1049969o5280g0317o     XY01    151129  1059616 update  update.LAP.PV3
    1050970o5280g0316o     XY01    151128  1059817 update  update.LAP.PV3
    1051 o6091g0094o     XY22    498248  1060186 update  update.LAP.PV3 
     971o6091g0094o     XY22    498248  1060186 update  update.LAP.PV3
     972
    1052973Complain about a missing mdc config file:
    1053974{{{
    … …  
    1056977For the first 3 it also complains about burntool state not matching that in the chip_imfile.pl
    1057978
    1058 Sent them to revert...perhaps they will clear themselves up.
    1059 TODO TOMORROW MORNING:
    1060 Perhaps revert these to sort them out, or reprocess.
     979Sent them to revert...perhaps they will clear themselves up ('''update''': they did not, tune in to Monday's czar log to find out what happened)
    1061980
    1062981warps:
    1063 
     982{{{
    1064983SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault,rawExp.camera FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.label like 'update.LAP.PV3' LIMIT 80 OFFSET 0;
    1065984
    … …  
    1073992| o5649g0216o | 318039 | 1048195 | 1003988 | skycell.0839.048 | update | update.LAP.PV3 |     4 | GPC1   |
    1074993+-------------+--------+---------+---------+------------------+--------+----------------+-------+--------+
    1075 
     994}}}
    1076995
    1077996
    … …  
    1080999    warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id XXXXX
    10811000}}}
    1082 check that is has cleaned succesfully (query the database). Then set it to update:
     1001check that is has cleaned successfully by querying the database e.g.
     1002{{{
     1003SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault,rawExp.camera FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.warp_id IN (1004746,1004795,1004800,1004812,1003988);
     1004}}}
     1005
     1006Then set it to update:
    10831007{{{
    10841008warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id XXXXX
    10851009}}}
    10861010
    1087 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1004746
    1088 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1004795
    1089 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1004800
    1090 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1004812
    1091 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1003988
    1092 
    1093 SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault,rawExp.camera FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.warp_id IN (1004746,1004795,1004800,1004812,1003988);
    1094 
    1095 warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1004746
    1096 warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1004795
    1097 warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1004800
    1098 warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1004812
    1099 warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1003988
    1100