| Version 41 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.03.25 - 2019.03.31
(Up to PS1 IPP Czar Logs)
Czar of the week: CCL, with EAM as czar-lord (previously: JRF + MEH)
Monday : 2019.03.25
- MEH: ganglia again missing many nodes reporting -- Gavin cleared and notes to restart from scratch/defunct state order
ippc18: restart gmond -- sudo /etc/init.d/gmond restart ippc19: restart gmond all other nodes: restart gmond (ganglia-monitor on ubuntu/systemd -- sudo systemctl status ganglia-monitor.service) restart gmetad on ippops2 when page not running or items not displaying or multiple copies displaying on web page -- sudo /etc/init.d/gmetad restart
LAP.PV3 faulted: by JRF
The following are broken:
o5280g0301o XY01 151113 1059611 update update.LAP.PV3 LAP.PV3.20140730.20140901 LAP.ThreePi 2 o5280g0317o XY01 151129 1059616 update update.LAP.PV3 LAP.PV3.20140730.20140901 LAP.ThreePi 2 o5280g0316o XY01 151128 1059817 update update.LAP.PV3 LAP.PV3.20140730.20140901 LAP.ThreePi 2 o6091g0094o XY22 498248 1060186 update update.LAP.PV3 LAP.PV3.20140730.20140901 LAP.ThreePi 2 o5582g0644o XY44 283730 1062952 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5582g0647o XY45 283733 1062954 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5582g0665o XY44 283751 1062956 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5582g0649o XY45 283735 1062992 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5719g0065o XY73 344295 1063341 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5282g0287o XY01 152148 1063529 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2
All of these are complaining about some missing mdc files, so it fails with a poor burntool state. I will try to reprocess them:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name XXXXXX
Check they are done:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5280g0301o","o5280g0317o","o5280g0316o","o6091g0094o","o5582g0644o","o5582g0647o","o5582g0665o","o5582g0649o","o5719g0065o","o5282g0287o");
They are not done, and they contiue to fail... However, two of them made it through! o6091g0094o o5719g0065o
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o6091g0094o","o5719g0065o");
These two had slightly different errors. They did not complain about the burntool, so reprocessing worked for them.
Instead, they are complaining about a missing psf file. Take a look at the missing file using:
neb-stat -validate
we can see that is does not exist:
instance location:
0 NON-EXISTANT file:///data/ipp120.1/nebulous/32/5c/9229835926.gpc1:LAP.PV3.20140730:2014:09:01:o6091g0094o.498248:o6091g0094o.498248.ch.1060186.XY22.psf
Indeed, doing 'ls' on the file location tells us that it does not exist.
The best way to fix this is to move that neb location to bad.
neb-mv neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf.bad
Then we can reprocess the file and it should hopefully make a new psf file:
chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1060186
burntool issues by JRF
The remaining issues complain about the burntool state. We have little information, it just says that:
burntool state vs burntoolStateGood : 0 vs 14
Image burntool version does not match current accepted version. at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834
main::my_die('Image burntool version does not match current accepted version.', 151113, 2131394, 'XY01', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 328
Check out the uri:
neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl
Note that the neb file it the same as the fits file in the error log, but with .burn.tbl (I didn't include the fits part above)
We find that this burn.tbl file has 0 size, and it's located on an available node ipp070.0
Check out: https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20180521 for info on how to fix this:
Ok, so we are going to fix this as ippitc, as it's probably safer. Run the burntool fix:
ipp_apply_burntool_fix.pl --exp_name o5280g0301o --class_id XY01 --verbose --dbname gpc1
Make sure to give it the correct class_id.
It will spew out some useful information on what's happening e.g. that it has removed the old stuff, and made some new ones. It's best to check that it created some new burn tables, which can be done by looking up the table again:
neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl
Running this we see two new copies, in new places, they are the same size, and that size is non-zero!
Repeat for the remaining:
ipp_apply_burntool_fix.pl --exp_name o5280g0316o --class_id XY01 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0644o --class_id XY44 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0647o --class_id XY45 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0665o --class_id XY44 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0649o --class_id XY45 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5719g0065o --class_id XY73 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5282g0287o --class_id XY01 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0608o --class_id XY44 --verbose --dbname gpc1
Missing PSF files by JRF
This was originally solved by some reprocessing (see the two that worked above). However, the correct way to fix it is as follows:
The errors complain about a config file, psf file, and burn tool. So look at all of them:
$> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.ppImage.mdc
no instances
$> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.psf
no instances
$> neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
object id: 3915331061
key: gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
read lock: 0
write lock:
epoch: 2014-08-31 18:02:51
mtime: 0000-00-00 00:00:00
md5sum count: 1
requested instances: 1
available instances: 1
existing instances: 1
total instances: 1
instances location:
1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp069.0/nebulous/b3/3f/5048738853.gpc1:20100325:o5280g0317o:o5280g0317o.ota01.burn.tbl
No instances for the first two, and the burn.tbl has the magic 0 size MD5 sum d41d8cd98f00b204e9800998ecf8427e
So, first off let's make new burn.tbls and then revert:
ipp_apply_burntool_fix.pl --exp_name o5280g0317o --class_id XY01 --verbose --dbname gpc1
neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1059616
MOPS No Masks by JRF
MOPS have requested some exposures to be reprocessed with no mask:
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0145o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0126o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0107o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0088o
Check they are done:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o8347g0145o","o8347g0126o","o8347g0107o","o8347g0088o");
They're done
Prep for nightly by JRF
Have created a config for targeting:
ipphosts.mhpcc.config.190325.ready_for_nightly_and_more
which contains the config from earlier in the month before we were targeting some new nodes for updates. It also includes 4 of the new nodes, ipp127-ipp130, as these seem ok to be used (the ones after this cannot be accessed from the console yet).
CCL: Some pending jobs on pstamp were sent in title of [ps-ipp-ops] Cron <ippitc@ippc30> /bin/bash /data/ippc64.1/ippitc/check_pstamp_stalled.bat Following previous steps Czar Logs 2019-03-18 to solve the problem.
email contents: count(*) label req_id name timestamp fault exp_id dtime utc_timestamp() 176 MOPS 919001 weryk_5C994224 2019-03-25 21:10:32 0 1330887 02:15:39 2019-03-25 23:20:01 Under terminal, check the warp id with “pstamptool”, warp id is the 4th column. pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple 10961873 new warp 1018709 skycell.0925.068 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961873.log 500 10961874 new warp 1018709 skycell.0925.079 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961874.log 500 10961875 new warp 1018709 skycell.0925.069 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961875.log 500 10961876 new warp 1018709 skycell.0925.078 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961876.log 500 10961880 new warp 1018708 skycell.0925.068 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961880.log 500 10961881 new warp 1018708 skycell.0925.079 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961881.log 500 10961882 new warp 1018708 skycell.0925.069 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961882.log 500 10961883 new warp 1018708 skycell.0925.078 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961883.log 500 Under mysql, check their labels. SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708); +-------------+--------+---------+----------------+--------+---------+----------------+--------+ | exp_name | exp_id | chip_id | chipLabel | state | warp_id | warpLabel | state | +-------------+--------+---------+----------------+--------+---------+----------------+--------+ | o6864g0069o | 770608 | 1070075 | update.LAP.PV3 | update | 1018708 | update.LAP.PV3 | update | | o6864g0070o | 770607 | 1070076 | update.LAP.PV3 | update | 1018709 | update.LAP.PV3 | update | +-------------+--------+---------+----------------+--------+---------+----------------+--------+ 2 rows in set (0.00 sec) Under terminal, set their label to ps_ud_MOPS (right?) chiptool -dbname gpc1 -updaterun -chip_id 1070075 -set_label ps_ud_MOPS chiptool -dbname gpc1 -updaterun -chip_id 1070076 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1018708 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1018709 -set_label ps_ud_MOPS Under mysql, check their label SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708); +-------------+--------+---------+------------+--------+---------+------------+--------+ | exp_name | exp_id | chip_id | chipLabel | state | warp_id | warpLabel | state | +-------------+--------+---------+------------+--------+---------+------------+--------+ | o6864g0069o | 770608 | 1070075 | ps_ud_MOPS | update | 1018708 | ps_ud_MOPS | update | | o6864g0070o | 770607 | 1070076 | ps_ud_MOPS | update | 1018709 | ps_ud_MOPS | update | +-------------+--------+---------+------------+--------+---------+------------+--------+ 2 rows in set (0.00 sec) Few minutes later, the state changed to full and problem solved! SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708); +-------------+--------+---------+------------+-------+---------+------------+-------+ | exp_name | exp_id | chip_id | chipLabel | state | warp_id | warpLabel | state | +-------------+--------+---------+------------+-------+---------+------------+-------+ | o6864g0069o | 770608 | 1070075 | ps_ud_MOPS | full | 1018708 | ps_ud_MOPS | full | | o6864g0070o | 770607 | 1070076 | ps_ud_MOPS | full | 1018709 | ps_ud_MOPS | full | +-------------+--------+---------+------------+-------+---------+------------+-------+ 2 rows in set (0.01 sec)
CCL: put ipp134-ipp139 repair, not ready for new data yet.
neb-host --state repair --host ipp134 --note 'CCL: up --> repair, not for new data yet' neb-host --state repair --host ipp135 --note 'CCL: up --> repair, not for new data yet' neb-host --state repair --host ipp136 --note 'CCL: up --> repair, not for new data yet' neb-host --state repair --host ipp137 --note 'CCL: up --> repair, not for new data yet' neb-host --state repair --host ipp138 --note 'CCL: up --> repair, not for new data yet' neb-host --state repair --host ipp139 --note 'CCL: up --> repair, not for new data yet'
Tuesday : 2019.03.26
- JRF: Don't forget about these that I couldn't get to update again from last Sunday:
SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipRun.chip_id IN (1047387,1047394,1047433,1047440,1047444,1047445,1047446,1047451,1047462,1047465,1047468,1047470,1047471,1047472,1047473,1047474,1047475);
- CCL: pstamp stalled last night
2 MOPS 919024 weryk_5C997837 2019-03-26 01:12:03 26 686123 00:12:59 2019-03-26 01:25:02 mysql> SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1771288, 1771274, 1771255, 1771238, 1769120, 1769119, 1769102, 1769100, 1769087, 1769082, 1769068, 1769066, 1768305, 1768287, 1768269, 1768250, 1763443, 1763422, 1763401, 1763379); warp and chip labels were changed from update.hold to ps_ud_MOPS
- NOTE : ippc71 and ippc74 have DISK WARNING - free space: / 19316 MB (20% inode=95%):
Both log files /var/log/apache2/access_log have grown to 13G Will wait for the logrotate to take care of these files tonight, if the problems still show up then try the short-term solution by EAM:
1) stop the pantasks 2) stop the apache server [sudo /etc/init.d/apache2 stop] 3) move the file to a new name 4) start the apache server [sudo /etc/init.d/apache2 start] 5) compress the saved file (the apache restart is needed because even if you compress or remove the log file, if apache is still running the file system will keep the data around).
- MEH: need to setup ipp134-139 neb-host up during day/every day for large update processing
neb-host ipp134 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here" neb-host ipp135 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here" neb-host ipp136 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here" neb-host ipp137 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here" neb-host ipp138 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here" neb-host ipp139 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here"
- CCL 17:43: set ipp134-ipp139 repair
- CCL: try to fix some ps_ud_MOPS issue
chip stage o6490g0215o XY57 634056 1391686 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20150107 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1 o6007g0018o XY12 468033 1446000 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20150125 LAP.ThreePi 3 chiptool -revertprocessedimfile -fault 3 -label ps_ud_MOPS -dbname gpc1 o6007g0041o XY12 468057 1446001 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20150125 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1 o6007g0066o XY01 468082 1446949 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20150125 LAP.ThreePi 3 chiptool -revertprocessedimfile -fault 3 -label ps_ud_MOPS -dbname gpc1 o6007g0089o XY02 468105 1446950 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20150125 LAP.ThreePi 3 chiptool -revertprocessedimfile -fault 3 -label ps_ud_MOPS -dbname gpc1 o5516g0684o XY65 256903 1521531 update ps_ud_MOPS LAP.PV3.20140730.ipp.20150214 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1 o7819g0491o XY20 1212613 1909513 update ps_ud_MOPS OSS.20170307 SweetSpot 3 chiptool -revertprocessedimfile -fault 3 -label ps_ud_MOPS -dbname gpc1 o7819g0531o XY20 1212652 1909552 update ps_ud_MOPS OSS.20170307 SweetSpot 3 chiptool -revertprocessedimfile -fault 3 -label ps_ud_MOPS -dbname gpc1 o7819g0551o XY20 1212671 1909572 update ps_ud_MOPS OSS.20170307 SweetSpot 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1 first 6 missing log files due to they are located at neb://stsci* machines, 3 others are at regular ipp machines. try these command, but it looks like just move the problem to update.xxxx label? chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxxxx chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id xxxxx chiptool -dbname gpc1 -setimfiletoupdate -set_label update.OSS -chip_id xxxxx similar to warp id warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1284689 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1848779 warptool -dbname gpc1 -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id 1284689 warptool -dbname gpc1 -setskyfiletoupdate -set_label update.OSS -warp_id 1848779 fault 3, set it clean chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1260304 chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1260304
- mdc file missing
try reprocess chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5567g0335o, o5719g0065o, o5582g0607o, o5250g0479o, o5251g0472o, o5213g0291o check mysql -hxxx -uxxx -pxxx gpc1 -e "SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ('o5567g0335o', 'o5719g0065o', 'o5582g0607o', 'o5250g0479o', 'o5251g0472o', 'o5213g0291o');" +-------------+--------+---------+--------------------+------------+ | exp_name | exp_id | chip_id | chip_label | chip_state | +-------------+--------+---------+--------------------+------------+ | o5213g0291o | 123613 | 58242 | goto_cleaned_redo | cleaned | | o5213g0291o | 123613 | 529146 | goto_cleaned | cleaned | | o5213g0291o | 123613 | 1072875 | update.LAP.PV3 | update | | o5213g0291o | 123613 | 2131410 | mopsfix | new | | o5250g0479o | 139869 | 63020 | goto_cleaned.rerun | cleaned | | o5250g0479o | 139869 | 81096 | goto_cleaned_redo | cleaned | | o5250g0479o | 139869 | 542480 | goto_cleaned | cleaned | | o5250g0479o | 139869 | 1066434 | update.LAP.PV3 | update | | o5250g0479o | 139869 | 2131408 | mopsfix | new | | o5251g0472o | 140461 | 66088 | goto_cleaned.rerun | cleaned | | o5251g0472o | 140461 | 82466 | goto_cleaned_redo | cleaned | | o5251g0472o | 140461 | 542936 | goto_cleaned_redo | cleaned | | o5251g0472o | 140461 | 1067512 | update.LAP.PV3 | update | | o5251g0472o | 140461 | 2131409 | mopsfix | new | | o5567g0335o | 274405 | 176834 | goto_cleaned_redo | cleaned | | o5567g0335o | 274405 | 527098 | goto_cleaned | cleaned | | o5567g0335o | 274405 | 1060209 | update.LAP.PV3 | update | | o5567g0335o | 274405 | 2131405 | mopsfix | new | | o5582g0607o | 283693 | 181103 | goto_cleaned_redo | cleaned | | o5582g0607o | 283693 | 541962 | goto_cleaned | cleaned | | o5582g0607o | 283693 | 1063669 | update.LAP.PV3 | update | | o5582g0607o | 283693 | 2131407 | mopsfix | new | | o5719g0065o | 344295 | 239213 | goto_cleaned_redo | cleaned | | o5719g0065o | 344295 | 527546 | goto_cleaned | cleaned | | o5719g0065o | 344295 | 1063341 | update.LAP.PV3 | update | | o5719g0065o | 344295 | 2131402 | mopsfix | full | | o5719g0065o | 344295 | 2131406 | mopsfix | new | +-------------+--------+---------+--------------------+------------+ set them clean chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1072875 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1066434 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1067512 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1060209 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1063669 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1063341 +-------------+--------+---------+--------------------+------------+ | exp_name | exp_id | chip_id | chip_label | chip_state | +-------------+--------+---------+--------------------+------------+ | o5213g0291o | 123613 | 58242 | goto_cleaned_redo | cleaned | | o5213g0291o | 123613 | 529146 | goto_cleaned | cleaned | | o5213g0291o | 123613 | 1072875 | goto_cleaned | cleaned | | o5213g0291o | 123613 | 2131410 | mopsfix | new | | o5250g0479o | 139869 | 63020 | goto_cleaned.rerun | cleaned | | o5250g0479o | 139869 | 81096 | goto_cleaned_redo | cleaned | | o5250g0479o | 139869 | 542480 | goto_cleaned | cleaned | | o5250g0479o | 139869 | 1066434 | goto_cleaned | cleaned | | o5250g0479o | 139869 | 2131408 | mopsfix | new | | o5251g0472o | 140461 | 66088 | goto_cleaned.rerun | cleaned | | o5251g0472o | 140461 | 82466 | goto_cleaned_redo | cleaned | | o5251g0472o | 140461 | 542936 | goto_cleaned_redo | cleaned | | o5251g0472o | 140461 | 1067512 | goto_cleaned | cleaned | | o5251g0472o | 140461 | 2131409 | mopsfix | new | | o5567g0335o | 274405 | 176834 | goto_cleaned_redo | cleaned | | o5567g0335o | 274405 | 527098 | goto_cleaned | cleaned | | o5567g0335o | 274405 | 1060209 | goto_cleaned | cleaned | | o5567g0335o | 274405 | 2131405 | mopsfix | new | | o5582g0607o | 283693 | 181103 | goto_cleaned_redo | cleaned | | o5582g0607o | 283693 | 541962 | goto_cleaned | cleaned | | o5582g0607o | 283693 | 1063669 | goto_cleaned | cleaned | | o5582g0607o | 283693 | 2131407 | mopsfix | new | | o5719g0065o | 344295 | 239213 | goto_cleaned_redo | cleaned | | o5719g0065o | 344295 | 527546 | goto_cleaned | cleaned | | o5719g0065o | 344295 | 1063341 | goto_cleaned | cleaned | | o5719g0065o | 344295 | 2131402 | mopsfix | full | | o5719g0065o | 344295 | 2131406 | mopsfix | new | +-------------+--------+---------+--------------------+------------+ why did not see it became full? Reminded by MEH, need to add mopsfix label to pantask for processing. And they are actually from burntool files problems so try these to solve pending_burntool state. ipp_apply_burntool_fix.pl --exp_name o5567g0335o --class_id XY44 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5582g0607o --class_id XY44 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5250g0479o --class_id XY45 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5251g0472o --class_id XY02 --verbose --dbname gpc1 update: I set this two chip id 2131405, 2131407 to clean mode too early for cam stage. Set them back to update for pantasks and then cam stage can go through.
- CCL: pstamp stalled tonight
mysql -hxxx -uxxx -p gpc1 -e "SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1955483, 1955460, 1955446, 1955427);" +-------------+---------+---------+-------------+--------+---------+-------------+--------+ | exp_name | exp_id | chip_id | chipLabel | state | warp_id | warpLabel | state | +-------------+---------+---------+-------------+--------+---------+-------------+--------+ | o8059g0095o | 1324697 | 2012061 | update.hold | update | 1955427 | update.hold | update | | o8059g0112o | 1324714 | 2012078 | update.hold | update | 1955446 | update.hold | update | | o8059g0129o | 1324731 | 2012095 | update.hold | update | 1955460 | update.hold | update | | o8059g0146o | 1324748 | 2012112 | update.hold | update | 1955483 | update.hold | update | +-------------+---------+---------+-------------+--------+---------+-------------+--------+ chiptool -dbname gpc1 -updaterun -chip_id 2012061 -set_label ps_ud_MOPS chiptool -dbname gpc1 -updaterun -chip_id 2012078 -set_label ps_ud_MOPS chiptool -dbname gpc1 -updaterun -chip_id 2012095 -set_label ps_ud_MOPS chiptool -dbname gpc1 -updaterun -chip_id 2012112 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1955427 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1955446 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1955460 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1955483 -set_label ps_ud_MOPS
- update.LAP.PV3
some skycells not found issue, their size are none zero so try to revert to see what will happen warptool -revertwarped -fault 2 -label update.LAP.PV3 -dbname gpc1 warptool -revertwarped -fault 4 -label update.LAP.PV3 -dbname gpc1 did not work with revert, so set clean and update to solve. warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1284689, 1003988, 1020989 warptool -dbname gpc1 -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id 1284689, 1003988, 1020989
Wednesday : 2019.03.27
- MEH: manually clearing out and restarting cleanup pantasks
- CCL 08:30: ipp134-139 repair to up and fix some update.LAP.PV3 errors
warptool -revertwarped -fault 4 -label update.LAP.PV3 -dbname gpc1 did not work so try clean 1027567, 1027638, 1027645, 1027695, 1027697, 1027705, 1027711, 1027781, 1027785, 1027828, 1142465 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxxx warptool -dbname gpc1 -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id xxxxx
- CCL 16:30: ipp134-139 up --> repair, not for new data yet
- MEH: mops having trouble accessing data on ipp127-ipp130 so set to repair for now
Thursday : 2019.03.28
- MEH: ipp127-130 repair->up -- mops has was work around for now and nightly processing finished
- CCL 10:00: ipp134-139 repair->up for updating data
- CCL 10:00: add update.LAP.PV3 label to pantask
- CCL 14:00: fix some problems on ps_ud_MOPS
warp id 1469446, 1469448, 1472184, 1472186, 1485402, 1485404 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxx after cleaned warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id xxx if it gets stucked at warp stage, maybe need to update chip id as well so 1519049, 1519051, 1523937, 1523939, 1524637, 1524639 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxx after cleaned... chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id xxxx
- CCL 16:00: fix some problems on update.LAP.PV3
chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1
- CCL 16:30: stop updating update.LAP.PV3 (del.label update.LAP.PV3) for nightly processing
- CCL 16:30: set ipp134-139 repair to avoid saving new data
- CCL 17:00: ippmonitor loading lag, need to be fix while offline.
- CCL : add.label update.LAP.PV3 for updating since no observing tonight (shutter did not work)
neb-host --state up --host ipp134 --note 'CCL: repair->up, for updating run' neb-host --state up --host ipp135 --note 'CCL: repair->up, for updating run' neb-host --state up --host ipp136 --note 'CCL: repair->up, for updating run' neb-host --state up --host ipp137 --note 'CCL: repair->up, for updating run' neb-host --state up --host ipp138 --note 'CCL: repair->up, for updating run' neb-host --state up --host ipp139 --note 'CCL: repair->up, for updating run'
Friday : 2019.03.29
- CCL 09:30: manually restart pantasks Processing#Stoppingandstartingthepantasksservers
- CCL 10:00: fix burntool 0 size file issue
* check burn.tbl neb-ls -p -l neb://ipp047.0/gpc1/20100320/o5275g0337o/o5275g0337o.ota01.burn.tbl -rw-rw-r-- 1 apache nebulous 0 Sep 17 2014 /data/ipp070.0/nebulous/ff/f2/5114551853.gpc1:20100320:o5275g0337o:o5275g0337o.ota01.burn.tbl * run fix program ipp_apply_burntool_fix.pl --exp_name o5275g0337o --class_id XY01 --verbose --dbname gpc1 * check again neb-ls -p -l gpc1/20100320/o5275g0337o/o5275g0337o.ota01.burn.tbl -rw-rw-r-- 1 apache nebulous 14723 Mar 29 10:02 /data/ipp137.0/nebulous/ff/f2/11434470340.gpc1:20100320:o5275g0337o:o5275g0337o.ota01.burn.tbl neb-stat --validate gpc1/20100320/o5275g0337o/o5275g0337o.ota01.burn.tbl instance location: 1 a69525aef9121e0b0fbc8ec506c98e3e file:///data/ipp137.0/nebulous/ff/f2/11434470340.gpc1:20100320:o5275g0337o:o5275g0337o.ota01.burn.tbl 1 a69525aef9121e0b0fbc8ec506c98e3e file:///data/ippb23.0/nebulous/ff/f2/11434471371.gpc1:20100320:o5275g0337o:o5275g0337o.ota01.burn.tbl * run revert processed imfile with chiptool chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1 * There is usually unnecessary to set update.LAP.PV3 label to ps_ud_MOPS for burntool 0 size issue unless it requires reprocessing immediately. * same for others without verbose ipp_apply_burntool_fix.pl --exp_name o5629g0376o --class_id XY10 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5636g0580o --class_id XY22 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5634g0360o --class_id XY22 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5631g0175o --class_id XY22 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5631g0198o --class_id XY22 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5585g0614o --class_id XY22 --dbname gpc1 * then chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1
- CCL 15:40: set ippb04 & ippb05 from down to repair so that pantask can retrive the data for reprocessing.
o5329g0560o XY06 167027 1385535 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20150105 LAP.ThreePi 3 chiptool -revertprocessedimfile -fault 3 -label ps_ud_MOPS -dbname gpc1 o7036g0274o XY04 853551 1405323 update ps_ud_MOPS OSS.20150114 SweetSpot 3 chiptool -revertprocessedimfile -fault 3 -label ps_ud_MOPS -dbname gpc1 o6841g0142o XY31 764605 1022216 update ps_ud_MOPS ThreePi.20140703 ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1 * set them goto_cleaned and then set them update chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1385535 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1405323 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1022216 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1385535 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1405323 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1022216
Saturday : 2019.03.30
Sunday : 2019.03.31
Note:
See TracWiki
for help on using the wiki.
