IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:PS1_IPP_Czarlog_20190325

Version 12 (modified by cclin33, 7 years ago) ( diff )

--

PS1 IPP Czar Logs for the week 2019.03.25 - 2019.03.31

    1. PS1 IPP Czar Logs for the week 2019.03.25 - 2019.03.31
      1. Monday : 2019.03.25
      2. Tuesday : 2019.03.26
      3. Wednesday : 2019.03.27
      4. Thursday : 2019.03.28
      5. Friday : 2019.03.29
      6. Saturday : 2019.03.30
      7. Sunday : 2019.03.31

(Up to PS1 IPP Czar Logs)

Czar of the week: CCL, with EAM as czar-lord (previously: JRF + MEH)

Monday : 2019.03.25

  • MEH: ganglia again missing many nodes reporting -- Gavin cleared and notes to restart from scratch/defunct state order 
    ippc18: restart gmond -- sudo /etc/init.d/gmond restart
ippc19: restart gmond
all other nodes: restart gmond (ganglia-monitor on ubuntu/systemd -- sudo systemctl status ganglia-monitor.service)

restart gmetad on ippops2 when page not running or items not displaying or multiple copies displaying on web page -- sudo /etc/init.d/gmetad restart

LAP.PV3 faulted: by JRF

The following are broken: 

o5280g0301o XY01    151113  1059611 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2 
o5280g0317o XY01    151129  1059616 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2 
o5280g0316o XY01    151128  1059817 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2 
o6091g0094o XY22    498248  1060186 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2 
o5582g0644o XY44    283730  1062952 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5582g0647o XY45    283733  1062954 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5582g0665o XY44    283751  1062956 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5582g0649o XY45    283735  1062992 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5719g0065o XY73    344295  1063341 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5282g0287o XY01    152148  1063529 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2

All of these are complaining about some missing mdc files, so it fails with a poor burntool state. I will try to reprocess them: 

chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name XXXXXX

Check they are done: 

SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5280g0301o","o5280g0317o","o5280g0316o","o6091g0094o","o5582g0644o","o5582g0647o","o5582g0665o","o5582g0649o","o5719g0065o","o5282g0287o");

They are not done, and they contiue to fail... However, two of them made it through! o6091g0094o o5719g0065o 

SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o6091g0094o","o5719g0065o");

These two had slightly different errors. They did not complain about the burntool, so reprocessing worked for them.

Instead, they are complaining about a missing psf file. Take a look at the missing file using: 

neb-stat -validate

we can see that is does not exist: 

instance location:
      0                     NON-EXISTANT file:///data/ipp120.1/nebulous/32/5c/9229835926.gpc1:LAP.PV3.20140730:2014:09:01:o6091g0094o.498248:o6091g0094o.498248.ch.1060186.XY22.psf

Indeed, doing 'ls' on the file location tells us that it does not exist.

The best way to fix this is to move that neb location to bad. 

neb-mv neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf.bad

Then we can reprocess the file and it should hopefully make a new psf file: 

    chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1060186

burntool issues by JRF

The remaining issues complain about the burntool state. We have little information, it just says that: 

    burntool state vs burntoolStateGood : 0 vs 14
Image burntool version does not match current accepted version. at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834
    main::my_die('Image burntool version does not match current accepted version.', 151113, 2131394, 'XY01', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 328

Check out the uri: 

neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl

Note that the neb file it the same as the fits file in the error log, but with .burn.tbl (I didn't include the fits part above)

We find that this burn.tbl file has 0 size, and it's located on an available node ipp070.0

Check out: https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20180521 for info on how to fix this:

Ok, so we are going to fix this as ippitc, as it's probably safer. Run the burntool fix: 

ipp_apply_burntool_fix.pl --exp_name o5280g0301o --class_id XY01 --verbose --dbname gpc1

Make sure to give it the correct class_id.

It will spew out some useful information on what's happening e.g. that it has removed the old stuff, and made some new ones. It's best to check that it created some new burn tables, which can be done by looking up the table again: 

    neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl

Running this we see two new copies, in new places, they are the same size, and that size is non-zero!

Repeat for the remaining: 

    ipp_apply_burntool_fix.pl --exp_name o5280g0316o --class_id XY01 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0644o --class_id XY44 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0647o --class_id XY45 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0665o --class_id XY44 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0649o --class_id XY45 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5719g0065o --class_id XY73 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5282g0287o --class_id XY01 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0608o --class_id XY44 --verbose --dbname gpc1

Missing PSF files by JRF

This was originally solved by some reprocessing (see the two that worked above). However, the correct way to fix it is as follows:

The errors complain about a config file, psf file, and burn tool. So look at all of them: 

    $> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.ppImage.mdc
    no instances

    $> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.psf
    no instances

    $> neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
    object id:             3915331061
    key:                   gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
    read lock:             0
    write lock:            
    epoch:                 2014-08-31 18:02:51
    mtime:                 0000-00-00 00:00:00
    md5sum count:          1
    requested instances:   1
    available instances:   1
    existing instances:    1
    total instances:       1
    instances location:  
      1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp069.0/nebulous/b3/3f/5048738853.gpc1:20100325:o5280g0317o:o5280g0317o.ota01.burn.tbl

No instances for the first two, and the burn.tbl has the magic 0 size MD5 sum d41d8cd98f00b204e9800998ecf8427e

So, first off let's make new burn.tbls and then revert: 

    ipp_apply_burntool_fix.pl --exp_name o5280g0317o --class_id XY01 --verbose --dbname gpc1
    neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
    chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1059616

MOPS No Masks by JRF

MOPS have requested some exposures to be reprocessed with no mask: 

chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0145o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0126o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0107o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0088o

Check they are done: 

SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o8347g0145o","o8347g0126o","o8347g0107o","o8347g0088o");

They're done

Prep for nightly by JRF

Have created a config for targeting: 

    ipphosts.mhpcc.config.190325.ready_for_nightly_and_more

which contains the config from earlier in the month before we were targeting some new nodes for updates. It also includes 4 of the new nodes, ipp127-ipp130, as these seem ok to be used (the ones after this cannot be accessed from the console yet).

CCL: Some pending jobs on pstamp were sent in title of [ps-ipp-ops] Cron <ippitc@ippc30> /bin/bash /data/ippc64.1/ippitc/check_pstamp_stalled.bat Following previous steps Czar Logs 2019-03-18 to solve the problem. 

email contents:
count(*) label    req_id            name            timestamp fault        exp_id    dtime            utc_timestamp()
176       MOPS    919001  weryk_5C994224  2019-03-25 21:10:32     0       1330887 02:15:39        2019-03-25 23:20:01

Under terminal, check the warp id with “pstamptool”, warp id is the 4th column.
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
10961873 new warp 1018709 skycell.0925.068 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961873.log 500
10961874 new warp 1018709 skycell.0925.079 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961874.log 500
10961875 new warp 1018709 skycell.0925.069 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961875.log 500
10961876 new warp 1018709 skycell.0925.078 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961876.log 500
10961880 new warp 1018708 skycell.0925.068 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961880.log 500
10961881 new warp 1018708 skycell.0925.079 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961881.log 500
10961882 new warp 1018708 skycell.0925.069 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961882.log 500
10961883 new warp 1018708 skycell.0925.078 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961883.log 500

Under mysql, check their labels.
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708);
+-------------+--------+---------+----------------+--------+---------+----------------+--------+
| exp_name    | exp_id | chip_id | chipLabel      | state  | warp_id | warpLabel      | state  |
+-------------+--------+---------+----------------+--------+---------+----------------+--------+
| o6864g0069o | 770608 | 1070075 | update.LAP.PV3 | update | 1018708 | update.LAP.PV3 | update |
| o6864g0070o | 770607 | 1070076 | update.LAP.PV3 | update | 1018709 | update.LAP.PV3 | update |
+-------------+--------+---------+----------------+--------+---------+----------------+--------+
2 rows in set (0.00 sec)

Under terminal, set their label to ps_ud_MOPS (right?)
chiptool -dbname gpc1 -updaterun -chip_id 1070075 -set_label ps_ud_MOPS
chiptool -dbname gpc1 -updaterun -chip_id 1070076 -set_label ps_ud_MOPS
warptool -dbname gpc1 -updaterun -warp_id 1018708 -set_label ps_ud_MOPS
warptool -dbname gpc1 -updaterun -warp_id 1018709 -set_label ps_ud_MOPS

Under mysql, check their label
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708);
+-------------+--------+---------+------------+--------+---------+------------+--------+
| exp_name    | exp_id | chip_id | chipLabel  | state  | warp_id | warpLabel  | state  |
+-------------+--------+---------+------------+--------+---------+------------+--------+
| o6864g0069o | 770608 | 1070075 | ps_ud_MOPS | update | 1018708 | ps_ud_MOPS | update |
| o6864g0070o | 770607 | 1070076 | ps_ud_MOPS | update | 1018709 | ps_ud_MOPS | update |
+-------------+--------+---------+------------+--------+---------+------------+--------+
2 rows in set (0.00 sec)

Few minutes later, the state changed to full and problem solved!
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708);
+-------------+--------+---------+------------+-------+---------+------------+-------+
| exp_name    | exp_id | chip_id | chipLabel  | state | warp_id | warpLabel  | state |
+-------------+--------+---------+------------+-------+---------+------------+-------+
| o6864g0069o | 770608 | 1070075 | ps_ud_MOPS | full  | 1018708 | ps_ud_MOPS | full  |
| o6864g0070o | 770607 | 1070076 | ps_ud_MOPS | full  | 1018709 | ps_ud_MOPS | full  |
+-------------+--------+---------+------------+-------+---------+------------+-------+
2 rows in set (0.01 sec)

CCL: put ipp134-ipp139 repair, not ready for new data yet. 

neb-host --state repair --host ipp134 --note 'CCL: up --> repair, not for new data yet'
neb-host --state repair --host ipp135 --note 'CCL: up --> repair, not for new data yet'
neb-host --state repair --host ipp136 --note 'CCL: up --> repair, not for new data yet'
neb-host --state repair --host ipp137 --note 'CCL: up --> repair, not for new data yet'
neb-host --state repair --host ipp138 --note 'CCL: up --> repair, not for new data yet'
neb-host --state repair --host ipp139 --note 'CCL: up --> repair, not for new data yet'

Tuesday : 2019.03.26

JRF: Don't forget about these that I couldn't get to update again from last Sunday: 

    SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipRun.chip_id IN (1047387,1047394,1047433,1047440,1047444,1047445,1047446,1047451,1047462,1047465,1047468,1047470,1047471,1047472,1047473,1047474,1047475);

CCL: pstamp stalled last night 

2       MOPS    919024  weryk_5C997837  2019-03-26 01:12:03     26      686123  00:12:59        2019-03-26 01:25:02

mysql> SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1771288, 1771274, 1771255, 1771238, 1769120, 1769119, 1769102, 1769100, 1769087, 1769082, 1769068, 1769066, 1768305, 1768287, 1768269, 1768250, 1763443, 1763422, 1763401, 1763379);

warp and chip labels were changed from  update.hold to ps_ud_MOPS

NOTE : ippc71 and ippc74 have DISK WARNING - free space: / 19316 MB (20% inode=95%):

Both log files /var/log/apache2/access_log have grown to 13G

Will wait for the logrotate to take care of these files tonight,

if the problems still show up then try the short-term solution by EAM: 

1) stop the pantasks 
2) stop the apache server [sudo /etc/init.d/apache2 stop]
3) move the file to a new name 
4) start the apache server [sudo /etc/init.d/apache2 start]
5) compress the saved file
(the apache restart is needed because even if you compress or remove the log file, if apache is still running the file system will keep the data around).
  • MEH: need to setup ipp134-139 neb-host up during day/every day for large update processing 
    neb-host ipp134 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here"
neb-host ipp135 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here"
neb-host ipp136 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here"
neb-host ipp137 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here"
neb-host ipp138 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here"
neb-host ipp139 up --note "MEH: repair->up during day after nightly processing finished so massive update load goes here"

CCL 17:43: set ipp134-ipp139 repair

Wednesday : 2019.03.27

Thursday : 2019.03.28

Friday : 2019.03.29

Saturday : 2019.03.30

Sunday : 2019.03.31

Note: See TracWiki for help on using the wiki.