PS1 IPP Czar Logs for the week 2019.04.22 - 2019.04.28

Czar of the week: TdB + EAM (czar-lord), previously CCL

(Up to PS1 IPP Czar Logs)

Monday : 2019.04.22

• CCL: these are full stage on chip, but still show up on red flags

      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182888
      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182889
      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182890
      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182891
      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182892
      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182893
      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182894
      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182895
      chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182896
  
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182888
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182889
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182890
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182891
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182892
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182893
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182894
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182895
      chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182896
  --> can't be updated due to ippb nodes are shutdown.
  

• CCL: most of them have burntool files on ippb node without copies on ITC nodes.

  SELECT rawExp.exp_name,chipProcessedImfile.class_id,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipRun.data_group,chipRun.dist_group,chipProcessedImfile.fault FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipProcessedImfile.fault != 0 AND chipRun.label like 'update.LAP.PV3' group by chip_id;
  +-------------+----------+--------+---------+--------+----------------+-------------------------------+-------------+-------+
  | exp_name    | class_id | exp_id | chip_id | state  | label          | data_group                    | dist_group  | fault |
  +-------------+----------+--------+---------+--------+----------------+-------------------------------+-------------+-------+
  | o5464g0207o | XY75     | 230361 | 1179692 | update | update.LAP.PV3 | LAP.PV3.20140730.20141024     | LAP.ThreePi |     2 |
  | o5444g0007o | XY76     | 219509 | 1179773 | update | update.LAP.PV3 | LAP.PV3.20140730.20141024     | LAP.ThreePi |     2 |
  | o5444g0024o | XY75     | 219526 | 1179774 | update | update.LAP.PV3 | LAP.PV3.20140730.20141024     | LAP.ThreePi |     2 |
  | o5444g0009o | XY75     | 219511 | 1181481 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5444g0026o | XY75     | 219528 | 1181482 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5027g0338o | XY75     |  87852 | 1181498 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5462g0101o | XY26     | 229028 | 1181500 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5343g0321o | XY73     | 173486 | 1181536 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5343g0336o | XY76     | 173502 | 1181539 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5487g0040o | XY76     | 242068 | 1181546 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5487g0052o | XY75     | 242080 | 1181547 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5343g0330o | XY74     | 173494 | 1181573 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5343g0333o | XY73     | 173498 | 1181605 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5343g0372o | XY76     | 173537 | 1181945 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5362g0293o | XY75     | 181552 | 1182008 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5390g0030o | XY75     | 191840 | 1182127 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5390g0067o | XY76     | 191878 | 1182376 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5390g0068o | XY74     | 191879 | 1182678 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5365g0588o | XY73     | 183722 | 1182737 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5365g0590o | XY12     | 183730 | 1182738 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5365g0609o | XY74     | 183764 | 1182742 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025     | LAP.ThreePi |     2 |
  | o5369g0248o | XY75     | 185907 | 1182788 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5369g0251o | XY27     | 185910 | 1182873 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5371g0256o | XY76     | 186536 | 1183057 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5371g0258o | XY75     | 186538 | 1183093 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5405g0451o | XY75     | 198470 | 1183305 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5405g0470o | XY74     | 198489 | 1183308 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5444g0013o | XY76     | 219515 | 1184003 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5444g0016o | XY76     | 219518 | 1184043 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5343g0353o | XY75     | 173517 | 1184369 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5343g0367o | XY74     | 173532 | 1184371 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5343g0369o | XY74     | 173534 | 1184372 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5404g0151o | XY75     | 197586 | 1184373 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5404g0172o | XY76     | 197607 | 1184374 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o6435g0293o | XY30     | 613989 | 1184406 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o6525g0133o | XY74     | 646008 | 1184407 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o6513g0165o | XY04     | 641174 | 1184566 | update | update.LAP.PV3 | LAP.PV3.20140730.ipp.20141026 | LAP.ThreePi |     2 |
  | o5343g0351o | XY47     | 173516 | 1184668 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  | o5344g0252o | XY75     | 174158 | 1184675 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026     | LAP.ThreePi |     2 |
  +-------------+----------+--------+---------+--------+----------------+-------------------------------+-------------+-------+
  

• MEH: adding QUB.2 to pstamp server for bulk QUB requests (similar to MOPS.2, large/long request not needed ASAP), ps_ud_QUB.2 label and updates done in ippqub:stdscience_ws pantask -- lots of faults, czar not need to worry about at this time and due to the ongoing issue of multiple copies of needed files only on the ippb nodes and none on the actual IPP-ITC cluster...

• TdB: Around 21:00 I noticed that data was no longer downloading from the summit. A quick look at the summitcopy pantask stdout log showed that MySQL was down, which was causing the problem. Gene checked the process list for mysql:

  ps aux | grep mys
  

  it was not listed, so it had crashed or had been shutdown. Looking at /var/log/mysql/mysqld.err showed the messages about the crash. Gene restarted MySQL on ippdb09 as follows:

  restarting MySQL:
  
  /etc/init.d/mysql start
  
  (it complains about the PID file being left behind, but tries to start anyway.  The start also takes too long, and the startup script gives up and claims the start fails.  NOTE this is not true: you need to check the process list (ps aux | grep mys) and check for mysql.  It will probably be there, it is just taking a long time to start up.  You can check the log file (tail -f /var/log/mysql/mysqld.err) and wait for it to say "ready for connections.").
  

  Once mysql was back up, nebulous started working again and downloads proceeded OK.

Also, I put ippb10-11 back to up to deal with the storage there. Temperature in the ARTC IT seems to be holding, as processing picked up.

Tuesday : 2019.04.23

Wednesday : 2019.04.24

Thursday : 2019.04.25

Friday : 2019.04.26

Saturday : 2019.04.27

Sunday : 2019.04.28