PS1 IPP Czar Logs for the week 2019.04.01 - 2019.04.07

(Up to PS1 IPP Czar Logs)

Czar of the week: TdB, with MEH as czar-lord (previously: CCL + EAM)

Monday : 2019.04.01

• EAM 09:00: powered-off ipp001, ipp003, ipp022
• CCL 09:20: restart stdscience manually
• CCL: not fixed chip or warp ids, they are related to stsci servers, mask files, psf files, or memory block, etc.

  * chip
  mysql -hippdb08 -uippuser -p gpc1 -e "SELECT chipRun.chip_id,chipRun.state,chipRun.label,chipRun.data_group,chipRun.dist_group FROM chipRun WHERE chipRun.label like 'update.LAP.PV3.broke'"
  
  * warp
  mysql -hippdb08 -uippuser -pippuser gpc1 -e \
  "SELECT chipRun.chip_id,camRun.cam_id,fakeRun.fake_id,warpRun.warp_id,warpRun.state,warpRun.label,warpRun.data_group \
  FROM warpRun, fakeRun, camRun, chipRun \
  WHERE warpRun.fake_id = fakeRun.fake_id \
  AND fakeRun.cam_id = camRun.cam_id \
  AND camRun.chip_id = chipRun.chip_id \
  AND warpRun.label LIKE 'ps_ud_MOPS' \
  AND warpRun.warp_id IN (1027567 , 1027638 , 1027645 , 1027695 , 1027697 , 1027705 , 1027711 , 1027781 , 1027785 , 1027828 , 1050735 , 1142465 , 1281048 , 1284676 , 1284689 , 1391868 , 1392030 , 1392058 , 1392250 , 1413405 , 1413407 , 1414496 , 1414848 , 1848779)"
  

• EAM 11:00 : Gavin has switch ippops1 & ippops2 to use ippops3 for homedirs and ippc18 for NIS

• TdB: around lunch time, Haydn asked for ipp091 and ipp127 to be put to down, in order for him to look at it, following a bad BBU and subsequent high load. They were both put to neb-host down and ipp091 was subsequently put back to up after repairs. During his look at ipp127, Haydn found a blown fuse on the PDU. He will look into the full rack tomorrow, and as a precaution we put all the nodes in that rack to repair, to prevent this ngihts data from going there and not being available to MOPS tomorrow.

• TdB: John, Roger, Mark and myself met to look at some of the errors on IPPmonitor. There were some errors left over from the weekend, in particular errors due to missing weight files on Sunday. We sent the chip_IDs for those exposures to cleanup and then set their imfiles back to update, followed by skyfiles back to update:

  chiptool -dbname gpc1 -updaterun -chip_id XXXXX -set_state goto_cleaned -set_label goto_cleaned
  chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXX
  warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id XXXXX
  

  That solved the issue. Unfortunately, some exposures were faulting due to ipp127 not being available yet, at the time. I will follow-up more on these, once the rack is back up.

Tuesday : 2019.04.02

Wednesday : 2019.04.03

Thursday : 2019.04.04

Friday : 2019.04.05

Saturday : 2019.04.06

Sunday : 2019.04.07