PS1 IPP Czar Logs for the week 2019.05.10 - 2019.05.16

(Up to PS1 IPP Czar Logs)

Friday : 2019.05.10

CZAR HANDOVER by CCL

• Haydn mentioned he will check them on 5/13 (remind him maybe)
• ipp128: Device: /dev/bus/0 [megaraid_disk_15], Read SMART Self-Test Log Failed
• ipp139: Device: /dev/bus/0 [megaraid_disk_37], Read SMART Self-Test Log Failed
• ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
• ippc129, ippc135, ippc145, ippc149, ippc150: drive issue

• update.LAP.PV3: 15k warp prcessing jobs on ippmonitor, some of them might have been processed to be "full" state by queue their data_state in the warpSkyfile table. Their data quality is not equal to 0 which may affect the state keep "update".
• QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).
• these huge jobs may cause ipp restart unsuccessfully (stdscience happened twice), make sure all server are running by check_system.sh

• TdB: Taking over as czar. Following the weekly IPP meeting, it was mentioned having too many jobs queued for updates might lead to issues in communication between the servers and the pantasks. Potentially, this is affecting the stdscience restart that Roger mentioned. To assess the situation, I removed 10k warp updates from current processing to a hold label, which will be fed back in as processing continues:

  mysql -hX -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.LAP.PV3" and warpRun.state like "update" group by warp_id limit 10000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3.lazy -warp_id "$2 }' | tcsh
  

  I took the opportunity to restart the stdscience pantask after clearing them as well. Restart went fine, and check_system shows it is running fine as well.

• TdB: Around 18:00 we hit a situation where chips were still updating but warps were not. I did a bit of digging:

  mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "update" group by chip_id limit 500; '
  

There are a lot of exps that have corresponding warps in the hold label, so the ones currently in warp update are waiting on their chip updates to finish. Send some of the warps in the hold label back to the regular label if their corresponding chip state is already in full, they can run through:

mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "full" and warpRun.label LIKE "update.LAP.PV3.lazy" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$2 }' | tcsh

Saturday : 2019.05.11

Sunday : 2019.05.12

Monday : 2019.05.13

Tuesday : 2019.05.14

Wednesday : 2019.05.15

Thursday : 2019.05.16