== PS1 IPP Czar Logs for the week 2019.05.10 - 2019.05.16 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Friday : 2019.05.10 ===
'''CZAR HANDOVER''' by CCL

 * Haydn mentioned he will check them on 5/13 (remind him maybe)
 - ipp128: Device: /dev/bus/0 [megaraid_disk_15], Read SMART Self-Test Log Failed
 - ipp139: Device: /dev/bus/0 [megaraid_disk_37], Read SMART Self-Test Log Failed
 - ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
 - ippc129, ippc135, ippc145, ippc149, ippc150: drive issue

 - update.LAP.PV3: 15k warp prcessing jobs on ippmonitor, some of them might have been processed to be "full" state by queue their data_state in the warpSkyfile table.  Their data quality is not equal to 0 which may affect the state keep "update".
 - QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).
 - these huge jobs may cause ipp restart unsuccessfully (stdscience happened twice), make sure all server are running by check_system.sh

 * TdB: Taking over as czar. Following the weekly IPP meeting, it was mentioned having too many jobs queued for updates might lead to issues in communication between the servers and the pantasks. Potentially, this is affecting the stdscience restart that Roger mentioned. To assess the situation, I removed 10k warp updates from current processing to a hold label, which will be fed back in as processing continues: 
{{{
mysql -hX -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.LAP.PV3" and warpRun.state like "update" group by warp_id limit 10000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3.lazy -warp_id "$2 }' | tcsh
}}}
 I took the opportunity to restart the stdscience pantask after clearing them as well. Restart went fine, and check_system shows it is running fine as well.

=== Saturday : 2019.05.11 ===

=== Sunday : 2019.05.12 ===

=== Monday : 2019.05.13 ===
  
=== Tuesday : 2019.05.14 ===

=== Wednesday : 2019.05.15 ===

=== Thursday : 2019.05.16 ===