PS1 IPP Czar Logs for the week 2017.03.13 - 2017.03.19

(Up to PS1 IPP Czar Logs)

Monday : 2017.03.13

• 21:00 EAM : we shutdown 1/3 of the cluster today to move to ITC. I set up the pantasks host to avoid machines in the POD. I also added in ipp121 and ipp078, which had been behaving themselves. Unfortunately, around 20:00 ipp121 had a kernel panic and started to have nfs problems. I rebooted ipp121 and took and ipp078 out of processing.
• 23:59 EAM : load on s3 & s5 was a bit heavy, while m1 & c2 were light. adjusted loading so all of those groups have 5 connections

Tuesday : 2017.03.14

• 17:00 CZW: I've set up a pantasks to run rsync jobs between the low-numbered storage volumes and the high-numbered storage volumes (ipp118/119/120 for now). These jobs should automatically stop at night (6:30PM), and restart in the morning (8AM), to ensure that nothing impacts nightly processing. It will likely need additional tuning, as the server takes a long time to load the input file, but it is at least running. Server directory /data/stare00.1/watersc1/itc_rsync_lists/rsync_run, running as ipptest.
• 19:30 EAM : stopping and restarting pantasks

Wednesday : 2017.03.15

• 14:20 CZW: I've chagned the Nebulous-Server max_used_space parameter to 0.989 on ippc70-ippc75, and restarted their apache servers. This change will allow the ITC nodes to be used at a higher fill factor than before, allowing processing to shift there.
• 16:20 CZW: With the shift of processing to ITC, I've disabled the trange limits on the rsync pantasks (/data/stare00.1/watersc1/itc_rsync_lists/rsync_run). This will allow jobs to run internal to the MRTCB over night, as the target hosts aren't used for nightly processing.

Thursday : 2017.03.16

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD