PS1 IPP Czar Logs for the week 2017.03.13 - 2017.03.19

(Up to PS1 IPP Czar Logs)

Monday : 2017.03.13

• 21:00 EAM : we shutdown 1/3 of the cluster today to move to ITC. I set up the pantasks host to avoid machines in the POD. I also added in ipp121 and ipp078, which had been behaving themselves. Unfortunately, around 20:00 ipp121 had a kernel panic and started to have nfs problems. I rebooted ipp121 and took and ipp078 out of processing.
• 23:59 EAM : load on s3 & s5 was a bit heavy, while m1 & c2 were light. adjusted loading so all of those groups have 5 connections

Tuesday : 2017.03.14

• 17:00 CZW: I've set up a pantasks to run rsync jobs between the low-numbered storage volumes and the high-numbered storage volumes (ipp118/119/120 for now). These jobs should automatically stop at night (6:30PM), and restart in the morning (8AM), to ensure that nothing impacts nightly processing. It will likely need additional tuning, as the server takes a long time to load the input file, but it is at least running. Server directory /data/stare00.1/watersc1/itc_rsync_lists/rsync_run, running as ipptest.
• 19:30 EAM : stopping and restarting pantasks

Wednesday : 2017.03.15

• 14:20 CZW: I've chagned the Nebulous-Server max_used_space parameter to 0.989 on ippc70-ippc75, and restarted their apache servers. This change will allow the ITC nodes to be used at a higher fill factor than before, allowing processing to shift there.
• 16:20 CZW: With the shift of processing to ITC, I've disabled the trange limits on the rsync pantasks (/data/stare00.1/watersc1/itc_rsync_lists/rsync_run). This will allow jobs to run internal to the MRTCB over night, as the target hosts aren't used for nightly processing. I've also increased the loading slightly (3x instead of 1x, still running 10MB/s rsync jobs), and may move it up a bit more (5x max).
• 18:20 MEH: shutting down pantasks @MRTC-B and starting summitcopy+registration+stdscience @ITC --

Thursday : 2017.03.16

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD