PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2016-04-11

• 15:00 CZW: Processing was slow last night, but none of the main tasks seemed to be failing at an excessive rate. There are more timeouts in the pztool -pendingexp task, so perhaps that was the limiting step. I've rebuilt the ipp user's ippTasks and ippTools to pick up the r39527 changes I've made to this task/tool. There is now a -dateobs_begin/end argument to restrict the dates considered. The task uses a 30-day look back to set dateobs_begin, which was the agreed upon length on the email chain from last week. A quick test suggests a factor of four speed improvement with this restriction. I plan on relaunching the ipp pantasks at 16:00 today, at which point this change will be active.
• 15:14 CZW: Saw that there were some daytime exposures being downloaded by summitcopy, so I restarted that to get the change early to check for problems before tonight. Seems to work fine.
• 16:25 CZW: Cleanup took some time to finish up. Restarting ipp pantasks.

Tuesday : 2016.04.12

• 14:22 MEH: revised the pantasks_hosts.input to use s5 again since Gene reported dvo calibration mostly finished, added s6 as well -- also updating the data targets -- will restart nightly pantasks to use these ~1600
  • seeing one ipp008 off and others on... looks like some conflict in the hosts_ignore_storage group use.. -- if in the _ignore_ group it needs to be commented out of the active group... probably has been a problem for a while...
  • summitcopy/registration now are about @72 nodes -- probably excessive for summit but provides a sample of poor nodes to prune from (possibly/likely the set w/ rsyncs..) -- include a pruning of the rack set that likes to power cycle is down to 67 nodes
  • stdscience ~@560 after pruning the rack set that likes to power cycle (some of which already were supposed to be and actually weren't... -- ipp008,012,013,014,016,018,037)
• 15:10 CZW: Started wave3 rsyncs on stare00. I'm starting fairly slowly, with 5x4x10MB transfers to get an idea of the speed and loading of this.
• 19:12 MEH: ipp050 crash, nothing on console -- not responding to power cycle yet so neb-host down (from repair) before it messes up nightly processing..
• 20:13 MEH: appears ipp100-104 were NOT fully setup to run pantasks jobs... missing /local/ipp link etc..
  • oddly so is ipp078 -- /local/ipp symlink to /export/ipp078.0/ipp -- why aren't things like this being checked... probably explains some faults in the past when ipp078 was in use before dvo use...
• 23:10 MEH: things running better tonight, only ~40 exposures behind @2230 when >70 the past few nights (so similar to earlier last week and still not back to ~60 exposures/hr as normal several weeks ago) -- noticed another apache server had NOT been turned on to replace ippc05 being removed since disk/raid issues, ippc02 was fixed when disk/raid and power supply issue at end of 2015 so have uncommented that for ~ipp, ~ippqub/.tcshrc
  • ippc02 also originally hosted stdscience pantasks, probably should also be returned since seen ippc01 having a good amount of load at times...
  • wondering if there are other pantasks running not using the same apache for nebulous -- maybe should stop apache on those nodes to block any cases of that... -- yep... ~ipptest and ~ipplanl were both out of alignment with ~ipp/ippqub using ippc02 rather than ippc05, so using ippc02 again helps those use the proper same apache servers..

Wednesday : 2016-04-13

• 12:10 CZW: Started additional rsync processes on ipp033 to clear that host faster. It's already ignored in pantasks, so this shouldn't cause processing issues.
• 17:58 HAF: daily restart of pantasks.

  summary of haydn work (from various emails):
  I've got ipp048 back online. Please exercise it vigorously if possible.
  I'll have ipp050 back online with one less CPU soon.
  I'd like to power cycle ippc05 to swap a disk in it, and then it might work again.
  ippc05 is back online, and will be rebuilding its RAID for the next hour or so.
  ipp085 is back online with a different RAID battery, which I tested earlier this week.
  
  

Thursday : 2016.04.14

• 20:00 EAM : stopping pantasks for restart

Friday : YYYY.MM.DD

Saturday : 2016.04.16

• 19:55 EAM : stopping pantasks for restart

Sunday : YYYY.MM.DD