== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : 2016-04-11 === * 15:00 CZW: Processing was slow last night, but none of the main tasks seemed to be failing at an excessive rate. There are more timeouts in the pztool -pendingexp task, so perhaps that was the limiting step. I've rebuilt the ipp user's ippTasks and ippTools to pick up the r39527 changes I've made to this task/tool. There is now a -dateobs_begin/end argument to restrict the dates considered. The task uses a 30-day look back to set dateobs_begin, which was the agreed upon length on the email chain from last week. A quick test suggests a factor of four speed improvement with this restriction. I plan on relaunching the ipp pantasks at 16:00 today, at which point this change will be active. * 15:14 CZW: Saw that there were some daytime exposures being downloaded by summitcopy, so I restarted that to get the change early to check for problems before tonight. Seems to work fine. * 16:25 CZW: Cleanup took some time to finish up. Restarting ipp pantasks. === Tuesday : 2016.04.12 === * 14:22 MEH: revised the pantasks_hosts.input to use s5 again since Gene reported dvo calibration mostly finished, added s6 as well -- also updating the data targets -- will restart nightly pantasks to use these ~1600 * seeing one ipp008 off and others on... looks like some conflict in the hosts_ignore_storage group use.. -- if in the _ignore_ group it needs to be commented out of the active group... probably has been a problem for a while... * summitcopy/registration now are about @72 nodes -- probably excessive for summit but provides a sample of poor nodes to prune from (possibly/likely the set w/ rsyncs..) -- include a pruning of the rack set that likes to power cycle is down to 67 nodes * stdscience ~@560 after pruning the rack set that likes to power cycle (some of which already were supposed to be and actually weren't... -- ipp008,012,013,014,016,018,037) * 15:10 CZW: Started wave3 rsyncs on stare00. I'm starting fairly slowly, with 5x4x10MB transfers to get an idea of the speed and loading of this. * 19:12 MEH: ipp050 crash, nothing on console -- not responding to power cycle yet so neb-host down (from repair) before it messes up nightly processing.. * 20:13 MEH: appears ipp100-104 were fully setup to run jobs... missing /local/ipp dir etc.. * oddly so is ipp078 -- /local/ipp symlink to /export/ipp078.0/ipp -- why aren't things like this being checked... probably explains some faults in the past when ipp078 was in use before dvo use.. * 23:10 MEH: things running better tonight, only ~40 exposures behind @2230 when >70 the past few nights -- noticed another apache server had '''NOT''' been turned on to replace ippc05 being removed since down, ippc02 was fixed when disk/raid and power supply issue at end of 2015 so have uncommented that for ~ipp, ~ippqub/.tcshrc * ippc02 also originally hosted stdscience pantasks, probably should also be returned since seen ippc01 having a good amount of load at times... * wondering if there are other pantasks running not using the same apache for nebulous -- maybe should stop apache on those nodes to block any cases of that... === Wednesday : YYYY.MM.DD === === Thursday : YYYY.MM.DD === === Friday : YYYY.MM.DD === === Saturday : YYYY.MM.DD === === Sunday : YYYY.MM.DD ===