PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.09.15

• MEH: until the new consortium starts (10/2014?), all WS diffims to be turned off -- comment out label and turn off query to create in stdsci auto-startup, sending all to cleanup
• 13:40 MEH: Gene asked for pv3 stdlocal stack poll bump to 200 and adding in more data nodes for processing -- adding 4x ippsXX for ~60 for now since they are unused (easier to deal with when nightly starts)
  • ipp058 getting hammered when trying to clean up ~1TB of test processing -- putting into repair for now
  • same for ipp060, both have a nice chunk of space available again
• 16:00 MEH: adding 1x s3 manually to lanl stdlocal -- notice many in ignore_s3 running, so the auto on/off script not properly turning off??
  • cleanup still running, if too much loading, will just take that much longer -- adding 1x s3 see clear increase in cleanup time for jobs
• 20:15 MEH: ipp058 behaving badly.. console login up, but get errors -- may need to power cycle

  Cannot execute /bin/tcsh: Too many open files in system
  
  login: failure forking: Cannot allocate memory
  

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD