== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.09.15 ===
 * MEH: until the new consortium starts (10/2014?), all WS diffims to be turned off -- comment out label and turn off query to create in stdsci auto-startup, sending all to cleanup  
 * 13:40 MEH: Gene asked for pv3 stdlocal stack poll bump to 200 and adding in more data nodes for processing -- adding 4x ippsXX for ~60 for now since they are unused (easier to deal with when nightly starts)
  * ipp058 getting hammered when trying to clean up ~1TB of test processing -- putting into repair for now
  * same for ipp060, both have a nice chunk of space available again
 * 16:00 MEH: adding 1x s3 manually to lanl stdlocal -- notice many in ignore_s3 running, so the auto on/off script not properly turning off??
  * cleanup still running, if too much loading, will just take that much longer -- adding 1x s3 see clear increase in cleanup time for jobs
 * 20:15 MEH: ipp058 behaving badly.. console login up, but get errors -- may need to power cycle
{{{
Cannot execute /bin/tcsh: Too many open files in system

login: failure forking: Cannot allocate memory
}}}
  * finally cleared some after a bit and neb-host down -- many many [2014-09-15 20:27:07] VFS: file-max limit 4945516 reached
 * 20:45 MEH: unclear if gpc1 down for the night, but lanl stdlocal should be fine w/ ippsXX added at least
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===