PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.09.15

• MEH: until the new consortium starts (10/2014?), all WS diffims to be turned off -- comment out label and turn off query to create in stdsci auto-startup, sending all to cleanup
• 13:40 MEH: Gene asked for pv3 stdlocal stack poll bump to 200 and adding in more data nodes for processing -- adding 4x ippsXX for ~60 for now since they are unused (easier to deal with when nightly starts)
  • ipp058 getting hammered when trying to clean up ~1TB of test processing -- putting into repair for now
  • same for ipp060, both have a nice chunk of space available again
• 16:00 MEH: adding 1x s3 manually to lanl stdlocal -- notice many in ignore_s3 running, so the auto on/off script not properly turning off??
  • cleanup still running, if too much loading, will just take that much longer -- adding 1x s3 see clear increase in cleanup time for jobs
• 20:15 MEH: ipp058 behaving badly.. console login up, but get errors -- may need to power cycle

  Cannot execute /bin/tcsh: Too many open files in system
  
  login: failure forking: Cannot allocate memory
  

  • finally cleared some after a bit and neb-host down -- many many [2014-09-15 20:27:07] VFS: file-max limit 4945516 reached
• 20:45 MEH: unclear if gpc1 down for the night, but lanl stdlocal should be fine w/ ippsXX added at least
• 21:30 MEH: nightly started, ippsXX off in lanl stdlocal, ipp058 heavy load so into repair until morning
• 22:00 MEH: seeing file issue faults in warp stage for ipp060 -- also repair until morning -- may not be robust enough with so few data nodes available

  Unable to perform warp.
   -> psFitsOpen (psFits.c:217): I/O error
       Failed to delete a previously-existing file (/data/ipp060.0/nebulous/e4/40/5106938020.gpc1:OSS.nt:2014:09:16:o6916g0050o.795035:o6916g0050o.795035.wrp.1030977.skycell.1140.098.mask.fits), error 2: No such file or directory
   -> pmFPAfileOpen (pmFPAfileIO.c:816): I/O error
       error opening file /data/ipp060.0/nebulous/e4/40/5106938020.gpc1:OSS.nt:2014:09:16:o6916g0050o.795035:o6916g0050o.795035.wrp.1030977.skycell.1140.098.mask.fits
   -> pmFPAfileWrite (pmFPAfileIO.c:415): I/O error
       failed to open /data/ipp060.0/nebulous/e4/40/5106938020.gpc1:OSS.nt:2014:09:16:o6916g0050o.795035:o6916g0050o.795035.wrp.1030977.skycell.1140.098.mask.fits (PSWARP.OUTPUT.MASK)
   -> pmFPAfileIOChecks (pmFPAfileIO.c:91): I/O error
       failed WRITE in FPA_AFTER block for PSWARP.OUTPUT.MASK
   -> pswarpCleanup (pswarpCleanup.c:84): I/O error
       Unable to write files.
  

  • also seeing warp stage log having issue log warning with new keyword (Unable to find (null) --- assuming UTC), will need to clean up
• 22:10 MEH: stdsci having timeout flurry for chiptool.. ippdb01 has load ~13

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD