(Up to PS1 IPP Czar Logs)

Monday : 2012.08.06

Mark is czar

• 07:40 MEH: nightly science slowly continues to download/process, ~100 exposures behind. restarted summitcopy to see if that helps.
  • ipp027.0 continues to decrease, 100% disk used for ~140GB remaining.
  • reports that 'IPP datastore is down, can't connect to backend MySQL server', i'm getting proxy error. can't find any notes on the proxy setup other than now on ippops1. /data/ippc17.0 was full, cleaned out /local/ipp/tmp and mysql happy again. pstamp pantasks having trouble now.
• 08:55 (Serge): pubtool -revert -dbname gpc1 -fault 2 -label ThreePi.nightlyscience after ippc17 full disk incident
• 09:10 (Serge): Killed rogue jobs and restarted publishing
• 09:30 (Serge): Fixed LAP fault 2: gpc1/20100530/o5346g0039o/o5346g0039o.ota47.burn.tbl, gpc1/20100624/o5371g0063o/o5371g0063o.ota64.fits, gpc1/20100529/o5345g0048o/o5345g0048o.ota26.fits
• 10:20 MEH: 12-18hr restarting of stdscience. seems to have picked up the remaining 3 3PI chips..
• 10:30 rebooting ippc17 to try and possibly clear some weird behavior. pstamp and datastore back online. oddly, looks like publishing pushed out some results to datastore with 8/5 date.
• 11:00 nightly science finished. Chris fixed LAP issue and full runs continuing.
• 11:10 (Serge): ippc63 is 68965 behind its master
• 14:20: MEH: odd behavior, stdscience mostly stopped suspect due to book filling up with CRASH entries from ippc29,ippc61 with a very odd error. restarted and slowly put in ippc29,c61 and seems to be fine again. stack was also having CRASH with ippc29. (meeting discussion suggestes possibility stale NFS to ippc18 psLib may have been the problem from rebuild of tag today)

  --> pantasks.stdout.log.20120806.142251 
  crash for: warp_skycell.pl --threads @MAX_THREADS@ --warp_id 489012 --warp_skyfile_id 40697774 --skycell_id skycell.1456.049 --tess_dir RINGS.V3 --camera GPC1 --outroot neb://stsci01.2/gpc1/LAP.ThreePi.20120706/2012/08/04/o5566g0752o.273732/o5566g0752o.273732.wrp.489012.skycell.1456.049 --redirect-output --run-state new  --reduction LAP_SCIENCE --dbname gpc1 --verbose
  job exit status: CRASH
  job host: ippc61
  job dtime: 0.104959
  job exit date: Mon Aug  6 14:19:14 2012
  hostname: ippc61
  
  --> pantasks.stderr.log.20120806.142251
   -> addprocessedimfileMode (chiptool.c:539): unknown psLib error
       -exp_id is required
  
  
  --> ippc29 also in stack -- nothing in stderr log though
  crash for: stack_skycell.pl --threads @MAX_THREADS@ --stack_id 1120678 --outroot neb://any/gpc1/LAP.ThreePi.20120706/2012/08/06/RINGS.V3/skycell.1276.094/RINGS.V3.skycell.1276.094.stk.1120678 --redirect-output --run-state new --reduction THREEPI_STACK --dbname gpc1 --verbose
  job exit status: CRASH
  job host: ippc29
  job dtime: 0.114992
  job exit date: Mon Aug  6 14:13:14 2012
  hostname: ippc29
  

• 16:50 MEH: shutdown and restarted all pantasks.

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD