(Up to PS1 IPP Czar Logs)

Monday : 2012.08.06

Mark is czar

• 07:40 MEH: nightly science slowly continues to download/process, ~100 exposures behind. restarted summitcopy to see if that helps.
  • ipp027.0 continues to decrease, 100% disk used for ~140GB remaining.
  • reports that 'IPP datastore is down, can't connect to backend MySQL server', i'm getting proxy error. can't find any notes on the proxy setup other than now on ippops1. /data/ippc17.0 was full, cleaned out /local/ipp/tmp and mysql happy again. pstamp pantasks having trouble now.
• 08:55 (Serge): pubtool -revert -dbname gpc1 -fault 2 -label ThreePi.nightlyscience after ippc17 full disk incident
• 09:10 (Serge): Killed rogue jobs and restarted publishing
• 09:30 (Serge): Fixed LAP fault 2: gpc1/20100530/o5346g0039o/o5346g0039o.ota47.burn.tbl, gpc1/20100624/o5371g0063o/o5371g0063o.ota64.fits, gpc1/20100529/o5345g0048o/o5345g0048o.ota26.fits
• 10:20 MEH: 12-18hr restarting of stdscience. seems to have picked up the remaining 3 3PI chips..
• 10:30 rebooting ippc17 to try and possibly clear some weird behavior. pstamp and datastore back online. oddly, looks like publishing pushed out some results to datastore with 8/5 date.
• 11:00 nightly science finished. Chris fixed LAP issue and full runs continuing.
• 11:10 (Serge): ippc63 is 68965 behind its master
• 14:20: MEH: odd behavior, stdscience mostly stopped suspect due to book filling up with CRASH entries from ippc29,ippc61 with a very odd error. restarted and slowly put in ippc29,c61 and seems to be fine again. stack was also having CRASH with ippc29. (meeting discussion suggestes possibility stale NFS to ippc18 psLib may have been the problem from rebuild of tag today)

  --> pantasks.stdout.log.20120806.142251 
  crash for: warp_skycell.pl --threads @MAX_THREADS@ --warp_id 489012 --warp_skyfile_id 40697774 --skycell_id skycell.1456.049 --tess_dir RINGS.V3 --camera GPC1 --outroot neb://stsci01.2/gpc1/LAP.ThreePi.20120706/2012/08/04/o5566g0752o.273732/o5566g0752o.273732.wrp.489012.skycell.1456.049 --redirect-output --run-state new  --reduction LAP_SCIENCE --dbname gpc1 --verbose
  job exit status: CRASH
  job host: ippc61
  job dtime: 0.104959
  job exit date: Mon Aug  6 14:19:14 2012
  hostname: ippc61
  
  --> pantasks.stderr.log.20120806.142251
   -> addprocessedimfileMode (chiptool.c:539): unknown psLib error
       -exp_id is required
  
  
  --> ippc29 also in stack -- nothing in stderr log though
  crash for: stack_skycell.pl --threads @MAX_THREADS@ --stack_id 1120678 --outroot neb://any/gpc1/LAP.ThreePi.20120706/2012/08/06/RINGS.V3/skycell.1276.094/RINGS.V3.skycell.1276.094.stk.1120678 --redirect-output --run-state new --reduction THREEPI_STACK --dbname gpc1 --verbose
  job exit status: CRASH
  job host: ippc29
  job dtime: 0.114992
  job exit date: Mon Aug  6 14:13:14 2012
  hostname: ippc29
  

• 16:50 MEH: shutdown and restarted all pantasks.
• 20:20 and the fun continues. ipp055 raid is degraded, setting to repair.. failed.. neb-host stalls
• 20:25 all summitcopy, processing has stalled, nebulous problem?

  500 read timeout at /home/panstarrs/ipp/psconfig//ipp-20120802.lin64/lib/Nebulous/Client.pm line 1351
  Unable to perform dsget: 9 at /home/panstarrs/ipp/psconfig//ipp-20120802.lin64/bin/summit_copy.pl line 236.
  
  failure for: summit_copy.pl --uri http://otis3.ifa.hawaii.edu/ds/skyprobe/o6146i0041o01/o6146i0041o01.fits --filename neb://any/isp/20120807/o6146i0041o01/o6146i0041o01.chip01.fits --summit_id 728008 --exp_name o6146i0041o01 --inst isp --telescope ps1 --class chip --class_id chip01 --bytes 8602560 --md5 6d88e816d69f885f0966ecd5af3bd814 --dbname isp --timeout 600 --verbose --copies 2 --nebulous
  

  • doh! /dev/sda4 1.9T 1.9T 20K 100% /export/ippdb00.0
  • /export/ippdb00.0/ipp/gpc1/MD08.20091110 is from 2009, assuming doesn't need to be on ippdb00.0 and rsync'd to ipp060.0 for 8.8GB

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD