(Up to PS1 IPP Czar Logs)

Monday : 2012.08.06

Mark is czar

• 07:40 MEH: nightly science slowly continues to download/process, ~100 exposures behind. restarted summitcopy to see if that helps.
  • ipp027.0 continues to decrease, 100% disk used for ~140GB remaining.
  • reports that 'IPP datastore is down, can't connect to backend MySQL server', i'm getting proxy error. can't find any notes on the proxy setup other than now on ippops1. /data/ippc17.0 was full, cleaned out /local/ipp/tmp and mysql happy again. pstamp pantasks having trouble now.
• 08:55 (Serge): pubtool -revert -dbname gpc1 -fault 2 -label ThreePi.nightlyscience after ippc17 full disk incident
• 09:10 (Serge): Killed rogue jobs and restarted publishing
• 09:30 (Serge): Fixed LAP fault 2: gpc1/20100530/o5346g0039o/o5346g0039o.ota47.burn.tbl, gpc1/20100624/o5371g0063o/o5371g0063o.ota64.fits, gpc1/20100529/o5345g0048o/o5345g0048o.ota26.fits
• 10:20 MEH: 12-18hr restarting of stdscience. seems to have picked up the remaining 3 3PI chips..
• 10:30 rebooting ippc17 to try and possibly clear some weird behavior. pstamp and datastore back online. oddly, looks like publishing pushed out some results to datastore with 8/5 date.
• 11:00 nightly science finished. Chris fixed LAP issue and full runs continuing.
• 11:10 (Serge): ippc63 is 68965 behind its master
• 14:20: MEH: odd behavior, stdscience mostly stopped suspect due to book filling up with CRASH entries from ippc29,ippc61 with a very odd error. restarted and slowly put in ippc29,c61 and seems to be fine again. stack was also having CRASH with ippc29. (meeting discussion suggestes possibility stale NFS to ippc18 psLib may have been the problem from rebuild of tag today)

  --> pantasks.stdout.log.20120806.142251 
  crash for: warp_skycell.pl --threads @MAX_THREADS@ --warp_id 489012 --warp_skyfile_id 40697774 --skycell_id skycell.1456.049 --tess_dir RINGS.V3 --camera GPC1 --outroot neb://stsci01.2/gpc1/LAP.ThreePi.20120706/2012/08/04/o5566g0752o.273732/o5566g0752o.273732.wrp.489012.skycell.1456.049 --redirect-output --run-state new  --reduction LAP_SCIENCE --dbname gpc1 --verbose
  job exit status: CRASH
  job host: ippc61
  job dtime: 0.104959
  job exit date: Mon Aug  6 14:19:14 2012
  hostname: ippc61
  
  --> pantasks.stderr.log.20120806.142251
   -> addprocessedimfileMode (chiptool.c:539): unknown psLib error
       -exp_id is required
  
  
  --> ippc29 also in stack -- nothing in stderr log though
  crash for: stack_skycell.pl --threads @MAX_THREADS@ --stack_id 1120678 --outroot neb://any/gpc1/LAP.ThreePi.20120706/2012/08/06/RINGS.V3/skycell.1276.094/RINGS.V3.skycell.1276.094.stk.1120678 --redirect-output --run-state new --reduction THREEPI_STACK --dbname gpc1 --verbose
  job exit status: CRASH
  job host: ippc29
  job dtime: 0.114992
  job exit date: Mon Aug  6 14:13:14 2012
  hostname: ippc29
  

• 16:50 MEH: shutdown and restarted all pantasks.
• 20:20 and the fun continues. ipp055 raid is degraded, setting to repair.. failed.. neb-host stalls
• 20:25 all summitcopy, processing has stalled, nebulous problem?

  500 read timeout at /home/panstarrs/ipp/psconfig//ipp-20120802.lin64/lib/Nebulous/Client.pm line 1351
  Unable to perform dsget: 9 at /home/panstarrs/ipp/psconfig//ipp-20120802.lin64/bin/summit_copy.pl line 236.
  
  failure for: summit_copy.pl --uri http://otis3.ifa.hawaii.edu/ds/skyprobe/o6146i0041o01/o6146i0041o01.fits --filename neb://any/isp/20120807/o6146i0041o01/o6146i0041o01.chip01.fits --summit_id 728008 --exp_name o6146i0041o01 --inst isp --telescope ps1 --class chip --class_id chip01 --bytes 8602560 --md5 6d88e816d69f885f0966ecd5af3bd814 --dbname isp --timeout 600 --verbose --copies 2 --nebulous
  

  • doh! /dev/sda4 1.9T 1.9T 20K 100% /export/ippdb00.0
  • /export/ippdb00.0/ipp/gpc1/MD08.20091110 is from 2009, assuming doesn't need to be on ippdb00.0 and rsync'd to ipp060.0 for 8.8GB
  • neb-host ipp055 repair -- worked
  • summitcopy on and running, registration on, stdscience on w/o LAP
  • pstamp+update also on for MOPS, not sure how long 8GB will last
• 21:10 ipp012+ipp021 raid rebuilding, putting to neb-repair? was old email, put back up.
• 21:55 (Serge) Purged 590 GB of binlogs on ippdb00 (full disk)

  PURGE BINARY LOGS BEFORE '2012-07-20';
  

• 21:56 (Serge) ippc63 is about 50000 behind its master

Tuesday : 2012-08-07

Serge is czar

• 06:40 (Serge): Still 600-409 = 191 exposures to be registered but registration activity seems fine (as far as I can tell).
• 09:10 (Serge): Fixed gpc1/20110315/o5635g0567o/o5635g0567o.ota75.burn.tbl (LAP)
• 09:15 (Serge): 68 exposures to register/download
• 09:40 (Serge): ipp016 set to repair: neb-host ipp016 repair
• 09:50 (Serge): ~ipp/check_system.sh hostoff ipp016 run 8 times. ./check_system.sh hostcheck ipp016 tells that it's OFF in all pantasks
• 10:00 (Serge): No ipp process running on ipp016. neb-host ipp016 down
• 10:03 (Serge): Shutting down ipp016
• 10:35 (Serge): ippc63 is 40000 seconds behind its master
• 11:15 (Serge): All observations downloaded and registered
• 11:30 MEH: MOPS would like WS diffims from STS last night, adding to stdscience label STS.nightlyscience.mops
• 11:45 MEH: stealing ippc57,c58,c59,c60,ipp060 from stdscience for testruns
• 13:15 (Serge): ipp016 is back
• 14:10 (Serge): Fixed gpc1/20100530/o5346g0142o/o5346g0142o.ota14.burn.tbl (LAP)
• 21:30 MEH: restarting stdscience to turn on MD01, tweaking ssdiff to then run.

Wednesday : 2012-08-08

Serge is czar

• 04:20 (Serge): About 80 exposures behind. Let's see if labeltool -updatelabel -set_inactive -label LAP.ThreePi.20120706 -dbname gpc1 helps...
• 07:30 (Serge): All but 20 darks missing: labeltool -updatelabel -set_inactive -label LAP.ThreePi.20120706 -dbname gpc1
• 09:05 (Serge): Nightly science over
• 09:10 (Serge): Setting LAP label to inactive was of no help. I ran source ~bills/db01/tz.sql and source ~bills/db01/latency_query.sql which showed that the time difference between the end of observation and the end of registration in the IPP is slightly increasing.
• 09:20 (Serge): Fixed gpc1/20100530/o5346g0144o/o5346g0144o.ota14.burn.tbl and gpc1/20100530/o5346g0164o/o5346g0164o.ota61.burn.tbl in LAP
• 09:22 (Serge): ippc63 is 600 seconds behind its master
• 10:15 (Serge): Killed 3 ppImage instances on ipp039 and one on ipp047. Fixed gpc1/20100530/o5346g0182o/o5346g0182o.ota15.burn.tbl.
• 11:00 (Serge): ippc63 is synchronized with ippdb00.
• 11:00 MEH: ran wrong command in stdscience, need to restart..
• 11:00 (Serge): Recovering space (should be around 500GB) on ipp027 by deleting the various /export/ipp027.0/backup/snapshots/daily.?/alala/data/alala.0/elixir. On ipp022, I'm archiving /export/ipp022.0/alala.0/elixir. It will be copied by rsnapshot.
• 14:15 (Serge): Fixed gpc1/20100530/o5346g0104o/o5346g0104o.ota13.burn.tbl and gpc1/20100118/o5214g0253o/o5214g0253o.ota22.burn.tbl
• 14:30 (Serge): Killed ppImage on ipp039 involving (o4991g0100o, xy60) and (o4985g0063o, xy60)

  chiptool -dbname gpc1 -updateprocessedimfile -fault 0 -set_quality 42 -set_state full -chip_id 534179 -class_id XY60
  regtool -dbname gpc1 -updateprocessedimfile -set_state corrupt -class_id XY60 -exp_id 75621
  

and

chiptool -dbname gpc1 -updateprocessedimfile -fault 0 -set_quality 42 -set_state full -chip_id 534138 -class_id XY60
regtool -dbname gpc1 -updateprocessedimfile -set_state corrupt -class_id XY60 -exp_id 73255 

• 15:15 (Serge): Killed ppImage on ipp039 involving (o4985g0082o, XY60) and (o4991g0100o, XY60)

  chiptool -dbname gpc1 -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 534139  -class_id XY60
  chiptool -dbname gpc1 -updateprocessedimfile -set_state full -chip_id 534139  -class_id XY60
  regtool -dbname gpc1 -updateprocessedimfile -set_state corrupt -class_id XY60 -exp_id 73272 
  

and

chiptool -dbname gpc1 -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 534179 -class_id XY60
chiptool -dbname gpc1 -updateprocessedimfile -set_state full -chip_id 534179 -class_id XY60
regtool -dbname gpc1 -updateprocessedimfile -set_state corrupt -class_id XY60 -exp_id 75621

• 15:55 (Serge): Restarted rgistration after bias images painted the czartool in red
• 16:40 (Bill): Fixed and installed ppBackground to apply the correct sign with the pattern continuity correction. Queued 120 new M31 chipBackgroundRuns
• 17:15 MEH: ipp016 raid issues today, disk swapped and raid rebuilding so setting neb-host repair to harass it less

Thursday : 2012-08-09

Serge is czar

• 07:30 (Serge): Summit copy and registration finished
• 09:00 (Serge): Nightly science finished (but 4 STS at warp stage and one M31 at stack)
• 09:05 (Serge): Killed the usual bunch of mad ppImage. Dropped (class_id, exp_id, chip_id)
  • on ipp039: XY60 73268 535777; XY60 73279 535850 ; XY60 73239 536023.
  • on ipp043: XY60 73266 536066
  • on ipp047: XY67 73268 535777 ; XY67 73279 535850
  • on ippc49: XY60 73264 535799
• 09:10 (Serge): Fixed gpc1/20100118/o5214g0268o/o5214g0268o.ota16.burn.tbl; gpc1/20100530/o5346g0092o/o5346g0092o.ota47.burn.tbl
• 09:15 (Serge): Recovered not-so-lost ota gpc1/20100530/o5346g0113o/o5346g0113o.ota26.fits

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD