PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2012.03.12

Mark is czar

• 07:15 nightly science looks downloaded and processed. One DM07 stack (828514, skycell.011) has completely bad PSF and probably needs to be dropped. Fixed couple XY26 LAP chips.
• 08:30 stack ahoy.. cleared the 5 stalled LAP runs held up by error_cleaned in warp stage (common skycell_ids 2349.041,2349.042,2350.019,2350.053) with

  warptool -dbname gpc1 -tocleanedskyfile -warp_id 383567 -skycell_id skycell.2350.053
  warptool -dbname gpc1  -setskyfiletoupdate -warp_id 383567 -skycell_id skycell.2350.053 -set_label LAP.ThreePi.20110809
  

• 08:50 while most of LAP is in stack, going to restart stdscience.
• 10:50 LAP stack on ippc24 seems stalled >2ks. a suggestion of Chris, logging into the data host ipp008 and running sync seems to have cleared.

  -- stack pantasks
    0    ippc24    BUSY   2349.08 0.0.0.20e5  0 stack_skycell.pl --threads @MAX_THREADS@ --stack_id 829996 --outroot neb://any/gpc1/LAP.ThreePi.20110809/2012/03/12/RINGS.V3/skycell.2572.042/RINGS.V3.skycell.2572.042.stk.829996 --redirect-output --run-state new --reduction THREEPI_STACK --dbname gpc1 --verbose
  
  -- ps on ippc24
  ipp      15381 11400  0 10:38 ?        00:00:00 glockfile /data/ipp008.0/nebulous/e2/07/1954880509.gpc1:LAP.ThreePi.20110809:2012:03:12:RINGS.V3:skycell.2572.042:RINGS.V3.skycell.2572.042.stk.829996.ppStack.mdc xcld 0
  
  

• 12:10 many pending LAP jobs want data on ipp041 (only 1 CPU still and out of processing), Chris has set unwant=7 (5 is default) and 7 remote tasks running. Chris going to try bumping up to 10, system very busy and going to watch to see if anything breaks.
• 13:55 Serge fixed replication issue Replication_Issues
• 17:00 non-existent LAP OTA to drop

  neb://ipp040.0/gpc1/20100921/o5460g0133o/o5460g0133o.ota15.fits 
  
  chiptool -dropprocessedimfile -set_quality 42 -chip_id 426078 -class_id XY15 -dbname gpc1
  regtool -updateprocessedimfile -set_ignored -exp_id 228082 -class_id XY15 -dbname gpc1
  

• 17:30 Chris/Gene investigated ipp015 disk issues. ended up rebooting.
• 18:00 ipp015 back up but the disk not being exported (correctly?). ran exportfs -f. however, ganglia seems to still thinks it is down.
• 19:00 more xy26 LAP chips repaired. oops, actually have 18 more lap_id left to go..

Tuesday : 2012.03.13

Mark is czar

• 07:00 ipp015 is down, and not rebooting... looks like needs someone to physically go check out.
• 07:30 setting neb-host down ipp015, stopping and shutting down most pantasks while attempting to untangle ipp015
• 08:00 turning summitcopy back on (while going into office), seems to be downloading somewhat.
• 09:10 Serge: Nebulous snafu. Restarted the mysql server on ippdb00.
• 10:05 Serge: All apache/nebulous servers have been restarted.
• 11:00 still killing off, cleaning things up
• 11:30 restarting things, putting ipp015 into the hosts_ignore_wave1 list (do we want to comment it out completely in the wave1 list?)

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD