(Up to PS1 IPP Czar Logs)

Monday : 2012-07-23

Serge is czar

• 00:10 keeping chip.revert off to avoid reverting the chip runs that use excessively large memory that can bring down machines by overloading swap
• 09:00 Serge. LAP failed chips.
  • gpc1/20100605/o5352g0430o/o5352g0430o.ota25.burn.tbl fixed (same procedure as on Friday)
  • gpc1/20100618/o5365g0104o/o5365g0104o.ota52.fits fixed
  • gpc1/20100618/o5365g0103o/o5365g0103o.ota52.fits fixed. Manually replicated to ippb00.2. See entry at 12:10 in this page
  • gpc1/20100618/o5365g0116o/o5365g0116o.ota26.fits recovered
  • gpc1/20100524/o5340g0106o/o5340g0106o.ota35.burn.tbl fixed
  • gpc1/20100524/o5340g0120o/o5340g0120o.ota55.burn.tbl fixed
  • gpc1/20100602/o5349g0037o/o5349g0037o.ota31.fits fixed
  • gpc1/20100524/o5340g0096o/o5340g0096o.ota17.burn.tbl fixed
  • gpc1/20100524/o5340g0096o/o5340g0096o.ota72.burn.tbl fixed
  • gpc1/20100622/o5369g0059o/o5369g0059o.ota26.fits recovered
• 09:50 MEH: setup SAS_v8 to distribution (probably will also send chips since didnt un-target those in the SAS dist_group..)
• 09:51 Serge: manually reverted failed chips for LAP
• 10:10 Serge. LAP again
  • gpc1/20100228/o5255g0497o/o5255g0497o.ota47.burn.tbl fixed I ran (see Mark's entry on 2012.05.07)

    ipp_apply_burntool_single.pl --exp_id 143264 --class_id XY47 --this_uri neb://ipp030.0/gpc1/20100228/o5255g0497o/o5255g0497o.ota47.fits 
      --continue 10 --previous_uri neb://ipp005.0/gpc1/20100228/o5255g0496o/o5255g0496o.ota47.fits --dbname gpc1 --verbose
    

  • gpc1/20100714/o5391g0011o/o5391g0011o.ota60.burn.tbl fixed
  • gpc1/20100714/o5391g0012o/o5391g0012o.ota60.burn.tbl fixed
  • gpc1/20100228/o5255g0533o/o5255g0533o.ota35.burn.tbl fixed (using ipp_apply_burntool_single.pl)
  • gpc1/20100228/o5255g0487o/o5255g0487o.ota12.burn.tbl fixed (using ipp_apply_burntool_single.pl)
  • gpc1/20100228/o5255g0491o/o5255g0491o.ota10.burn.tbl fixed (using ipp_apply_burntool_single.pl)
  • gpc1/20100228/o5255g0496o/o5255g0496o.ota33.burn.tbl fixed (using ipp_apply_burntool_single.pl) -- Manually replicated to ippb00.2.
  • gpc1/20100228/o5255g0510o/o5255g0510o.ota40.burn.tbl fixed (using ipp_apply_burntool_single.pl) -- Manually replicated to ippb00.2.
  • gpc1/20100228/o5255g0466o/o5255g0466o.ota22.burn.tbl fixed
  • gpc1/20100228/o5255g0532o/o5255g0532o.ota35.burn.tbl fixed
  • gpc1/20100228/o5255g0534o/o5255g0534o.ota45.burn.tbl fixed
  • gpc1/20100228/o5255g0538o/o5255g0538o.ota53.burn.tbl fixed
  • gpc1/20100228/o5255g0527o/o5255g0527o.ota40.burn.tbl fixed
  • gpc1/20100714/o5391g0027o/o5391g0027o.ota60.burn.tbl fixed
  • gpc1/20100714/o5391g0028o/o5391g0028o.ota60.burn.tbl fixed
  • gpc1/20100714/o5391g0030o/o5391g0030o.ota60.burn.tbl fixed
  • gpc1/20100728/o5405g0348o/o5405g0348o.ota15.fits fixed
• 12:00 MEH: killing some ppImages taking up >50% RAM on machines before it incaps them
• 12:10 Serge: Manually reverted LAP chips
• 13:50 Serge: Killed a bunch of crazy ppImage
• 14:26 Serge: Restarted stdscience

Tuesday : 2012-07-24

• 08:55 Serge: fixing some LAP (Note: for recovered ota to replicate at atrc, I temporarily set neb-host ippb00 to up).
  • gpc1/20100626/o5373g0248o/o5373g0248o.ota65.fits fixed
  • gpc1/20100626/o5373g0246o/o5373g0246o.ota76.fits fixed
  • gpc1/20100626/o5373g0245o/o5373g0245o.ota26.fits recovered
  • gpc1/20100626/o5373g0249o/o5373g0249o.ota26.fits recovered
  • gpc1/20100518/o5334g0175o/o5334g0175o.ota14.burn.tbl fixed
• 09:20 Serge: Reverted LAP chips with fault 2: chiptool -dbname gpc1 -label LAP.ThreePi.20120706 -revertprocessedimfile -fault 2
• 12:15 Serge: Started ingestion of nebulous dump (mysql-neb-ippdb02-2012-07-23T00:30:02.dump.bz) on ippc63... Screen session "nebulous_ingestion".
• 12:40 Serge: Started ingestion of nebulous dump (mysql-neb-ippdb02-2012-07-23T00:30:02.dump.bz) on ippc62... Screen session "nebulous_ingestion".

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD