PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2012.03.12

Mark is czar

• 07:15 nightly science looks downloaded and processed. One DM07 stack (828514, skycell.011) has completely bad PSF and probably needs to be dropped. Fixed couple XY26 LAP chips.
• 08:30 stack ahoy.. cleared the 5 stalled LAP runs held up by error_cleaned in warp stage (common skycell_ids 2349.041,2349.042,2350.019,2350.053) with

  warptool -dbname gpc1 -tocleanedskyfile -warp_id 383567 -skycell_id skycell.2350.053
  warptool -dbname gpc1  -setskyfiletoupdate -warp_id 383567 -skycell_id skycell.2350.053 -set_label LAP.ThreePi.20110809
  

• 08:50 while most of LAP is in stack, going to restart stdscience.
• 10:50 LAP stack on ippc24 seems stalled >2ks. a suggestion of Chris, logging into the data host ipp008 and running sync seems to have cleared.

  -- stack pantasks
    0    ippc24    BUSY   2349.08 0.0.0.20e5  0 stack_skycell.pl --threads @MAX_THREADS@ --stack_id 829996 --outroot neb://any/gpc1/LAP.ThreePi.20110809/2012/03/12/RINGS.V3/skycell.2572.042/RINGS.V3.skycell.2572.042.stk.829996 --redirect-output --run-state new --reduction THREEPI_STACK --dbname gpc1 --verbose
  
  -- ps on ippc24
  ipp      15381 11400  0 10:38 ?        00:00:00 glockfile /data/ipp008.0/nebulous/e2/07/1954880509.gpc1:LAP.ThreePi.20110809:2012:03:12:RINGS.V3:skycell.2572.042:RINGS.V3.skycell.2572.042.stk.829996.ppStack.mdc xcld 0
  
  

• 12:10 many pending LAP jobs want data on ipp041 (only 1 CPU still and out of processing), Chris has set unwant=7 (5 is default) and 7 remote tasks running. Chris going to try bumping up to 10, system very busy and going to watch to see if anything breaks.
• 13:55 Serge fixed replication issue Replication_Issues
• 17:00 non-existent LAP OTA to drop

  neb://ipp040.0/gpc1/20100921/o5460g0133o/o5460g0133o.ota15.fits 
  
  chiptool -dropprocessedimfile -set_quality 42 -chip_id 426078 -class_id XY15 -dbname gpc1
  regtool -updateprocessedimfile -set_ignored -exp_id 228082 -class_id XY15 -dbname gpc1
  

• 17:30 Chris/Gene investigated ipp015 disk issues. ended up rebooting.
• 18:00 ipp015 back up but the disk not being exported (correctly?). ran exportfs -f. however, ganglia seems to still thinks it is down.
• 19:00 more xy26 LAP chips repaired. oops, actually have 18 more lap_id left to go..

Tuesday : 2012.03.13

Mark is czar

• 07:00 ipp015 is down, and not rebooting... looks like needs someone to physically go check out.
• 07:30 setting neb-host down ipp015, stopping and shutting down most pantasks while attempting to untangle ipp015
• 08:00 turning summitcopy back on (while going into office), seems to be downloading somewhat.
• 09:10 Serge: Nebulous snafu. Restarted the mysql server on ippdb00.
• 10:05 Serge: All apache/nebulous servers have been restarted.
• 11:00 still killing off, cleaning things up
• 11:30 restarting things, putting ipp015 into the hosts_ignore_wave1 list (do we want to comment it out completely in the wave1 list?)
• 12:13 CZW: I've modified a copy of repair_bad_instance, and fed it with an input list generated from Serge's slocate databases of ota26 entries on ipp064. It's currently running singly threaded, and checking all of these raw fits files and repairing bad instances when possible. This is copying good copies from the ATRC, but as it should not be doing more than one at a time, I do not expect this to overly tax the ATRC computers/ATRC AC.
• 12:15 Serge: Killed ippdb03 scheduled mysqldump to perform a manual one that I intend to ingest on the broken ipp001. Replication on ippdb03 has been stopped on purpose.
• 12:40 - nightly science warp faults mostly due to processed chip data needed on ipp015.
• 12:50 replication pantasks back running, target.on only.
• 13:00 Gavin setup access to the power control for us on compute3 nodes
• 14:10 Serge: Restarted replication on ippdb03. CHANGE MASTER TO MASTER_LOG_FILE='mysqld-bin.020936', MASTER_LOG_POS=73345738;
• 14:30 registration has been very unhappy. ipp011 often having lockd server not responding for ipp008,ipp014 so set those two to repair.
• 15:00 all data downloaded and registered from last night finally.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD