PS1 IPP Czar Logs for the week 2012.06.11 - 2012.06.18

(Up to PS1 IPP Czar Logs)

Monday : 2012.06.11

• 21:20 leaving postage stamp server running along with nightly science. seems to do fine with active MOPS jobs past couple nights while being monitored, but will turn update off later. some QUB stamps seem to have stalled even with update on and no other stamps running, will need looking into.

Tuesday : 2012.06.12

Mark is czar

• 07:20 nightly science data downloaded and processed. postage stamp running still, MOPS jobs almost finished but more QUB jobs seems to have become stuck and am looking into.
• 09:20 tracing out the PSS stalling, one looks like wanting/waiting to update a diffim that is in the goto_cleaned state (never actually cleaned), MD06 diff_id=242933
  • turning on cleanup pantasks and watching processing/ippdb03 closely to avoid crashing, try and "clean-up" -- looks like will take a couple hours. want to turn back off before nightly science?
  • 12:30 QUB from past few days moving through now as well as some web ones too.
• 10:00 Serge turning replication/balance back on since doesn't use gpc1
• 10:30 removing ippc12 from pantasks and from loaded compute list since being swapped in for ippdb01.
• 10:35 did ippc12 going down stall a distribution entry for MD07 SSdiff?
  • do the easy restart of distribution and see if picks up again -- yes and cleared.
• 12:30 MOPS has ~36k stamps requests, running ~200/minute, so about 2-3hr finish with this batch for a gpc1 dump? no reason to wait, sooner is better since not replicated/no copy to fall back on..
• 13:35 Serge: manually set the correct time and restarted ntpd on ippdb01
• 13:40 Mark: stopping all processing, czartool etc for ippdb03:gpc1 dump
• 13:53 Serge: dumping gpc1 on ippdb03 to /export/ippdb03.0/mysql-dumps/backup/gpc1_20120612.sql.bz2. Coordinates of ippdb03 as a master are:

  mysql> SHOW MASTER STATUS;
  +-------------------+----------+--------------+------------------+
  | File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
  +-------------------+----------+--------------+------------------+
  | mysqld-bin.000004 |       98 |              |                  | 
  +-------------------+----------+--------------+------------------+
  1 row in set (0.00 sec)
  

• 15:40 Serge: end of dump. I also performed a czardb dump
• 15:45 Mark: processing restarted, stdscience+stack+pstamp+update fully shutdown and restarted to make sure ippc12 was dropped.
• 16:15 Serge: ippdb01 is "clean". Dropped the following databases: MD01, MD02, MD03, MD04, Rw10BH96, beaumont, czardb_2, czwtest, detection_efficiency, eamMC, eamtest, haf_addtest, heather, heather2, ipptopsps_test, simtestMEH, ippadmin. I also dropped czardb and gpc1 for a full reingestion (czardb has actually been ingested while I write these lines and gpc1 ingestion has begun).
• 20:50 Mark: appears registration is stalled. trying a restart.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD