== PS1 IPP Czar Logs for the week 2012.06.11 - 2012.06.18 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.06.11 ===
  * 21:20 leaving postage stamp server running along with nightly science. seems to do fine with active MOPS jobs past couple nights while being monitored, but will turn update off later. some QUB stamps seem to have stalled even with update on and no other stamps running, will need looking into. 
=== Tuesday : 2012.06.12 ===
 Mark is czar
 * 07:20 nightly science data downloaded and processed. postage stamp running still, MOPS jobs almost finished but more QUB jobs seems to have become stuck and am looking into.
 * 09:20 tracing out the PSS stalling, one looks like wanting/waiting to update a diffim that is in the goto_cleaned state (never actually cleaned), MD06 diff_id=242933
  * turning on cleanup pantasks and watching processing/ippdb03 closely to avoid crashing, try and "clean-up" -- looks like will take a couple hours. want to turn back off before nightly science? 
  * 12:30 QUB from past few days moving through now as well as some web ones too.
 * 10:00 Serge turning replication/balance back on since doesn't use gpc1
 * 10:30 removing ippc12 from pantasks and from loaded compute list since being swapped in for ippdb01.
 * 10:35 did ippc12 going down stall a distribution entry for MD07 SSdiff?
  * do the easy restart of distribution and see if picks up again -- yes and cleared.
 * 12:30 MOPS has ~36k stamps requests, running ~200/minute, so about 2-3hr finish with this batch for a gpc1 dump? no reason to wait, sooner is better since not replicated/no copy to fall back on..
 * 13:35 Serge: manually set the correct time and restarted ntpd on ippdb01
 * 13:40 Mark: stopping all processing, czartool etc for ippdb03:gpc1 dump
 * 13:53 Serge: dumping gpc1 on ippdb03 to {{{/export/ippdb03.0/mysql-dumps/backup/gpc1_20120612.sql.bz2}}}. Coordinates of ippdb03 as a master are:
{{{
mysql> SHOW MASTER STATUS;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| mysqld-bin.000004 |       98 |              |                  | 
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)
}}}
 * 15:40 Serge: end of dump. I also performed a czardb dump
 * 15:45 Mark: processing restarted, stdscience+stack+pstamp+update fully shutdown and restarted to make sure ippc12 was dropped.
 * 16:15 Serge: ippdb01 is "clean". Dropped the following databases: MD01, MD02, MD03, MD04, Rw10BH96, beaumont, czardb_2, czwtest, detection_efficiency, eamMC, eamtest, haf_addtest, heather, heather2, ipptopsps_test, simtestMEH, ippadmin. I also dropped czardb and gpc1 for a full reingestion (czardb has actually been ingested while I write these lines and gpc1 ingestion has begun).
 * 20:50 Mark: appears registration is stalled. trying a restart -- that worked...
=== Wednesday : 2012-06-13 ===
 Mark is czar
 * 06:44 Serge: Mysql replication of ippdb03 to ippdb01 started and looking fine (right now: 45000 seconds behind).
 * 07:40 Serge: ippdb03 has caught up. We can switch back to the normal configuration.
 * 07:50 Mark: nightly science downloaded and processed.
 * 08:10 all services shutdown to flip back over to primary DB scidbm (ippdb01)
 * 09:45 Serge: ippdb03 is replicating ippdb01. To avoid problems I removed all access to ippdb03 by the 'ipp' user. Other users are unchanged.
 * 10:00 Mark: restarting pantasks, starting with pstamp since will have MOPS requests to check. then update, then stdscience for the MD SSdiffs, then distribution
  * pstamp running, SSdiffs running, update failing -- needed ippadmin DB on scidbm/ippdb01 even though just queried. -- now SSdiffs and updates running
 * Heather added back isp to summitcopy - it seems to be working, heather will verify later (it's several days behind). Heather will also add to input files so isp is again automatic . The magic command for summitcopy/registration) since there are no labels in those pantasks is to do add.database isp or del.database isp
 * 12:10 czartool page not fully updating, investigating -- czarpages on ippMonitor use scidbs (db03) and replication broken right now.
=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===