== PS1 IPP Czar Logs for the week 2011.06.27 - 2011.07.03 == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday: 2011-06-27 === * 07:17 CZW: marked exposure impacted by loss of summit computer as drop to allow registration to proceed: {{{pztool -updatepzexp -exp_name o5739g0084o -inst gpc1 -telescope ps1 -set_state drop -summit_id 348983}}} * 08:00 (Roy) 3 exposures not registered. Looks like one or all of summitcopy/burntool/registration were down most of the night. * 10:58 serge: in stdscience: {{{ del.label LAP.ThreePi.20110621 }}} * 15:17 serge stopped gpc1 pantasks servers. heather stopped addstar and isp related ones * 15:53 serge started dump of gpc1 czardb isp ippadmin ipptopsps to /export/ippdb01.0/mysql_gpc1.backup/gpc1.20110627.sql.bz2 {{{ mysqldump [hidden connection information] --databases gpc1 czardb isp ippadmin | ~ipp/local/bin/pbzip2 > gpc1.20110627.sql.bz2 }}} About the master: {{{ mysql> SHOW MASTER STATUS; +-------------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +-------------------+----------+--------------+------------------+ | mysqld-bin.020077 | 84637877 | | | +-------------------+----------+--------------+------------------+ }}} * 17:09 Patnasks servers have been restarted * 17:45 CZW: Finally remembered to queue the cleanup of LAP/20110505 distribution bundles for bad stacks. * 18:26 CZW: Rebuilt psModules to pick up stacking speed improvements. This may not be sufficient, however, so more work is likely needed. === Tuesday : 2011-06-28 === * 05:30 Serge: gpc1 is replicated on ippdb03 (ingestion on ipp001 is not finished yet) * 08:13 Roy: all exposures downloaded. Will add {{{LAP.ThreePi.20110621}}} label back-in when 3PI is done * 09:15 Serge: gpc1 is replicated on ipp001 * 09:35 Serge: gpc1, ippadmin, and isp dumps are distributed as they should be. I will keep an eye on them though * 12:00 roy: ipp033 is down on Ganglia, but has network. Gavin is on the case. * 13:50 roy: ipp033 is back, but /data/ipp033.0/ is not visible from other machines * 14:00 Serge: replication broken on both hosts. ippdb01 is caching the hostname of ippdb03 with the old hostname value (ippc00). I don't think it's the explanation for the failure though. I'm ingesting yesterday's dump in ippdb03 once again. I will not ingest it in ipp001. I leave the old ingestion schema activated (ingestion in gpc1_0 and gpc1_1). * 15:15 - 15:40 Bill: regenerated several corrupt files using the scripts in ipp/tools {{{ corrupt file on ipp031 caused diff failure --warp_id 215094 --skycell_id skycell.1276.049 corrupt file on ipp018 caused diff failure --warp_id 215061 --skycell_id skycell.0962.042 corrupt file on ipp051 caused warp failure --cam_id 226915 corrupt file on ipp008 caused magic failure --diff_id 141410 --skycell_id skycell.1992.136 corrupt file on ipp012 caused magic failure --diff_id 141396 --skycell_id skycell.1992.143 corrupt file on ipp012 caused magic failure --diff_id 141386 --skycell_id skycell.1992.134 corrupt file on ipp012 caused distribution failure. Fixing this required faulting the corrupted magicDSFile and setting the states in the DB back to new. }}} * 15:35 turned magic revert and dist revert off to track down some corrupt files * 19:38 Serge: Replication started on ippdb03 * 19:47 Bill: turned off diff processing temporarily. * 19:43-53 Serge: Replication seems to have failed again. Last exp_id on ippdb01 is 354060 while on ippdb03 it's 353683. The SHOW SLAVE STATUS command shows that everything's fine on the replication host: {{{ Slave_IO_State: Waiting for master to send event Master_Host: ippdb01.ifa.hawaii.edu Master_User: repl_gpc1 Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysqld-bin.020091 Read_Master_Log_Pos: 99203082 Relay_Log_File: mysqld-relay-bin.000031 Relay_Log_Pos: 99203220 Relay_Master_Log_File: mysqld-bin.020091 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: gpc1,czardb,ippadmin,isp Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 99203082 Relay_Log_Space: 99203220 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 1 row in set (0.00 sec) }}} On the master: {{{ | 204458295 | repl_gpc1 | ippdb03.ifa.hawaii.edu:35103 | NULL | Binlog Dump | 574 | Has sent all binlog to slave; waiting for binlog to be updated | }}} ippdb03 log does not show any error: {{{ 110628 19:37:42 [Note] Slave I/O thread: connected to master 'repl_gpc1@ippdb01.ifa.hawaii.edu:3306', replication started in log 'mysqld-bin.020077' at position 84637877 110628 19:37:42 [Note] Slave SQL thread initialized, starting replication in log 'mysqld-bin.020077' at position 84637877, relay log './mysqld-relay-bin.000001' position: 4 EOF }}} ... and the ippdb01 log is empty (size 0) (it doesn't seem to be redirected to some other file according to /etc/mysql/my.cnf) Note: While looking at the PROCESSLIST table of ippdb01, I saw that: {{{ | 204467208 | unauthenticated user | ipp034.ifa.hawaii.edu:59761 | NULL | Connect | NULL | Reading from net | NULL | }}} (It's likely unrelated but I wanted to write it down) J'aime pas ça... === Wednesday : YYYY.MM.DD === === Thursday : YYYY.MM.DD === === Friday : YYYY.MM.DD === === Saturday : YYYY.MM.DD === === Sunday : YYYY.MM.DD ===