PS1 IPP Czar Logs for the week 2011.06.27 - 2011.07.03

(Up to PS1 IPP Czar Logs)

Monday: 2011-06-27

• 07:17 CZW: marked exposure impacted by loss of summit computer as drop to allow registration to proceed:

pztool -updatepzexp -exp_name o5739g0084o -inst gpc1 -telescope ps1 -set_state drop -summit_id 348983

• 08:00 (Roy) 3 exposures not registered. Looks like one or all of summitcopy/burntool/registration were down most of the night.
• 10:58 serge: in stdscience:

  del.label LAP.ThreePi.20110621
  

• 15:17 serge stopped gpc1 pantasks servers. heather stopped addstar and isp related ones
• 15:53 serge started dump of gpc1 czardb isp ippadmin ipptopsps to /export/ippdb01.0/mysql_gpc1.backup/gpc1.20110627.sql.bz2

  mysqldump [hidden connection information] --databases gpc1 czardb isp ippadmin | ~ipp/local/bin/pbzip2 > gpc1.20110627.sql.bz2
  

About the master:

mysql> SHOW MASTER STATUS;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| mysqld-bin.020077 | 84637877 |              |                  | 
+-------------------+----------+--------------+------------------+

• 17:09 Patnasks servers have been restarted
• 17:45 CZW: Finally remembered to queue the cleanup of LAP/20110505 distribution bundles for bad stacks.
• 18:26 CZW: Rebuilt psModules to pick up stacking speed improvements. This may not be sufficient, however, so more work is likely needed.

Tuesday : 2011-06-28

• 05:30 Serge: gpc1 is replicated on ippdb03 (ingestion on ipp001 is not finished yet)
• 08:13 Roy: all exposures downloaded. Will add LAP.ThreePi.20110621 label back-in when 3PI is done
• 09:15 Serge: gpc1 is replicated on ipp001
• 09:35 Serge: gpc1, ippadmin, and isp dumps are distributed as they should be. I will keep an eye on them though
• 12:00 roy: ipp033 is down on Ganglia, but has network. Gavin is on the case.
• 13:50 roy: ipp033 is back, but /data/ipp033.0/ is not visible from other machines
• 14:00 Serge: replication broken on both hosts. ippdb01 is caching the hostname of ippdb03 with the old hostname value (ippc00). I don't think it's the explanation for the failure though. I'm ingesting yesterday's dump in ippdb03 once again. I will not ingest it in ipp001. I leave the old ingestion schema activated (ingestion in gpc1_0 and gpc1_1).
• 15:15 - 15:40 Bill: regenerated several corrupt files using the scripts in ipp/tools

corrupt file on ipp031 caused diff failure --warp_id 215094 --skycell_id skycell.1276.049
corrupt file on ipp018 caused diff failure --warp_id 215061 --skycell_id skycell.0962.042
corrupt file on ipp051 caused warp failure --cam_id 226915
corrupt file on ipp008 caused magic failure --diff_id 141410 --skycell_id skycell.1992.136
corrupt file on ipp012 caused magic failure --diff_id 141396 --skycell_id skycell.1992.143
corrupt file on ipp012 caused magic failure --diff_id 141386 --skycell_id skycell.1992.134
corrupt file on ipp012 caused distribution failure. Fixing this required faulting the corrupted magicDSFile and setting the states in the DB back to new.

• 15:35 turned magic revert and dist revert off to track down some corrupt files
• 19:38 Serge: Replication started on ippdb03
• 19:47 Bill: turned off diff processing temporarily.
• 19:43-53 Serge: Replication seems to have failed again. Last exp_id on ippdb01 is 354060 while on ippdb03 it's 353683.

The SHOW SLAVE STATUS command shows that everything's fine on the replication host:

            Slave_IO_State: Waiting for master to send event
                Master_Host: ippdb01.ifa.hawaii.edu
                Master_User: repl_gpc1
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: mysqld-bin.020091
        Read_Master_Log_Pos: 99203082
             Relay_Log_File: mysqld-relay-bin.000031
              Relay_Log_Pos: 99203220
      Relay_Master_Log_File: mysqld-bin.020091
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB: gpc1,czardb,ippadmin,isp
        Replicate_Ignore_DB: 
         Replicate_Do_Table: 
     Replicate_Ignore_Table: 
    Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
                 Last_Errno: 0
                 Last_Error: 
               Skip_Counter: 0
        Exec_Master_Log_Pos: 99203082
            Relay_Log_Space: 99203220
            Until_Condition: None
             Until_Log_File: 
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File: 
         Master_SSL_CA_Path: 
            Master_SSL_Cert: 
          Master_SSL_Cipher: 
             Master_SSL_Key: 
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

On the master:

| 204458295 | repl_gpc1 | ippdb03.ifa.hawaii.edu:35103 | NULL   | Binlog Dump |   574 | Has sent all binlog to slave; waiting for binlog to be updated |

ippdb03 log does not show any error:

110628 19:37:42 [Note] Slave I/O thread: connected to master 'repl_gpc1@ippdb01.ifa.hawaii.edu:3306',  replication started in log 'mysqld-bin.020077' at position 84637877
110628 19:37:42 [Note] Slave SQL thread initialized, starting replication in log 'mysqld-bin.020077' at position 84637877, relay log './mysqld-relay-bin.000001' position: 4
EOF

... and the ippdb01 log is empty (size 0) (it doesn't seem to be redirected to some other file according to /etc/mysql/my.cnf)

Note: While looking at the PROCESSLIST table of ippdb01, I saw that:

| 204467208 | unauthenticated user | ipp034.ifa.hawaii.edu:59761  | NULL   | Connect     |  NULL | Reading from net                                               | NULL                                                                                                 | 

(It's likely unrelated but I wanted to write it down)

J'aime pas ça...

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD