PS1 IPP Czar Logs for the week 2011.06.27 - 2011.07.03

(Up to PS1 IPP Czar Logs)

Monday: 2011-06-27

• 07:17 CZW: marked exposure impacted by loss of summit computer as drop to allow registration to proceed:

pztool -updatepzexp -exp_name o5739g0084o -inst gpc1 -telescope ps1 -set_state drop -summit_id 348983

• 08:00 (Roy) 3 exposures not registered. Looks like one or all of summitcopy/burntool/registration were down most of the night.
• 10:58 serge: in stdscience:

  del.label LAP.ThreePi.20110621
  

• 15:17 serge stopped gpc1 pantasks servers. heather stopped addstar and isp related ones
• 15:53 serge started dump of gpc1 czardb isp ippadmin ipptopsps to /export/ippdb01.0/mysql_gpc1.backup/gpc1.20110627.sql.bz2

  mysqldump [hidden connection information] --databases gpc1 czardb isp ippadmin | ~ipp/local/bin/pbzip2 > gpc1.20110627.sql.bz2
  

About the master:

mysql> SHOW MASTER STATUS;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| mysqld-bin.020077 | 84637877 |              |                  | 
+-------------------+----------+--------------+------------------+

• 17:09 Patnasks servers have been restarted
• 17:45 CZW: Finally remembered to queue the cleanup of LAP/20110505 distribution bundles for bad stacks.
• 18:26 CZW: Rebuilt psModules to pick up stacking speed improvements. This may not be sufficient, however, so more work is likely needed.

Tuesday : 2011-06-28

• 05:30 Serge: gpc1 is replicated on ippdb03 (ingestion on ipp001 is not finished yet)
• 08:13 Roy: all exposures downloaded. Will add LAP.ThreePi.20110621 label back-in when 3PI is done
• 09:15 Serge: gpc1 is replicated on ipp001
• 09:35 Serge: gpc1, ippadmin, and isp dumps are distributed as they should be. I will keep an eye on them though
• 12:00 roy: ipp033 is down on Ganglia, but has network. Gavin is on the case.
• 13:50 roy: ipp033 is back, but /data/ipp033.0/ is not visible from other machines
• 14:00 Serge: replication broken on both hosts. ippdb01 is caching the hostname of ippdb03 with the old hostname value (ippc00). I don't think it's the explanation for the failure though. I'm ingesting yesterday's dump in ippdb03 once again. I will not ingest it in ipp001. I leave the old ingestion schema activated (ingestion in gpc1_0 and gpc1_1).
• 15:15 - 15:40 Bill: regenerated several corrupt files using the scripts in ipp/tools

corrupt file on ipp031 caused diff failure --warp_id 215094 --skycell_id skycell.1276.049
corrupt file on ipp018 caused diff failure --warp_id 215061 --skycell_id skycell.0962.042
corrupt file on ipp051 caused warp failure --cam_id 226915
corrupt file on ipp008 caused magic failure --diff_id 141410 --skycell_id skycell.1992.136
corrupt file on ipp012 caused magic failure --diff_id 141396 --skycell_id skycell.1992.143
corrupt file on ipp012 caused magic failure --diff_id 141386 --skycell_id skycell.1992.134
corrupt file on ipp012 caused distribution failure. Fixing this required faulting the corrupted magicDSFile and setting the states in the DB back to new.

• 15:35 turned magic revert and dist revert off to track down some corrupt files
• 19:38 Serge: Replication started on ippdb03

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD