(Up to PS1 IPP Czar Logs)

Monday : 2012.08.27

• 09:05 EAM : processing was running a bit slow. there were no long-running jobs outstanding. I'm doing a quick test to see if simply restarting the controller improves the throughput. I stopped all jobs pantasks: stop, then waited until the controller was empty. I then exited the controller: pantasks: control exit true, and then restarted it by reloading all of the hosts: pantasks: hosts.stdscience. I am waiting for the hosts to get connected before re-starting the processing.
• 12:45 (Serge): Stopped all processing. All pantasks server shut down.
• 13:05 (Serge): Stopped mysql server on ippdb01. Rsyncing ippdb01:/var/lib/mysql to ippdb03:/var/lib/ippdb01
• 13:10 (Serge): Stopped mysql server on ippdb03
• 15:00 (Serge): Restarted replication on ippdb03 and CzarPoll on ipp009

Tuesday : 2012.08.28

• 08:45 (Bill): turned off chip revert so that we can see if any chips are failing due to missing files
• 09:15 (Serge): Stopped replication on ippdb03. Dumping to /export/ippdb03.0/backups/dump_ippdb01.20120828.sql.bz2

                  Master_Host: ippdb01.ifa.hawaii.edu
                  Master_User: repl_gpc1
              Master_Log_File: mysqld-bin.023064
          Read_Master_Log_Pos: 44362377
              Replicate_Do_DB: czardb,gpc1,ippadmin,ipptopsps,megacam,ssp,uip
  

• 11:25 (Serge): Replication restarted on ippdb03. Copying and then ingesting the dump into ipp001.

Wednesday : 2012.08.29

Bill is acting czar today.

• 07:20 regenerated serveral burntool tables. See tools/fixburntool. Use with care.
• 07:30 stopping stdscience in preparation for restart. (throughput had dropped to 50 per hour)
• 08:31 stopped warp revert to debug a repeating assertion failure
• 09:52 stopping processing to pick up bug fix in psLib
• 10:03 processing set to run warp revert back on
• 13:35 (Serge): Replication restarted on ipp001

Thursday : 2012.08.30

Mark is Czar

• 06:20 small amount of nightly science almost through, then will do the sub-daily restart of stdscience to increase LAP rate back up. Based on the rate plot from yesterday, restart probably needs to be done every 9-10hrs (120/hr to reach ~100k in pantasks) to keep the rate >100/hr, so ~5pm at the latest.
• 07:30 LAP thin on chips to run, set.poll 500 to keep pantasks mostly busy even if just warps.
• 14:03 (Bill) rebuilt ppStack with chisq rejection fix
• 14:50 MEH: Chris noticed stsci nodes running pretty hard. 99% jobs are warps/stacks, so dropping poll in stdscience from 600 to 400 to see if overdriving systems/NFS (so if no chips, not all of stdscience 550 nodes will be processing warps).
• 15:45 restarted stdscience, only warps to do and barely kept 200/400 loaded.. start re-monitor warp rate from here.

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD