== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.04.15 ===
 mark is czar
 * 07:10 MEH: nightly downloaded (before the morning shutdown again) and processed -- tweak_ssdiff to run early before any system work needs to be done today
 * 09:00 Bill: postage stamp pantasks was running very slowly (pcontrol spin) restarted it. Also restarted update.
 * 10:25 Bill: mysqld on ippc17 is not running. 
  * MEH: ippc17 having several port/disk error messages..
 * 10:40 MEH: stsci01 crashed over weekend even with neb-host repair. was rebooted into new 3.7.6 kernel, will put neb-host up as seems to be running okay like stsci06 -- stsci01 was one of the stsci machines that had not crashed since the power failure 4/2 and in neb-host repair, will probably want to start sequentially reboot all stsci machine soon into the new kernel..
 * 10:50 MEH: shutting down majority of processing for ippdb work
 * 13:00 MEH: Serge et al finished ippdb swap and cleanup of apache node logs, system back up. ipp home directory <1G space left.. people need to clean home dir. moving ~ipp pantasks logs to archive and compressing.
 * 16:30 Gavin/Rita report that ippc17 looks to have experienced a motherboard failure, will need to be looked at again tomorrow. Datastore and postage server (as it uses the datastore to distribute images) will likely be down for 24 hours. 
 * 17:00 Serge: Commented out ippc19 related backup on ipp@ipp001: /home/panstarrs/ipp/mysql-dump/ops_dump.csh
 * 17:10 Bill set rcserver.off as it uses the datastore, but distribution can be left running to do the preparation work.
 * 17:40 MEH: removing pstamp and publishing from roboczar until ippc17/datastore back
 * 23:50 MEH: looks like ipp025 has given up around 23:14.. SMP oops.. cannot log in, power cycled.. [wiki:ipp025_log]
=== Tuesday : 2013.04.16 ===
 * 06:17 Bill: postage stamp and update pantasks have been moved to ippc14 and started up. 
 * 06:30 MEH: looks like registration may have had hangups over the early morning again.. may not make it through all processing before network down ~@0900. MD done so tweak_ssdiff
 * 8:13 Bill: set rcDestination.dbhost = 'ippc19' and restarted distribution pantasks.
 * 09:50 MEH: all processing stopped/shutdown for Rita/Gavin to swap switch connecting ippc01-ippc16
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===