== PS1 IPP Czar Logs for the week 2015.03.02 - 2015.03.09 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.03.02 ===
 * 08:50 MEH: ipp001 seems to be unresponsive -- Gavin tried manually power cycling, no response and likely hardware fault somewhere.
  * QUB and Durham use this as a replication of gpc1 DB -- will need to setup for ipp002 or ipp003 -- ipp003 is already used for TSS by QUB so should have open ports.
  * ipp0012 is also the external access node?
  * Gavin+Haydn got back up w/ 1 CPU -- '''all critical data needs to be moved off ASAP like for ipp002'''
  * ipp001.0 99% full, also needs old mysql dumps cleared
 * 12:45 MEH: restart pstamp 
 * 17:00 MEH: regular restart of summitcopy+registration+stdsci for nightly processing
  * summitcopy having repeat 503 timeout faults for a set of files --
{{{
http://conductor.ifa.hawaii.edu/ds/gpc1/o7084g0037d/o7084g0037d10.fits
http://conductor.ifa.hawaii.edu/ds/gpc1/o7084g0037d/o7084g0037d73.fits
http://conductor.ifa.hawaii.edu/ds/gpc1/o7084g0037d/o7084g0037d62.fits

http://conductor.ifa.hawaii.edu/ds/gpc1/o7084g0036d/o7084g0036d37.fits
http://conductor.ifa.hawaii.edu/ds/gpc1/o7084g0036d/o7084g0036d27.fits
http://conductor.ifa.hawaii.edu/ds/gpc1/o7084g0036d/o7084g0036d42.fits
http://conductor.ifa.hawaii.edu/ds/gpc1/o7084g0036d/o7084g0036d75.fits
}}}
  * Craig found t20 behaving strangely, rebooted and files downloaded 
 * 17:30 MEH: ippdb00 down to 2.5G.. not good going into the night -- Gene clearing binlogs to date that should be replicated to backup ippdb06,02,c63
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===