== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.10.07 ===
 * 09:20 Bill restarted pstamp pantasks
 * 17:30 CZW: ipp051 and ipp063 had to be power cycled, as they had memory issues due to galactic plane.
 * 22:50 MEH: surprise, registration stuck.. 70 exposures behind -- restarting registration seems to have cleared
 * 00:10 MEH: then 2 OSS chips stalling on ipp028 for ~3ks.. restarted stdsci, STS and LAP label out
  * stsci10-19 also down and not off.. something was odd..
 * 01:20 MEH:  OSS warps MIA.. restarting stdsci again found them.  also five OSS had fault 5
{{{
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1501.041 -diff_id 483695 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1413.083 -diff_id 483713 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1413.095 -diff_id 483713 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1322.020 -diff_id 483726 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1322.090 -diff_id 483726 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1414.007 -diff_id 483727 -fault 0
}}}
=== Tuesday : 2013.10.08 ===
 * 07:00 MEH: nightly mostly done, STS back in.  3PI fault 5 diffim 
{{{
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1324.041 -diff_id 483889 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1332.035 -diff_id 483964 -fault 0

difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1233.080 -diff_id  483855 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1234.079 -diff_id  483859 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1235.098 -diff_id  483863 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1331.020 -diff_id  483978 -fault 0
}}}
 * 09:20 MEH: ipp034 back up, put back into neb-host repair.  ipp0016 put into repair over weekend because problematic, put back up now after rebooted sunday..
 * 09:25 adding LAP label back in now -- 
 * 09:40 MEH: continuing general preventative maintenance  -- restarting long running pantasks (summit, cleanup, distribution, publishing, stack; did registration last night) -- been month+ since last log archive to clear up space on ippc18.0
  * ippc18 disk is overused, cannot rsync the logs ippc18.0 to ippc18.1 w/o over-driving load and killing processing.. suspect too many accounts are writing too many logs to home disk.  have to set rsync to <800KPS
 * 09:50 MEH: ipp049 had long running (>2hr) pswarp and ppImage, killed and reverted but looks like another pair ppImage stalling -- 
 * 11:15 MEH: still no LAP stacks, putting 2x compute3 from stack into pstamp for MOPS (18.5k jobs)
  * possibly a lot of data for MOPS on ipp017 requested at once? driving load >200.. settling down now.  need to check if also was from access by UMD for high throughput tests at that time
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===