== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.12.22 ===
 * 07:40 MEH: warp fault -- cannot build growth curve (psf model is invalid everywhere)
{{{
warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1297048 -skycell_id skycell.2460.067
}}}
 * 13:05 MEH: ipp006-031 neb-host repair, limit space available and try to help limit data across 10G link w/ note reflecting such (as discussed at morning meeting). ipp033-053 already in repair for similar reasons.
  * all except   ipp008,012,013,014,016,018,109,020,021,037 which are actually on ippcore
 * 18:00 CZW: restarting stdlocal, as it's old and not spawning jobs efficiently. -- '''be sure to re-adjust the poll to not overload the 10G link with chip processing'''
 * 23:40 MEH: doesn't look like balance was done? poll>400 for chips and 10G overload harassing other processing (good thing no nightly), rate lower than before with fewer stacks.  8x:x nodes (not sure when changed) looks to be trying 6 stacks at a time.. probably not a good default to have.. -- setting back to 6x and reducing poll for chips to where 10G link isn't being as overloaded


=== Tuesday : 2014.12.23 ===
 * 09:05 MEH: ipp015 unresponsive, nothing on console, minimal load -- power cycle

=== Wednesday : YYYY.MM.DD ===
 * 01:40 MEH: regtool on ipp062 stalling registration for past 1.5hr.. 20 exposures behind -- killed job and moving again
 * 2:10 MEH: ippc28 crash -- GPF and something odd w/ swap /dev/sdb? -- try to powercycle but taking out of stdsci
{{{
<Dec/24 01:42 am>[1507866.922239] end_request: I/O error, dev sdb, sector 198065
<Dec/24 01:42 am>[1507866.928211] Read-error on swap-device (8:16:198073)
}}}
  * power cycled but not fully booting, stalling on  * Starting local ...
 * 06:15 MEH: fixed fault 5 diff
{{{
difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 625690 -skycell_id skycell.1243.051 -dbname gpc1
}}}
=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===