== PS1 IPP Czar Logs for the week 2015.02.02 - 2015.02.08 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.02.02 ===
 * 08:50 EAM: stopped and restarted stdlocal, stdscience is basically done, so i've added in the storage hosts as well  
 * 17:54 HAF: stop and restarting stdsci + friends for tonight.
=== Tuesday : 2015.02.03 ===
 * 06:25 EAM: stopping and restarting stdlocal
 * 06:55 EAM: stsci13 hung on reads from lmsensors.  I tried to use reboot, but it just hung.  I power cycled and it is coming up
 * 10:55 MEH: '''reminder''' -- because we are an aggressive cleanup cycle of 2 days, any faults that need manual clearing have to be done ~daily or they won't clear w/o having to then also manually put the chips+warps back through updates -- for example faulted WS from 2/2 will have its warps cleaned up today at 2pm, so these need to get fixed now
{{{
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 654186
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 654213
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 654235
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 654286

difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 655221
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 655250
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 655267
}}}
 * 12:40 MEH: another modification to the data host targeting for nightly processing, adding nodes w/ poor BBU to accept raw OTA since worked with systems <ipp066 (ipp-20141024.lin64/share/pantasks/modules/ipphosts.mhpcc.config). will be used when summitcopy+registration+stdsci are restarted
 * 16:00 CZW stdlocal stopped to avoid possible race conditions as I batch run some LAP scripts.
 * 16:30 CZW stdlocal back up, stdlanl stopped to do the same LAP push, and also clear out runs that have stacks that finished today.
=== Wednesday : YYYY.MM.DD ===

 * 07:35 EAM: we have still 5200 stdlocal stacks to do.  since we are only doing stacks, I've (a) turned off half of the xnode connections (down to 5 from 9) and (b) bumped up the stack poll (to 750 from 500).  I've also turned on the storage hosts.  

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===