== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.12.15 ===
 * 08:00 MEH: cleanup must be turned back on, the few number of nodes available for data is killing processing again  
 * 11:00 CZW: I've updated the database to prevent the exposures from the dead pixelserver from clogging up summitcopy:
{{{
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832987;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832988;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832989;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832990;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832991;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832992;
}}}
 * 14:00 CZW: Restarting stdlocal.  I'm going to see what happens with the "default" set of hosts.
  * 14:07 CZW: default set is underpowered.  Turning on xnodes with xnode.hosts.on macro.
 * 14:50 MEH: summitcopy stop -- camera group reporting ramping overload -- anyone else doing summitcopy type stuff?
 * 14:45 CZW: I noticed more failing exposures in pzDownloadExp table, and dropped the camera controlled exposures in state 'run':
{{{
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834453;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834454;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834455;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834456;
UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834457;
}}}
 * 14:55 MEH: Haydn working on ipp077, will be up/down and shouldn't be used until he gives the all-clear
  * back online -- setting to repair only so not to overload until we catchup
 * 15:30 MEH: cleanup diff on only, doing double loading and poll 120 -- seems okay, will help move the larger size diffs from past few days out of the way 
 * 16:00 CZW: With ipp077 back online, the LANL remote stack prep commands are no longer failing.  I've reverted the LAP.PV3.20140730.bigmem stacks locally, to see if adding stacks impacts the processing rate.
 * 16:25 CZW: Restarting the ipp user pantasks.  I will add 4x c2 to stdscience and ensure that cleanup is prioritizing the larger diffs.
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===