== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.02.15 ===
 * 13:00 EAM : I am going to launch relphot using most ippx nodes and the ipps nodes, and probably some of the ipp080 - ipp104 nodes.  I'm restarting stdscience pantasks now to ensure nightly jobs are completed before launching relphot.  
 * 13:30 EAM : I have started relphot for the full PV3 master database.  this will likely take 24-48 hours.

=== Tuesday : YYYY.MM.DD ===

 * 10:30 CZW: ipp/replication pantasks running ippb03.0 OTA replication.
=== Wednesday : YYYY.MM.DD ===

=== Thursday : 2016.02.18 ===
 * 18:00 EAM : dvo operations are using most of the memory for the ipp0XX nodes.  I'm stopping the pantaskses and I will remove the storage nodes from the processing set for standard processing.
 * 21:20 EAM : summitcopy was stuck for a while and I had some trouble tracking down the reason.  apparently, two exposures had a failure in summitcopy advance.  using the command `pztool -toadvance -dbname gpc1 -limit 10` showed the problem exposures.  in summitcopy pantasks, the command `show.books` and then `show.book pzPendingAdvance` revealed the two exposures still on the books as if they were running -- they must have died in a way which pantasks dropped.  stopping summitcopy and calling `reset.advance` cleared that book and made things right again.

=== Friday : 2016.02.19 ===
 * 09:40 MEH: clearing unrecoverable fault 4 (warp curve of growth) and fault 5 (diff)
{{{
warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.0534.058 -warp_id 1688791 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.1550.056 -diff_id 1341246  -fault 0
}}}
 * 11:30 MEH: will do regular restart of nightly pantasks once WSdiffs finish -- including registration which seems to have crashed on its own?
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===