== PS1 IPP Czar Logs for the week 2015.03.20 - 2015.04.05 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.03.30 ===
  
 * HAF: {{{ warptool -revertwarped -fault 4 -label OSS.nightlyscience -dbname gpc1  }}}
 * HAF: still broken:
   {{{
cannot build growth curve (psf model is invalid everywhere)
Backtrace depth: 11
Backtrace 0: p_psAssert
Backtrace 1: pmGrowthCurveGenerate
Backtrace 2: psphotMakeGrowthCurve
Backtrace 3: psphotChoosePSFReadout
Backtrace 4: psphotChoosePSF
Backtrace 5: psphotReadoutFindPSF
Backtrace 6: (unknown)
Backtrace 7: (unknown)
Backtrace 8: (unknown)
Backtrace 9: __libc_start_main
Backtrace 10: (unknown)

Unable to perform pswarp: 4 at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/warp_skycell.pl line 563.
Running [/home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/warptool -addwarped -tess_id RINGS.V3 -path_base neb://ipp081.0/gpc1/OSS.nt/2015/03/30//o7111g0405o.894133/o7111g0405o.894133.wrp.1520051.skycell.0849.005 -hostname ippc56 -dtime_script 19.999971985817 -warp_id 1520051 -skycell_id skycell.0849.005 -fault 4 -dbname gpc1]...

}}}
 * HAF manually setting quality for those 2 to 42:
{{{
warptool -revertwarped -fault 4 -label OSS.nightlyscience -dbname gpc1
warptool -addwarped -tess_id RINGS.V3 -path_base neb://ipp081.0/gpc1/OSS.nt/2015/03/30//o7111g0405o.894133/o7111g0405o.894133.wrp.1520051.skycell.0849.005 -hostname ippc56 -dtime_script 19.999971985817 -warp_id 1520051 -skycell_id skycell.0849.005 -quality 42 -fault 0 -dbname gpc1
warptool -addwarped -tess_id RINGS.V3 -path_base neb://ipp081.0/gpc1/OSS.nt/2015/03/30//o7111g0539o.894269/o7111g0539o.894269.wrp.1520184.skycell.1027.036 -hostname ipp088 -dtime_script 17.9999828338623 -warp_id 1520184 -skycell_id skycell.1027.036 -fault 0 -quality 42 -dbname gpc1

}}}
 * 1605 Bill: restarted postage stamp server pantasks. Modified pstamp_job_run.pl to look for staticsky cmf for stacks if a skycal result is not given by the parser.

 * 20:00 EAM: stopping and restarting stdscience (80k+jobs)

=== Tuesday : 2015.03.31 ===
 * 04:00 Bill: mysqld on ippdb05 crashed. Log file saved to ~ipp restarted it
  * 04:40 The burntool process was quite messed up by the DB crashed. Reset the states of about 20 rawfiles.
  * 04:45 restarted summitcopy pantasks so that the error counts are reset
 * 05:00 Bill restarted czarpoll and roboczar on ippc11. they exited when the database went down
 * 16:16 CZW: ipplanl/pv3stacksummary is running repair stack summary jobs.  This pantasks uses storage nodes, but the jobs are very lightweight, and did not bother things last time they ran.  
=== Wednesday : 2015.04.01 ===

=== Thursday : 2015.04.02 ===

=== Friday : 2015.04.03 ===

=== Saturday : 2015.04.04 ===

=== Sunday : 2015.04.05 ===