PS1 IPP Czar Logs for the week 2015.03.20 - 2015.04.05

(Up to PS1 IPP Czar Logs)

Monday : 2015.03.30

• HAF: warptool -revertwarped -fault 4 -label OSS.nightlyscience -dbname gpc1
• HAF: still broken:

  cannot build growth curve (psf model is invalid everywhere)
  Backtrace depth: 11
  Backtrace 0: p_psAssert
  Backtrace 1: pmGrowthCurveGenerate
  Backtrace 2: psphotMakeGrowthCurve
  Backtrace 3: psphotChoosePSFReadout
  Backtrace 4: psphotChoosePSF
  Backtrace 5: psphotReadoutFindPSF
  Backtrace 6: (unknown)
  Backtrace 7: (unknown)
  Backtrace 8: (unknown)
  Backtrace 9: __libc_start_main
  Backtrace 10: (unknown)
  
  Unable to perform pswarp: 4 at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/warp_skycell.pl line 563.
  Running [/home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/warptool -addwarped -tess_id RINGS.V3 -path_base neb://ipp081.0/gpc1/OSS.nt/2015/03/30//o7111g0405o.894133/o7111g0405o.894133.wrp.1520051.skycell.0849.005 -hostname ippc56 -dtime_script 19.999971985817 -warp_id 1520051 -skycell_id skycell.0849.005 -fault 4 -dbname gpc1]...
  
  

• HAF manually setting quality for those 2 to 42:

  warptool -revertwarped -fault 4 -label OSS.nightlyscience -dbname gpc1
  warptool -addwarped -tess_id RINGS.V3 -path_base neb://ipp081.0/gpc1/OSS.nt/2015/03/30//o7111g0405o.894133/o7111g0405o.894133.wrp.1520051.skycell.0849.005 -hostname ippc56 -dtime_script 19.999971985817 -warp_id 1520051 -skycell_id skycell.0849.005 -quality 42 -fault 0 -dbname gpc1
  warptool -addwarped -tess_id RINGS.V3 -path_base neb://ipp081.0/gpc1/OSS.nt/2015/03/30//o7111g0539o.894269/o7111g0539o.894269.wrp.1520184.skycell.1027.036 -hostname ipp088 -dtime_script 17.9999828338623 -warp_id 1520184 -skycell_id skycell.1027.036 -fault 0 -quality 42 -dbname gpc1
  
  

• 1605 Bill: restarted postage stamp server pantasks. Modified pstamp_job_run.pl to look for staticsky cmf for stacks if a skycal result is not given by the parser.

• 20:00 EAM: stopping and restarting stdscience (80k+jobs)

Tuesday : 2015.03.31

• 04:00 Bill: mysqld on ippdb05 crashed. Log file saved to ~ipp restarted it
  • 04:40 The burntool process was quite messed up by the DB crashed. Reset the states of about 20 rawfiles.
  • 04:45 restarted summitcopy pantasks so that the error counts are reset
• 05:00 Bill restarted czarpoll and roboczar on ippc11. they exited when the database went down
• 16:16 CZW: ipplanl/pv3stacksummary is running repair stack summary jobs. This pantasks uses storage nodes, but the jobs are very lightweight, and did not bother things last time they ran.

Wednesday : 2015.04.01

Thursday : 2015.04.02

Friday : 2015.04.03

Saturday : 2015.04.04

Sunday : 2015.04.05