PS1 IPP Czar Logs for the week 2015.05.04 - 2015.05.10

(Up to PS1 IPP Czar Logs)

Monday : 2015.05.04

• ippc18 is off - this causes cron jobs to not run as well as ganglia. (so this is a wrning)
• haf : 21:00 registration sucks:

      21	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY37 -set_state pending_burntool -dbname gpc1
      22	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY42 -set_state pending_burntool -dbname gpc1
      23	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY62 -set_state pending_burntool -dbname gpc
      32	21:50	regtool -updateprocessedimfile -exp_id 909566 -class_id XY37 -set_state pending_burntool -dbname gpc1
  

Tuesday : 2015.05.05

• 03:50 Bill: Registration is falling behind. There are 3 register_imfile.pl processes that have been running for 10 hours. I am beginning to investigate.
  • those processes were not an issue They were all from one dark exposure that had already finished registration. The scripts must have got stuck while exiting.
  • However exp_id 909635 XY31 was stuck in check_burntool data_state. Set it back to pending_burntool and registration is now catching up.
• 05:40 Bill: registration is stuck again 3 apply_burntool single processes have been running for > 6000 seconds XY42, XY75, and XY17. The hosts running the processes ipp004, ipp010, and ipp054 are the same ones that were stuck previously. There is probably an nfs problem.
  • killed the burntool program processes that were inexplicably stuck and the system fixed the burntool_state on it's own.
  • set hosts ipp004, ipp010, and ipp054 to off in registration pantasks. They seem to keep hanging.
• 12;29 Bill: set pstamp pantasks to stop in preparation for a restart. Lots of faults in the status that are distracting.
  • 12:36 restarted pstamp pantasks
• 12:40 CZW: restarting pv3fflt/rt.
• 20:12 HAF: registration stuck
  • regtool -updateprocessedimfile -exp_id 909868 -class_id XY02 -set_state pending_burntool -dbname gpc1

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD