PS1 IPP Czar Logs for the week 2015.05.04 - 2015.05.10

(Up to PS1 IPP Czar Logs)

Monday : 2015.05.04

• ippc18 is off - this causes cron jobs to not run as well as ganglia. (so this is a wrning)
• haf : 21:00 registration sucks:

      21	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY37 -set_state pending_burntool -dbname gpc1
      22	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY42 -set_state pending_burntool -dbname gpc1
      23	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY62 -set_state pending_burntool -dbname gpc
      32	21:50	regtool -updateprocessedimfile -exp_id 909566 -class_id XY37 -set_state pending_burntool -dbname gpc1
  

Tuesday : 2015.05.05

• 03:50 Bill: Registration is falling behind. There are 3 register_imfile.pl processes that have been running for 10 hours. I am beginning to investigate.
  • those processes were not an issue They were all from one dark exposure that had already finished registration. The scripts must have got stuck while exiting.
  • However exp_id 909635 XY31 was stuck in check_burntool data_state. Set it back to pending_burntool and registration is now catching up.
• 05:40 Bill: registration is stuck again 3 apply_burntool single processes have been running for > 6000 seconds XY42, XY75, and XY17. The hosts running the processes ipp004, ipp010, and ipp054 are the same ones that were stuck previously. There is probably an nfs problem.
  • killed the burntool program processes that were inexplicably stuck and the system fixed the burntool_state on it's own.
  • set hosts ipp004, ipp010, and ipp054 to off in registration pantasks. They seem to keep hanging.
• 12;29 Bill: set pstamp pantasks to stop in preparation for a restart. Lots of faults in the status that are distracting.
  • 12:36 restarted pstamp pantasks
• 12:40 CZW: restarting pv3fflt/rt.
• 20:12 HAF: registration stuck
  • regtool -updateprocessedimfile -exp_id 909868 -class_id XY02 -set_state pending_burntool -dbname gpc1
  • regtool -updateprocessedimfile -exp_id 909877 -class_id XY06 -set_state pending_burntool -dbname gpc1 *

    regtool -updateprocessedimfile -exp_id 909885 -class_id XY06 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 909887 -class_id XY06 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 909890 -class_id XY06 -set_state pending_burntool -dbname gpc1
    
    

• 21:35 HAF : ipp010 / ipp054 / ipp029 had stck mounts, gene fixed them (mostly) - ipp010 , ipp029 are fixed, ipp054, ippc19 are unhappy still, but better than we were..
• 00:45 MEH: registration >200 behind, doesn't look like any gpc1 data has processed in a while (though has been doing gpc2). seems to be stuck on

  o7148g0073o  XY06 -14 pending_burntool neb://ipp091.0/gpc1/20150506/o7148g0073o/o7148g0073o.ota06.fits  
  

  • looks like this is full but got set to pending -- doing

    regtool -updateprocessedimfile -exp_id 909890 -class_id XY06 -set_state full -dbname gpc1
    

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD