PS1 IPP Czar Logs for the week 2015.05.04 - 2015.05.10

(Up to PS1 IPP Czar Logs)

Monday : 2015.05.04

• ippc18 is off - this causes cron jobs to not run as well as ganglia. (so this is a wrning)
• haf : 21:00 registration sucks:

      21	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY37 -set_state pending_burntool -dbname gpc1
      22	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY42 -set_state pending_burntool -dbname gpc1
      23	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY62 -set_state pending_burntool -dbname gpc
      32	21:50	regtool -updateprocessedimfile -exp_id 909566 -class_id XY37 -set_state pending_burntool -dbname gpc1
  

Tuesday : 2015.05.05

• 03:50 Bill: Registration is falling behind. There are 3 register_imfile.pl processes that have been running for 10 hours. I am beginning to investigate.
  • those processes were not an issue They were all from one dark exposure that had already finished registration. The scripts must have got stuck while exiting.
  • However exp_id 909635 XY31 was stuck in check_burntool data_state. Set it back to pending_burntool and registration is now catching up.
• 05:40 Bill: registration is stuck again 3 apply_burntool single processes have been running for > 6000 seconds XY42, XY75, and XY17. The hosts running the processes ipp004, ipp010, and ipp054 are the same ones that were stuck previously. There is probably an nfs problem.
  • killed the burntool program processes that were inexplicably stuck and the system fixed the burntool_state on it's own.
  • set hosts ipp004, ipp010, and ipp054 to off in registration pantasks. They seem to keep hanging.
• 12;29 Bill: set pstamp pantasks to stop in preparation for a restart. Lots of faults in the status that are distracting.
  • 12:36 restarted pstamp pantasks
• 12:40 CZW: restarting pv3fflt/rt.
• 20:12 HAF: registration stuck
  • regtool -updateprocessedimfile -exp_id 909868 -class_id XY02 -set_state pending_burntool -dbname gpc1
  • regtool -updateprocessedimfile -exp_id 909877 -class_id XY06 -set_state pending_burntool -dbname gpc1 *

    regtool -updateprocessedimfile -exp_id 909885 -class_id XY06 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 909887 -class_id XY06 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 909890 -class_id XY06 -set_state pending_burntool -dbname gpc1
    
    

• 21:35 HAF : ipp010 / ipp054 / ipp029 had stck mounts, gene fixed them (mostly) - ipp010 , ipp029 are fixed, ipp054, ippc19 are unhappy still, but better than we were..
• 00:45 MEH: registration >200 behind, doesn't look like any gpc1 data has processed in a while (though has been doing gpc2). seems to be stuck on

  o7148g0073o  XY06 -14 pending_burntool neb://ipp091.0/gpc1/20150506/o7148g0073o/o7148g0073o.ota06.fits  
  

  • looks like this is full but got set to pending -- doing

    regtool -updateprocessedimfile -exp_id 909890 -class_id XY06 -set_state full -dbname gpc1
    

• 01:10 MEH: since here, also going to do a restart of registration, been running for a few days

Wednesday : 2015.05.06

• 03:50 Bill: registration still proceeding very slowly. It is 258 exposures behind with burntool. Each burntool processs for the 5 or so chips that are behind seems to take several minutes to complete. However there are 30 or so chipsRuns in the queue so the system isn't completely stuck at this time. Processing rate is only about 20-25 per hour though
• 06:00 EAM: still way behind, and Ken Smith reports problems with postage stamps. I'm going to stop all processing and attempt to clear out hung mounts throughout the system and restart apache servers as needed.
• 06:30 EAM: problem hosts (hung df): ipp054 (rebooting now), ippc11, ippc13, ippc19, ippc29, ippc32, ippc33. i'll try to clear mounts on these and reboot as needed (ex ippc19). update: also stsci00, stsci01,
  • MEH: remember ippc29 isn't a normal processing machine, please give notification before rebooting
• 6:45 HAF: came late to the game, noticed all the pantasks are off - I assume they will get restarted once stuff is cleared up (by gene?) - i'll stay out of the way now..
• 07:10 EAM : only ippc13 (failed to reboot) and ipp19 (hung mounts) are now a problem. I'm restarting just the ipp services
• 07:11 EAM : update : some ippx noddes are also still having a problem -- they do not see ippc19.
• 07:40 EAM : i forgot to remove ipp004 from the pantasks hosts. it is being used by haydn for ippc18 recovery work and has an excessive load. i'm stopping things now to take it out cleanly
• 07:55 EAM : everything for ipp user is up and running again
• 10:50 EAM : burntool got stuck at exp 348, but i've cleared the problem and things are moving along now.
• 15:00 CZW: regtool -updateprocessedimfile -set_state pending_burntool -exp_id 910276 -class_id XY23
• 17:45 EAM: ippc30 crashed; i am rebooting now

Thursday : 2015.05.07

Friday : 2015.05.08

Saturday : 2015.05.09

Sunday : 2015.05.10