PS1 IPP Czar Logs for the week 2011.10.17 - 2011.10.23

(Up to PS1 IPP Czar Logs)

Monday : 2011.10.17

• 11:00 - 11:15 update pantasks died. Restated by bills. distribution is backed up. Restarted pantasks
• 12:45 Mark: set ipp029 down in nebulous for Cindy to install new ram.
• 13:40 Added back to nebulous in repair state initially. Will set to up in nebulous and add back in gradually for testing.
• 14:15 One stdscience job started on ipp029 ran for ~15min before it crashed.
• 20:45 stdscience seems to be under-running, restarting.

Tuesday : 2011.10.18

• 06:00 registration stalled. restarted pantasks and stalled. started running again after ~20 minutes.
• 06:20 ippc13 down for 5hr. rebooted.
• 06:25 warprun stalled

  Reading FITS file /data/ipp027.0/nebulous/c7/38/1467947204.gpc1:ThreePi.nt:2011:10:18:o5852g0316o.408936:o5852g0316o.408936.ch.327264.XY44.ch.wt.fits failed.
  
  perl ~ipp/src/ipp-20110622/tools/runchipimfile.pl --chip_id  327264  --class_id XY44 --redirect-output 
  

• 06:30 registration now stuck with ipp033 mount to ipp028, running again after

  /usr/local/sbin/force.umount ipp028
  

• 06:35 stdscience pantasks down. restarted...
• 06:50 registration stalled again... restarted. summitcopy stalled with 26 images left. restarted. looks like o5852g0500o/o5852g0500o46.fits got stalled with mount problem and failed with

  failure for: summit_copy.pl --uri http://conductor.ifa.hawaii.edu/ds/gpc1/o5852g0500o/o5852g0500o46.fits --filename neb://ipp029.0/gpc1/20111018/o5852g0500o/o5852g0500o.ota46.fits --summit_id 404735 --exp_name o5852g0500o --inst gpc1 --telescope ps1 --class chip --class_id ota46 --bytes 51831360 --md5 7e8c954ded69e0537d465a1f457ebc26 --dbname gpc1 --timeout 600 --verbose --copies 2 --compress --nebulous
  

  summitcopy still not loading and running manually fails to insert row now. Entry looks okay, no fault. Talking with Bill to trace out why/how to fix.
• 09:50 Bill traced it to variable CLASS_ID not found in summit_copy.pro and put in a temporary fix. Remaining exposures look to make it through to fake stage okay. In future would help to properly use the start_server.sh so logs get renamed.
• 10:00 will delay shutdown for nebulous repair until remaining data looks to make it through and MOPS gets postage stamps.
• 11:00 systems shutdown, apache off for nebulous/ippdb02 repair.
• 11:20 nebulous mysqldump started (/export/ippdb00.0/sch/nebulous-20111018.sql). Master coordinates: mysqld-bin.003345 ; 30002581
• 11:30 Cleaned ippdb02. Backup of user table in /export/ippdb02.0/mysql-dumps/backups/mysql.user.sql; cd /var/lib/mysql && rm -rf *. Archived /var/log/mysql contents (to /export/ippdb02.0/mysql-dumps/backups/20111018)
• 16:30 dump not going to finish before 5pm. stopping it and restarting IPP processing.
• 16:50 crashed ipp036 after processing restarted Ipp036-crash-20111018T165000 (looks like kernel panic like ipp026?). had 482 days w/o the root filesystem being checked so check done.
• 17:30 processing running smoothly again.
• 21:00 MD04 g-band deep stack sample running. the deepstack pantasks now uses only the compute2 group (ippc20-c29), the compute group has been returned to distribution.

Wednesday : 2011.10.19

• 00:39 ippc06 down. rebooted.
• 09:20 stopped processing for nebulous mysqldump. Master coordinates mysqld-bin.003352 / 277806591
• 09:21 Dump started /export/ippdb00.0/sch/nebulous_20111019.sql.bz2
• 15:00 Cindy had reported a possible 5 drives in ippb00 failed/faulted according to the manufacturer in past 2 weeks. ippb00 is going to be shut down to avoid any data loss until she had more information from the manufacturer tomorrow.
• 15:16 End of nebulous dump (5h57m to make the dump)
• 16:00 Mark: ippc22 went down ~1 hr ago while system idle (during nebulous DB dump). trying to bring back up
• 16:05 Processing restarted. Copied nebulous dump on ippdb02 (/export/ippdb02.0/ipp/nebulous_20111019.sql.bz2) and ipp001 (/export/ipp001.0/ipp/mysql-dumps/nebulous_20111019.sql.bz2)

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD