PS1_IPP_Czarlog_20120924 – Pan-STARRS IPP

Context Navigation

Version 16 (modified by bills, 14 years ago) ( diff )
--

CZW: 12:00 Stopped system for stdscience reboot.
CZW: 12:04 Rebooted ipp020 due to mounting issues that were resistant to force.umount. Rebooted using the call:
```
sync
sync
sync
reboot -f
```
CZW: 12:10 autofs was not correct on ipp020. /etc/init.d/autofs restart resolved the problem.
CZW: 12:12 Repeated this procedure on ipp063, once ipp020 returned.
CZW: 12:16 Restarted stdscience.
CZW: 13:50 Restarted ipp020/gmond to fix the "ganglia purple memory bug" that Mark pointed out.

08:00 Bill We got some data last night. registration has a stuck job on ipp010. It is having the "ganglia purple memory bug" Killed the register_imfile. Rebooting ipp010
08:05 reset registrations pending book. set fault for the ipp010 victim 521411:ota26 then reverted it. burntool is now feverishly working to catch up.

Serge is czar

09:00 (Serge): rsync mysql@ippdb02 to ippc63 complete. Starting rsync of mysql@ippc63 to ippc61
15:10 MEH: turning off the deepstack compute3 group from stack pantasks for a local/trunk deepstack pantask setup for MD09.GR0 staticsky run
16:25 CZW: restarting stdscience because it looks slow.
23:40 MEH: looks like ipp018 has lost some mounts, removing from processing to try and get nightly science back online and avoid rebooting until tomorrow. Also putting ipp018 into repair, restarting summitcopy and registration.
- ipp018 is bypassed, registration going again but will see how long it lasts. ipp018 needs a reboot in the morning.

04:30 Bill We have our nightly registration hangs. tonight's troublemakers are ipp011 and ipp012
05:00 rebooted ipp011 and ipp012. Restarted registration and stdscience. Burntool immediately fell behind on ipp012 because stdscience loaded it up with several ppImage and pswarp processes. Took the node out of stdscience for a bit.
05:10 restarted update and pstamp pantasks
05:15 ipp018 has nfs hangs rebooted

Note: See TracWiki for help on using the wiki.