PS1 IPP Czar Logs for the week 2015.01.05 - 2015.01.11

(Up to PS1 IPP Czar Logs)

Monday : 2015.01.05

• 09:15 EAM : stdlocal is sluggish, restarting it now.
• 10:20 EAM : ipp071 seems to be responding to logins again, so i've put it in neb repair (not down)
• 19:20 MEH: w/ ipp071 back available, the half faulted deepstacks can hopefully finish -- ~ippmd/deepstack/ptolemy.rc -- found possible problem, deepstack stop now

Tuesday : 2015.01.06

• 20:20 MEH: try md stacks again -- ~ippmd/deepstack/ptolemy.rc
• 20:50 EAM: summit is getting heavy winds so I'm leaving stdlocal at high load levels. I'll check again after the KP3 telecon (11pm).
• 21:15 EAM: stdlocal was at 120k warps so I'm restarting it.

Wednesday : 2015.01.07

• 10:44 HAF unjammed registration

   regtool -updateprocessedimfile -exp_id 848923 -class_id XY55 -set_state pending_burntool -dbname gpc1
  regtool -updateprocessedimfile -exp_id 848925 -class_id XY55 -set_state pending_burntool -dbname gpc1
  

• restarted registration (11:12, haf)

Thursday : 2015.01.08

• 03:45 Bill stdscience not making progress. Jobs in pcontrol queue without associated processes on the target host. No ppImage processes in 'who uses cluster page' Shutting down stdscience for a restart
  • unfortunately the who uses cluster page is not working with all of the hosts. There are some ppImage processes still running See for example ippc26
  • ganglia shows ipp059 is down (should have loooked there first. Unfortunately the console app won't accept my password when I try to connect with it
  • I am hesitant to start up a new stdscience pantasks with things in this state
  • ran neb-host ipp059 down

• 05:30 MEH: power cycling ipp059 but then need to go to aas booth and won't be able to check on until later
  • only info on console was

    <Jan/07 10:48 pm>ipp059 login: [381619.298227] Disabling IRQ #78
    

• 06:20 EAM: ipp059 is back up and happy, i've restarted stdscience.
• 07:45 EAM: it looks like cab1 had a hiccup last night. all machines in that cab, ex ipp009 and ipp019, rebooted about 9pm. I stopped all processing to clear out some mount point troubles. then it turned out that ipp018 was having xfs problems on /export/ipp018.0. I tried to repair, but needed to reboot to umount. on boot, /dev/sda4 refused to mount. I ran xfs_check /dev/sda4 and it said the metadata needed to be replayed by mounting. mounting gave an error that "Structure needs cleaning". I ran xfs_repair /dev/sda4 as advised, and it said that the log was unrecoverable. I am now running xfs_repair -L /dev/sda4 as advised, and it is trying to clean up the disk partition. The output is going to /root/sda4.repair.log. I have since restarted the processing with ipp018 in neb 'down'. there will probably be some errors until ipp018 is up if anything put output there (was it in neb up earlier? not sure).
• 09:10 EAM: xfs_repair finally finished and /export/ipp018.0 mounted successfully. nfs exports it just fine now.
• 10:43 CZW: processing is working, but there are a lot of faults caused by failures to read detrends on ipp018. The files exist, have the correct md5sum, and funpack correctly. A closer look suggests that we have mounts on some of the x-nodes that can't correctly read from ipp018. ippx090 is an example of this: attempting to read a file results in 'no such file' errors, and the mount point doesn't mount correctly. I'm going to set ipp018 to down to try and get nightly science to clear out, and when ipp018 is quiet, see if a reboot makes it behave again.
• 16:45 CZW: ipp051 to neb-host down so Haydn can work on a disk.
• 16:50 CZW: restarting stdlocal pantasks.
• 17:45 CZW: ipp051 repair, 53,48,47,45,39 down for fast work.
• 18:20 CZW: all storage hosts back online and into repair (as they were before the work), as is ipp018 (which now responds to NFS requests from the hosts it was ignoring before).

Friday : 2015.01.09

• 04:50 MEH: ipp083 has odd battery state and went into WT state yesterday afternoon, doesn't behave well in that state so to neb-host repair it goes
• 04:55 MEH: registration >200 behind, 4 imfiles needed to be reverted but was 50 exposures from those still -- is stdload overloading things?
  • ippx nodes are active in 2x in stdsci and 6-8x in stdlocal (normal allocation), was this intended? going to assume not since wasn't logged and turn them off in stdsci -- ippc nodes also actively mixed, 4x normal in stdsci and 6x in stdlocal -- with many extra nodes in stdsci (~600) there are order 200 sitting IDLE and far from the ~100k job issue state..

Saturday : 2015.01.10

Sunday : 2015.01.11