I was exploring a couple of <code>fault = 4</code> ('programming') errors in the stacks I'm currently processing, and found that both processes died due to memory corruption, in different places in the code. There are no known or suspected bugs in the code that would cause this. Then I noticed that they both occurred on ipp025, which also recently crashed with the do_IRQ error.

I suggest that the memory corruption I experienced is roughly equivalent to a memtest86 error, and conclude that ipp025 has bad memory.

Paul.Price 13:34, 3 November 2008 (HST)

2008-11-03 21:27:12Z|ipp025              |I|ppStack
    Time to perform rejection on image 7: 20.196237 sec
Memory Block ID: 16273299 @ 0x7f841cd084c0
        Previous Block: 0x7f841cd08540 Next Block: 0x7f841cd08400
        Free function: (nil)
        Size: 8 Reference count: 1 Persistent: No
        Posts: deadbeef deadbeef deadbcef
        Allocated in vectorAlloc at (psVector.c:78)
                by Thread ID 1127573840
 
        Memory block is corrupted; buffer overflow detected.
        Caught in vectorFree at (psVector.c:44).
 
p_psMemDecrRefCounter (psMemory.c:800) Unsafe to Continue

2008-11-03 21:30:29Z|ipp025              |I|psModules.imcombine
    Mean deviation from 98 stamps: 0.188637 +/- 0.197773
Memory Block ID: 4541300 @ 0x7f0272aba770
        Previous Block: 0x7f0272aba900 Next Block: 0x7f0272aba7f0
        Free function: 0x7f028e2d1210
        Size: 16 Reference count: 1 Persistent: No
        Posts: deadbeef deadbeef deadbcef
        Allocated in pmSubtractionConvolve at (pmSubtraction.c:1282)
                by Thread ID 139648987760368
 
        Memory block is corrupted; buffer overflow detected.
        Caught in psArrayElementsFree at (psArray.c:239).
 
p_psMemDecrRefCounter (psMemory.c:800) Unsafe to Continue
 
Aborting.  Error stack:

mysql> select hostname, path_base from stackSumSkyfile join stackRun
using(stack_id) where label = 'pap_md_nov2008' and fault = 4;
+----------+-------------------------------------------------------------------------------------+
| hostname | path_base                  |
+----------+-------------------------------------------------------------------------------------+
| ipp025   | neb://ipp010/md_demo_Nov2008//FIXNS/skycell.02874.13/FIXNS.skycell.02874.13.stk.375 |
| ipp025   | neb://ipp010/md_demo_Nov2008//FIXNS/skycell.02878.62/FIXNS.skycell.02878.62.stk.434 |
+----------+-------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)