DL380 G7 Crash minidump

nutcase · 21 Jan 2017 at 19:17

Is anyone familiar with these? It's the first one I've looked at. I'm thinking it's a RAM issue but can't be certain?

Code:

Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\administrator\Desktop\012117-42416-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: srv*
Executable search path is: 
Windows 7 Kernel Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: LanManNt, suite: TerminalServer SingleUserTS
Built by: 7601.23572.amd64fre.win7sp1_ldr.161011-0600
Machine Name:
Kernel base = 0xfffff800`02015000 PsLoadedModuleList = 0xfffff800`02257730
Debug session time: Sat Jan 21 15:25:41.750 2017 (UTC + 0:00)
System Uptime: 0 days 2:18:41.426
Loading Kernel Symbols
...............................................................
................................................................
.............................
Loading User Symbols
Loading unloaded module list
......
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck D1, {10, 2, 0, fffff88001216208}

Probably caused by : storport.sys ( storport!RaidUnitReleaseIrp+38 )

Followup:     MachineOwner
---------

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000010, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff88001216208, address which referenced memory

Debugging Details:
------------------


DUMP_CLASS: 1

DUMP_QUALIFIER: 400

BUILD_VERSION_STRING:  7601.23572.amd64fre.win7sp1_ldr.161011-0600

SYSTEM_MANUFACTURER:  HP

SYSTEM_PRODUCT_NAME:  ProLiant DL380 G7

SYSTEM_SKU:  470065-375      

BIOS_VENDOR:  HP

BIOS_VERSION:  P67

BIOS_DATE:  01/30/2011

DUMP_TYPE:  2

BUGCHECK_P1: 10

BUGCHECK_P2: 2

BUGCHECK_P3: 0

BUGCHECK_P4: fffff88001216208

READ_ADDRESS: GetPointerFromAddress: unable to read from fffff800022c1100
Unable to get MmSystemRangeStart
GetUlongPtrFromAddress: unable to read from fffff800022c12e8
GetUlongPtrFromAddress: unable to read from fffff800022c1498
GetPointerFromAddress: unable to read from fffff800022c10b8
 0000000000000010 

CURRENT_IRQL:  2

FAULTING_IP: 
storport!RaidUnitReleaseIrp+38
fffff880`01216208 488b5610        mov     rdx,qword ptr [rsi+10h]

CPU_COUNT: 8

CPU_MHZ: 854

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 1a

CPU_STEPPING: 5

CPU_MICROCODE: 6,1a,5,0 (F,M,S,R)  SIG: 19'00000000 (cache) 15'00000000 (init)

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT_SERVER

BUGCHECK_STR:  0xD1

PROCESS_NAME:  System

ANALYSIS_SESSION_HOST:  CHUBBS

ANALYSIS_SESSION_TIME:  01-21-2017 18:47:22.0071

ANALYSIS_VERSION: 10.0.14321.1024 x86fre

TRAP_FRAME:  fffff8000351f8a0 -- (.trap 0xfffff8000351f8a0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffffa803dbee6d0 rbx=0000000000000000 rcx=0000000000000000
rdx=fffffa803dbee600 rsi=0000000000000000 rdi=0000000000000000
rip=fffff88001216208 rsp=fffff8000351fa30 rbp=fffffa803dbee600
 r8=fffff8000351f9d0  r9=0000000000000000 r10=fffffa801945db10
r11=fffff8000351faa0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
storport!RaidUnitReleaseIrp+0x38:
fffff880`01216208 488b5610        mov     rdx,qword ptr [rsi+10h] ds:00000000`00000010=????????????????
Resetting default scope

DPC_STACK_BASE:  FFFFF80003525FB0

STACK_OVERFLOW: Stack Limit: fffff8000351ffb0. Use (kF) and (!stackusage) to investigate stack usage.

LAST_CONTROL_TRANSFER:  from fffff800020849a9 to fffff80002085400

STACK_TEXT:  
fffff800`0351f758 fffff800`020849a9 : 00000000`0000000a 00000000`00000010 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
fffff800`0351f760 fffff800`02083620 : 00000000`00000000 00000000`00000000 00000000`00000000 fffffa80`1d7f8160 : nt!KiBugCheckDispatch+0x69
fffff800`0351f8a0 fffff880`01216208 : fffffa80`3dbee600 00000000`00000000 fffffa80`3dbee600 00000000`00000000 : nt!KiPageFault+0x260
fffff800`0351fa30 fffff880`0121bfd1 : fffffa80`1945db10 00000000`000000fa fffffa80`1d7f8160 fffffa80`1d7f8160 : storport!RaidUnitReleaseIrp+0x38
fffff800`0351fa60 fffff880`01204ee5 : fffffa80`1d7f8160 00000000`00000001 fffff880`01250300 00000000`000000fe : storport!RaidUnitProcessBusyRequest+0x81
fffff800`0351fad0 fffff880`01202866 : fffffa80`1945db10 fffff800`02605895 fffff800`000000fb fffffa80`18eea040 : storport! ?? ::FNODOBFM::`string'+0x1606
fffff800`0351fbb0 fffff880`012094be : 00000000`00000001 fffff800`000000fb 00000000`001fc7d5 00000000`00000001 : storport!RaidUnitCompleteRequest+0x3a6
fffff800`0351fc90 fffff800`0209007c : fffff800`02203e80 fffffa80`18eea000 00000004`00000001 00000000`000000ad : storport!RaidpAdapterRedirectDpcRoutine+0x4e
fffff800`0351fcd0 fffff800`0207d10a : fffff800`02203e80 fffff800`02211cc0 00000000`00000000 fffff880`01209470 : nt!KiRetireDpcList+0x1bc
fffff800`0351fd80 00000000`00000000 : fffff800`03520000 fffff800`0351a000 fffff800`0351fd40 00000000`00000000 : nt!KiIdleLoop+0x5a


STACK_COMMAND:  kb

THREAD_SHA1_HASH_MOD_FUNC:  af46fc2a0695d8dd5e1f50cce6d342e2c711319d

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  c0309e8c5ff5d6b15111d72c1bc63b337f9c1e6e

THREAD_SHA1_HASH_MOD:  26f2c98289d095ee5edc151846c8346ca735bed1

FOLLOWUP_IP: 
storport!RaidUnitReleaseIrp+38
fffff880`01216208 488b5610        mov     rdx,qword ptr [rsi+10h]

FAULT_INSTR_CODE:  10568b48

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  storport!RaidUnitReleaseIrp+38

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: storport

IMAGE_NAME:  storport.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  52f04432

IMAGE_VERSION:  6.1.7601.18386

FAILURE_BUCKET_ID:  X64_0xD1_storport!RaidUnitReleaseIrp+38

BUCKET_ID:  X64_0xD1_storport!RaidUnitReleaseIrp+38

PRIMARY_PROBLEM_CLASS:  X64_0xD1_storport!RaidUnitReleaseIrp+38

TARGET_TIME:  2017-01-21T15:25:41.000Z

OSBUILD:  7601

OSSERVICEPACK:  1000

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  2

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 7

OSEDITION:  Windows 7 LanManNt (Service Pack 1) TerminalServer SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  2016-10-11 15:57:55

BUILDDATESTAMP_STR:  161011-0600

BUILDLAB_STR:  win7sp1_ldr

BUILDOSVER_STR:  6.1.7601.23572.amd64fre.win7sp1_ldr.161011-0600

ANALYSIS_SESSION_ELAPSED_TIME: 631

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:x64_0xd1_storport!raidunitreleaseirp+38

FAILURE_ID_HASH:  {0bada84e-b975-e7ac-60de-9bdf17817e88}

Followup:     MachineOwner
---------

Thanks

Caged · 21 Jan 2017 at 19:51

It's not really possible to tell what the fault is other than it's hardware/driver related. Try leaving memtest running.

nutcase · 21 Jan 2017 at 20:43

Thanks

Not much point to that then. Memtest needs downtime? Took exchaneg offline and figured I'd give the HP diags another go. Bingo:

Code:

Serial presence detect (SPD) information - DIMM 9, Card 2
Memory type	DDR3
Memory DRAM type	RDIMM
Memory DRAM Speed	1333 Mbits
DDR3 Standard Voltage Capability	YES
DDR3 Low Voltage Capability	NO
DIMM Description	DIMM 4GB PC3-10600R 256Mx4 RoHS
Additional DIMM Description	HP 4GB 2Rx4 PC3-10600R-9 Kit
Spare Part Number	501534-001
Manufacturer Name	Hyundai Electronics
Manufacturer Location	01
Manufacturer Date	1110
Manufacturer Module Serial Number	1abbb64a
Correctable Error Threshold Exceeded Status	Correctable error threshold exceeded
Uncorrectable Error Status	No DIMM errors detected
Correctable Error Threshold Count	1
Uncorrectable Error Count	0

Caged · 21 Jan 2017 at 20:59

Sounds like you're running Exchange on one box that is old

Time to embrace the cloudy future.

nutcase · 21 Jan 2017 at 21:16

Tell me something I don't know

THis server used to run EVERYTHING. I've been (slowly) moving roles off - now it just runs Exchange, mailmarshall, webmarshall, sophos enterprise console and DC/DHCP/DNS.

It's got a relatively easy life but until I'm allowed to spend lots of money I'll be making use of the 24/7 carepack we keep maintained on it.

nutcase · 21 Jan 2017 at 22:22

New stick of RAM being delivered about 07:30 Monday. Could have had it sooner but unless it starts flaking out more frequently I don't really want to go in over the weekend and I can get away with a few mins downtime early on a Monday.

First time we've used the 24/7 support - until I got involved we were 8/5 (well actually for about 8 months it was out of any support!).

Ricimer · 21 Jan 2017 at 23:43

What did the diagnostics say? The memory is ECC, so it causing a crash without being flagged as failed is unusual. The memory dump suggests it was the storage driver.

nutcase · 22 Jan 2017 at 08:51

Apart from the RAM issue it didn't tell me anything. I just double checked and the array firmware and drivers are up to date.

nutcase · 23 Jan 2017 at 09:29

RAM installed. Fingers crossed!

Showboat · 24 Jan 2017 at 20:51

IMAGE_NAME: storport.sys

Update that driver, it caused the crash. Did you have any disk issues (failed / about to fail) around the time of the crash?

nutcase · 25 Jan 2017 at 10:59

Not a sausage - no sign of any issues before it beetrooted.

I just found this (surprised HPE didn't tell me about it) http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0122663

Both the storport.sys and MPIO.sys are newer versions of what I have so I will try that hotfix I guess. The only storage controllers in the box are the P410i that comes as standard. A 10Gbe NIC was added a few weeks ago but that's the only hardware change in years.

Armageus · 25 Jan 2017 at 11:09

Is your P410i firmware and drivers up to date (and also drive firmwares e.g. if you have recently replaced a drive?)

As I understand it storport.sys is the main storage driver that then delegates to HP's miniport driver for the P410i.

I would double check what versions you are on, and look at the release notes for any subsequent versions (if there are any) and see if there are any critical fixes mentioned (have seen the release notes when upgrading some of our servers before, and some of the fixes are quite random but potentially scary situations).

nutcase · 25 Jan 2017 at 14:05

Yes - assuming the drives haven't had a new release in the last 3 months, as I went through them then. I don't believe we've had to replace a drive in over 12 months now. I have double checked the controller and that's still up to date. No idea how I missed the BIOS though