Intel has a Pretty Big Problem..

Rroff · 20 Jul 2024 at 11:32

StarShock said:
Do you think intel will cover people's motherboard costs if they do end up recalling stuff? I have a feeling they won't want to do that.

Either (or maybe both) fob people off with replacement 14th gen and hope the problem goes away or try and delay/stall people until they've got Bartlett chips coming through which hopefully aren't affected.

KompuKare · 20 Jul 2024 at 11:35

EDIT very big post all messed up. Will delete this one and try it later. Seems some tags messed things up, it intercepted [ SKU ] as a tag

KompuKare · 20 Jul 2024 at 11:46

tommoT.Striker said:
This is a disaster, I wonder if there has been any precedence of this happening before? If the manufacturing fault theory is true, did they only identify there was an issue when they got reports of instability this year, or was it a case of we know but hopefully nobody will notice / chances of issues are remote? I will send the info of the failed 14900KF I had to GamersNexus as they look like they are trying to identify a date range of affected CPUs.

Would love to know if from the codes it is possible to not only get the date but also the fab as at least two fabs are supposed to be involved.
High-res image from a recent view:

Looks like Intel are not keen to tell anyone where it was diffused.

KompuKare · 20 Jul 2024 at 11:47

Pity there is no print copy of the video.

Youtube transcribers do work but when they are accurate, they leave you with no paragraphs etc. - okay for subtitles and searching but harder to quote.

And there are plenty of quotable things in that GN video:

09:09 ...with that said first we have a contact at a major Intel customer - I think it might be it's possible it's their biggest customer but it's definitely one of the top three - uh and the contact has informed us that the
current estimate points toward over 8 million 13th gen CPUs which may be affected for that customer loone of those 8 million 13th gen CPUs 6.1 million are narrowed down to a list of these 11 SKU we'll put it on the screen [i9-13900T, i9-13900, i9-13900F, i7-13700K, i7-13700KF, i7-13700, i7-13700T, i5-13600K, i5-13600KF, i5-13900K, i5-13900KF]

No 14th in that list because:

09:39 ... the current known failure rate for that customer is a range so the actual affected units versus the possible total affected units the numbers I just gave you actual is a range of 10% to 25%. The reason for the range is because some groups within the it's a very large company some groups within the company have seen a higher rate of failure others are estimating lower depends on where they work. Taking these numbers
then 10% to 25% you just do a simple multiply across the whole company assume uh the same failure rate they would be in the range of 600,000 to 2 million plus or minus failed units out of those units in the SKU batch that was on the screen now critically the customer does not yet have information on 14th gen.
... products that is 13th gen only and uh one of our contacts was nervous about how that's going to develop going forward so we don't have newss on the 14th gen side of things it is uh expected to be affected though based on.

Hiring a chip analysis labs (cost: "five figures per CPU"!):

10:55 ... one lead claims that Intel has an issue and its fabrication process where its antioxidation cating uh was improperly applied leading to oxidized Vias for this one Gan [Gamer's Nexus?] is hiring a failure analysis lab to try and get the CPS analyze for this defect uh we don't currently have a timeline it will take some time to do is possible Intel puts out a statement and and hopefully likely

19:33 ... that potential problem and uh for this lead we will not be able to progress until the fa lab we're hiring completes its analysis the lab has warned us that it only has about a 50% success rate on root causing chips of this level of complexity normally the way they work around that is by doing multiple samples I'm still kind of waiting on quotes we're going to send them as many as we can afford and uh it's possible this
kind of gets into five figures territory per CPU not really sure yet so depending on that we may not be able to do more than one to two uh but I'm going do my best and like I said it is extremely likely that this story develops further before we get an answer back and that's fine and we're fine with that because either way this would tell us a lot and if this is a stone that's otherwise left unturned as micro code updates come out

List of affected 13th gen SKU which Intel allegedly told one OEM:

13:55 ... who we're speaking to we also received a leak that Intel has confirmed the following processors are allegedly known to be affected from the 13 series:

The I9 13900 T 13900 13900 F

The i7 13700 K 13700 KF 13700 13700 T

The I5 13600 K 13600 KF

and the I9 13900 K and 13900 KF

these comprise 6.1 million of the 8 plus million total units uh from one large Intel customer we spoke with and of those you get the 10 to 25% possible failure rate number that we mentioned earlier the uh possible total
affected unit is not necessarily units that are currently affected with instability is units that the company is worried might be subject to it so then we get back down to that sort of 600,000 to 2 million or so plus or minus of actual known in the field failures at this point we heard that the uh the 14 series should map to this but we don't have those yet the 13700 T here is actually particularly important because the 13700 T kind of like Wendel's findings with the servers side motherboards uh is is a CPU with a low power envelope and it won't be overclocked and we have at least one of those that's failed and on the way so

Intel's figures:

21:04 ... concerns some don't one source told us that Intel is telling OEMs that it has observed a 0.035% worldwide failure rate this is in
conflict with the OEM we spoke with uh which stated that it has observed a 10 to 25% failure rate and the 25% aligns pretty closely with servers from Wendell's list of contacts uh where

Intake QA from one integrator:

21:50 ... One Source stated this quote either Intel's lying or they don't know the real failure rate until last month they reported to us that 10% of their production was still having the oxidation issue GN has been
informed by multiple parties that Intel is beginning uh process it calls vendor remediation for its OEM customers our understanding is that OEMs are reimbursed for the replacement cost of chips that uh are defective as a result of an Intel fault - specifically working with a medium-sized American System integrator GN learned that the SI is failing 12% of its Intel CPUs during intake quality assurance

Someone the AT thread was speculating that this might easily eclipse Microsoft Xbox 360's Ring of Death, but I think the overall cost to Intel customers may be closer to Capacitor plague.

If it is this huge, then no wonder Intel haven't made a proper statement yet. (An other poster on AT - possibly a shareholder - basically said that a class action might fail and Intel could just ignore everything; what I called "doing an Nvidia Bumbgate" in reply.)

G J · 20 Jul 2024 at 11:49

Aretak said:
Steve's joined the ranks of the AMD uberfans.

I wouldnt confuse the two as Steve seems to actually care, esp so as hes willing to spend a lot of his own money to find out what the problem is.

TrixP10 said:
Nope. He's stating the bleeding obvious.

For whatever reason these CPU's don't work as intended. Also, all these views on why it doesn't work are pointless - it does not work as intended. It's not for the customer/user to find a fix.

CaptainRAVE said:
It’s hardly a fanboy issue when the hardware doesnt work and they can’t recommend it.

As a concerned 13600K owner here where's the statement/information that this effects all CPU's please as the GN video even says that not all CPU's are effected.

Tetras · 20 Jul 2024 at 11:56

G J said:
I wouldnt confuse the two as Steve seems to actually care, esp so as hes willing to spend a lot of his own money to find out what the problem is.

As a concerned 13600K owner here where's the statement/information that this effects all CPU's please as the GN video even says that not all CPU's are effected.

In the event that it is a contamination problem during manufacturing then it would presumably impact:
Any i9 and any i7
i5-K/KF
i5-14600
i5-14400/14400F/13400/13400F (B0 only)

It is suggested that a lower frequency and voltage (on CPU and/or memory) might be able to slow the issues, or maybe it wouldn't even appear at all.

Edit: I think the tip said it happened between somewhere in 2023 and 2024.

Finners · 20 Jul 2024 at 11:57

tommoT.Striker said:
This is a disaster, I wonder if there has been any precedence of this happening before? If the manufacturing fault theory is true, did they only identify there was an issue when they got reports of instability this year, or was it a case of we know but hopefully nobody will notice / chances of issues are remote? I will send the info of the failed 14900KF I had to GamersNexus as they look like they are trying to identify a date range of affected CPUs.

Not as big but intel had to recall early P67 chipset boards as there was a manufacturing defect I remember.

CaptainRAVE · 20 Jul 2024 at 12:00

Tetras said:
In the event that it is a contamination problem during manufacturing then it would presumably impact:
Any i9 and any i7
i5-K/KF
i5-14600
i5-14400/14400F/13400/13400F (B0 only)

It is suggested that a lower frequency and voltage (on CPU and/or memory) might be able to slow the issues, or maybe it wouldn't even appear at all.

And their silence on the matter isn’t exactly helping. Until Intel provides clarification then it could indeed be a wider problem and therefore no, I wouldn’t go anywhere near them nor would any reputable source recommend them.

humbug · 20 Jul 2024 at 12:03

Aretak said:
"We can't recommend Intel CPUs right now. Not until there's either some level of first-party transparency and a support assurance from Intel itself, or until we or some other third party are able to verify why its CPUs are not stable in some situations, which CPUs might be affected, what those conditions are and how you might resolve it."

"If there is no statement and we're just left in the dark on what it might be, then we have no assurance or confidence in the product or the company behind it. And that means right now our blanket statement is we cannot comfortably recommend Intel CPUs and it's gonna be that simple. Until they say something so that we know a little bit more about what the scope of this is."

Steve's joined the ranks of the AMD uberfans.

What do you want Steve to do? do you want him to tell his audience this is fine, nothing to see here? Whose the Uberfan?

Tetras · 20 Jul 2024 at 12:04

CaptainRAVE said:
And their silence on the matter isn’t exactly helping. Until Intel provides clarification then it could indeed be a wider problem and therefore no, I wouldn’t go anywhere near them nor would any reputable source recommend them.

Yeah, I won't be including them in any specs anymore either, I don't want to suggest someone buy a CPU that has a premature failure baked-in. If it is contamination, I guess some CPUs will be a lot more impacted than others, but there could be any kind of fault popping up and any level of instability is going to be frustrating for the user.

CuriousTomCat · 20 Jul 2024 at 12:12

Don't Intel have a bug bounty program where you can get up to $250,000 for reporting a bug with a CPU?

humbug · 20 Jul 2024 at 12:14

CuriousTomCat said:
Don't Intel have a bug bounty program where you can get up to $250,000 for reporting a bug with a CPU?

That prise should go to Nvidia shouldn't it?

Rroff · 20 Jul 2024 at 12:15

Rroff said:
I've found some interesting behaviour with some 13th and 14th gen CPUs and FFMPEG but I've not extensively tested it enough yet to know if it is a software/OS environment issue or this issue or whether it happens with other CPUs or not. Hoping to get my hands on a couple of AMD systems later to see if I can reproduce it there.

Update on this - managed to reproduce the same out of memory error with AMD CPUs - so it isn't a symptom of a 13th/14th gen specific failing, probably a bug in the software. For some reason doesn't happen on my 14700K or older CPUs I have here though.

EDIT: Also happens on my Lenovo Legion Go - overlooked that I had that.

EDIT2: So back to square one in any attempt to get a better understanding of this problem, none of the 13th or 14th gen CPUs I have access to, or my own 14700K, appear to be showing any signs of this issue - no problems with UE5 games, etc. no issues with CineBench including the older versions which were tripping it for some people.

humbug · 20 Jul 2024 at 12:16

Rroff said:
Update on this - managed to reproduce the same out of memory error with AMD CPUs - so it isn't a symptom of a 13th/14th gen specific failing, probably a bug in the software. For some reason doesn't happen on my 14700K or older CPUs I have here though.

Good Grief.... :cry:

No no no... its not Intel!!!!

humbug · 20 Jul 2024 at 12:20

@Rroff if you replicated the same error code in another CPU doesn't mean Intel are in the clear, it just means you deliberately made another CPU unstable by messing with it.

cliffy · 20 Jul 2024 at 13:26

If the issue covers 2 generatiosn (which it pretty obviously does) that means it includes multiple batches, multiple manufacturing dates and presumably more thnen 1 fab. Also it raises the question of did Intel know about this before they started producing the 14th gen, considering the size of the issue i would be surprised if they didn't and did nothing.

Also if a recall happens how the hell do they work out which batches to recall (it would have to surely be all of them) but more importantly how would they know if the replacements were good?

This seems like Intel are majorly FUBAR'd

Rroff · 20 Jul 2024 at 13:31

cliffy said:
If the issue covers 2 generatiosn (which it pretty obviously does) that means it includes multiple batches, multiple manufacturing dates and presumably more thnen 1 fab. Also it raises the question of did Intel know about this before they started producing the 14th gen, considering the size of the issue i would be surprised if they didn't and did nothing.

Also if a recall happens how the hell do they work out which batches to recall (it would have to surely be all of them) but more importantly how would they know if the replacements were good?

This seems like Intel are majorly FUBAR'd

Seen someone who profiled a ton of these CPUs note that 13900s are running almost 100mv higher average VID than the first few batches of 14th gen chips and later batches of 14th gen chips have again ~100mv higher average VID - with some reports claiming the earlier batches of 14th gen chips aren't affected but again so little real information so far.

But I've not so far heard of widespread issues with 13th gen chips predating people starting to notice some 14900s weren't stable. So I don't know what is going on.

uscool · 20 Jul 2024 at 13:31

Would be great to be fly on the wall, what intel are discussing right now

McManicMan · 20 Jul 2024 at 13:34

I wonder what OCUK store has to say about this!. My rig was bought from them with 3yr warranty, 14 months have passed with no issues YET, BUT not particularly happy that something may be wrong with the CPU

TrixP10 · 20 Jul 2024 at 13:35

uscool said:
Would be great to be fly on the wall, what intel are discussing right now

sex, drugs, money. no particular order.