Excel And My Calculator Disagree?

DAnDan · 18 Nov 2009 at 19:00

Quite a strange one this, doing some calculations at the moment, I found that doing the same calculation on excel, and my calculator (Casio scientific jobby) give different answers, and not just down to rounding as far as I can tell.

Is this a know thing, or am I overlooking something, or simply being crap and pressing buttons?

Jez · 18 Nov 2009 at 19:01

You are pressing the buttons wrong innit

Jotun · 18 Nov 2009 at 19:01

What calculations are you doing?

Post the eqns you've entered in to excel and the values you've used. There shouldn't be any difference really.

DAnDan · 18 Nov 2009 at 19:04

On my calculator I have (2.997*10^-4)/0.018, and excel has the exact same, yet I am getting different answers.

Calc - 0.01665
Excel - 0.016538889 Which will round to 0.01654

Cybermyk · 18 Nov 2009 at 19:05

If .9999 = 1 then its probably OK

div0 · 18 Nov 2009 at 19:09

Excel gives exactly 0.01665, as does Calc.

Both your answers are wrong - you're doing something stupid.

kibblerok · 18 Nov 2009 at 19:09

my excel gives:
0.0166500000000

Jotun · 18 Nov 2009 at 19:09

DAnDan said:
On my calculator I have (2.997*10^-4)/0.018, and excel has the exact same, yet I am getting different answers.

Calc - 0.01655
Excel - 0.016538889 Which will round to 0.01654

In excel I get 1.6650000000E-02
(set it to scientific mode with 10dp to be sure it wasn't rounding)

Wolfram Alpha and Google both agree that's the right answer so the problem must be to do with your excel file.

knowledge123 · 18 Nov 2009 at 19:10

google: 0.01665
wolfram alpha: 0.01665
spotlight: 0.01665

Rich_L · 18 Nov 2009 at 19:10

Strange, when I do 0.0002997 / 0.018, I get .01665

You need to put brackets round the 10^-4 (maybe? lol)

DAnDan · 18 Nov 2009 at 19:15

Thanks guys, I entered the first part into a different cell, and used that for the equation and it now gives the correct answer.

Moredhel · 18 Nov 2009 at 19:33

Note that this sort of discrepancy on a computer or calculator can often be due to floating point precision errors.

http://msdn.microsoft.com/en-us/library/c151dt3s(VS.80).aspx

lopkinfop · 18 Nov 2009 at 19:58

always use more brackets

so (2.997*10^-4)/0.018 becomes (2.997*(10^-4))/0.018

BigglesPiP · 18 Nov 2009 at 20:04

Derive gives 0.01665 when told to approximate it to 10 significant figures.