Bug 51684 - Math constant rounding error
Summary: Math constant rounding error
Status: RESOLVED FIXED
Alias: None
Product: konqueror
Classification: Applications
Component: kjs (show other bugs)
Version: 3.0.3
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: Harri Porten
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2002-12-09 15:34 UTC by Simon Ejsing
Modified: 2003-10-02 01:37 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Ejsing 2002-12-09 15:34:26 UTC
Version:           3.0.3 (using KDE KDE 3.0.3)
Installed from:    Compiled From Sources
Compiler:          gcc 2.95.3 
OS:          Linux

This little test alerts a non-rounded number due to floating point conversion, but the last division by 10, should be integer division...

        alert(Math.floor(21 * 3.65 * 10) / 10);

(This is the only method of rounding to one decimal that I've found searching the Internet, and it seems to be a widely used one)
Comment 1 George Staikos 2003-02-20 23:05:23 UTC
Looks valid to me 
Comment 2 Peter Kelly 2003-04-26 03:01:59 UTC
This is the correct behaviour... all numerical operations in ecmascript   
are based on floating point operations. If you try the same code in IE or 
Mozilla you will get a floating point number. 
   
Use Math.round() if you want an integer value 
 
Comment 3 Simon Ejsing 2003-04-28 08:45:57 UTC
I've tested it in both Mozilla and IE, they do _not_ return a floating point value. It's a 
commonly used rounding method, so many pages will display odd number values in 
Konqueror 
Comment 4 Simon Ejsing 2003-05-21 10:26:47 UTC
well, then try: 
alert(Math.round(21 * 3.65 * 10) / 10);  
 
same behaviour. 
Comment 5 Harri Porten 2003-05-21 11:29:24 UTC
When I run this code in Mozilla I get 76.6. kjs gives 76.59999999999999. The
Mozilla engine probably has its own floating point calculus implementation. The
difference is not nice but the error is within the machine's precision.

I don't see your point of an integer division. Which browser gives you a rounded
result ?
Comment 6 Simon Ejsing 2003-05-21 13:26:53 UTC
My point in doing this, is because I want to limit the decimal count to one. 
 
The correct output would be 76.6 as Mozilla shows it. MS IE gives the same result, 
however Konqueror produces 76.59999999999999, obviously because it does not 
convert from float to int by either Math.floor or Math.round... 
 
I believe it is a bug that neither floor nor round converts to integers. 
Comment 7 Simon Ejsing 2003-05-21 13:31:05 UTC
"I don't see your point of an integer division. Which browser gives you a rounded  
 result ?" 
 
Ahhhh, I think I got your point now... 
 
No you are right, it's not an integer division, but if 766 was returned as an integer 766 / 
10 would give 76.6, right? 
Comment 8 Harri Porten 2003-05-21 14:04:47 UTC
Subject: Re:  Math constant rounding error         

On 21 May 2003, Simon Ejsing wrote:

> No you are right, it's not an integer division, but if 766 was returned as an integer 766 / 
> 10 would give 76.6, right?

Whether or not 766 is an integer or floating point number doesn't matter.
Neither for kjs or other browser. All consistantly return the same no
matter wheter you write 766/10 or 766.0/10.0. The difference is just that
others appearantly do some rounding up, e.g. flip the last bit.

Harri.
 

Comment 9 Simon Ejsing 2003-05-21 14:33:55 UTC
Yeah, it seems that both Mozilla and IE somehow interpretes the division and fixes the 
decimal count to avoid "buggy" results. 
Comment 10 Thiago Macieira 2003-05-21 17:29:40 UTC
Or, rather, that they round up the representation of 76.5999999999999 or somehow 
do the operation in another way so that the rounding error does not show up. 
 
Whether or not Math.round(766.6) becomes 766 is irrelevant. The result of 766 / 10 
is 76.6 or 76.59999999999 given the precision. 
 
For instance, when trying to print a number with that many decimal places, if it's 
shrunk to, say, 6, rounding would occur and 76.6 would be shown. 
Comment 11 zack-weg 2003-07-04 16:01:40 UTC
Note that 76.59999999999999 == 76.6 is true in JavaScript. Both have a binary
representation of 1001100.1001100110011001100110011001100110011001100110.

The ECMAScript algorithm to construct a string from a number is:

--8<----

9.8.1 ToString Applied to the Number Type

The operator ToString converts a number m to string format as follows:

[...]

5. Otherwise, let n, k, and s be integers such that k >= 1,
   10^(k-1) <= s < 10^k, the number value for s × 10^(n-k) is m, and k is as
   small as possible. Note that k is the number of digits in the decimal
   representation of s, that s is not divisible by 10, and that the least
   significant digit of s is not necessarily uniquely determined by these
   criteria.

[...]

7. If 0 < n <= 21, return the string consisting of the most significant n digits
   of the decimal representation of s, followed by a decimal point '.', followed
   by the remaining k-n digits of the decimal representation of s.

--8<----

n= 2, k= 16, s= 7659999999999999 (as kjs chooses), satisfies most of the
requirements, but k is not as small as possible. The correct choice would be
n= 2, k= 3, s= 766.

If the choice of s is still ambiguous, ECMAScript allows both representations,
but recommends to choose the one with an even last digit.
So 76.59999999999997.toString() should give "76.59999999999996", but kjs does
the opposite and gives "76.59999999999997" for 76.59999999999996.toString().
Comment 12 zack-weg 2003-07-04 16:50:21 UTC
BTW, the normal method of rounding to one decimal is (766/10).toFixed(1),
but kjs hasn't implemented this important method yet.
Comment 13 Harri Porten 2003-07-04 23:27:10 UTC
Subject: Re:  Math constant rounding error

zack-weg@gmx.de wrote:
> 
> ------- Additional Comments From zack-weg@gmx.de  2003-07-04 16:50 -------
> BTW, the normal method of rounding to one decimal is (766/10).toFixed(1),
> but kjs hasn't implemented this important method yet.

It's implemented in the current version in CVS.

Harri.

Comment 14 Harri Porten 2003-10-02 01:37:07 UTC
*** Bug has been marked as fixed ***.