Overflow problem [VB.net]

[Zeller]

When given the values (all intagers)

a=650820
b=4974863
c=41
d=165

I get an overflow error on this line

Code Select Expand


Dim myInt As Integer = ((a - b) * 500) / c / d


I cant change these values going into the formula. When I do this in the calculater it comes out to -319589 which should fit into an integer. I need a way to make this formula work. help

[Adron]

When given the values (all intagers)

a=650820
b=4974863
c=41
d=165

I get an overflow error on this line

Code Select Expand


Dim myInt As Integer = ((a - b) * 500) / c / d


I cant change these values going into the formula. When I do this in the calculater it comes out to -319589 which should fit into an integer. I need a way to make this formula work. help

I'm not sure about VB.NET, but in regular VB, an integer is only 16 bits. It has a range from -32768 to 32767.

[Zeller]

The Int64 value type represents integers with values ranging from negative 9,223,372,036,854,775,808 through positive 9,223,372,036,854,775,807


Even when I made myInt an 64-bit integer I still got on overflow error.

EDIT: I fixed the problem by making a and b int64's. I wonder why that fixed it though  

[Adron]

The Int64 value type represents integers with values ranging from negative 9,223,372,036,854,775,808 through positive 9,223,372,036,854,775,807


Even when I made myInt an 64-bit integer I still got on overflow error.

EDIT: I fixed the problem by making a and b int64's. I wonder why that fixed it though  

The intermediate results of the calculation have to be in range too.

(a - b) * 500 = 2162021500 > 2147483647 > 32767

2147483647 is the limit for a 32-bit signed integer and 32767 is the limit for a 16-bit signed integer. Your value is just above the limit for 32-bit signed. If you want to an exact integer calculation, you need either unsigned 32-bit integers or signed 64-bit integers.

[K]

I don't think it would be wise to use unsigned 32bit integers here, since we're subtracting b from a where b > a.

[Adron]

Oh, right. Nothing to stop him from swapping them and implying the sign though.

[MyndFyre]

The problem is due to the fact that the middle operation (mult. by 500) results in an overflow on the Int32 type (which is the underlying type mapped to by Integer).

-2162021500

The valid range for Int32 is:

-2147483648 to 2147483647

Use the type Long.

Alternatively, you could calculate (a - b) * (500 / c / d).  However, since you have rounding, you'll probably come up with 0.  I suggest using type Decimal if you want to approach it this way.