Integer overflow. The result of an integer operation caused the most significant bit of the result to carry.


Short explanation:

If you use fixed precision datatypes (smallint, integer, bigint, decimal and numeric), it is possible that the result of calculation doesn't fit the datatype. Try casting the values in complex expressions as double precision and see whether the error goes away. If it works and you don't care about being too precise, you can leave it at that. Otherwise you need to check every operation and calculate the result.

Details:

Here's an example: if you multiply 9.12 with 8.11 (both numeric(18,2)) you would get 73.9632. If Firebird would store that into numeric(18,2) datatype, we would lose 0.0032. Doesn't look much, but when you have complex calculations, you can easily loose thousands (dollars or euros). Therefore, the result is stored in numeric(18,4).

Problems are rarely seen with such low precision as 2. Let's use some bigger precision. For example, numeric(18,6) times numeric(18,6) yields numeric(18,12) result, meaning that maximal value it can store is 9223372.036854775807. If (for example) you wish to keep only 6 digits of precision, you could use something like:

cast(value1 as numeric(18,3)) * cast(value2 as numeric(18,3))

which would yield numeric(18,6) result, but it is quite possible that you would get more accurate result by casting to double:

cast(cast(value1 as double precision) * cast(value2 as double precision) as numeric(18,6))


Also, if you have mixed multiplications and divisions it helps to change the order of operations, so that the overflow doesn't happen.


Do you find this FAQ incorrect or incomplete? Please e-mail us what needs to be changed. To ensure quality, each change is checked by our editors (and often tested on live Firebird databases), before it enters the main FAQ database. If you desire so, the changes will be credited to your name. To learn more, visit our add content page.



All contents are copyright © 2007-2024 FirebirdFAQ.org unless otherwise stated in the text.


Links   Firebird   News   FlameRobin   Powered by FB: Home Inventory   Euchre  
Add content   About  

Categories
 Newbies
 SQL
 Installation and setup
 Backup and restore
 Performance
 Security
 Connectivity and API
 HOWTOs
 Errors and error codes
 Miscellaneous