What Is Integer Overflow

An integer overflow is a type of an arithmetic overflow error when the result of an integer operation does not fit within the allocated memory space. Instead of an error in the program, it usually causes the result to be unexpected. Integer overflows have been listed as the number 8 most dangerous software error in the most recent CWE 2019 list, mostly because they often lead to buffer overflows, which are currently the number 1 most dangerous software error according to that list.

How Integer Overflows Happen

In most programming languages, integer values are usually allocated a certain number of bits in memory. For example, space reserved for a 32-bit integer data type may store an unsigned integer between 0 and 4,294,967,295 or a signed integer between −2,147,483,648 and 2,147,483,647. In the case of signed integers, the most significant (first) bit usually signifies whether the integer is a positive value or a negative value.

However, what happens when you perform the calculation 4,294,967,295 + 1 and attempt to store the result that is greater than the maximum value for the integer type? It depends completely on the language and the compiler. And, unfortunately, most languages and most compilers raise no error at all and simply perform a modulo operation, wraparound, or truncation, or they have other undefined behavior. For the above example, the result is most often 0.

Results can be even more unexpected for signed integers. When you go above the maximum value of the signed integer, the result usually becomes a negative number. For example, 2,147,483,647 +1 is usually −2,147,483,648. When you go below the minimum value (underflow), the result usually becomes a positive number. For example, −2,147,483,648 − 1 is usually 2,147,483,647.

In addition to typical operations such as addition, subtraction, or multiplication, integer overflows may also happen due to typecasting. For example, one operation may treat an integer as an unsigned one and another operation may treat exactly the same integer as a signed one, therefore interpreting the value incorrectly.

Integer Overflow Risks

Most integer overflow conditions simply lead to erroneous program behavior but do not cause any vulnerabilities. However, in some cases, integer overflows may have severe consequences:

• If an integer overflow happens when you calculate the length of a buffer, you may end up with a buffer overflow. A buffer overflow lets the attacker gain shell access and attempt further privilege escalation.
• If an integer overflow happens during financial calculations, it may, for example, result in the customer receiving credit instead of paying for a purchase or may cause a negative account balance to become positive.

An excellent example of an integer overflow that leads to a buffer overflow can be found in an older version of OpenSSH (3.3):

nresp = packet_get_int();
if (nresp > 0) {
 response = xmalloc(nresp*sizeof(char*));
 for (i = 0; i < nresp; i++)
  response[i] = packet_get_string(NULL);
} 

If nresp is 1073741824 and sizeof(char*) is 4 (which is typical), then nresp*sizeof(char*) results in an overflow. Therefore, xmalloc() receives and allocates a 0-byte buffer. The subsequent loop causes a heap buffer overflow, which may, in turn, be used by an attacker to execute arbitrary code.

Preventing Integer Overflows

The biggest issue with even the most basic integer overflows is that they are very hard to discover and prevent. There is no error, there is no warning, you simply get a wrong result of the operation. The only way to discover them is to examine the operands before the operation or examine the result after (for example, checking whether the addition result for two positive numbers is smaller than the operands).

Depending on the language, you may be able to come across libraries or mechanisms that help you prevent and discover integer overflows. In the case of the GCC compiler, there are built-in functions that check for integer overflows. In the case of C++ programs, there is a library called SafeInt that performs safe operations. Unfortunately, enforcing the use of this library in all your code for all arithmetic operations may not be easy.

If you want to know more about integer overflows, we recommend having a look at the extensive Phrack article by blexim.

THE AUTHOR

Tomasz Andrzej Nidecki
Principal Cybersecurity Writer
LinkedIn

Tomasz Andrzej Nidecki (also known as tonid) is a Primary Cybersecurity Writer at Invicti, focusing on Acunetix. A journalist, translator, and technical writer with 25 years of IT experience, Tomasz has been the Managing Editor of the hakin9 IT Security magazine in its early years and used to run a major technical blog dedicated to email security.