51

Everything in Python is an object. So the size of an int in Python will be larger than usual.

>>> sys.getsizeof(int())
24

OK, but why does it take 12 more bytes for 2⁶³ compared too 2⁶³ - 1 and not just one?

>>> sys.getsizeof(2**63)
36
>>> sys.getsizeof(2**62)
24

I get that 2⁶³ is a long and 2⁶³-1 an int, but why 12 bytes of difference?

No more intuitive, I tried some other things:

>>> a = 2**63
>>> a -= 2**62
>>> sys.getsizeof(a)
36

a is still stored as a long even if it could be in an int now. So that's not surprising. But:

>>> a -= (2**63 - 1)
>>> a = 2**63
>>> a -= (2**63 - 1)
>>> a
1L
>>> sys.getsizeof(a)
28

A new size.

>>> a = 2**63
>>> a -= 2**63
>>> a
0L
>>> sys.getsizeof(a)
24

Back to 24 bytes, but still with a long.

Last thing I got:

>>> sys.getsizeof(long())
24

Question:

How does the memory storage work in those scenarios?

Sub-questions:

Why is there a gap of 12 bytes to add what our intuition tells us is just 1 bit?

Why are int() and long() 24 bytes, but long(1) is already 28 bytes and int(2⁶²)?

NB: Python 3.X is working a bit differently, but not more intuitively. Here I focused on Python 2.7; I did not test on prior versions.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
T.Nel
  • 1,540
  • 2
  • 18
  • 34
  • 5
    To one of those questions: Same reason `2.5 - 1.5` is a float, and not an int. Result is always the more complex type of the involved operands. In the case of `2**63 - 2**62` or any calculation invloving `int` and `long` the resulting type will be `long` – user2390182 Mar 07 '18 at 14:08
  • 1
    Very related if not dupe: [Understanding memory allocation for large integers in Python](https://stackoverflow.com/questions/40344159/understanding-memory-allocation-for-large-integers-in-python) – Dimitris Fasarakis Hilliard Mar 07 '18 at 19:17
  • 3
    @NH. Shame about the formatting of your comment. :( – Draco18s no longer trusts SE Mar 07 '18 at 20:59
  • @JimFasarakisHilliard: maybe a better dup target would be [Why do ints require three times as much memory in Python?](https://stackoverflow.com/questions/23016610/why-do-ints-require-three-times-as-much-memory-in-python), which has an answer showing the C source for the python interpreter's data structures. – Peter Cordes Mar 08 '18 at 05:42
  • OP, you seem to be confusing (2^63) - 1 and 2^(63-1). Or at least after mentioning 2⁶³ - 1 you then work with 2\*\*62, which is **not** the same thing. – NH. Mar 08 '18 at 16:21
  • @A.I.Breveleri, but now that I deleted my comment (to make the new one), you and Draco have to delete yours... – NH. Mar 08 '18 at 16:22

2 Answers2

62

why does it get 12 more bytes for 2⁶³ compared too 2⁶³ - 1 and not just one?

On an LP64 system1, a Python 2 int consists of exactly three pointer-sized pieces:

  • type pointer
  • reference count
  • actual value, a C long int

That's 24 bytes in total. On the other hand, a Python long consists of:

  • type pointer
  • reference count
  • digit count, a pointer-sized integer
  • inline array of value digits, each holding 30 bits of value, but stored in 32-bit units (one of the unused bits gets used for efficient carry/borrow during addition and subtraction)

2**63 requires 64 bits to store, so it fits in three 30-bit digits. Since each digit is 4 bytes wide, the whole Python long will take 24+3*4 = 36 bytes.

In other words, the difference comes from long having to separately store the size of the number (8 additional bytes) and from it being slightly less space-efficient about storing the value (12 bytes to store the digits of 2**63). Including the size, the value 2**63 in a long occupies 20 bytes. Comparing that to the 8 bytes occupied by any value of the simple int yields the observed 12-byte difference.

It is worth noting that Python 3 only has one integer type, called int, which is variable-width, and implemented the same way as Python 2 long.

1 64-bit Windows differs in that it retains a 32-bit long int, presumably for source compatibility with a large body of older code that used char, short, and long as "convenient" aliases for 8, 16, and 32-bit values that happened to work on both 16 and 32-bit systems. To get an actual 64-bit type on x86-64 Windows, one must use __int64 or (on newer compiler versions) long long or int64_t. Since Python 2 internally depends on Python int fitting into a C long in various places, sys.maxint remains 2**31-1, even on 64-bit Windows. This quirk is also fixed in Python 3, which has no concept of maxint.
user4815162342
  • 141,790
  • 18
  • 296
  • 355
  • Is it correct that the 30-bit digits (4 bytes each) is exclusive to 64 bit systems? Would 32 bit systems use 15 bits, each taking up 2 bytes of memory? – user3483203 Mar 07 '18 at 14:16
  • 3
    @chrisz You should treat that as an implementation detail; the only way to tell is to look at the source of the interpreter/compiler in question and see what it does. – chepner Mar 07 '18 at 14:18
  • @chrisz Not necessarily, since 64-bit types and instructions are available on many 32-bit systems. As chepner said, it's an implementation details; but if you're curious, you can consult the source code. – user4815162342 Mar 07 '18 at 14:19
  • Where can the data types implementation be found? – CIsForCookies Mar 07 '18 at 14:28
  • 4
    @CIsForCookies Take a look at the [headers](https://github.com/python/cpython/blob/master/Include/longintrepr.h) and the [implementation](https://github.com/python/cpython/blob/master/Objects/longobject.c). – user4815162342 Mar 07 '18 at 14:31
  • 1
    @CIsForCookies: See also [Why do ints require three times as much memory in Python?](https://stackoverflow.com/questions/23016610/why-do-ints-require-three-times-as-much-memory-in-python) for a look at some of the internals. Although I still haven't found why Python only uses 30 bits of each "digit", and whether the digits are a base 2^30 or base-10^9 representation. Either way requires manual carry propagation that can't compile to `add`/`adc`, but rather `add` / `cmp` / `adc` or similar. (e.g. [like this extended-precision asm Fibonacci](https://codegolf.stackexchange.com/a/135618/30206)) – Peter Cordes Mar 08 '18 at 05:46
  • @PeterCordes What makes you think the base is 10^9? `_PyLong_BASE` is defined as `1 << PyLong_SHIFT` (`PyLong_SHIFT` being 15 or 30), so it's definitely a power of 2. If you're referring to `_PyLong_DECIMAL_BASE`, that is only used for efficient conversion to decimal. – user4815162342 Mar 08 '18 at 07:10
  • @user4815162342: thanks, I saw the decimal stuff and assumed it was always used. The comments say the value of a number is `SUM(for i=0 through abs(ob_size)-1) ob_digit[i] * 2**(SHIFT*i)`, so you're right; it's base 2^30. I don't understand why they wouldn't use all 32 bits, though, if using a power-of-2 radix. That enables significantly more efficient arbitrary-precision operations, because the CPU generates carry-out on its own, so a chain of `add` / `adc` / `adc` ... works. (That's the other reason I assumed decimal.) It's clunky to express in C, though, and compilers aren't great. – Peter Cordes Mar 08 '18 at 07:17
  • 1
    @PeterCordes Retaining a bit for carry is a fairly standard trick for efficient addition of bignums in portable C. It's true that it doesn't compile to assembler that would use the processor's own carry flag, but it does lead to much faster code than the alternative, which is to precede each digit addition/subtractions with comparisons and branches. Instead, carry is simply obtained as [`sum >> PyLong_SHIFT`](https://github.com/python/cpython/blob/dd431b32f4a599fff9c9cddfe9d48cc66b347481/Objects/longobject.c#L2995). – user4815162342 Mar 08 '18 at 07:18
  • @user4815162342 that makes some sense if they decided not to use http://gmplib.org/ (asm optimized) functions for numbers that don't fit in a single `long`. But using `sum >> shift` doesn't compile much more efficiently than `sum = a+b; carry_out = sum – Peter Cordes Mar 08 '18 at 07:23
  • @PeterCordes They don't use gmplib because Python is as old as (or even older tnan) gmplib, and possibly also because Python has specific allocation requirements. I remember reports of some people trying gmplib and not getting speed increases, except on extremely large numbers. For that use case there is [a dedicated module](https://pypi.python.org/pypi/gmpy). – user4815162342 Mar 08 '18 at 07:31
  • @PeterCordes Even if branch prediction isn't involved, it's not obvious that shift+mask (with constants) would be slower than the comparison to a non-constant. But I guess that can be easily checked. – user4815162342 Mar 08 '18 at 07:32
  • @user4815162342: Ah, I'm not too surprised that the extra overhead of a function call isn't worth it, even to [GMP's lightweight `mpn` functions](https://gmplib.org/manual/Low_002dlevel-Functions.html) that don't do any memory allocation. The functions have lots of pointer args that the compiler has to put in registers instead of accessing with whatever addressing mode, and even just a function call at all clobbers registers. – Peter Cordes Mar 08 '18 at 07:35
  • 2
    @user48: Agreed that it's not *obvious*, but remember that the non-constant was an operand for `add`, so it's typically still in a register. Turns out I actually already wrote an SO answer about [getting compilers to generate `adc`](https://stackoverflow.com/questions/46701364/how-to-access-the-carry-flag-while-adding-two-64-bit-numbers-using-asm-in-c). It's worse than I was remembering, and only clang actually used `adc` at all. Also, **[`sum – Peter Cordes Mar 08 '18 at 08:00
  • 1
    So TL;DR: the current shift design with base 2^30 is probably best if they aren't going to use (inline) asm or intrinsics. Although base 2^60 or 2^63 would be twice as fast on 64-bit CPUs. IDK how common it is for Python integers to be too large for a `long`, but only needing 90 bits or less. – Peter Cordes Mar 08 '18 at 08:04
5

While I didn't find it in the documentation, here is my explanation.

Python 2 promotes int to long implicitly, when the value exceeds the value that can be stored in int. The size of the new type (long) is the default size of long, which is 32. From now on, the size of your variable, will be determined by its value, which can go up and down.

from sys import getsizeof as size
a = 1
n = 32

# going up
for i in range(10):
    if not i:
        print 'a = %100s%13s%4s' % (str(a), type(a), size(a))
    else:
        print 'a = %100s%14s%3s' % (str(a), type(a), size(a))
    a <<= n

# going down
for i in range(11):
    print 'a = %100s%14s%3s' % (str(a), type(a), size(a))
    a >>= n


a =                                                                                                    1 <type 'int'>  24
a =                                                                                           4294967296 <type 'long'> 32
a =                                                                                 18446744073709551616 <type 'long'> 36
a =                                                                        79228162514264337593543950336 <type 'long'> 40
a =                                                              340282366920938463463374607431768211456 <type 'long'> 44
a =                                                    1461501637330902918203684832716283019655932542976 <type 'long'> 48
a =                                           6277101735386680763835789423207666416102355444464034512896 <type 'long'> 52
a =                                 26959946667150639794667015087019630673637144422540572481103610249216 <type 'long'> 56
a =                       115792089237316195423570985008687907853269984665640564039457584007913129639936 <type 'long'> 60
a =              497323236409786642155382248146820840100456150797347717440463976893159497012533375533056 <type 'long'> 64
a =    2135987035920910082395021706169552114602704522356652769947041607822219725780640550022962086936576 <type 'long'> 68
a =              497323236409786642155382248146820840100456150797347717440463976893159497012533375533056 <type 'long'> 64
a =                       115792089237316195423570985008687907853269984665640564039457584007913129639936 <type 'long'> 60
a =                                 26959946667150639794667015087019630673637144422540572481103610249216 <type 'long'> 56
a =                                           6277101735386680763835789423207666416102355444464034512896 <type 'long'> 52
a =                                                    1461501637330902918203684832716283019655932542976 <type 'long'> 48
a =                                                              340282366920938463463374607431768211456 <type 'long'> 44
a =                                                                        79228162514264337593543950336 <type 'long'> 40
a =                                                                                 18446744073709551616 <type 'long'> 36
a =                                                                                           4294967296 <type 'long'> 32
a =                                                                                                    1 <type 'long'> 28

As you can see, the type stays long after it first became too big for an int, and the initial size was 32, but the size changes with the value (can be higher or lower [or equal, obviously] to 32)

So, to answer your question, the base size is 24 for int, and 28 for long, while long has also the space for saving large values (which starts as 4 bytes - hence 32 bytes for long, but can go up and down according to the value)

As for your sub-question, creating a unique type (with a unique size) for a new number is impossible, so Python has "sub classes" of long type, which deal with a range of numbers, therefore, once you over the limit of your old long you must use the newer, which accounts for much larger numbers too, therefore, it has a few bytes more.

CIsForCookies
  • 12,097
  • 11
  • 59
  • 124
  • The last paragraph is incorrect, or at least imprecise. There are no (Python-level) subclasses of `long`, just like there are no subclasses of `str` and `tuple`. `long` is simply one of those types whose each instance stores its size, much like `str`. `long` doesn't expose its size in `__len__`, but it's there nonetheless. The C API calls those [`PyVarObject`](https://docs.python.org/2/c-api/structures.html#c.PyVarObject). – user4815162342 Mar 07 '18 at 15:30
  • Yeah, that's what I meant. I just didn't know how to say that so I used sub-class with `"`... – CIsForCookies Mar 07 '18 at 15:52