281

I have the following code:

r = numpy.zeros(shape = (width, height, 9))

It creates a width x height x 9 matrix filled with zeros. Instead, I'd like to know if there's a function or way to initialize them instead to NaNs in an easy way.

ibodi
  • 1,543
  • 3
  • 21
  • 40
devoured elysium
  • 101,373
  • 131
  • 340
  • 557
  • 2
    One caveat is that NumPy doesn't have an integer NA value (unlike R). See [pandas list of gotchas](http://pandas.pydata.org/pandas-docs/stable/gotchas.html). Hence `np.nan` goes wrong when converted to int. – smci Jul 28 '13 at 03:31
  • smci is right. For NumPy there is no such NaN value. So it depends on the type and on NumPy which value will be there for NaN. If you are not aware of this, it will cause troubles – MasterControlProgram Nov 04 '16 at 15:38
  • It would seem like there is scope for a np.nans function to mimic np.zeros and np.ones in fact, but I suppose np.full is a generalization that precludes the need for all the specialized functions. Nice question. – ClimateUnboxed Jan 21 '22 at 11:25

11 Answers11

390

You rarely need loops for vector operations in numpy. You can create an uninitialized array and assign to all entries at once:

>>> a = numpy.empty((3,3,))
>>> a[:] = numpy.nan
>>> a
array([[ NaN,  NaN,  NaN],
       [ NaN,  NaN,  NaN],
       [ NaN,  NaN,  NaN]])

I have timed the alternatives a[:] = numpy.nan here and a.fill(numpy.nan) as posted by Blaenk:

$ python -mtimeit "import numpy as np; a = np.empty((100,100));" "a.fill(np.nan)"
10000 loops, best of 3: 54.3 usec per loop
$ python -mtimeit "import numpy as np; a = np.empty((100,100));" "a[:] = np.nan" 
10000 loops, best of 3: 88.8 usec per loop

The timings show a preference for ndarray.fill(..) as the faster alternative. OTOH, I like numpy's convenience implementation where you can assign values to whole slices at the time, the code's intention is very clear.

Note that ndarray.fill performs its operation in-place, so numpy.empty((3,3,)).fill(numpy.nan) will instead return None.

Princy
  • 333
  • 3
  • 11
u0b34a0f6ae
  • 48,117
  • 14
  • 92
  • 101
  • 10
    I agree that your code's intention is clearer. But thanks for the unbiased timings (or rather, the fact that you still posted them), I appreciate it :) – Jorge Israel Peña Nov 10 '09 at 07:19
  • 12
    I like this one: `a = numpy.empty((3, 3,)) * numpy.nan`. It timed faster than `fill` but slower than the assignment method, but it is a oneliner!! – heltonbiker Apr 30 '12 at 14:09
  • 2
    Please look at this answer: http://stackoverflow.com/questions/10871220/making-a-matrix-square-and-padding-it-with-desired-value-in-numpy – Ivan Jun 03 '12 at 15:15
  • 4
    I prefer the `.fill()` method, but the difference in speeds reduces to practically nothing as the arrays get larger. – naught101 Mar 24 '14 at 11:13
  • Why not combining the two? `np.empty([2, 5]).fill(np.nan)`? This returns `None`, and I'm not sure why... – Atcold Apr 23 '16 at 04:51
  • 4
    ... because `np.empty([2, 5])` creates an array, then `fill()` modifies that array in-place, but does not return a copy or a reference. If you want to call `np.empty(2, 5)` by a name ("assign is to a variable"), you have to do so before you do in-place operations on it. Same kinda thing happens if you do `[1, 2, 3].insert(1, 4)`. The list is created and a 4 is inserted, but it is impossible to get a reference to the list (and thus it can be assumed to have been garbage collected). On immutable data like strings, a copy is returned, because you can't operate in-place. Pandas can do both. – flutefreak7 Jun 02 '16 at 21:26
  • Be aware of NaN in NumPy. There is no such number. So, not checking it can mess it up. – MasterControlProgram Nov 04 '16 at 15:41
273

Another option is to use numpy.full, an option available in NumPy 1.8+

a = np.full([height, width, 9], np.nan)

This is pretty flexible and you can fill it with any other number that you want.

datu-puti
  • 1,306
  • 14
  • 33
Pietro Biroli
  • 2,866
  • 1
  • 13
  • 8
  • 22
    I'd consider this as the *most correct* answer since it is eactly what `full` is meant for. `np.empy((x,y))*np.nan` is a good runner-up (and compatibility for old versions of numpy). – travc Sep 21 '15 at 19:38
  • 1
    this is slower that `fill` ```python -mtimeit "import numpy as np; a = np.empty((100,100));" "a.fill(np.nan)" 100000 loops, best of 3: 13.3 usec per loop python -mtimeit "import numpy as np; a = np.full((100,100), np.nan);" 100000 loops, best of 3: 18.5 usec per loop``` – Farnabaz Oct 19 '16 at 13:29
  • 7
    @Farnabaz If you put the equivalent code insiding the timing loop they are about the same. The two methods are basically equal, you've just got the "np.empty" outside the timer in the first one. `python -mtimeit "import numpy as np; a = np.empty((1000,1000)); a.fill(np.nan)" 1000 loops, best of 3: 381 usec per loop $ python -mtimeit "import numpy as np; a = np.full((1000,1000), np.nan);" 1000 loops, best of 3: 383 usec per loop` – Scott Staniewicz Oct 28 '18 at 01:35
81

I compared the suggested alternatives for speed and found that, for large enough vectors/matrices to fill, all alternatives except val * ones and array(n * [val]) are equally fast.

enter image description here

Code to reproduce the plot:

import numpy
import perfplot

val = 42.0


def fill(n):
    a = numpy.empty(n)
    a.fill(val)
    return a


def colon(n):
    a = numpy.empty(n)
    a[:] = val
    return a


def full(n):
    return numpy.full(n, val)


def ones_times(n):
    return val * numpy.ones(n)


def list(n):
    return numpy.array(n * [val])


b = perfplot.bench(
    setup=lambda n: n,
    kernels=[fill, colon, full, ones_times, list],
    n_range=[2 ** k for k in range(20)],
    xlabel="len(a)",
)
b.save("out.png")
Nico Schlömer
  • 53,797
  • 27
  • 201
  • 249
  • 1
    Strange that `numpy.full(n, val)` is slower than `a = numpy.empty(n) .. a.fill(val)` since it does the same thing internally – endolith Aug 03 '19 at 17:38
  • I did the same experient and got roughly the same result as Nico. I use MacBook Pro (16-inch, 2019), 2.6 GHz 6-Core Intel Core i7. Strange that full is slower then fill. – halfmoonhalf Sep 03 '22 at 18:21
27

Are you familiar with numpy.nan?

You can create your own method such as:

def nans(shape, dtype=float):
    a = numpy.empty(shape, dtype)
    a.fill(numpy.nan)
    return a

Then

nans([3,4])

would output

array([[ NaN,  NaN,  NaN,  NaN],
       [ NaN,  NaN,  NaN,  NaN],
       [ NaN,  NaN,  NaN,  NaN]])

I found this code in a mailing list thread.

Jorge Israel Peña
  • 36,800
  • 16
  • 93
  • 123
  • 1
    Seems like overkill. – Mad Physicist Jul 19 '16 at 16:42
  • 2
    @MadPhysicist That depends entirely on your situation. If you have to initialize only one single NaN array, then yes, a custom function is probably overkill. However if you have to initialize a NaN array at dozens of places in your code, then having this function becomes quite convenient. – Xukrao Sep 28 '18 at 14:47
  • 1
    @Xukaro. Not really, given that a more flexible and efficient version of such a function already exists and is mentioned in multiple other answers. – Mad Physicist Sep 28 '18 at 15:20
13

You can always use multiplication if you don't immediately recall the .empty or .full methods:

>>> np.nan * np.ones(shape=(3,2))
array([[ nan,  nan],
       [ nan,  nan],
       [ nan,  nan]])

Of course it works with any other numerical value as well:

>>> 42 * np.ones(shape=(3,2))
array([[ 42,  42],
       [ 42,  42],
       [ 42, 42]])

But the @u0b34a0f6ae's accepted answer is 3x faster (CPU cycles, not brain cycles to remember numpy syntax ;):

$ python -mtimeit "import numpy as np; X = np.empty((100,100));" "X[:] = np.nan;"
100000 loops, best of 3: 8.9 usec per loop
(predict)laneh@predict:~/src/predict/predict/webapp$ master
$ python -mtimeit "import numpy as np; X = np.ones((100,100));" "X *= np.nan;"
10000 loops, best of 3: 24.9 usec per loop
Community
  • 1
  • 1
hobs
  • 18,473
  • 10
  • 83
  • 106
8

Yet another possibility not yet mentioned here is to use NumPy tile:

a = numpy.tile(numpy.nan, (3, 3))

Also gives

array([[ NaN,  NaN,  NaN],
       [ NaN,  NaN,  NaN],
       [ NaN,  NaN,  NaN]])

update: I did a speed comparison, and it's not very fast :/ It's slower than the ones_times by a decimal.

JHBonarius
  • 10,824
  • 3
  • 22
  • 41
6

As said, numpy.empty() is the way to go. However, for objects, fill() might not do exactly what you think it does:

In[36]: a = numpy.empty(5,dtype=object)
In[37]: a.fill([])
In[38]: a
Out[38]: array([[], [], [], [], []], dtype=object)
In[39]: a[0].append(4)
In[40]: a
Out[40]: array([[4], [4], [4], [4], [4]], dtype=object)

One way around can be e.g.:

In[41]: a = numpy.empty(5,dtype=object)
In[42]: a[:]= [ [] for x in range(5)]
In[43]: a[0].append(4)
In[44]: a
Out[44]: array([[4], [], [], [], []], dtype=object)
Mad Physicist
  • 107,652
  • 25
  • 181
  • 264
ntg
  • 12,950
  • 7
  • 74
  • 95
  • Aside from having virtually nothing to do with the original question, neat. – Mad Physicist Jul 19 '16 at 16:44
  • 1
    Well, It's about "Initializing numpy matrix to something other than zero or one", in the case "something other" is an object :) (More practically, google led me here for initializing with an empty list ) – ntg Aug 28 '17 at 01:32
6

Another alternative is numpy.broadcast_to(val,n) which returns in constant time regardless of the size and is also the most memory efficient (it returns a view of the repeated element). The caveat is that the returned value is read-only.

Below is a comparison of the performances of all the other methods that have been proposed using the same benchmark as in Nico Schlömer's answer.

enter image description here

Giancarlo Sportelli
  • 1,219
  • 1
  • 17
  • 20
3

Just a warning that initializing with np.empty() without subsequently editing the values can lead to (memory allocation?) problems:

arr1 = np.empty(96)
arr2 = np.empty(96)
print(arr1)
print(arr2)

# [nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan  1.  1.
#   1.  1.  2.  2.  2.  2. nan nan nan nan nan nan nan nan  0.  0.  0.  0.
#   0.  0.  0.  0. nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan]
#
# [nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan  1.  1.
#   1.  1.  2.  2.  2.  2. nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan]

The floats initialized in the array are used somewhere else in my script but are not associated with variables arr1 or arr2 at all. Spooky.

Answer from user @JHBonarius solved this problem:

arr = np.tile(np.nan, 96)
print(arr)

# [nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
#  nan nan nan nan nan nan]
Novorodnas
  • 85
  • 6
  • 2
    np.empty allocates memory for the array without overwriting the existing data in those bytes, so it will contain arbitrary (not random) data. – markemus May 18 '22 at 11:27
2
>>> width = 2
>>> height = 3

>>> r = np.full((width, height, 9), np.nan)

>>> print(r)

array([[[nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan]],

       [[nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan]]])

>>> r.shape
(2, 3, 9)
user19409802
  • 29
  • 1
1

Pardon my tardiness, but here is the fastest solution for large arrays, iff single-precision (f4 float32) is all you need. And yes, np.nan works as well.

def full_single_prec(n):
    return numpy.full(n, val, dtype='f4')

Perfplot Comparison Of All Approaches (Tested in 3.8.13)

Larry Panozzo
  • 191
  • 2
  • 7