-2

As opposed to the queue.Queue the multiprocessing.Queue does not seem to have the maxsize attribute. As I try to get the maxsize by calling queue_object.maxsize I get the following exception: AttributeError: 'Queue' object has no attribute 'maxsize'.

Is this an implementation fault on my side or does it actually not exist? And if it does not exists is there another way to get this information?

How to reproduce:

Python 3.7.6 (v3.7.6:43364a7ae0, Dec 18 2019, 14:18:50) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> q = Queue(maxsize=10)
>>> q.maxsize
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Queue' object has no attribute 'maxsize'

EDIT: As explained in the answers below this attribute is not supported, which leads me to the question why this is the case? I believe the multiprocessing.Queue is based on the queue.Queue which does support the attribute (which led to the confusion in the first place).

Tom Stock
  • 1,098
  • 1
  • 12
  • 26
  • I'm actually wondering why you would want to get the maxsize. What is the use case for this? A queue is generally something where one process puts things in, and another process gets it out of it. The size is irrelevant at these places, especially considering that a process cannot reliably determine how many elements are in it (and how much space is left). – wovano Aug 18 '20 at 09:09
  • @wovano The maxsize of the queues are configurable in my use case, therefore it could change from time to time. I am not using the maxsize attribute now in the rest of the application, however it could be useful in autoscaling different modules within a memory constraint. E.g. add an additional module in a pipeline with the same input and output if you signal that the output queue is not always at X capacity. – Tom Stock Aug 18 '20 at 09:18

2 Answers2

2

multiprocessing.Queue replicates the documented API of queue.Queue. It makes no effort to replicate the implementation details.

The maxsize attribute of a queue.Queue instance is an undocumented implementation detail. The lack of a preceding underscore doesn't make it public. Users were never intended to interact with this attribute. There is no guarantee that it will exist in all Python implementations, or that it will exist in future releases. (For example, if the queue module ever gets a C implementation, this attribute is quite likely to go away.)

The intended way to get the max size of a multiprocessing.Queue is to keep track of the value you passed when the object was created. If you're determined to get it out of the queue object directly and you're fine with accessing implementation details (as you did with maxsize in queue.Queue), you can access the private attribute the current implementation uses:

queue._maxsize

There is no guarantee this will work in future versions or other Python implementations.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • 1
    Hmm, I guess that makes sense though in my opinion it would be a handy addition to keep track of it in the Queue object instead of having to manually track it (as it is used over multiple different processes). – Tom Stock Aug 18 '20 at 09:02
0

I have found a workaround, as can be seen in the source code of the multiprocessing, the attribute is set as a private attribute. So in order to acces it one could simply do queue_object._maxsize. This is not an elegant solution though and I wonder why it is implemented this way.

Python 3.7.6 (v3.7.6:43364a7ae0, Dec 18 2019, 14:18:50) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> q = Queue(maxsize=10)
>>> q._maxsize
10
Tom Stock
  • 1,098
  • 1
  • 12
  • 26
  • 1
    It's implemented this way because they don't want you to do this. There's no guarantee that this will work after the next update. Why do you want to read it anyway? You've set the size to 10 yourself, so you know it's 10. – wovano Aug 18 '20 at 08:49
  • As I use multiple queues over different processes it is nice to not have to pass all that information multiple times (as it is present in the object anyways). I can live without it but was wondering why this was not supported and why the implementation differed between the multiprocessing and normal queue. – Tom Stock Aug 18 '20 at 09:05