0

I'm currently testing some things with tensorflow and I built a little neural network. To train it, I'm repeatedly running the session on the optimizer object:

error = 0.5 * tf.square(tf.subtract(output_layer, supervised_layer))
optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(error)

session.run(tf.global_variables_initializer())

for i in range(1000):
    session.run([optimizer, error], feed_dict={
        input_layer: [[0, 1], [1, 0], [0, 0], [1, 1]],
        supervised_layer: [[1], [1], [0], [0]]
    })

I feel like calling session.run() very often from python doesn't make good use of the way tensorflow works and probably can't use GPU processing advantages.

So my question now is quite simple: Is there any overhead on session.run() and is there a better way for running a session very often.

Ian Rehwinkel
  • 2,486
  • 5
  • 22
  • 56

2 Answers2

1

I see a problem when using sess.run() and not using the close() afterwords in general case.

A session may own resources, such as tf.Variable and it is important to release these resources when they are no longer required. To do this, either invoke the tf.Session.close() method on the session, or use the session as a context manager.

The following two examples are equivalent:

# Using the `close()` method.
sess = tf.Session()
sess.run(...)
sess.close()

# Using the context manager.
with tf.Session() as sess:
sess.run(...)

On calling the session.run() to often...

The Sessionclass is for running TensorFlow operations and the session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.

If you can achieve your computations without creating the environment you can say it would be smarter to go without the session.run().

Note: Session.run() method runs one "step" of TensorFlow computation, by running the necessary graph fragment to execute every Operation and evaluate every Tensor in fetches, substituting the values in feed_dict for the corresponding input values.

The fetches argument may be a single graph element, or an arbitrarily nested list, tuple, namedtuple, dict, or OrderedDict containing graph elements at its leaves. A graph element can be one of the following types:

  • An tf.Operation. The corresponding fetched value will be None.
  • A tf.Tensor. The corresponding fetched value will be a numpy ndarray containing the value of that tensor.
  • A tf.SparseTensor. The corresponding fetched value will be a tf.SparseTensorValue containing the value of that sparse tensor.
  • A get_tensor_handle op. The corresponding fetched value will be a numpy ndarray containing the handle of that tensor.
  • A string which is the name of a tensor or operation in the graph.

Update: Interactive session will not help you either. The only difference comparing to the normal Session, InteractiveSession makes itself the default session so you can call run() or eval() on variables explicitly using the session object.

#start the session
sess = tf.InteractiveSession()

# stop the session
sess.stop()
ops.reset_default_graph()
prosti
  • 42,291
  • 14
  • 186
  • 151
  • Wouldn't `session.close()` reset the values stored in all instances of `tf.Variable`? If so, I would lose all training progress from the last steps and that would be totally counter-productive. – Ian Rehwinkel Dec 23 '18 at 19:13
  • Yes, `session.close()` will free all resources associated with the session. – prosti Dec 23 '18 at 19:26
  • So from what you said in your answer, I assume there is no way to run a session several times with one tensorflow function call? – Ian Rehwinkel Dec 23 '18 at 21:09
  • Yes, as defined inside c_api.cc https://github.com/tensorflow/tensorflow/blob/ec6df8e4fb10b25737295bcb49791842eb478400/tensorflow/c/c_api.cc#L2600 – prosti Dec 23 '18 at 21:27
1

In a typical scenario a single session.run takes up to several seconds so the cost of executing the python loop is negligible. So the short answer is that you should not worry, because you will certainly benefit from GPU acceleration when you need it. You already perform operations in batches. I would suggest to simply increase the batch size.

However, if you were absolutely certain that you need to perform multiple updates in one session.run call there are ways of implementing a loop in tensorflow (tf.while_loop, autograph). So you can actually call arbitrary code multiple times within a single session.run. Be advised that it is quite difficult in general, but in most practical cases you could find simpler solutions, e.g. for RNNs.

pkubik
  • 780
  • 6
  • 19