Useful or not, from you.
tensorflow NotImplementedError: Cannot convert a symbolic Tensor (args_0:0) to a numpy array.

System information

  • This is custom code
  • Running Google Colab on Mac
  • TensorFlow version 2.3.0
  • Python version 3.6.9
  • XLA_GPU hosted by Colab; memory_limit = 15695549568

Describe the current behavior I have a custom augmentation function written in Colab that worked normally until today. The last time I ran the entire code through was 09/08/2020 and the augmentation functioned performed the operations normally. Now, I receive an error that pertains to symbolic tensors, which I have never seen before.

Describe the expected behavior Augmentation function is meant to map over a batch of images and masks, taking in samples one at a time, converting them to NumPy arrays, performing some sort of augmentation, then returning them back as tensors.

Standalone code to reproduce the issue A link to the Colab notebook: https://colab.research.google.com/drive/1ujShezPjcveG2kg019c7zqweLtu1ESuW?usp=sharing A link to the training data I use in the notebook: https://drive.google.com/drive/folders/1ZS1wKjo692Lg7Vuhtr3akz0sYSOmZBKl

The particular section of code where the error stems from:

# Function used to perform "on-the-fly" augmentation during training.
# UPDATED ON 09/21/2020.

def augmentation(img, msk):

  # Call in skimage package, which will be used for transformations.
  from skimage.transform import rotate, AffineTransform, warp
  
  # Create some random floats, which will be used in augmentation steps.
  tilt = tf.random.uniform(shape = [], minval = -90, maxval = 90, dtype = tf.float32)
  dx = tf.random.uniform(shape = [], minval = -20, maxval = 20, dtype = tf.float32)
  dy = tf.random.uniform(shape = [], minval = -20, maxval = 20, dtype = tf.float32)
  
  # Cast image and mask to numpy arrays.
  img = np.array(img)
  msk = np.array(msk)

  # Use TensforFlow-style if conditionals, used to flip image and mask.
  img = tf.cond(tilt > 0, lambda: np.fliplr(img), lambda: np.flipud(img))
  msk = tf.cond(tilt > 0, lambda: np.fliplr(msk), lambda: np.flipud(msk))

  # Rotate the image and mask to some degree.
  img = rotate(img, angle = tilt, mode = 'reflect')
  msk = rotate(msk, angle = tilt, mode = 'reflect')

  # Write the conditions for an affine transformation.
  transform = AffineTransform(translation = (dx,dy))

  # Perform the affine transformation.
  img = warp(img, inverse_map = transform, mode = 'reflect')
  msk = warp(msk, inverse_map = transform, mode = 'reflect')
 
  # Convert the inputs back into tensors, put back into a tuple.
  finalTuple = (tf.convert_to_tensor(img), tf.convert_to_tensor(msk))

  return finalTuple

# Callback for data augmentation.
class aug(tf.keras.callbacks.Callback):
  def on_training_batch_begin(self, batch, logs = None):
    batch.map(augmentation, num_parallel_calls = 5)
    batch.shuffle(10)
    
# Callback for CSV logger (used for charting).
csv = tf.keras.callbacks.CSVLogger(f'/content/gdrive/My Drive/{today}_metrics.csv', separator=',', append=False)

# Callback for saving the model.
save_model_path = f'/content/gdrive/My Drive/{today}_wellpad_model_.h5'
cp = tf.keras.callbacks.ModelCheckpoint(filepath=save_model_path,
                                        monitor='val_loss',
                                        mode='min',
                                        save_best_only=True)
# Test the augmentation function on a batch of images.
test_batch = evaluation.take(1)

test_batch = test_batch.map(augmentation,num_parallel_calls=5) # ERROR OCCURS HERE.

# For plotting below.
test_batch = [(image,mask) for image, mask in test_batch]

plt.figure(figsize=(10,10))
plt.subplot(1,2,1)
# Show the original image.
plt.imshow(test_batch[0][0][0])
# Show the masked image.
plt.subplot(1,2,2)
plt.imshow(tf.squeeze(test_batch[0][1][0]))

Other info / logs Here is the full error I receive:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-17-81c1086f76f0> in <module>()
      2 test_batch = evaluation.take(1)
      3 
----> 4 test_batch = test_batch.map(augmentation,num_parallel_calls=5)
      5 
      6 test_batch = [(image,mask) for image, mask in test_batch]

10 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in map(self, map_func, num_parallel_calls, deterministic)
   1700           num_parallel_calls,
   1701           deterministic,
-> 1702           preserve_cardinality=True)
   1703 
   1704   def flat_map(self, map_func):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, input_dataset, map_func, num_parallel_calls, deterministic, use_inter_op_parallelism, preserve_cardinality, use_legacy_function)
   4082         self._transformation_name(),
   4083         dataset=input_dataset,
-> 4084         use_legacy_function=use_legacy_function)
   4085     if deterministic is None:
   4086       self._deterministic = "default"

/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs)
   3369       with tracking.resource_tracker_scope(resource_tracker):
   3370         # TODO(b/141462134): Switch to using garbage collection.
-> 3371         self._function = wrapper_fn.get_concrete_function()
   3372         if add_to_graph:
   3373           self._function.add_to_graph(ops.get_default_graph())

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in get_concrete_function(self, *args, **kwargs)
   2937     """
   2938     graph_function = self._get_concrete_function_garbage_collected(
-> 2939         *args, **kwargs)
   2940     graph_function._garbage_collector.release()  # pylint: disable=protected-access
   2941     return graph_function

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _get_concrete_function_garbage_collected(self, *args, **kwargs)
   2904       args, kwargs = None, None
   2905     with self._lock:
-> 2906       graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
   2907       seen_names = set()
   2908       captured = object_identity.ObjectIdentitySet(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3211 
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3073             arg_names=arg_names,
   3074             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075             capture_by_value=self._capture_by_value),
   3076         self._function_attributes,
   3077         function_spec=self.function_spec,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985 
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987 
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in wrapper_fn(*args)
   3362           attributes=defun_kwargs)
   3363       def wrapper_fn(*args):  # pylint: disable=missing-docstring
-> 3364         ret = _wrapper_helper(*args)
   3365         ret = structure.to_tensor_list(self._output_structure, ret)
   3366         return [ops.convert_to_tensor(t) for t in ret]

/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in _wrapper_helper(*args)
   3297         nested_args = (nested_args,)
   3298 
-> 3299       ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
   3300       # If `func` returns a list of tensors, `nest.flatten()` and
   3301       # `ops.convert_to_tensor()` would conspire to attempt to stack

/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    256       except Exception as e:  # pylint:disable=broad-except
    257         if hasattr(e, 'ag_error_metadata'):
--> 258           raise e.ag_error_metadata.to_exception(e)
    259         else:
    260           raise

NotImplementedError: in user code:

    <ipython-input-16-3d72fc8870c2>:15 augmentation  *
        img = np.array(img)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:848 __array__  **
        " a NumPy call, which is not supported".format(self.name))

    NotImplementedError: Cannot convert a symbolic Tensor (args_0:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
That's a useful answer
Without any help

@bjarrell15, As you would be filing a separate bug next week, can you please confirm if we can close this issue. Thanks!