Type Error when exporting TF2 OD API to SavedModel using efficientdet_d0 #8841
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- I am using the latest TensorFlow Model Garden release and TensorFlow 2.
- I am reporting the issue to the correct repository. (Model Garden official or research directory)
- I checked to make sure that this issue has not already been filed.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py
2. Describe the bug
I'm trying to export a TF2 OD API efficientdet_d0 that I just trained (using pretrained weights from model zoo 2 and 2GPUs in the tf2 docker image from OD API repo) with exporter_main_v2.py in order to serve this model with Tensorflow Serving.
When exporting, I'm getting a type error.
3. Steps to reproduce
Take the result of training a ssd_efficientdet_d0_512x512 model.
Inside TF2 container, (https://github.com/tensorflow/models/blob/master/research/object_detection/dockerfiles/tf2/Dockerfile), run :
python object_detection/exporter_main_v2.py \
--input_type encoded_image_string_tensor \
--trained_checkpoint_dir /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/model_dir/ \
--output_directory /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/export/ \
--pipeline_config_path /home/tensorflow/pipeline.config4. Expected behavior
The model is exported in /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/export/ as described in https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py
5. Additional context
Logs :
I0711 17:37:09.737181 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:144] EfficientDet EfficientNet backbone version: efficientnet-b0
I0711 17:37:09.737298 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:145] EfficientDet BiFPN num filters: 64
I0711 17:37:09.737352 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:147] EfficientDet BiFPN num iterations: 3
I0711 17:37:09.745369 139828628232000 efficientnet_model.py:146] round_filter input=32 output=32
I0711 17:37:10.747392 139828628232000 efficientnet_model.py:146] round_filter input=32 output=32
I0711 17:37:10.747502 139828628232000 efficientnet_model.py:146] round_filter input=16 output=16
I0711 17:37:10.829591 139828628232000 efficientnet_model.py:146] round_filter input=16 output=16
I0711 17:37:10.829701 139828628232000 efficientnet_model.py:146] round_filter input=24 output=24
I0711 17:37:11.092894 139828628232000 efficientnet_model.py:146] round_filter input=24 output=24
I0711 17:37:11.093004 139828628232000 efficientnet_model.py:146] round_filter input=40 output=40
I0711 17:37:11.314886 139828628232000 efficientnet_model.py:146] round_filter input=40 output=40
I0711 17:37:11.314993 139828628232000 efficientnet_model.py:146] round_filter input=80 output=80
I0711 17:37:11.654193 139828628232000 efficientnet_model.py:146] round_filter input=80 output=80
I0711 17:37:11.654304 139828628232000 efficientnet_model.py:146] round_filter input=112 output=112
I0711 17:37:11.993307 139828628232000 efficientnet_model.py:146] round_filter input=112 output=112
I0711 17:37:11.993419 139828628232000 efficientnet_model.py:146] round_filter input=192 output=192
I0711 17:37:12.449521 139828628232000 efficientnet_model.py:146] round_filter input=192 output=192
I0711 17:37:12.449632 139828628232000 efficientnet_model.py:146] round_filter input=320 output=320
I0711 17:37:12.555558 139828628232000 efficientnet_model.py:146] round_filter input=1280 output=1280
I0711 17:37:12.599392 139828628232000 efficientnet_model.py:459] Building model efficientnet with params ModelConfig(width_coefficient=1.0, depth_coefficient=1.0, resolution=224, dropout_rate=0.2, blocks=(BlockConfig(input_filters=32, output_filters=16, kernel_size=3, num_repeat=1, expand_ratio=1, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=16, output_filters=24, kernel_size=3, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=24, output_filters=40, kernel_size=5, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=40, output_filters=80, kernel_size=3, num_repeat=3, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=80, output_filters=112, kernel_size=5, num_repeat=3, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=112, output_filters=192, kernel_size=5, num_repeat=4, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=192, output_filters=320, kernel_size=3, num_repeat=1, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise')), stem_base_filters=32, top_base_filters=1280, activation='simple_swish', batch_norm='default', bn_momentum=0.99, bn_epsilon=0.001, weight_decay=5e-06, drop_connect_rate=0.2, depth_divisor=8, min_depth=None, use_se=True, input_channels=3, num_classes=1000, model_name='efficientnet', rescale_input=False, data_format='channels_last', dtype='float32')
WARNING:tensorflow:From /home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py:101: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated and will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.map_fn(fn, elems, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems))
W0711 17:37:16.879535 139828628232000 deprecation.py:573] From /home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py:101: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated and will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.map_fn(fn, elems, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems))
WARNING:tensorflow:Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f2b7c1ac5f8>, because it is not built.
W0711 17:37:39.926661 139828628232000 save_impl.py:76] Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f2b7c1ac5f8>, because it is not built.
Traceback (most recent call last):
File "object_detection/exporter_main_v2.py", line 126, in <module>
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "object_detection/exporter_main_v2.py", line 122, in main
FLAGS.output_directory)
File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py", line 180, in export_inference_graph
signatures=concrete_function)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 951, in save
obj, export_dir, signatures, options, meta_graph_def)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 1022, in _build_meta_graph
_ = _SaveableView(checkpoint_graph_view)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 173, in __init__
self.checkpoint_view.objects_ids_and_slot_variables())
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/training/tracking/graph_view.py", line 415, in objects_ids_and_slot_variables
trackable_objects, path_to_root = self._breadth_first_traversal()
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/training/tracking/graph_view.py", line 199, in _breadth_first_traversal
for name, dependency in self.list_dependencies(current_trackable):
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 108, in list_dependencies
extra_dependencies = self.list_extra_dependencies(obj)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 137, in list_extra_dependencies
self._serialization_cache)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2746, in _list_extra_dependencies_for_serialization
.list_extra_dependencies_for_serialization(serialization_cache))
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/base_serialization.py", line 74, in list_extra_dependencies_for_serialization
return self.objects_to_serialize(serialization_cache)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 73, in objects_to_serialize
serialization_cache).objects_to_serialize)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 92, in _get_serialized_attributes
serialization_cache)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/model_serialization.py", line 53, in _get_serialized_attributes_internal
serialization_cache))
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 101, in _get_serialized_attributes_internal
functions = save_impl.wrap_layer_functions(self.obj, serialization_cache)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 163, in wrap_layer_functions
'{}_layer_call_and_return_conditional_losses'.format(layer.name))
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 503, in add_function
self.add_trace(*self._input_signature)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 418, in add_trace
trace_with_training(True)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 416, in trace_with_training
fn.get_concrete_function(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function
return super(LayerCall, self).get_concrete_function(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function
concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 865, in _get_concrete_function_garbage_collected
self._initialize(args, kwargs, add_initializers_to=initializers)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize
*args, **kwds))
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn
return weak_wrapped_fn().__wrapped__(*args, **kwds)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 524, in wrapper
ret = method(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 170, in wrap_with_training_arg
lambda: replace_training_and_call(False))
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 65, in smart_cond
pred, true_fn=true_fn, false_fn=false_fn, name=name)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond
return true_fn()
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 169, in <lambda>
lambda: replace_training_and_call(True),
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 165, in replace_training_and_call
return wrapped_call(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 566, in call_and_return_conditional_losses
return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/meta_architectures/ssd_meta_arch.py", line 251, in call
return self._extract_features(inputs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 224, in _extract_features
list(zip(self._output_layer_alias, base_feature_maps)))
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in __call__
outputs = self.call(cast_inputs, *args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 71, in return_outputs_and_add_losses
outputs, losses = fn(inputs, *args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 170, in wrap_with_training_arg
lambda: replace_training_and_call(False))
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 65, in smart_cond
pred, true_fn=true_fn, false_fn=false_fn, name=name)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond
return true_fn()
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 169, in <lambda>
lambda: replace_training_and_call(True),
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 165, in replace_training_and_call
return wrapped_call(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 541, in __call__
self.call_collection.add_trace(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 418, in add_trace
trace_with_training(True)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 416, in trace_with_training
fn.get_concrete_function(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function
return super(LayerCall, self).get_concrete_function(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function
concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 865, in _get_concrete_function_garbage_collected
self._initialize(args, kwargs, add_initializers_to=initializers)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize
*args, **kwds))
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn
return weak_wrapped_fn().__wrapped__(*args, **kwds)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 524, in wrapper
ret = method(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 484, in wrap_with_training_arg
return call_fn(*args, **kwargs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 566, in call_and_return_conditional_losses
return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1627, in get_losses_for
reachable = tf_utils.get_reachable_from_inputs(inputs, losses)
File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 140, in get_reachable_from_inputs
raise TypeError('Expected Operation, Variable, or Tensor, got ' + str(x))
TypeError: Expected Operation, Variable, or Tensor, got level_5
6. System information
- OS Platform and Distribution : Linux Ubuntu 16.04, docker 19.3, nvidia-docker
- Mobile device name if the issue happens on a mobile device: N/A
- TensorFlow installed from (source or binary): Docker image, binary
- TensorFlow version (use command below): 2.2.0
- Python version: 3.6
- Bazel version (if compiling from source):N/A
- GCC/Compiler version (if compiling from source):N/A
- CUDA/cuDNN version: N/A
- GPU model and memory: N/A
v2.2.0-rc4-8-g2b96f3662b 2.2.0