8000 Type Error when exporting TF2 OD API to SavedModel using efficientdet_d0 · Issue #8841 · tensorflow/models · GitHub
[go: up one dir, main page]

Skip to content
Type Error when exporting TF2 OD API to SavedModel using efficientdet_d0 #8841
@m-brgs

Description

@m-brgs

Prerequisites

Please answer the following questions for yourself before submitting an issue.

  • I am using the latest TensorFlow Model Garden release and TensorFlow 2.
  • I am reporting the issue to the correct repository. (Model Garden official or research directory)
  • I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py

2. Describe the bug

I'm trying to export a TF2 OD API efficientdet_d0 that I just trained (using pretrained weights from model zoo 2 and 2GPUs in the tf2 docker image from OD API repo) with exporter_main_v2.py in order to serve this model with Tensorflow Serving.

When exporting, I'm getting a type error.

3. Steps to reproduce

Take the result of training a ssd_efficientdet_d0_512x512 model.

Inside TF2 container, (https://github.com/tensorflow/models/blob/master/research/object_detection/dockerfiles/tf2/Dockerfile), run :

python object_detection/exporter_main_v2.py \
--input_type encoded_image_string_tensor \
--trained_checkpoint_dir /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/model_dir/ \
--output_directory /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/export/ \
--pipeline_config_path /home/tensorflow/pipeline.config

4. Expected behavior

The model is exported in /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/export/ as described in https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py

5. Additional context

Logs :

I0711 17:37:09.737181 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:144] EfficientDet EfficientNet backbone version: efficientnet-b0
I0711 17:37:09.737298 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:145] EfficientDet BiFPN num filters: 64
I0711 17:37:09.737352 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:147] EfficientDet BiFPN num iterations: 3
I0711 17:37:09.745369 139828628232000 efficientnet_model.py:146] round_filter input=32 output=32
I0711 17:37:10.747392 139828628232000 efficientnet_model.py:146] round_filter input=32 output=32
I0711 17:37:10.747502 139828628232000 efficientnet_model.py:146] round_filter input=16 output=16
I0711 17:37:10.829591 139828628232000 efficientnet_model.py:146] round_filter input=16 output=16
I0711 17:37:10.829701 139828628232000 efficientnet_model.py:146] round_filter input=24 output=24
I0711 17:37:11.092894 139828628232000 efficientnet_model.py:146] round_filter input=24 output=24
I0711 17:37:11.093004 139828628232000 efficientnet_model.py:146] round_filter input=40 output=40
I0711 17:37:11.314886 139828628232000 efficientnet_model.py:146] round_filter input=40 output=40
I0711 17:37:11.314993 139828628232000 efficientnet_model.py:146] round_filter input=80 output=80
I0711 17:37:11.654193 139828628232000 efficientnet_model.py:146] round_filter input=80 output=80
I0711 17:37:11.654304 139828628232000 efficientnet_model.py:146] round_filter input=112 output=112
I0711 17:37:11.993307 139828628232000 efficientnet_model.py:146] round_filter input=112 output=112
I0711 17:37:11.993419 139828628232000 efficientnet_model.py:146] round_filter input=192 output=192
I0711 17:37:12.449521 139828628232000 efficientnet_model.py:146] round_filter input=192 output=192
I0711 17:37:12.449632 139828628232000 efficientnet_model.py:146] round_filter input=320 output=320
I0711 17:37:12.555558 139828628232000 efficientnet_model.py:146] round_filter input=1280 output=1280
I0711 17:37:12.599392 139828628232000 efficientnet_model.py:459] Building model efficientnet with params ModelConfig(width_coefficient=1.0, depth_coefficient=1.0, resolution=224, dropout_rate=0.2, blocks=(BlockConfig(input_filters=32, output_filters=16, kernel_size=3, num_repeat=1, expand_ratio=1, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=16, output_filters=24, kernel_size=3, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=24, output_filters=40, kernel_size=5, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=40, output_filters=80, kernel_size=3, num_repeat=3, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=80, output_filters=112, kernel_size=5, num_repeat=3, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=112, output_filters=192, kernel_size=5, num_repeat=4, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=192, output_filters=320, kernel_size=3, num_repeat=1, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise')), stem_base_filters=32, top_base_filters=1280, activation='simple_swish', batch_norm='default', bn_momentum=0.99, bn_epsilon=0.001, weight_decay=5e-06, drop_connect_rate=0.2, depth_divisor=8, min_depth=None, use_se=True, input_channels=3, num_classes=1000, model_name='efficientnet', rescale_input=False, data_format='channels_last', dtype='float32')
WARNING:tensorflow:From /home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py:101: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated and will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.map_fn(fn, elems, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems))
W0711 17:37:16.879535 139828628232000 deprecation.py:573] From /home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py:101: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated and will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.map_fn(fn, elems, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems))
WARNING:tensorflow:Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f2b7c1ac5f8>, because it is not built.
W0711 17:37:39.926661 139828628232000 save_impl.py:76] Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f2b7c1ac5f8>, because it is not built.
Traceback (most recent call last):
  File "object_detection/exporter_main_v2.py", line 126, in <module>
    app.run(main)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "object_detection/exporter_main_v2.py", line 122, in main
    FLAGS.output_directory)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py", line 180, in export_inference_graph
    signatures=concrete_function)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 951, in save
    obj, export_dir, signatures, options, meta_graph_def)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 1022, in _build_meta_graph
    _ = _SaveableView(checkpoint_graph_view)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 173, in __init__
    self.checkpoint_view.objects_ids_and_slot_variables())
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/training/tracking/graph_view.py", line 415, in objects_ids_and_slot_variables
    trackable_objects, path_to_root = self._breadth_first_traversal()
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/training/tracking/graph_view.py", line 199, in _breadth_first_traversal
    for name, dependency in self.list_dependencies(current_trackable):
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 108, in list_dependencies
    extra_dependencies = self.list_extra_dependencies(obj)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 137, in list_extra_dependencies
    self._serialization_cache)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2746, in _list_extra_dependencies_for_serialization
    .list_extra_dependencies_for_serialization(serialization_cache))
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/base_serialization.py", line 74, in list_extra_dependencies_for_serialization
    return self.objects_to_serialize(serialization_cache)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 73, in objects_to_serialize
    serialization_cache).objects_to_serialize)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 92, in _get_serialized_attributes
    serialization_cache)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/model_serialization.py", line 53, in _get_serialized_attributes_internal
    serialization_cache))
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 101, in _get_serialized_attributes_internal
    functions = save_impl.wrap_layer_functions(self.obj, serialization_cache)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 163, in wrap_layer_functions
    '{}_layer_call_and_return_conditional_losses'.format(layer.name))
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 503, in add_function
    self.add_trace(*self._input_signature)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 418, in add_trace
    trace_with_training(True)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 416, in trace_with_training
    fn.get_concrete_function(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function
    return super(LayerCall, self).get_concrete_function(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function
    concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 865, in _get_concrete_function_garbage_collected
    self._initialize(args, kwargs, add_initializers_to=initializers)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize
    *args, **kwds))
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 524, in wrapper
    ret = method(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 170, in wrap_with_training_arg
    lambda: replace_training_and_call(False))
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 65, in smart_cond
    pred, true_fn=true_fn, false_fn=false_fn, name=name)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond
    return true_fn()
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 169, in <lambda>
    lambda: replace_training_and_call(True),
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 165, in replace_training_and_call
    return wrapped_call(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 566, in call_and_return_conditional_losses
    return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/meta_architectures/ssd_meta_arch.py", line 251, in call
    return self._extract_features(inputs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 224, in _extract_features
    list(zip(self._output_layer_alias, base_feature_maps)))
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 71, in return_outputs_and_add_losses
    outputs, losses = fn(inputs, *args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 170, in wrap_with_training_arg
    lambda: replace_training_and_call(False))
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 65, in smart_cond
    pred, true_fn=true_fn, false_fn=false_fn, name=name)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond
    return true_fn()
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 169, in <lambda>
    lambda: replace_training_and_call(True),
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 165, in replace_training_and_call
    return wrapped_call(*a
6F38
rgs, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 541, in __call__
    self.call_collection.add_trace(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 418, in add_trace
    trace_with_training(True)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 416, in trace_with_training
    fn.get_concrete_function(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function
    return super(LayerCall, self).get_concrete_function(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function
    concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 865, in _get_concrete_function_garbage_collected
    self._initialize(args, kwargs, add_initializers_to=initializers)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize
    *args, **kwds))
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 524, in wrapper
    ret = method(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 484, in wrap_with_training_arg
    return call_fn(*args, **kwargs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 566, in call_and_return_conditional_losses
    return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1627, in get_losses_for
    reachable = tf_utils.get_reachable_from_inputs(inputs, losses)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 140, in get_reachable_from_inputs
    raise TypeError('Expected Operation, Variable, or Tensor, got ' + str(x))
TypeError: Expected Operation, Variable, or Tensor, got level_5

6. System information

  • OS Platform and Distribution : Linux Ubuntu 16.04, docker 19.3, nvidia-docker
  • Mobile device name if the issue happens on a mobile device: N/A
  • TensorFlow installed from (source or binary): Docker image, binary
  • TensorFlow version (use command below): 2.2.0
  • Python version: 3.6
  • Bazel version (if compiling from source):N/A
  • GCC/Compiler version (if compiling from source):N/A
  • CUDA/cuDNN version: N/A
  • GPU model and memory: N/A

v2.2.0-rc4-8-g2b96f3662b 2.2.0

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    0