Skip to content

Inconsistent Error Handling in tf.raw_ops.SparseSegmentSumGradV2 Between CPU and GPU Implementations #94151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SilentTester73 opened this issue May 26, 2025 · 2 comments
Assignees
Labels
comp:ops OPs related issues TF 2.19 type:bug Bug

Comments

@SilentTester73
Copy link

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

binary

TensorFlow version

tf 2.20.0-dev20250526

Custom code

Yes

OS platform and distribution

Linux CPU & GPU

Mobile device

No response

Python version

3.13.2

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

CUDA 12.5.1

GPU model and memory

No response

Current behavior?

The tf.raw_ops.SparseSegmentSumGradV2 function exhibits inconsistent behavior between CPU and GPU implementations.

When provided with out-of-range negative index, the CPU version raise an exception normally while the GPU version acquiesces to negative indexing or causes an abort.

Standalone code to reproduce the issue

import tensorflow as tf

with tf.device('/cpu:0'):
    try:
        grad = tf.constant([0, 0, 0, 0], dtype=tf.float64, shape=[4])
        indices = tf.constant([-1], dtype=tf.int64, shape=[1])
        segment_ids = tf.constant([-1], dtype=tf.int64, shape=[1])
        dense_output_dim0 = tf.constant([1], dtype=tf.int32, shape=[])

        tf.raw_ops.SparseSegmentSumGradV2(
            grad=grad,
            indices=indices,
            segment_ids=segment_ids,
            dense_output_dim0=dense_output_dim0,
            name=None
        )
        print("SparseSegmentSumGradV2 executed successfully on CPU")
    except Exception as e:
        print(f"Exception on CPU: {e}")
        
with tf.device('/gpu:0'):
    try:
        grad = tf.constant([0, 0, 0, 0], dtype=tf.float64, shape=[4])
        indices = tf.constant([-1], dtype=tf.int64, shape=[1])
        segment_ids = tf.constant([-1], dtype=tf.int64, shape=[1])
        dense_output_dim0 = tf.constant([1], dtype=tf.int32, shape=[])

        tf.raw_ops.SparseSegmentSumGradV2(
            grad=grad,
            indices=indices,
            segment_ids=segment_ids,
            dense_output_dim0=dense_output_dim0,
            name=None
        )
        print("SparseSegmentSumGradV2 executed successfully on GPU")
    except Exception as e:
        print(f"Exception on GPU: {e}")

Relevant log output

Output:


I0000 00:00:1748278378.253343  624834 gpu_device.cc:2018] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2033 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.9
2025-05-27 00:52:58.280430: I tensorflow/core/framework/local_rendezvous.cc:407] Local rendezvous is aborting with status: INVALID_ARGUMENT: Index -1 out of range [0, 1).
Exception on CPU: {{function_node __wrapped__SparseSegmentSumGradV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Index -1 out of range [0, 1). [Op:SparseSegmentSumGradV2] name: 
SparseSegmentSumGradV2 executed successfully on GPU
@sharktide
Copy link

We could probably include a check for negaive indicies before doing continuing with the job. To me, this seems like a we-overlooked-this kinda issue.

@Venkat6871
Copy link
Contributor

I was able to reproduce the same issue using TensorFlow 2.19.0 as well as the nightly version. Please find the gist for your reference.
Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:ops OPs related issues TF 2.19 type:bug Bug
Projects
None yet
Development

No branches or pull requests

4 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy