_scaled_dot_product_efficient_attention implementation

arunava5764 · May 28, 2025, 1:46pm

I was profiling a ViT model using pytorch profiler. It shows “_scaled_dot_product_efficient_attention” function called 12 times. I want to see the implementation of this function in pytorch github repo. The function name is mentioned in line no 766 of the following link:

github.com/pytorch/pytorch

aten/src/ATen/native/transformers/attention.cpp

main

#include <ATen/core/TensorBody.h>
#define TORCH_ASSERT_ONLY_METHOD_OPERATORS
#include <ATen/core/Tensor.h>
#include <ATen/TensorOperators.h>
#include <ATen/AccumulateType.h>
#include <ATen/Dispatch.h>
#include <ATen/OpMathType.h>
#include <ATen/mps/MPSDevice.h>
#include <ATen/native/DispatchStub.h>
#include <ATen/NestedTensorImpl.h>
#include <ATen/TensorIndexing.h>
#include <ATen/TensorSubclassLikeUtils.h>
#include <ATen/native/transformers/attention.h>
#include <ATen/native/transformers/sdp_utils_cpp.h>
#include <c10/util/typeid.h>
#include <c10/core/DeviceType.h>
#include <c10/core/SymInt.h>
#include <c10/core/SymIntArrayRef.h>
#include <c10/util/Logging.h>
#include <c10/core/DispatchKey.h>

This file has been truncated. show original

Also, in the same file the header file associated with this function is included in line no 40. But I cannot find any file where the function is implemented. Same goes for “_efficient_attention_forward” function showing in the profiler output. Is there any way to see those implementation in pytorch github?