How to extarct Memory profiling?

JUNSUNGKIM99 · July 13, 2021, 5:38am

Hello I’m using Resnet18, Resnet50 as an AOT bundle now. When the test image is inputted, the inference time and the confidence can be confirmed to be output. Furthermore, I want to check memory usage or memory transfer. What should I do to check? Do I have to implement it directly on the code?

jfix · July 16, 2021, 4:28am

You should be able to use these structs that are autogenerated and should be included in the main function you write in in the C++ that drives inference:

github.com

pytorch/glow/blob/b0de69a7639567bfec9081ab6a8bcb792638cba9/lib/LLVMIRCodeGen/BundleSaver.cpp#L106-L135

    
      
          /// Header file common definitions for dynamic API.
          static const char *dynamicApiCommonDefines = R"RAW(
          // Type describing a symbol table entry of a generated bundle.
          struct SymbolTableEntry {
            // Name of a variable.
            const char *name;
            // Offset of the variable inside the memory area.
            uint64_t offset;
            // The number of elements inside this variable.
            uint64_t size;
            // Variable kind: 1 if it is a mutable variable, 0 otherwise.
            char kind;
          };
          
          
// Type describing the config of a generated bundle.
          struct BundleConfig {
            // Size of the constant weight variables memory area.
            uint64_t constantWeightVarsMemSize;
            // Size of the mutable weight variables memory area.
            uint64_t mutableWeightVarsMemSize;

This file has been truncated. show original

E.g. using the BundleConfig you can understand the total memory usage of the network itself, such as the size of constant weights, etc. Aside from that if you need more info about data transfer then yes you would need to take a look at the actual data you’re passing into the network in the C++ file that drives inference.