Partitioning multiple networks

I’m interested in using/extending glow to run multiple networks concurrently. From what I found out so far, the compilation process for some network works as follows:

  1. A caffe2 or onnx network is loaded into one glow::function inside a glow:module
  2. This module can be added to the HostManager where the compilation, optimizations and partitioning happen

Therefor, when multiple networks are added, the DAGs of the single networks are created independantly from each other, right? If that is correct, do the overall physically available resources need to be partitioned in advance for the networks?

What I want to achieve is to do the partitioning, taking all the loaded networks into account. Is that something that should even be supported by glow sooner or later, or is that not desired?
Maybe the most simple solution, concerning the current situation would be to just load multiple networks into one module, but that seems a bit hacky to me?

Hey @lpolari, you are correct, when multiple networks are added the DAGs for each network are independent from each other. Physical resources are taken into account while adding a network so advanced partitioning may not be necessary.
When loading partitions onto a device the Glow Runtime will try to evenly distribute used space on the devices.
Example:
3 Devices with 10GB of memory
Network A Partitioned: A1[6GB], A2[5GB]
Network B Partitioned B1[4GB], B2[8GB]
Assuming you load A then B.
The device loading will looks like:
Dev1: A1, B1
Dev2: A2
Dev3: B2
The limitation here is that Glow decides one network at a time so you might end up with a less than optimal assignment of resources.
To get “optimal” will depend a bit on your use case. In general glow only partitions a network if it will not fit on a single device. So you usually won’t end up with many different networks loaded on a single device.
If you are running many smaller networks then partitioning shouldn’t be an issue but you will have contention for devices at inference time.

2 Likes