in order to saving the inference time, I expect to use one gpu card to predict multi images concurrently, could someone give me some suggestion？many thanks.
By the way, I have tried to used more than one batch to predict, it will be ok. However, I want to use multiprocessing to predict multi images in parallel.
Using multiple batches is actually the best you can get. You can also use multiple threads but make sure to use locks when using model (to keep memory requirement same). Using multithread will make sure GPU is being used even when other threads are loading image or doing pre/post processing.