TypeError: image must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is <class 'NoneType'>

1

Hello!
I’m trying to replace the background of a picture using Stable Diffusion.
But I have the following problem:
TypeError: image must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is <class 'NoneType'>
Code

import os
from dotenv import load_dotenv
import torch
import numpy as np
from diffusers import AutoPipelineForInpainting, EulerDiscreteScheduler, ControlNetModel, AutoencoderKL
from PIL import Image, ImageOps
import rembg

load_dotenv()
SDV5_MODEL_PATH = os.getenv('SDV5_MODEL_PATH')
SAVE_PATH = os.getenv('SAVE_PATH')
KANDINSKY_MODEL_PATH=os.getenv('KANDINSKY_MODEL_PATH')
REV_ANIMATED_MODEL_PATH=os.getenv('REV_ANIMATED_MODEL_PATH')
CONTROLNET_MODEL_PATH=os.getenv('CONTROLNET_MODEL_PATH')
MMIX_MODEL_PATH=os.getenv('MMIX_MODEL_PATH')
SAM_MODEL_PATH = os.getenv('SAM_MODEL_PATH')
VAE_MODEL_PATH = os.getenv('VAE_MODEL_PATH')
CANNY_MODEL_PATH=os.getenv('CANNY_MODEL')

prompt = 'best quality, highres, high definition masterpiece, photorealistic, foggy winter forest'
negative_prompt = 'nsfw, people, worst quality, low quality, normal quality, lowres,watermark, monochrome, low resolutio, extra limbs'
num_inference_steps = 40
base_img = 'C:\\Users\\dovsy\\Downloads\\cat.png'
num_of_img_per_prompt = 1

controlnet = ControlNetModel.from_pretrained(
    CANNY_MODEL_PATH,
    torch_dtype=torch.float32,
    varient="fp32"
)

def binary_mask(init_image):
    #init_image = load_image(url).resize((512, 768))
    input_array = np.array(init_image)
    mask_array = rembg.remove(input_array, only_mask=True)
    mask_image = Image.fromarray(mask_array)
    mask_image = ImageOps.invert(mask_image)
    return mask_image

def toPill(image):
    minv = np.amin(image)
    maxv = np.amax(image)
    image = image - minv
    image = image / (maxv - minv)
    image = image * 255
    img = Image.fromarray(image.astype(np.uint8))
    return img

if __name__ == '__main__':
pipeline = AutoPipelineForInpainting.from_pretrained(
        REV_ANIMATED_MODEL_PATH,
        controlnet=controlnet,
torch_dtype=torch.float32,
        vae=AutoencoderKL.from_pretrained(
                 VAE_MODEL_PATH,
                 subfolder=None)
).to('cpu')

size = (1024, 704)
init_image = Image.open(base_img)
init_image = toPill(init_image)
bin_mask=binary_mask(init_image)

print(type(init_image))
    print(type(bin_mask))
    print(init_image.size)
    print(bin_mask.size)
image = pipeline(
                     prompt=prompt,
                     scheduler=EulerDiscreteScheduler.from_config(pipeline.scheduler.config),
                     negative_prompt=negative_prompt,
                     width=1024,
                     height=704,
                     num_inference_steps=70,
                     image=init_image,
                     mask_image=bin_mask,
                     #control_image=control_image,
                     guidance_scale=15,
                     strength=0.8
                     #generator=torch.manual_seed(189123)
                     ).images[0]

    image.save('test1.jpg')

My error:

<class 'PIL.Image.Image'>
<class 'PIL.Image.Image'>
(1024, 704)
(1024, 704)
Traceback (most recent call last):
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\background_change.py", line 155, in <module>
    image = pipeline(
            ^^^^^^^^^
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet_inpaint.py", line 1174, in __call__
    self.check_inputs(
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet_inpaint.py", line 673, in check_inputs
    self.check_image(image, prompt, prompt_embeds)
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet_inpaint.py", line 775, in check_image
    raise TypeError(
TypeError: image must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is <class 'NoneType'>

Process finished with exit code 1

Photo that i used:

cat
cat1024×704 28.6 KB

Thanks in advance for your reply!

2

Hi Dmitr!

Speculation on my part:

It looks like you are building pipeline with an explicit controlnet, but then calling
pipeline without an explicit control_image argument. I imagine that when you
don’t specify the control_image argument, it defaults to None. But then controlnet
in pipeline gets None passed on to it as its control_image and raises the NoneType
error.

To test this theory, try passing in an actual image, e.g., control_image = init_image,
and see if the error goes away, or try passing in some non-image object, e.g.,
control_image = scheduler, and see if you get an analogous wrong-type error,
but with some actual object type, rather than NoneType.

If this is what’s going on, you either have to pass in an appropriate control_image or
build a pipeline that doesn’t make use of a controlnet.

Good luck!

K. Frank