IndexError: index 1 is out of bounds for axis 0 with size 1

hi everyone
sorry for asking too much , i ma gitting this error and i do not seem to understand what does it mean

 for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                        dim2_stride*j:dim2_stride*j + crop_size] += image[index]
            index += 1
    return empty_array

it happend at this part and this part

def save_prediction_image(stacked_img, im_name, epoch, save_folder_name="result_images", save_im=True):
    """save images to save_path
    Args:
        stacked_img (numpy): stacked cropped images
        save_folder_name (str): saving folder name
    """
    div_arr = division_array(512, 2, 2, 512, 512)
    img_cont = image_concatenate(stacked_img.cpu().data.numpy(), 2, 2, 512, 512)
    img_cont = polarize((img_cont)/div_arr)*255
    img_cont_np = img_cont.astype('uint8')
    img_cont = Image.fromarray(img_cont_np)
    # organize images in every epoch
    desired_path = save_folder_name + '/epoch_' + str(epoch) + '/'
    # Create the path if it does not exist
    if not os.path.exists(desired_path):
        os.makedirs(desired_path)
    # Save Image!
    export_name = str(im_name) + '.png'
    img_cont.save(desired_path + export_name)
    return img_cont_np

Which method calls the nested loop?
Based on the error message, it seems dim0 of empty_array has only a size of 1.
Could you check its shape before passing it to the nested loop?

The shape is (512, 512)

def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
   """concatenate images
   Args :
       image : output images (should be square)
       crop_num2 (int) : number of crop in horizontal way (2)
       crop_num1 (int) : number of crop in vertical way (2)
       dim1(int) : vertical size of output (512)
       dim2(int) : horizontal size_of_output (512)
   Return :
       div_array : numpy arrays of numbers of 1,2,4
   """
   crop_size = image.shape[1]  # size of crop
   empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
   print(empty_array.shape)

   dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
   dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
   index = 0
   for i in range(crop_num1):
       for j in range(crop_num2):
           # add image to empty_array at specific position
           empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                       dim2_stride*j:dim2_stride*j + crop_size] += image[index]
           index += 1
   return empty_array

Could you check the shape of image then?

the shape of image is (1, 512, 512)

So your code will crash in the second iteration, since you are increasing index by 1. Based on your code it looks like image should have at least crop_num1*crop_num2 length.

@ptrblck should it be like this

def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
    """concatenate images
    Args :
        image : output images (should be square)
        crop_num2 (int) : number of crop in horizontal way (2)
        crop_num1 (int) : number of crop in vertical way (2)
        dim1(int) : vertical size of output (512)
        dim2(int) : horizontal size_of_output (512)
    Return :
        div_array : numpy arrays of numbers of 1,2,4
    """
    image= crop_num1*crop_num2
    crop_size = image # size of crop
    empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
    print(empty_array.shape)

    dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
    dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
    index = 0
    for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                        dim2_stride*j:dim2_stride*j + crop_size] += image[index]
            index += 1
    return empty_array

still crashing in the second iteration


TypeError: 'int' object is not subscriptable

also, the results of the first iteration, does not make sense to me the training loss and the accuracy are above 80 is it possible?

Epoch 1 Train loss: 0.820307083427906 Train acc 0.893951416015625
Epoch 2 Train loss: 0.8146810382604599 Train acc 0.8943264484405518
Epoch 3 Train loss: 0.8138034269213676 Train acc 0.8945329189300537
Epoch 4 Train loss: 0.8148394525051117 Train acc 0.894050121307373
Epoch 5 Train loss: 0.8261850923299789 Train acc 0.8950645923614502

No, I meant the size of dim0 should be at least that large.
You are passing image to the function, so you shouldn’t replace it with a int.
How would you like to index image if it’s only a single grayscale image?
Would you like to index in another dimension?

1 Like

Yes I want to index it in another array
Also before the crashing it gives me the input and target shape is different

But how would you like to index the image?
Currently it has a shape of [1, 512, 512].
If you call image[0], you will get the pixel value of the one and only channel.
However, inside the loop you are increasing index by one, so that the next call would be image[1], which does not exist.

Could you try to explain, how the indexing should work or maybe draw a sketch?

i think i am not understanding the idea of index , anyway if i am having 3 channels i can index the image correctly ?

Not using the current code.
It looks like you are trying to set some patches of en empty tensor to some input images.
Could you explain your workflow a bit so that we could maybe create some (pseudo-)code for it?

@ptrblck

my code is basically exploring whether the preprocessing and postprocessing for vessels segmentation using unet can improve the accuracy, i used ( Flip ,Gaussian_noise, Uniform_noise , Brightness ,Elastic distort {0: distort, 1:no distort} ,Crop the image
,Pad the image, Sanity Check for Cropped image , Normalize the image )
also dividing the training sets to val and train , but i am not getting a high accuracy also the index error

@ptrblck
this is the link for my colab drift
https://colab.research.google.com/drive/19W-h5lootgxCsNi6mZBXAdqiDroGOLfN

@ptrblck i just want to check if you have any idea how to fix it

The code is quite long, so it would be easier if you try to explain what the nested for loops are supposed to do.

ok, so the code is about data augmentation and pre-processing for vessels segmentation , the first part of the code is prepossessing and augmentation , then i train it using unet , it breaks on 5th epoch where the validation spouse to happen .

def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
    """concatenate images
    Args :
        image : output images (should be square)
        crop_num2 (int) : number of crop in horizontal way (2)
        crop_num1 (int) : number of crop in vertical way (2)
        dim1(int) : vertical size of output (512)
        dim2(int) : horizontal size_of_output (512)
    Return :
        div_array : numpy arrays of numbers of 1,2,4
    """
    crop_size = image.shape[1]  # size of crop
    empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
    dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
    dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
    index = 1
    for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i+ crop_size,
                        dim2_stride*j:dim2_stride*j+ crop_size] += image[index]
            index += 1
    return empty_array

this code is suposed to concatenate images by croping and vertical, horizontial dimentation
then used to save the predaction images


def save_prediction_image(stacked_img, im_name, epoch, save_folder_name="result_images", save_im=True):
    """save images to save_path
    Args:
        stacked_img (numpy): stacked cropped images
        save_folder_name (str): saving folder name
    """
    div_arr = division_array(256, 2, 2, 512, 512)
    img_cont = image_concatenate(stacked_img.cpu().data.numpy(), 2, 2, 512, 512)
    img_cont = polarize((img_cont)/div_arr)*255
    img_cont_np = img_cont.astype('uint8')
    img_cont = Image.fromarray(img_cont_np)
    # organize images in every epoch
    desired_path = save_folder_name + '/epoch_' + str(epoch) + '/'
    # Create the path if it does not exist
    if not os.path.exists(desired_path):
        os.makedirs(desired_path)
    # Save Image!
    export_name = str(im_name) + '.png'
    img_cont.save(desired_path + export_name)
    return img_cont_np

Thanks for the explanation.
It seems you would like to create the new image by concatenating 4 images from the tensor image.
In that case image should have the shape [4, height, width] in case you would like to use 4 different images.
Currently it seems image only contains 1 image. Would you like to use the same image 4 times?

If image only contains this single image only “sometimes”, it might be due to a remaining last batch, which can be smaller depending on your batch size.
If you are using a DataLoader, you could try to set drop_last=True and try it again.

@ptrblck Thank you so much , now it is breaks on 5 th epoch with error
“RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 1, 3, 3], but got 5-dimensional input of size [1, 1, 512, 512, 3] instead”

most of the answers for the question is abut using " unsqueeze(0) " with the data and thier mask i already used them

def validate_model(model, data_val, criterion, epoch, make_prediction=True, save_folder_name='prediction'):
    """
        Validation run
    """
    # calculating validation loss
    total_val_loss = 0
    total_val_acc = 0
    for batch, (images_v, masks_v, original_msk) in enumerate(data_val):
        stacked_img = torch.Tensor([]).unsqueeze(0).cuda()
        for index in range(images_v.size()[1]):
          
            with torch.no_grad():
                image_v = Variable(images_v[:, index, :, :].unsqueeze(0).float().cuda())
                mask_v = Variable(masks_v[:, index, :, :].unsqueeze(0).float().cuda())
                # print(image_v.shape, mask_v.shape)
                output_v = model(image_v)
                total_val_loss = total_val_loss + criterion(output_v, mask_v).cpu().item()
                # print('out', output_v.shape)
                output_v = torch.argmax(output_v, dim=1).float()
                stacked_img = torch.cat((stacked_img, output_v))
        if make_prediction:
            im_name = batch  # TODO: Change this to real image name so we know
            pred_msk = save_prediction_image(stacked_img, im_name, epoch, save_folder_name)
            acc_val = accuracy_check(original_msk, pred_msk)
            total_val_acc = total_val_acc + acc_val

    return total_val_acc/(batch + 1), total_val_loss/((batch + 1)*4)