IndexError: index 1 is out of bounds for axis 0 with size 1

(lavenderxx) #1

hi everyone
sorry for asking too much , i ma gitting this error and i do not seem to understand what does it mean

 for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                        dim2_stride*j:dim2_stride*j + crop_size] += image[index]
            index += 1
    return empty_array

it happend at this part and this part

def save_prediction_image(stacked_img, im_name, epoch, save_folder_name="result_images", save_im=True):
    """save images to save_path
    Args:
        stacked_img (numpy): stacked cropped images
        save_folder_name (str): saving folder name
    """
    div_arr = division_array(512, 2, 2, 512, 512)
    img_cont = image_concatenate(stacked_img.cpu().data.numpy(), 2, 2, 512, 512)
    img_cont = polarize((img_cont)/div_arr)*255
    img_cont_np = img_cont.astype('uint8')
    img_cont = Image.fromarray(img_cont_np)
    # organize images in every epoch
    desired_path = save_folder_name + '/epoch_' + str(epoch) + '/'
    # Create the path if it does not exist
    if not os.path.exists(desired_path):
        os.makedirs(desired_path)
    # Save Image!
    export_name = str(im_name) + '.png'
    img_cont.save(desired_path + export_name)
    return img_cont_np
#2

Which method calls the nested loop?
Based on the error message, it seems dim0 of empty_array has only a size of 1.
Could you check its shape before passing it to the nested loop?

(lavenderxx) #3

The shape is (512, 512)

def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
   """concatenate images
   Args :
       image : output images (should be square)
       crop_num2 (int) : number of crop in horizontal way (2)
       crop_num1 (int) : number of crop in vertical way (2)
       dim1(int) : vertical size of output (512)
       dim2(int) : horizontal size_of_output (512)
   Return :
       div_array : numpy arrays of numbers of 1,2,4
   """
   crop_size = image.shape[1]  # size of crop
   empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
   print(empty_array.shape)

   dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
   dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
   index = 0
   for i in range(crop_num1):
       for j in range(crop_num2):
           # add image to empty_array at specific position
           empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                       dim2_stride*j:dim2_stride*j + crop_size] += image[index]
           index += 1
   return empty_array
#4

Could you check the shape of image then?

(lavenderxx) #5

the shape of image is (1, 512, 512)

#6

So your code will crash in the second iteration, since you are increasing index by 1. Based on your code it looks like image should have at least crop_num1*crop_num2 length.

(lavenderxx) #7

@ptrblck should it be like this

def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
    """concatenate images
    Args :
        image : output images (should be square)
        crop_num2 (int) : number of crop in horizontal way (2)
        crop_num1 (int) : number of crop in vertical way (2)
        dim1(int) : vertical size of output (512)
        dim2(int) : horizontal size_of_output (512)
    Return :
        div_array : numpy arrays of numbers of 1,2,4
    """
    image= crop_num1*crop_num2
    crop_size = image # size of crop
    empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
    print(empty_array.shape)

    dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
    dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
    index = 0
    for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                        dim2_stride*j:dim2_stride*j + crop_size] += image[index]
            index += 1
    return empty_array

still crashing in the second iteration


TypeError: 'int' object is not subscriptable

also, the results of the first iteration, does not make sense to me the training loss and the accuracy are above 80 is it possible?

Epoch 1 Train loss: 0.820307083427906 Train acc 0.893951416015625
Epoch 2 Train loss: 0.8146810382604599 Train acc 0.8943264484405518
Epoch 3 Train loss: 0.8138034269213676 Train acc 0.8945329189300537
Epoch 4 Train loss: 0.8148394525051117 Train acc 0.894050121307373
Epoch 5 Train loss: 0.8261850923299789 Train acc 0.8950645923614502
#8

No, I meant the size of dim0 should be at least that large.
You are passing image to the function, so you shouldn’t replace it with a int.
How would you like to index image if it’s only a single grayscale image?
Would you like to index in another dimension?

1 Like
(lavenderxx) #9

Yes I want to index it in another array
Also before the crashing it gives me the input and target shape is different

#10

But how would you like to index the image?
Currently it has a shape of [1, 512, 512].
If you call image[0], you will get the pixel value of the one and only channel.
However, inside the loop you are increasing index by one, so that the next call would be image[1], which does not exist.

Could you try to explain, how the indexing should work or maybe draw a sketch?

(lavenderxx) #11

i think i am not understanding the idea of index , anyway if i am having 3 channels i can index the image correctly ?

#12

Not using the current code.
It looks like you are trying to set some patches of en empty tensor to some input images.
Could you explain your workflow a bit so that we could maybe create some (pseudo-)code for it?

(lavenderxx) #13

@ptrblck

my code is basically exploring whether the preprocessing and postprocessing for vessels segmentation using unet can improve the accuracy, i used ( Flip ,Gaussian_noise, Uniform_noise , Brightness ,Elastic distort {0: distort, 1:no distort} ,Crop the image
,Pad the image, Sanity Check for Cropped image , Normalize the image )
also dividing the training sets to val and train , but i am not getting a high accuracy also the index error

(lavenderxx) #14

@ptrblck
this is the link for my colab drift
https://colab.research.google.com/drive/19W-h5lootgxCsNi6mZBXAdqiDroGOLfN

(lavenderxx) #15

@ptrblck i just want to check if you have any idea how to fix it

#16

The code is quite long, so it would be easier if you try to explain what the nested for loops are supposed to do.

(lavenderxx) #17

ok, so the code is about data augmentation and pre-processing for vessels segmentation , the first part of the code is prepossessing and augmentation , then i train it using unet , it breaks on 5th epoch where the validation spouse to happen .

def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
    """concatenate images
    Args :
        image : output images (should be square)
        crop_num2 (int) : number of crop in horizontal way (2)
        crop_num1 (int) : number of crop in vertical way (2)
        dim1(int) : vertical size of output (512)
        dim2(int) : horizontal size_of_output (512)
    Return :
        div_array : numpy arrays of numbers of 1,2,4
    """
    crop_size = image.shape[1]  # size of crop
    empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
    dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
    dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
    index = 1
    for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i+ crop_size,
                        dim2_stride*j:dim2_stride*j+ crop_size] += image[index]
            index += 1
    return empty_array

this code is suposed to concatenate images by croping and vertical, horizontial dimentation
then used to save the predaction images


def save_prediction_image(stacked_img, im_name, epoch, save_folder_name="result_images", save_im=True):
    """save images to save_path
    Args:
        stacked_img (numpy): stacked cropped images
        save_folder_name (str): saving folder name
    """
    div_arr = division_array(256, 2, 2, 512, 512)
    img_cont = image_concatenate(stacked_img.cpu().data.numpy(), 2, 2, 512, 512)
    img_cont = polarize((img_cont)/div_arr)*255
    img_cont_np = img_cont.astype('uint8')
    img_cont = Image.fromarray(img_cont_np)
    # organize images in every epoch
    desired_path = save_folder_name + '/epoch_' + str(epoch) + '/'
    # Create the path if it does not exist
    if not os.path.exists(desired_path):
        os.makedirs(desired_path)
    # Save Image!
    export_name = str(im_name) + '.png'
    img_cont.save(desired_path + export_name)
    return img_cont_np
#18

Thanks for the explanation.
It seems you would like to create the new image by concatenating 4 images from the tensor image.
In that case image should have the shape [4, height, width] in case you would like to use 4 different images.
Currently it seems image only contains 1 image. Would you like to use the same image 4 times?

If image only contains this single image only “sometimes”, it might be due to a remaining last batch, which can be smaller depending on your batch size.
If you are using a DataLoader, you could try to set drop_last=True and try it again.

(lavenderxx) #19

@ptrblck Thank you so much , now it is breaks on 5 th epoch with error
“RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 1, 3, 3], but got 5-dimensional input of size [1, 1, 512, 512, 3] instead”

(lavenderxx) #20

most of the answers for the question is abut using " unsqueeze(0) " with the data and thier mask i already used them

def validate_model(model, data_val, criterion, epoch, make_prediction=True, save_folder_name='prediction'):
    """
        Validation run
    """
    # calculating validation loss
    total_val_loss = 0
    total_val_acc = 0
    for batch, (images_v, masks_v, original_msk) in enumerate(data_val):
        stacked_img = torch.Tensor([]).unsqueeze(0).cuda()
        for index in range(images_v.size()[1]):
          
            with torch.no_grad():
                image_v = Variable(images_v[:, index, :, :].unsqueeze(0).float().cuda())
                mask_v = Variable(masks_v[:, index, :, :].unsqueeze(0).float().cuda())
                # print(image_v.shape, mask_v.shape)
                output_v = model(image_v)
                total_val_loss = total_val_loss + criterion(output_v, mask_v).cpu().item()
                # print('out', output_v.shape)
                output_v = torch.argmax(output_v, dim=1).float()
                stacked_img = torch.cat((stacked_img, output_v))
        if make_prediction:
            im_name = batch  # TODO: Change this to real image name so we know
            pred_msk = save_prediction_image(stacked_img, im_name, epoch, save_folder_name)
            acc_val = accuracy_check(original_msk, pred_msk)
            total_val_acc = total_val_acc + acc_val

    return total_val_acc/(batch + 1), total_val_loss/((batch + 1)*4)