How to attach index from pandas DF image outputs from an autoencoder or CNN

Hello,

I’m pulling photos from a train directory by label (train/1 or train/0) using the datasets.ImageFolder command. I have a df or csv that has each image, target label and the location in the directory it can be found. That said, how would I attach an output of 1000 encoder features per image or a CNN prediction back to the index as I’m using an ID associated with picture as the index? The ultimate goal is to concatenate the output with other model outputs also associated with that id found in the index.

If you are using pandas to read the csv file, you might use pd.DataFrame.loc to write to the desired image ID.
Since pandas uses numpy arrays internally, you could access the array via tensor.numpy().

I’m not sure, if you are also asking about the PyTorch code to extract the features, so let me know, if you need more information.

Thank you for replying. I’m trying to encode two different sets of pictures at 274*274, 3 channels, down to 1000 features, then concatenation those features with a tabular data set to ultimately predict a binary outcome (kind of like the Ludwig model). The I’d is 1 to N pictures where 1 ID (set as the index) could have up to four pictures. I plan to average over the features for IDs with more than one picture.

The tabular dataset also has the IDs as the index. I’m just not sure how to join with the final feature sets from the encoded pictures without that ID on the photo side.

So, yes, any type of code would help to assign policy IDs with the picture. The csv’s (and dataframe) column headers are:
Index: ID
File name
Location
Label

The directory the images are in is split between the positive class and negative class.

So I’ve tried to follow this tutorial and I keep getting an error. My csv file has a few changes in the ordering of the columns:
ID
File name
description
target
root_dir (location)

This is my data class I tried to do following the tutorial:

class roofDataset(Dataset):
‘’’
roof data class

'''
def __init__(self, csv_file, transform = None):
    '''
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    '''
    self.roof_frame = pd.read_csv(csv_file)
    self.root_dir = self.roof_frame
    self.transform = transform
    
def __len__(self):
    return len(self.roof_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
        
    img_name = os.path.join(self.root_dir.iloc[idx,5], self.roof_frame.iloc[idx, 1])
    image = io.imread(img_name)
    roof = self.roof_frame.iloc[idx, 1:]
    roof = np.array([roof])
    sample = {'image': image, 'roof': roof}

    if self.transform:
        sample = self.transform(sample)

    return sample

Then I ran the following to test:

roof_dataset = roofDataset(csv_file=‘D:\CIS inspection images 0318\self_build\train\train_roof_images.csv’,
transform = train_transform)

fig = plt.figure()

for i in range(len(roof_dataset)):
sample = roof_dataset[i]

print(i, sample['image'].shape, sample['roof'].shape)

ax = plt.subplot(1, 4, i + 1)
plt.tight_layout()
ax.set_title('Sample #{}'.format(i))
ax.axis('off')
show_landmarks(**sample)

if i == 3:
    plt.show()
    break

And now I get this error:


TypeError Traceback (most recent call last)
in
5
6 for i in range(len(roof_dataset)):
----> 7 sample = roof_dataset[i]
8
9 print(i, sample[‘image’].shape, sample[‘roof’].shape)

in getitem(self, idx)
23 idx = idx.tolist()
24
—> 25 img_name = os.path.join(self.root_dir.iloc[idx,5], self.roof_frame.iloc[idx, 1])
26 image = io.imread(img_name)
27 roof = self.roof_frame.iloc[idx, 1:]

C:\ProgramData\Anaconda3\lib\ntpath.py in join(path, *paths)
113 return result_drive + result_path
114 except (TypeError, AttributeError, BytesWarning):
→ 115 genericpath._check_arg_types(‘join’, path, *paths)
116 raise
117

C:\ProgramData\Anaconda3\lib\genericpath.py in _check_arg_types(funcname, *args)
147 else:
148 raise TypeError(‘%s() argument must be str or bytes, not %r’ %
→ 149 (funcname, s.class.name)) from None
150 if hasstr and hasbytes:
151 raise TypeError(“Can’t mix strings and bytes in path components”) from None

TypeError: join() argument must be str or bytes, not ‘int64’

So, I think I have the class working correctly…

class roofDataset(Dataset):
‘’’
roof data class

'''
def __init__(self, csv_file, transform = None):
    '''
    Args:
        csv_file (string): Path to the csv file with annotations.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    '''
    self.roof_frame = pd.read_csv(csv_file)
    self.transform = transform
    
def __len__(self):
    return len(self.roof_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
     
    root_dir = self.roof_frame.iloc[idx,4]
    pic = self.roof_frame.iloc[idx, 1]
    label = self.roof_frame.iloc[idx, 3]
    img_name = os.path.join(root_dir, pic)
    image = Image.open(img_name)
    policy = self.roof_frame.iloc[idx, 0]
    sample = {'image': image, 'policy': policy, 'label':label}

    if self.transform:
        image = self.transform(image)

    return image, label, policy

So this returns the image, the label and the policy which is the ID I was referring to. I’ve made modifications to the tutorial and got everything to print as expected with no errors!

I’m not sure how I’ll now feed this into an auto encoder but, the first step is done! Thanks for the hints.