i’m working on distilbert for text classification, and want to save the tensors into a data frame so i can use it again as feature generator
to illustrate my question more:
below are token IDs for one sentence in a row in csv file i’m working on :
Token IDs: tensor([ 101, 11113, 2080, 25876, 2542, 15009, 11290, 22291, 3370, 2478,
7037, 22160, 3215, 1998, 5760, 5001, 10464, 24895, 15009, 2157,
2002, 17585, 6593, 16940, 1037, 2553, 3189, 1026, 1013, 1055,
1028, 1026, 1013, 1055, 1028, 11581, 2063, 2138, 1997, 1996,
15843, 1997, 10181, 17843, 2542, 15009, 11290, 22291, 3370, 25510,
7096, 2038, 2468, 1996, 2364, 7709, 2000, 7438, 5022, 2007,
2203, 1011, 2754, 102])
1- i want to replace the text in the csv file into their tokens by converting these tensors into np.array and save this CSV file
2- then i need to know how to convert this column back from np.array to tensors (input ids and attention masks) so i can use them for fine tuning tasks
1 To convert a tensor to a numpy array use a = tensor.numpy(), replace the values, and store it via e.g. np.save.
2. To convert a numpy array to a tensor use tensor = torch.from_numpy(a).
1 Like