Hello, everyone. I am new to pytorch and numpy. And I met two problems when I learn how to process the data.
first
I have a ndarray (the numpy array’s type) shaped (4, 2).
[
[np.random.rand(48, 48), int]
[np.random.rand(48, 48), int]
[np.random.rand(48, 48), int]
[np.random.rand(48, 48), int]
]
As you can see how I generate it:
def generate_fake_data():
"""
this is just for easy to show what I wanna do
"""
# the image data, shape is 48 * 48
a = np.random.rand(48, 48)
# the image data label, between 0 - 7
b = np.uint8(np.random.randint(7))
t = [a, b]
return np.array(t)
def create_data_set():
t = []
for i in range(100):
t.append(generate_fake_data())
return np.array(t)
data_set = create_data_set()
Then I defined a generator to generate the batch data.
def data_get(data, num):
"""
this is a generator to get 'num' element in data
:param data: a list
:param num: a num, which type is int
:return:
"""
if len(data) == 0:
raise StopIteration
yield data[0:num]
yield from data_get(data[num:], num)
Then I want to feed it to the pytorch model.
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(
1, 32, kernel_size=5, padding=2
)
self.conv2 = nn.Conv2d(
32, 32, kernel_size=4, padding=1
)
self.conv3 = nn.Conv2d(
32, 64, kernel_size=5, padding=2
)
self.pool = nn.MaxPool2d(
kernel_size=3,
stride=2
)
# full connection layer
self.func1 = nn.Linear(64 * 4 * 4, 2048)
self.func2 = nn.Linear(2048, 1024)
self.func3 = nn.Linear(1024, 7)
def forward(self, x):
# convolution layer
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = self.pool(F.relu(self.conv3(x)))
# convert the x to 2 dimension
x = x.view(-1, self.num_flat_features(x))
# full connection
x = F.relu(self.func1(x))
x = F.relu(self.func2(x))
x = F.softmax(self.func3(x))
return x
@staticmethod
def num_flat_features(x):
size = x.size()[1:]
num_features = 1
for s in size:
num_features *= s
return num_features
Then I want to get the data shaped (4, 48, 48), and the labels shaped (4, ).
for item in data_get(data_set, 4):
"""
every time you will get an item
"""
# shape is (4, ). which means it has four element, and every element's shape is (48, 48)
input_data = item[:, 0]
# shape is (4, )
labels = item[:, 1]
So the first question is how I can make the input_data shaped (4, 48, 48)? Does there any clean method to do that?
You can not only use numpy to solve, but also propose the pytorch method.(they will be all the same helpful to me or others with the problem)
Last night I found a stupid way to do that.
# reshape
result = np.ones((4, 48, 48))
tmp = []
for item in input_data:
item = item[np.newaxis, :]
tmp.append(tmp)
result = np.concatenate(tmp, 0)
# then I will use torch.from_numpy(result) get a torch tensor shaped (4, 48, 48)
# and use unsqueeze method get tensor shaped (4, 1, 48, 48)
Though I thought a solution, there may be some clean and clear method to solve the problem? If you know that, please let me know.
second
If you had read my code above, maybe you will have some question?
Shall we set the label before the img_data? (I had the question, too)
so I make some test.
# the operation is np.array(.....), and it's wrong，the wrong message is the followed line
# #ValueError: setting an array element with a sequence#
np.array([
np.uint8(4),
np.random.rand(4, 3)
])
# but it's right when you change the position between them
c = [np.random.rand(4, 3), np.uint8(3)]
np.array(c)
# it's also right, if you put the element into an array
np.array([
[np.uint8(4)],
np.random.rand(4, 3)
])
# some other examples
# the followed two is wrong!!
np.array([
[1, 2],
[[1, 2, 3], [1, 2, 4]]
])
np.array([
[1, 2],
[[1, 3], [1, 4]]
])
# the followed two is right
np.array([
[[1, 2, 3], [1, 2, 4]],
[1, 2]
])
np.array([
[[1, 3], [1, 4]],
[1, 2]
])
I have search the error message, reading them carefully, talking with my friends, but I can’t understand them.
So can someone explain the reason? Or talk the principle behind it?