Deeplearning on Yaohong

How backward and step are associated with model paramters update?

Wed, 17 Nov 2021 20:27:20 +0800

How are backward and step associated with model paramters update?

optimizer accept the paramters of the model, it can update the parameters, but how is loss function associated with paramters?

loss.backward()

optimizer.step()

REFERENCE:

1.pytorch - connection between loss.backward() and optimizer.step()

2.https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html#forward-and-backward-function-hooks

nn_Module

Tue, 16 Nov 2021 20:27:20 +0800

nn_Module

1.Where are module parameters configured?

The parameters are stored in the network node which is one of points of a network layer. Neural network layer is defined in init method of module and need to be defined as class variable;

import torch.nn as nn
import numpy as np
class TorchDNN(nn.Module):

 def __init__(self, input, hidden, output):
 super(TorchDNN, self).__init__();
 layer_hidden = nn.Linear(input, hidden, bias = True);
 def forward(self, input_data):
 pass

x = np.array([1, 2, 3])
torch_model = TorchDNN(len(x), 5, 3)
print(torch_model.state_dict())
# OUTPUT:
# OrderedDict()

Network layer should be defined as a variable of Module class;

Understanding arange, unsqueeze, repeat, stack methods in Pytorch

Fri, 30 Jul 2021 20:26:20 +0800

Understanding arange, unsqueeze, repeat, stack methods in Pytorch

torch.arange(start=0, end, step=1) return 1-D tensor of size (end-start)/step which value begin from start and each value take with common differences step.
torch.unsqueeze(input, dim) return a new tensor with a dimension of size one insterted at specified position; A dim value within the range [-input.dim() - 1, input.dim() + 1) can be used.
tensor.repeat(size*) return a tensor; the new shape of tensor is that original shape multiplied by arguments correspondingly, if the number of paramter don’t match the original shape, then last dimension of new shape = the last dimension of original shape * last paramter;

L1 L2 Regularization - Optimizer

Mon, 12 Jul 2021 20:26:20 +0800

Optimizer: L1 L2 Regularization

L1,L2 Loss function mean different type of loss function.

L1: sum(Y-f(x)) lasso
L2: sum(Y-f(x))^2 Ridge

L1, L2 regularization :

Y_predict = E(w_i(x_i)+b_i)

MES = E(Y-Y_predict)^2

L1: loss = MSE + 入E|w_i|
L2: loss = MES + 入E(w_i)^2

What does penalize the weights?

It means add another parameters to the loss function, so that the greater the weight, the higher the loss function value. That makes the weight parameters to be less or smaller.

How to Label Voice with Praat for Machine Learning

Sat, 10 Jul 2021 11:34:20 +0800

How to Label Voice with Praat for Machine Learning

1.Install

1.1 Download praat

1.Open Praat: doing Phonetics by Computer website;

2.Choose your OS system on download area in the upper left conner of website;

3.Then click the praat6150_mac.dmg or praat6150_win64.zip to download file;

For example, my os is MacOS, in my case I should download praat6150_mac.dmg and install it.

Option: You can also download the file from github, referce to Praat in github

1.2 Install Phonetic symbols

If you want to see good-quality phonetic characters on your screen and in your clipboard, you have to install the Charis SIL and/or the Doulos SIL font.

The Simple Implement of BatchNorm2D

Thu, 27 May 2021 12:30:20 +0800

The Simple Implement of BatchNorm2D

The first is that instead of whiteningthe features in layer inputs and outputs jointly, we will normalize each scalar feature independently, by making ithave the mean of zero and the variance of 1. For a layer with d-dimensional inputx = (x(1). . . x(d)), we will nor-malize each dimension

1.MyBatchNorm2D

import numpy as np;
class MyBatchNorm2D:

 def __init__(self):
 pass

 def forward(self, x):
 x = np.array(x);
 mean = np.mean(x);
 standard_deviation = np.sqrt(np.var(x) + 1e-05);
 x_norm = (x - mean) / standard_deviation;
 return x_norm;

input = [[[[ 1.1713, -10.7508],
 [-2.0155, -0.5290],
 [-0.2751, 1.0233]],
 [[-1.4446, -0.8337],
 [-1.0429, -0.8856],
 [ 5.3324, 7.6233]]],
 [[[ 2.1079, 1.6039],
 [-0.8938, 1.1655],
 [ 8.0355, -0.4911]],
 [[ 3.6337, 10.3400],
 [-1.5365, 0.7931],
 [ 0.8472, 1.1318]]]];
x = np.array(input);
bn = MyBatchNorm2D();
x_norm = bn.forward(input);
print("x_norm:", x_norm);

print("np.mean: ", np.mean(np.array(input)));
print("np.var: " , np.var(np.array(input)));
print("MyBatchNorm2D np.mean: ", np.mean(np.array(x_norm)));
print("MyBatchNorm2D np.var: " , np.var(np.array(x_norm)));

# OUTPUT:
# x_norm: [[[[ 0.0414345 -2.92181622]
# [-0.75064805 -0.38117689]
# [-0.31806977 0.00464894]]
# [[-0.60875025 -0.4569104 ]
# [-0.50890728 -0.4698102 ]
# [ 1.07568036 1.64508601]]]
# [[[ 0.27422743 0.14895769]
# [-0.47184832 0.0399929 ]
# [ 1.74753876 -0.3717568 ]]
# [[ 0.65346666 2.3203247 ]
# [-0.63159209 -0.05256752]
# [-0.0391209 0.03161673]]]]
# np.mean: 1.0045958333333334
# np.var: 16.18707780123264
# MyBatchNorm2D np.mean: 0.0
# MyBatchNorm2D np.var: 0.9999993822236513

2.Using BatchNorm2d in torch

input = [[[[ 1.1713, -10.7508],
 [-2.0155, -0.5290],
 [-0.2751, 1.0233]],
 [[-1.4446, -0.8337],
 [-1.0429, -0.8856],
 [ 5.3324, 7.6233]]],
 [[[ 2.1079, 1.6039],
 [-0.8938, 1.1655],
 [ 8.0355, -0.4911]],
 [[ 3.6337, 10.3400],
 [-1.5365, 0.7931],
 [ 0.8472, 1.1318]]]];

import torch
import torch.nn as nn
input = torch.tensor(input);
bn = nn.BatchNorm2d(2, momentum=None, affine=False, track_running_stats=None)
x_norm = bn(input)
print("BatchNorm2d new_x:", x_norm);

import numpy as np;
print("BatchNorm2d np.mean: " , np.mean(np.array(x_norm)));
print("BatchNorm2d np.var: " , np.var(np.array(x_norm)));

# OUTPUT:
# BatchNorm2d new_x: tensor([[[[ 0.2864, -2.6606],
# [-0.5013, -0.1339],
# [-0.0711, 0.2498]],
# [[-0.9184, -0.7553],
# [-0.8112, -0.7692],
# [ 0.8903, 1.5017]]],
# [[[ 0.5179, 0.3933],
# [-0.2241, 0.2850],
# [ 1.9831, -0.1245]],
# [[ 0.4369, 2.2267],
# [-0.9429, -0.3212],
# [-0.3067, -0.2308]]]])
# BatchNorm2d np.mean: -9.934108e-09
# BatchNorm2d np.var: 0.99999934

REFERENCE:

1.Torch nn.BatchNorm2d

model(x) vs model.forward(x)

Mon, 24 May 2021 11:00:20 +0800

model(x) vs model.forward(x)

__call__ magic method in nn.Module will invoke forward() method and take care of hooks and states that python allows, so we should use model(x) rather than call model.forward(x) directly.

REFERENCE:

1.Why there are different output between model.forward(input) and model(input)

2.Calling forward function without .forward()

3.torch.nn.module codes

DNN RNN CNN codes

Tue, 04 May 2021 08:27:20 +0800

Simple DNN RNN CNN example codes

1.DNN-Deep neural network


import numpy as np;
class myDNN:

 # 3 * 5 * 2
 def __init__(self, input, hidden, output):
 # hidden random weight 
 # Note: hidden_weight can be the shape of (input,hidden); correspondingly, `self.hidden_out` should equal `np.dot(input_data, self.hidden_weight)` to accord with hidden_weight shape.
 self.hidden_weight = np.random.rand(hidden, input); 

 self.hidden_bias = np.random.rand(hidden);

 # hidden random weight 
 self.output_weight = np.random.rand(output,hidden);
 self.output_bias = np.random.rand(output);

 # 
 def forward(self, input_data):
 self.hidden_out = np.dot(input_data, self.hidden_weight.T) + self.hidden_bias;

 self.output_out = np.dot(self.hidden_out, self.output_weight.T) + self.output_bias;

# Usage:
dnn = myDNN(3,5,3);
print("hidden_weight",dnn.hidden_weight)
print("hidden_bias:",dnn.hidden_bias)
print("output_weight",dnn.output_weight)
print("output_bias",dnn.output_bias)

x = np.array([1, 2, 3]) #inut
dnn.forward(x);
print("output_out",dnn.output_out)

# output:
# hidden_weight [[0.99663996 0.39342568 0.5312192 ]
# [0.0798744 0.50312289 0.86241405]
# [0.17138496 0.6761287 0.70645906]
# [0.61662379 0.69389404 0.16623206]
# [0.71213402 0.30800932 0.64149244]]
# hidden_bias: [0.81517457 0.56115705 0.3089624 0.84450962 0.93530796]
# output_weight [[0.34466034 0.31119367 0.12883636 0.34135026 0.43802589]
# [0.31553914 0.16063241 0.8179255 0.52314575 0.79439618]
# [0.86730239 0.25280671 0.20375421 0.78095429 0.67368635]]
# output_bias [0.5588883 0.98722366 0.21507382]
# output_out [ 6.8078659 11.30086755 11.16252686]

1.1 Use DNN in torch

import torch.nn as nn
class TorchDNN(nn.Module):

 def __init__(self, input, hidden, output):
 super(TorchDNN, self).__init__();
 self.layer_hidden = nn.Linear(input, hidden, bias = True);
 self.layer_output = nn.Linear(hidden, output, bias = True);

 # 
 def forward(self, input_data):
 self.hidden_out = self.layer_hidden(input_data);
 self.output_out = self.layer_output(self.hidden_out);

x = np.array([1, 2, 3])
torch_model = TorchDNN(len(x), 5, 3)
print(torch_model.state_dict())

# OrderedDict([('layer_hidden.weight', 
# tensor([[-0.5216, -0.5690, 0.4181],
# [-0.3142, 0.1489, 0.5071],
# [ 0.0295, 0.3381, 0.4401],
# [-0.4697, 0.0732, -0.0328],
# [ 0.5250, 0.1540, 0.2086]])), 
# ('layer_hidden.bias', tensor([-0.5134, 0.2645, -0.3366, -0.0597, 0.0159])), 
# ('layer_output.weight', 
# tensor([[ 0.2770, -0.3408, -0.3145, -0.3686, 0.1060],
# [ 0.1268, 0.0729, -0.3838, 0.2850, 0.1438],
# [ 0.1645, -0.0497, 0.1029, 0.1088, -0.0536]])), 
# ('layer_output.bias', tensor([ 0.0908, -0.1240, 0.2800]))])

2.RNN-Recurrent neural network


import numpy as np;
class myRNN:

 def __init__(self, input, hidden ):
 # random weight
 self.input_hidden_weight = np.random.randint(-10000,10000,(hidden, input))/10000;
 self.hidden_hidden_weight = np.random.randint(-10000,10000,(hidden))/10000;

 # random bias
 self.input_hidden_bias = np.random.randint(-10000,10000,(hidden))/10000;
 self.hidden_hidden_bias = np.random.randint(-10000,10000,(hidden))/10000;
 # self.input_hidden_bias = np.zeros(hidden);
 # self.hidden_hidden_bias = np.zeros(hidden);
 self.hidden_size = hidden

 
 def forward(self, input_data):
 self.last_hidden_output = np.zeros([self.hidden_size]);
 output = []
 for item in input_data:
 # ht=tanh(W_ih * x_t + b_ih  + W_hh*h_(t−1)+b_hh)
 hidden_cur = np.dot(item, self.input_hidden_weight.T) + self.input_hidden_bias;
 hidden_pre = np.dot(self.last_hidden_output, self.hidden_hidden_weight.T) + self.hidden_hidden_bias;

 hidden_output = np.tanh( hidden_cur + hidden_pre )
 output.append(hidden_output)
 self.last_hidden_output = hidden_output;

 return np.array(output), hidden_output;

# diy_model = myRNN(w_ih, w_hh, hidden_size)
x = np.array([[1, 2, 3], [3, 4, 5], [5, 6, 7]]) 
input_size = 3;
hidden_size = 4;
diy_model = myRNN(input_size,hidden_size)
output, hidden_output = diy_model.forward(x)
print("myRNN process output: ", output)
print("myRNN hidden_output:", hidden_output)

# output:
# myRNN process output: [[-0.62745049 -0.99314575 -0.96754221 -0.9965258 ]
# [-0.99542912 -0.99962783 -0.99965698 -0.99998354]
# [-0.99983032 -0.99992543 -0.9999868 -0.99999971]]
# myRNN hidden_output: [-0.99983032 -0.99992543 -0.9999868 -0.99999971]

2.1 Use RNN in torch

import torch.nn as nn;
import torch;
import numpy as np;
class TorchRNN(nn.Module):

 def __init__(self, input_size, hidden):
 super(TorchRNN,self).__init__();
 self.layer = nn.RNN(input_size, hidden, batch_first=True);

 def forward(self, x):
 return self.layer(x)


torch_model = TorchRNN(3, 4)
print(torch_model.state_dict())

x = np.array([[1, 2, 3], [3, 4, 5], [5, 6, 7]]) 
torch_x = torch.FloatTensor([x])
output, h = torch_model.forward(torch_x)
print("output:", output.detach().numpy())
print("h:",h.detach().numpy())

# output: 
# OrderedDict([('layer.weight_ih_l0', tensor([[ 0.0922, 0.2786, -0.4514],
# [ 0.3809, 0.2628, -0.4460],
# [-0.4951, -0.3599, -0.4961],
# [ 0.3794, 0.3397, 0.3185]])), ('layer.weight_hh_l0', tensor([[-0.1330, -0.1843, -0.2618, 0.4246],
# [ 0.4154, -0.3578, -0.4181, -0.4291],
# [ 0.3608, -0.2349, 0.4631, 0.4873],
# [ 0.4886, 0.0285, -0.0490, 0.2928]])), ('layer.bias_ih_l0', tensor([-0.1421, -0.3572, -0.2087, -0.0319])), ('layer.bias_hh_l0', tensor([-0.3799, 0.1126, -0.1766, 0.2630]))])
# output: [[[-0.8416229 -0.589054 -0.99585485 0.9778193 ]
# [-0.45533973 -0.3993926 -0.9999862 0.99957436]
# [-0.6221984 0.05736368 -0.99999994 0.9999955 ]]]
# h: [[[-0.6221984 0.05736368 -0.99999994 0.9999955 ]]]

3.CNN-Convolutional neural network

import numpy as np;
class MyCNN:

 # I don't know how do filters work.
 def __init__(self, in_channel, out_channel, kernel_size):
 # random weight
 # (out_channel, in_channel, kernel_size, kernel_size)
 # self.kernel_weight = np.random.randint(-10000,10000,(out_channel, in_channel, kernel_size, kernel_size))/10000;
 self.kernel_weight = np.array([[[[ 0.0106, -0.1561, 0.0984],
 [ 0.1468, 0.1580, -0.1404],
 [ 0.0856, 0.0780, 0.0636]],
 [[-0.1620, 0.2318, 0.0486],
 [-0.2214, -0.2046, 0.1070],
 [ 0.1609, 0.0160, -0.0374]]],
 [[[ 0.1876, -0.2056, 0.1858],
 [-0.1288, 0.0065, -0.0145],
 [-0.1080, 0.1519, 0.0581]],
 [[-0.0749, 0.2289, -0.0890],
 [ 0.0611, 0.0398, -0.1293],
 [ 0.0911, -0.0264, -0.2104]]]]);
 self.in_channel = in_channel;
 self.out_channel = out_channel;
 self.kernel_size = kernel_size;

 # c*h*w
 def forward(self, input_data):
 output = [];
 input_shape = input_data.shape;

 idx_start = np.int(np.floor( (self.kernel_size)/2)) ;
 width = input_shape[1];
 height = input_shape[2];
 in_channel = input_shape[0];

 for o_c in range(self.out_channel):
 piece_of_out_channel = np.zeros(( width-(idx_start*2), height-(idx_start*2) ));
 # print("piece_of_out_channel:", piece_of_out_channel.shape)
 # width
 for idx_height in range(idx_start, height - idx_start): 
 # height
 for idx_width in range(idx_start, width - idx_start ):
 # kernel_shape_input
 kernel_shape_input = input_data[:, idx_height-idx_start: idx_height+idx_start+1, idx_width-idx_start :idx_width+idx_start+1 ];

 out = self.kernel_weight[o_c] * kernel_shape_input;
 out = np.sum(out)

 # assign value
 idx_h = idx_height - idx_start;
 idx_w = idx_width - idx_start
 piece_of_out_channel[idx_h][idx_w] = out;

 output.append(piece_of_out_channel);
 return output;



# x = np.random.randint(0,10000,(2, 6, 6))/100;
# x = random.astype(int)
x = np.array([[[61,93,18,31,2,49]
 ,[12,62,32,60,58,30]
 ,[49,64,38,74,59,29]
 ,[71,34,29,88,59,41]
 ,[91,72,36,94,79,29]
 ,[17,15,86,29,84,53]]

 ,[[31,25,15,16,35,20]
 ,[76,45,82,88,49,99]
 ,[56,46,82,72,26,55]
 ,[7, 86,32,29,82,91]
 ,[76,68,17,50,19,53]
 ,[87,21,58,35,81,46]]
 ]);
# print(x);
print("x.shape:",x.shape)

myCNN = MyCNN(x.shape[0], 2, 3);
print(myCNN.kernel_weight)
output = myCNN.forward(x)
print("myCNN output:",output)

# output:
# x.shape: (2, 6, 6)
# myCNN output: [array([[-2.5093, 9.0098, -0.2033, 28.9 ],
# [ 6.5155, 27.3464, 0.7038, 14.5031],
# [24.2218, 16.1092, 23.2223, 16.9067],
# [29.6749, 2.0986, 16.8128, 45.025 ]]), 
# array([[-14.77 , -4.3335, 5.0665, 3.2378],
# [-28.2207, 18.1968, 11.889 , -27.3557],
# [ -2.748 , 22.5508, 10.6013, -19.0372],
# [ 22.0148, 9.1788, -22.0313, 9.5176]])]

3.1 Use CNN in torch



import torch;
import torch.nn as nn;
class TorchCNN(nn.Module):

 def __init__(self, in_channel, out_channel, kernel_size ):
 super().__init__();
 self.conv2d = nn.Conv2d(in_channel, out_channel, kernel_size, bias=False);

 def forward(self, input_data):
 return self.conv2d(input_data);

in_channel = x.shape[0];
torchcnn = TorchCNN(in_channel, 2, 3);
print("torchcnn weight:", torchcnn.state_dict())
print("torchcnn weight shape:", torchcnn.state_dict()['conv2d.weight'].numpy().shape)
# torchcnn weight shape: (2, 2, 3, 3) => (out_channel, in_channel, kernel_size, kernel_size)
torch_x = torch.FloatTensor([x])
out = torchcnn.forward(torch_x);
print("TorchCNN out: ", out)


# output
# torchcnn weight: OrderedDict([('conv2d.weight', tensor(
# [[[[ 0.0106, -0.1561, 0.0984],
# [ 0.1468, 0.1580, -0.1404],
# [ 0.0856, 0.0780, 0.0636]],
# [[-0.1620, 0.2318, 0.0486],
# [-0.2214, -0.2046, 0.1070],
# [ 0.1609, 0.0160, -0.0374]]],
# [[[ 0.1876, -0.2056, 0.1858],
# [-0.1288, 0.0065, -0.0145],
# [-0.1080, 0.1519, 0.0581]],
# [[-0.0749, 0.2289, -0.0890],
# [ 0.0611, 0.0398, -0.1293],
# [ 0.0911, -0.0264, -0.2104]]]]))])
# TorchCNN out: tensor([[[[ -2.5066, 9.0144, -0.1983, 28.9003],
# [ 6.5160, 27.3456, 0.7094, 14.5056],
# [ 24.2262, 16.1086, 23.2279, 16.9102],
# [ 29.6763, 2.1047, 16.8210, 45.0250]],
# [[-14.7564, -4.3273, 5.0752, 3.2491],
# [-28.2115, 18.1981, 11.8975, -27.3447],
# [ -2.7442, 22.5540, 10.6096, -19.0247],
# [ 22.0166, 9.1837, -22.0241, 9.5211]]]],
# grad_fn=<MkldnnConvolutionBackward>)

Averaging histograms

Tue, 29 Dec 2020 00:49:20 +0800

Averaging histograms

An image histogram is the number of each pixel value, which is displayed in the graph.

x axis of the graph is pixel value, range from 0 to 255;

y axis of the graph is the number of this pixel value;

1.How to averaging histograms?

Our goal is to generate a new image with a more even histogram distribution.

Some notes on Convolution Course

Thu, 21 May 2020 15:18:20 +0800

Some notes on Convolution course

What is padding?

Padding is to add some pixels to the border of the original image, such as a 6*6 image will become a 8*8 image if we add a pixel to its border.

valid convolution vs same convolution.

Valid convolution is on padding that means the actual pixels of the output image after we convole original image with filter.

Same convolution means adding padding so that the output image has the same size as its input image.

深度学习-第一个数字识别项目

Sun, 03 May 2020 09:15:20 +0800

深度学习-第一个数字识别项目

今天按Google官方推荐流程，整理了开发模板，不是所有深度学习都严格按这个模板来实现，不同项目步骤有所删减，但大体框架是差不多的，主要为以下过程:

Terms in machine learning

Mon, 01 Jan 0001 00:00:00 +0000

Terms in machine learning

FLOPS

FLOPS=Floating point operation per seconds
FLOPs=Floating point operations

REFERNECD: what-is-flops-in-field-of-deep-learning