Yaohong

为了真相不惜被羞辱

How backward and step are associated with model paramters update?

How are backward and step associated with model paramters update?

optimizer accept the paramters of the model, it can update the parameters, but how is loss function associated with paramters?

loss.backward()

optimizer.step()

REFERENCE:

1.pytorch - connection between loss.backward() and optimizer.step()

2.https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html#forward-and-backward-function-hooks

Posted by yaohong Wednesday, November 17, 2021

nn_Module

nn_Module

1.Where are module parameters configured?

The parameters are stored in the network node which is one of points of a network layer. Neural network layer is defined in init method of module and need to be defined as class variable;

import torch.nn as nn
import numpy as np
class TorchDNN(nn.Module):

    def __init__(self, input, hidden, output):
        super(TorchDNN, self).__init__();
        layer_hidden = nn.Linear(input, hidden, bias = True);
    def forward(self, input_data):
        pass

x = np.array([1, 2, 3])
torch_model = TorchDNN(len(x), 5, 3)
print(torch_model.state_dict())
# OUTPUT:
# OrderedDict()

Network layer should be defined as a variable of Module class;

Posted by yaohong Tuesday, November 16, 2021

Understanding arange, unsqueeze, repeat, stack methods in Pytorch

Understanding arange, unsqueeze, repeat, stack methods in Pytorch

  • torch.arange(start=0, end, step=1) return 1-D tensor of size (end-start)/step which value begin from start and each value take with common differences step.

  • torch.unsqueeze(input, dim) return a new tensor with a dimension of size one insterted at specified position; A dim value within the range [-input.dim() - 1, input.dim() + 1) can be used.

  • tensor.repeat(size*) return a tensor; the new shape of tensor is that original shape multiplied by arguments correspondingly, if the number of paramter don’t match the original shape, then last dimension of new shape = the last dimension of original shape * last paramter;

Posted by yaohong Friday, July 30, 2021

L1 L2 Regularization - Optimizer

Optimizer: L1 L2 Regularization

L1,L2 Loss function mean different type of loss function.

L1: sum(Y-f(x))     lasso
L2: sum(Y-f(x))^2   Ridge

L1, L2 regularization :

Y_predict = E(w_i(x_i)+b_i)

MES = E(Y-Y_predict)^2

L1: loss = MSE + 入E|w_i|
L2: loss = MES + 入E(w_i)^2

What does penalize the weights?

It means add another parameters to the loss function, so that the greater the weight, the higher the loss function value. That makes the weight parameters to be less or smaller.

Posted by yaohong Monday, July 12, 2021

How to Label Voice with Praat for Machine Learning

Praat

How to Label Voice with Praat for Machine Learning

1.Install

1.1 Download praat

1.Open Praat: doing Phonetics by Computer website;

2.Choose your OS system on download area in the upper left conner of website;

3.Then click the praat6150_mac.dmg or praat6150_win64.zip to download file;

For example, my os is MacOS, in my case I should download praat6150_mac.dmg and install it.

  • Option: You can also download the file from github, referce to Praat in github

1.2 Install Phonetic symbols

If you want to see good-quality phonetic characters on your screen and in your clipboard, you have to install the Charis SIL and/or the Doulos SIL font.

Posted by Yaohong Saturday, July 10, 2021

Anacode simple usage

Anacode simple usage

1.1 Download and install –Mac os

Download file: click to download

Install after download.

Run command in terminal to see your anconda version:

$conda -V
conda 4.10.1

Use conda info to see conda configuration:

(base) $ conda info

2.Anaconda Usage

2.1 List all enviroments

(base) $ conda info -e
# conda environments:
#
base                     /Users/Rhys/opt/anaconda3

2.1 create an enviroment

(base) $ conda create -n py36 python=3.6

2.2 activate an enviroment

(base) $ conda activate py36
(py36) $

The environment had changed after activating;

Posted by yaohong Saturday, June 26, 2021

Simple AI expert Enhanced Loop

Simple AI expert Enhanced Loop

Habit: Daily plan, weekly plan, month plan, 10 minute reading, Daily self-examination

Loop1: Assumption->design a experiment->do->feedback->conclusion

Loop2: Choose a subject->Weekly Share to my classmates->Feedback and update -> Make another share;

Posted by yaohong Monday, June 21, 2021

The Simple Implement of BatchNorm2D

The Simple Implement of BatchNorm2D

The first is that instead of whiteningthe features in layer inputs and outputs jointly, we will normalize each scalar feature independently, by making ithave the mean of zero and the variance of 1. For a layer with d-dimensional inputx = (x(1). . . x(d)), we will nor-malize each dimension

1.MyBatchNorm2D

import numpy as np;
class MyBatchNorm2D:

    def __init__(self):
        pass

    def forward(self, x):
        x = np.array(x);
        mean = np.mean(x);
        standard_deviation = np.sqrt(np.var(x) + 1e-05);
        x_norm = (x - mean) / standard_deviation;
        return x_norm;

input = [[[[ 1.1713, -10.7508],
          [-2.0155, -0.5290],
          [-0.2751,  1.0233]],
         [[-1.4446, -0.8337],
          [-1.0429, -0.8856],
          [ 5.3324,  7.6233]]],
        [[[ 2.1079,  1.6039],
          [-0.8938,  1.1655],
          [ 8.0355, -0.4911]],
         [[ 3.6337,  10.3400],
          [-1.5365,  0.7931],
          [ 0.8472,  1.1318]]]];
x = np.array(input);
bn = MyBatchNorm2D();
x_norm = bn.forward(input);
print("x_norm:", x_norm);

print("np.mean: ", np.mean(np.array(input)));
print("np.var: " , np.var(np.array(input)));
print("MyBatchNorm2D np.mean: ", np.mean(np.array(x_norm)));
print("MyBatchNorm2D np.var: " , np.var(np.array(x_norm)));

# OUTPUT:
# x_norm: [[[[ 0.0414345  -2.92181622]
#           [-0.75064805 -0.38117689]
#           [-0.31806977  0.00464894]]
#          [[-0.60875025 -0.4569104 ]
#           [-0.50890728 -0.4698102 ]
#           [ 1.07568036  1.64508601]]]
#         [[[ 0.27422743  0.14895769]
#           [-0.47184832  0.0399929 ]
#           [ 1.74753876 -0.3717568 ]]
#          [[ 0.65346666  2.3203247 ]
#           [-0.63159209 -0.05256752]
#           [-0.0391209   0.03161673]]]]
# np.mean:  1.0045958333333334
# np.var:  16.18707780123264
# MyBatchNorm2D np.mean:  0.0
# MyBatchNorm2D np.var:  0.9999993822236513

2.Using BatchNorm2d in torch

input = [[[[ 1.1713, -10.7508],
          [-2.0155, -0.5290],
          [-0.2751,  1.0233]],
         [[-1.4446, -0.8337],
          [-1.0429, -0.8856],
          [ 5.3324,  7.6233]]],
        [[[ 2.1079,  1.6039],
          [-0.8938,  1.1655],
          [ 8.0355, -0.4911]],
         [[ 3.6337,  10.3400],
          [-1.5365,  0.7931],
          [ 0.8472,  1.1318]]]];

import torch
import torch.nn as nn
input = torch.tensor(input);
bn = nn.BatchNorm2d(2, momentum=None, affine=False, track_running_stats=None)
x_norm = bn(input)
print("BatchNorm2d new_x:", x_norm);

import numpy as np;
print("BatchNorm2d np.mean: " , np.mean(np.array(x_norm)));
print("BatchNorm2d np.var: " , np.var(np.array(x_norm)));

# OUTPUT:
# BatchNorm2d new_x: tensor([[[[ 0.2864, -2.6606],
#                               [-0.5013, -0.1339],
#                               [-0.0711,  0.2498]],
#                              [[-0.9184, -0.7553],
#                               [-0.8112, -0.7692],
#                               [ 0.8903,  1.5017]]],
#                             [[[ 0.5179,  0.3933],
#                               [-0.2241,  0.2850],
#                               [ 1.9831, -0.1245]],
#                              [[ 0.4369,  2.2267],
#                               [-0.9429, -0.3212],
#                               [-0.3067, -0.2308]]]])
# BatchNorm2d np.mean:  -9.934108e-09
# BatchNorm2d np.var:  0.99999934

REFERENCE:

1.Torch nn.BatchNorm2d

Posted by yaohong Thursday, May 27, 2021

model(x) vs model.forward(x)

model(x) vs model.forward(x)

__call__ magic method in nn.Module will invoke forward() method and take care of hooks and states that python allows, so we should use model(x) rather than call model.forward(x) directly.

REFERENCE:

1.Why there are different output between model.forward(input) and model(input)

2.Calling forward function without .forward()

3.torch.nn.module codes

Posted by yaohong Monday, May 24, 2021

DNN RNN CNN codes

Simple DNN RNN CNN example codes

1.DNN-Deep neural network


import numpy as np;
class myDNN:

    # 3 * 5 * 2
    def __init__(self, input, hidden, output):
        # hidden random weight 
        # Note: hidden_weight can be the shape of (input,hidden); correspondingly, `self.hidden_out` should equal `np.dot(input_data, self.hidden_weight)` to accord with hidden_weight shape.
        self.hidden_weight     = np.random.rand(hidden, input); 

        self.hidden_bias       = np.random.rand(hidden);

        # hidden random weight 
        self.output_weight     = np.random.rand(output,hidden);
        self.output_bias       = np.random.rand(output);

    # 
    def forward(self, input_data):
        self.hidden_out = np.dot(input_data, self.hidden_weight.T) + self.hidden_bias;

        self.output_out = np.dot(self.hidden_out, self.output_weight.T) + self.output_bias;

# Usage:
dnn = myDNN(3,5,3);
print("hidden_weight",dnn.hidden_weight)
print("hidden_bias:",dnn.hidden_bias)
print("output_weight",dnn.output_weight)
print("output_bias",dnn.output_bias)

x = np.array([1, 2, 3])  #inut
dnn.forward(x);
print("output_out",dnn.output_out)

# output:
# hidden_weight [[0.99663996 0.39342568 0.5312192 ]
#  [0.0798744  0.50312289 0.86241405]
#  [0.17138496 0.6761287  0.70645906]
#  [0.61662379 0.69389404 0.16623206]
#  [0.71213402 0.30800932 0.64149244]]
# hidden_bias: [0.81517457 0.56115705 0.3089624  0.84450962 0.93530796]
# output_weight [[0.34466034 0.31119367 0.12883636 0.34135026 0.43802589]
#  [0.31553914 0.16063241 0.8179255  0.52314575 0.79439618]
#  [0.86730239 0.25280671 0.20375421 0.78095429 0.67368635]]
# output_bias [0.5588883  0.98722366 0.21507382]
# output_out [ 6.8078659  11.30086755 11.16252686]

1.1 Use DNN in torch

import torch.nn as nn
class TorchDNN(nn.Module):

    def __init__(self, input, hidden, output):
        super(TorchDNN, self).__init__();
        self.layer_hidden = nn.Linear(input, hidden, bias = True);
        self.layer_output = nn.Linear(hidden, output, bias = True);

    # 
    def forward(self, input_data):
        self.hidden_out = self.layer_hidden(input_data);
        self.output_out = self.layer_output(self.hidden_out);

x = np.array([1, 2, 3])
torch_model = TorchDNN(len(x), 5, 3)
print(torch_model.state_dict())

# OrderedDict([('layer_hidden.weight', 
#  tensor([[-0.5216, -0.5690,  0.4181],
#         [-0.3142,  0.1489,  0.5071],
#         [ 0.0295,  0.3381,  0.4401],
#         [-0.4697,  0.0732, -0.0328],
#         [ 0.5250,  0.1540,  0.2086]])), 
#         ('layer_hidden.bias', tensor([-0.5134,  0.2645, -0.3366, -0.0597,  0.0159])), 
#         ('layer_output.weight', 
# tensor([[ 0.2770, -0.3408, -0.3145, -0.3686,  0.1060],
#         [ 0.1268,  0.0729, -0.3838,  0.2850,  0.1438],
#         [ 0.1645, -0.0497,  0.1029,  0.1088, -0.0536]])), 
#         ('layer_output.bias', tensor([ 0.0908, -0.1240,  0.2800]))])

2.RNN-Recurrent neural network


import numpy as np;
class myRNN:

    def __init__(self, input, hidden ):
        # random weight
        self.input_hidden_weight     = np.random.randint(-10000,10000,(hidden, input))/10000;
        self.hidden_hidden_weight    = np.random.randint(-10000,10000,(hidden))/10000;

        # random bias
        self.input_hidden_bias     = np.random.randint(-10000,10000,(hidden))/10000;
        self.hidden_hidden_bias     = np.random.randint(-10000,10000,(hidden))/10000;
        # self.input_hidden_bias      = np.zeros(hidden);
        # self.hidden_hidden_bias     = np.zeros(hidden);
        self.hidden_size            = hidden

   
    def forward(self, input_data):
        self.last_hidden_output = np.zeros([self.hidden_size]);
        output = []
        for item in input_data:
            # ht​=tanh(W_ih​ * x_t​ + b_ih  ​ +   W_hh​*h_(t−1)​+b_hh​)
            hidden_cur = np.dot(item, self.input_hidden_weight.T) + self.input_hidden_bias;
            hidden_pre = np.dot(self.last_hidden_output, self.hidden_hidden_weight.T)   + self.hidden_hidden_bias;

            hidden_output =  np.tanh( hidden_cur + hidden_pre )
            output.append(hidden_output)
            self.last_hidden_output = hidden_output;

        return np.array(output), hidden_output;

# diy_model = myRNN(w_ih, w_hh, hidden_size)
x = np.array([[1, 2, 3], [3, 4, 5], [5, 6, 7]]) 
input_size = 3;
hidden_size = 4;
diy_model = myRNN(input_size,hidden_size)
output, hidden_output = diy_model.forward(x)
print("myRNN process output: ", output)
print("myRNN hidden_output:", hidden_output)

# output:
# myRNN process output:  [[-0.62745049 -0.99314575 -0.96754221 -0.9965258 ]
#  [-0.99542912 -0.99962783 -0.99965698 -0.99998354]
#  [-0.99983032 -0.99992543 -0.9999868  -0.99999971]]
# myRNN hidden_output: [-0.99983032 -0.99992543 -0.9999868  -0.99999971]

2.1 Use RNN in torch

import torch.nn as nn;
import torch;
import numpy as np;
class TorchRNN(nn.Module):

    def __init__(self, input_size, hidden):
        super(TorchRNN,self).__init__();
        self.layer = nn.RNN(input_size, hidden, batch_first=True);

    def forward(self, x):
        return self.layer(x)


torch_model = TorchRNN(3, 4)
print(torch_model.state_dict())

x = np.array([[1, 2, 3], [3, 4, 5], [5, 6, 7]]) 
torch_x = torch.FloatTensor([x])
output, h = torch_model.forward(torch_x)
print("output:", output.detach().numpy())
print("h:",h.detach().numpy())

# output: 
# OrderedDict([('layer.weight_ih_l0', tensor([[ 0.0922,  0.2786, -0.4514],
#         [ 0.3809,  0.2628, -0.4460],
#         [-0.4951, -0.3599, -0.4961],
#         [ 0.3794,  0.3397,  0.3185]])), ('layer.weight_hh_l0', tensor([[-0.1330, -0.1843, -0.2618,  0.4246],
#         [ 0.4154, -0.3578, -0.4181, -0.4291],
#         [ 0.3608, -0.2349,  0.4631,  0.4873],
#         [ 0.4886,  0.0285, -0.0490,  0.2928]])), ('layer.bias_ih_l0', tensor([-0.1421, -0.3572, -0.2087, -0.0319])), ('layer.bias_hh_l0', tensor([-0.3799,  0.1126, -0.1766,  0.2630]))])
# output: [[[-0.8416229  -0.589054   -0.99585485  0.9778193 ]
#   [-0.45533973 -0.3993926  -0.9999862   0.99957436]
#   [-0.6221984   0.05736368 -0.99999994  0.9999955 ]]]
# h: [[[-0.6221984   0.05736368 -0.99999994  0.9999955 ]]]

3.CNN-Convolutional neural network

import numpy as np;
class MyCNN:

    # I don't know how do filters work.
    def __init__(self, in_channel, out_channel, kernel_size):
        # random weight
        # (out_channel, in_channel, kernel_size, kernel_size)
        # self.kernel_weight    = np.random.randint(-10000,10000,(out_channel, in_channel, kernel_size, kernel_size))/10000;
        self.kernel_weight = np.array([[[[ 0.0106, -0.1561,  0.0984],
                                          [ 0.1468,  0.1580, -0.1404],
                                          [ 0.0856,  0.0780,  0.0636]],
                                         [[-0.1620,  0.2318,  0.0486],
                                          [-0.2214, -0.2046,  0.1070],
                                          [ 0.1609,  0.0160, -0.0374]]],
                                        [[[ 0.1876, -0.2056,  0.1858],
                                          [-0.1288,  0.0065, -0.0145],
                                          [-0.1080,  0.1519,  0.0581]],
                                         [[-0.0749,  0.2289, -0.0890],
                                          [ 0.0611,  0.0398, -0.1293],
                                          [ 0.0911, -0.0264, -0.2104]]]]);
        self.in_channel     = in_channel;
        self.out_channel    = out_channel;
        self.kernel_size    = kernel_size;

    # c*h*w
    def forward(self, input_data):
        output = [];
        input_shape = input_data.shape;

        idx_start   = np.int(np.floor( (self.kernel_size)/2)) ;
        width       = input_shape[1];
        height      = input_shape[2];
        in_channel  = input_shape[0];

        for o_c in range(self.out_channel):
            piece_of_out_channel = np.zeros(( width-(idx_start*2), height-(idx_start*2) ));
            # print("piece_of_out_channel:", piece_of_out_channel.shape)
            # width
            for idx_height in range(idx_start, height - idx_start): 
                # height
                for idx_width in range(idx_start, width - idx_start ):
                    # kernel_shape_input
                    kernel_shape_input = input_data[:, idx_height-idx_start: idx_height+idx_start+1, idx_width-idx_start :idx_width+idx_start+1 ];

                    out = self.kernel_weight[o_c] * kernel_shape_input;
                    out = np.sum(out)

                    # assign value
                    idx_h = idx_height - idx_start;
                    idx_w = idx_width - idx_start
                    piece_of_out_channel[idx_h][idx_w]  = out;

            output.append(piece_of_out_channel);
        return output;



# x = np.random.randint(0,10000,(2, 6, 6))/100;
# x = random.astype(int)
x = np.array([[[61,93,18,31,2,49]
            ,[12,62,32,60,58,30]
            ,[49,64,38,74,59,29]
            ,[71,34,29,88,59,41]
            ,[91,72,36,94,79,29]
            ,[17,15,86,29,84,53]]

            ,[[31,25,15,16,35,20]
            ,[76,45,82,88,49,99]
            ,[56,46,82,72,26,55]
            ,[7, 86,32,29,82,91]
            ,[76,68,17,50,19,53]
            ,[87,21,58,35,81,46]]
            ]);
# print(x);
print("x.shape:",x.shape)

myCNN = MyCNN(x.shape[0], 2, 3);
print(myCNN.kernel_weight)
output = myCNN.forward(x)
print("myCNN output:",output)

# output:
# x.shape: (2, 6, 6)
# myCNN output: [array([[-2.5093,  9.0098, -0.2033, 28.9   ],
#        [ 6.5155, 27.3464,  0.7038, 14.5031],
#        [24.2218, 16.1092, 23.2223, 16.9067],
#        [29.6749,  2.0986, 16.8128, 45.025 ]]), 
#         array([[-14.77  ,  -4.3335,   5.0665,   3.2378],
#        [-28.2207,  18.1968,  11.889 , -27.3557],
#        [ -2.748 ,  22.5508,  10.6013, -19.0372],
#        [ 22.0148,   9.1788, -22.0313,   9.5176]])]

3.1 Use CNN in torch



import torch;
import torch.nn as nn;
class TorchCNN(nn.Module):

    def __init__(self, in_channel, out_channel, kernel_size ):
        super().__init__();
        self.conv2d = nn.Conv2d(in_channel, out_channel, kernel_size, bias=False);

    def forward(self, input_data):
        return self.conv2d(input_data);

in_channel = x.shape[0];
torchcnn = TorchCNN(in_channel, 2, 3);
print("torchcnn weight:", torchcnn.state_dict())
print("torchcnn weight shape:", torchcnn.state_dict()['conv2d.weight'].numpy().shape)
# torchcnn weight shape: (2, 2, 3, 3) => (out_channel, in_channel, kernel_size, kernel_size)
torch_x = torch.FloatTensor([x])
out = torchcnn.forward(torch_x);
print("TorchCNN out: ", out)


# output
# torchcnn weight: OrderedDict([('conv2d.weight', tensor(
#       [[[[ 0.0106, -0.1561,  0.0984],
#           [ 0.1468,  0.1580, -0.1404],
#           [ 0.0856,  0.0780,  0.0636]],
#          [[-0.1620,  0.2318,  0.0486],
#           [-0.2214, -0.2046,  0.1070],
#           [ 0.1609,  0.0160, -0.0374]]],
#         [[[ 0.1876, -0.2056,  0.1858],
#           [-0.1288,  0.0065, -0.0145],
#           [-0.1080,  0.1519,  0.0581]],
#          [[-0.0749,  0.2289, -0.0890],
#           [ 0.0611,  0.0398, -0.1293],
#           [ 0.0911, -0.0264, -0.2104]]]]))])
# TorchCNN out:  tensor([[[[ -2.5066,   9.0144,  -0.1983,  28.9003],
#           [  6.5160,  27.3456,   0.7094,  14.5056],
#           [ 24.2262,  16.1086,  23.2279,  16.9102],
#           [ 29.6763,   2.1047,  16.8210,  45.0250]],
#          [[-14.7564,  -4.3273,   5.0752,   3.2491],
#           [-28.2115,  18.1981,  11.8975, -27.3447],
#           [ -2.7442,  22.5540,  10.6096, -19.0247],
#           [ 22.0166,   9.1837, -22.0241,   9.5211]]]],
#        grad_fn=<MkldnnConvolutionBackward>)

Posted by yaohong Tuesday, May 4, 2021

FEATURED TAGS
ai dart deeplearning flutter keras life network notes other php python rnn tech tensorflow
ABOUT ME
avatar

Software Developer, Lifelong learner