"> Pytorch简单笔记 | Stillwtm's Blog

Stillwtm's Blog
0%

Pytorch简单笔记

发表于 分类于 阅读数： 评论数：
Pytorch张量有关基本操作介绍

PyTorch简单笔记

张量(tensor)基础

构造张量

  • 可以从list构造，用[]访问和修改，下标访问直接得到的是PyTorch Scalar
  • 有.dim(), .shape方法，.item()可以将PyTorch scalar转换为普通的python scalar
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Create a two-dimensional tensor
b = torch.tensor([[1, 2, 3], [4, 5, 5]])
print('Here is b:')
print(b)
print('rank of b:', b.dim())
print('b.shape: ', b.shape)

# Access elements from a multidimensional tensor
print()
print('b[0, 1]:', b[0, 1])
print('b[1, 2]:', b[1, 2])

# Mutate elements of a multidimensional tensor
b[1, 1] = 100
print()
print('b after mutating:')
print(b)

内置的构造

  • torch.zeros: Creates a tensor of all zeros
  • torch.ones: Creates a tensor of all ones
  • torch.rand: Creates a tensor with uniform random numbers
  • torch.eye: 创建一个单位张量
  • torch.empty: 使用无用数据进行填充
  • torch.full: 所有元素相同的张量
  • 以及其他的选择 in the documentation

数据类型

  • large set of numeric datatypes
  • tensor的.dtype属性可以查看类型，也可以指定类型
  • 也可以使用.to(), .float()之类的方法进行类型转换
1
2
3
4
5
6
7
8
# Force a particular datatype
x0 = torch.tensor([1, 2], dtype=torch.float32)  # 32-bit float
y0 = torch.eye(3, dtype=torch.int64)

x1 = x0.float()  # Cast to 32-bit float
x2 = x0.double() # Cast to 64-bit float
x3 = x0.to(torch.float32) # Alternate way to cast to 32-bit float
x4 = x0.to(torch.float64) # Alternate way to cast to 64-bit float
  • 构造一些与已有tensor相同类型的tensor
1
2
3
4
x0 = torch.eye(3, dtype=torch.float64)  # Shape (3, 3), dtype torch.float64
x1 = torch.zeros_like(x0)               # Shape (3, 3), dtype torch.float64
x2 = x0.new_zeros(4, 5)                 # Shape (4, 5), dtype torch.float64
x3 = torch.ones(6, 7).to(x0)            # Shape (6, 7), dtype torch.float64)

tensor索引

切片索引

  • tensor支持[start:stop:step]式的索引
  • 切片得到的是原tensor的一个view，改变切片也会改变原tensor，规避方法是采用.clone()
1
2
3
4
5
6
7
# Create the following rank 2 tensor with shape (3, 4)
a = torch.tensor([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
print('Original tensor')
print(a)

row_r1 = a[1, :]    # Rank 1 view of the second row of a  
row_r2 = a[1:2, :]  # Rank 2 view of the second row of a

似乎采用a[[1], :]可以得到一个新的tensor而非原tensor的view，同时形式和a[1:2, :]类似

  • 支持a[1:3, 2: 4] = 2形式的赋值

整型索引

  • 使用索引数组得到全新的tensor

    • a[idx0, idx1]相当于

      1
      2
      3
      4
      5
      6
      torch.tensor([
        a[idx0[0], idx1[0]],
        a[idx0[1], idx1[1]],
        ...,
        a[idx0[N - 1], idx1[N - 1]]
      ])

布尔tensor索引

  • 使用一个torch.bool类型的mask进行索引
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
a = torch.tensor([[1,2], [3, 4], [5, 6]])
print('Original tensor:')
print(a)

# Find the elements of a that are bigger than 3. The mask has the same shape as
# a, where each element of mask tells whether the corresponding element of a
# is greater than three.
mask = (a > 3)
print('\nMask tensor:')
print(mask)

# We can use the mask to construct a rank-1 tensor containing the elements of a
# that are selected by the mask
print('\nSelecting elements with the mask:')
print(a[mask])
# tensor([4, 5, 6])

# We can also use boolean masks to modify tensors; for example this sets all
# elements <= 3 to zero:
a[a <= 3] = 0
print('\nAfter modifying with a mask:')
print(a)
# tensor([[0, 0],
#         [0, 4],
#         [5, 6]])

改变形状

View

  • .view()可以将tensor重塑形状，并返回该tensor的一个view

    • .view(-1)会令-1的维度大小保持和原来相同

    似乎是按行展开重塑的

交换维度

  • .t()直接进行转置
  • .transpose()选择任意两个维度进行交换
  • .permute()重新将维度进行排序
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# Create a tensor of shape (2, 3, 4)
x0 = torch.tensor([
     [[1,  2,  3,  4],
      [5,  6,  7,  8],
      [9, 10, 11, 12]],
     [[13, 14, 15, 16],
      [17, 18, 19, 20],
      [21, 22, 23, 24]]])

# Swap axes 1 and 2; shape is (2, 4, 3)
x1 = x0.transpose(1, 2)

# Permute axes; the argument (1, 2, 0) means:
# - Make the old dimension 1 appear at dimension 0;
# - Make the old dimension 2 appear at dimension 1;
# - Make the old dimension 0 appear at dimension 2
# This results in a tensor of shape (3, 4, 2)
x2 = x0.permute(1, 2, 0)

Contiguous

  • 一串改变tensor形状的操作连在一起可能会导致奇妙的错误，这是由于内部实现导致的
  • 可以通过.reshape()代替.view()或者在.view()之前添加.contiguous()来规避问题
1
2
3
4
5
6
7
8
9
10
11
12
13
x0 = torch.randn(2, 3, 4)

try:
  # This sequence of reshape operations will crash
  x1 = x0.transpose(1, 2).view(8, 3)
except RuntimeError as e:
  print(type(e), e)
  
# We can solve the problem using either .contiguous() or .reshape()
x1 = x0.transpose(1, 2).contiguous().view(8, 3)
x2 = x0.transpose(1, 2).reshape(8, 3)
print('x1 shape: ', x1.shape)
print('x2 shape: ', x2.shape)

张量运算

逐元素运算

  • 有重载的+ - * / **，并且也有对应的方法
  • .sqrt()，.sin()之类的方法，可以参考in the documentation

Reduction operation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
x = torch.tensor([[1, 2, 3], 
                  [4, 5, 6]], dtype=torch.float32)

print('\nSum over entire tensor:')
print(torch.sum(x))
print(x.sum())

# We can sum over each row:
print(torch.sum(x, dim=0))
print(x.sum(dim=0))

# Sum over each column:
print(torch.sum(x, dim=1))
print(x.sum(dim=1))

#Sum of each row:
#tensor([5., 7., 9.])
#tensor([5., 7., 9.])

#Sum of each column:
#tensor([ 6., 15.])
#tensor([ 6., 15.])
  • dim参数描述要沿着哪个维度做缩减，也就是说做完缩减这个维度会删除
  • 如果keepdim=True，就会仍然保留这个维度，且该维度的值为1
1
2
3
4
5
6
7
8
9
10
11
# Create a tensor of shape (128, 10, 3, 64, 64)
x = torch.randn(128, 10, 3, 64, 64)

# Take the mean over dimension 1; shape is now (128, 3, 64, 64)
x = x.mean(dim=1)
# Take the sum over dimension 2; shape is now (128, 3, 64)
x = x.sum(dim=2)

# Take the mean over dimension 1, but keep the dimension from being eliminated
# by passing keepdim=True; shape is now (128, 1, 64)
x = x.mean(dim=1, keepdim=True)

矩阵运算

Vectorization

  • 尽量避免显式地使用循环，而是调用Pytorch的运算来隐式完成循环操作，这通常会快得多，这种代码风格称为vectorization

广播

  • 广播的规则如下

    • 把低维度的张量扩展成高维度的，新增的维度大小为1
    • 如果两个张量在某维度上大小相同或者某个张量在该维度大小为1，称它们在这个维度上是可比较的
    • 当两个张量在所有维度都可比较时，可以进行广播
    • 广播之后，两个张量每个维度的大小都是最大的那个
    • 如果一个张量在某维度大小是1，另外一个大于1，广播的表现则是将前者沿着该维度进行复制
    • 详尽解释在documentation

    第一步中的添加维度似乎是从头（第0维）插入进去

  • 广播通常隐式进行，但也有显式的方法torch.broadcast_tensors

Out-of-place vs in-place

  • 大多数PyTorch运算被分成两类

    • Out-of-place operators: 返回一个新的tensor，大多数PyTorch运算表现如此

    • In-place operators: 修改并返回输入的tensor。带下划线的实例方法 （比如add_()）是in-place的。在 torch 命名空间内的运算可以用 out= 参数使其in-place

  • 通常应该避免使用in-place因为他们在自动求导计算梯度的时候可能会出问题

GPU加速

  • tensor有device属性用来表征tensor储存在哪里，CPU还是CUDA(for Nvidia GPUs)
  • .to()方法可以转换tensor的device属性，也可以用.cuda()和.cpu()，更加方便