将权重从Keras分配给Torch模型。

huangapple
go评论92阅读模式
英文:

Assign weights from keras to torch model

问题

我正在制作一个用于检查动画脸部是否相似的项目。我尝试自己训练一个模型，但进展不明显。因此，我决定使用已经训练好的模型并调整它以满足我的需求。我在GitHub上找到了这个模型。

模型结构如下：

  1. N_CATEGORIES = 203
  2. IMAGE_SIZE = 224
  4. input_tensor = Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3))
  5. base_model = VGG16(weights='imagenet', include_top=False,input_tensor=input_tensor)
  6. x = base_model.output
  7. x = GlobalAveragePooling2D()(x)
  8. x = Dense(1024, activation='relu')(x)
  9. predictions = Dense(N_CATEGORIES, activation='softmax')(x)
  10. model = Model(inputs=base_model.input, outputs=predictions)
  12. for layer in base_model.layers[:15]:
  13.    layer.trainable = False

Keras模型概要如下：

  1. Model: "model_1"
  2. Layer (type)                Output Shape              Param  
  3.  input_1 (InputLayer)        [(None, 224, 224, 3)]     0         
  4.                                                                  
  5.  block1_conv1 (Conv2D)       (None, 224, 224, 64)      1792      
  6.                                                                  
  7.  block1_conv2 (Conv2D)       (None, 224, 224, 64)      36928     
  8.                                                                  
  9.  block1_pool (MaxPooling2D)  (None, 112, 112, 64)      0         
  10.                                                                  
  11.  block2_conv1 (Conv2D)       (None, 112, 112, 128)     73856     
  12.                                                                  
  13.  block2_conv2 (Conv2D)       (None, 112, 112, 128)     147584    
  14.                                                                  
  15.  block2_pool (MaxPooling2D)  (None, 56, 56, 128)       0         
  16.                                                                  
  17.  block3_conv1 (Conv2D)       (None, 56, 56, 256)       295168    
  18.                                                                  
  19.  block3_conv2 (Conv2D)       (None, 56, 56, 256)       590080    
  20.                                                                  
  21.  block3_conv3 (Conv2D)       (None, 56, 56, 256)       590080    
  22.                                                                  
  23.  block3_pool (MaxPooling2D)  (None, 28, 28, 256)       0         
  24.                                                                  
  25.  block4_conv1 (Conv2D)       (None, 28, 28, 512)       1180160   
  26.                                                                  
  27.  block4_conv2 (Conv2D)       (None, 28, 28, 512)       2359808   
  28.                                                                  
  29.  block4_conv3 (Conv2D)       (None, 28, 28, 512)       2359808   
  30.                                                                  
  31.  block4_pool (MaxPooling2D)  (None, 14, 14, 512)       0         
  32.                                                                  
  33.  block5_conv1 (Conv2D)       (None, 14, 14, 512)       2359808   
  34.                                                                  
  35.  block5_conv2 (Conv2D)       (None, 14, 14, 512)       2359808   
  36.                                                                  
  37.  block5_conv3 (Conv2D)       (None, 14, 14, 512)       2359808   
  38.                                                                  
  39.  block5_pool (MaxPooling2D)  (None, 7, 7, 512)         0         
  40.                                                                  
  41.  global_average_pooling2d_1  (None, 512)               0         
  42.   (GlobalAveragePooling2D)                                       
  43.                                                                  
  44.  dense_1 (Dense)             (None, 1024)              525312    
  45.                                                                  
  46.  dense_2 (Dense)             (None, 203)               208075    
  47.                                                                  
  48. =================================================================

我下载了这个模型的.hdf5文件，但我无法将其转换为.pt格式。我在PyTorch中创建了类似于VGG16的结构：

  1. import torch
  2. import torch.nn as nn
  4. class CustomModel(nn.Module):
  5.     def __init__(self):
  6.         super(CustomModel, self).__init__()
  8.         self.block1 = nn.Sequential(
  9.             nn.Conv2d(3, 64, kernel_size=3, padding=1),
  10.             nn.ReLU(),
  11.             nn.Conv2d(64, 64, kernel_size=3, padding=1),
  12.             nn.ReLU(),
  13.             nn.MaxPool2d(kernel_size=2, stride=2)
  14.         )
  16.         # 你可以继续添加其他块...
  18.         self.fc1 = nn.Linear(512, 1024)
  19.         self.fc2 = nn.Linear(1024, 203)
  21.     def forward(self, x):
  22.         x = self.block1(x)
  23.         # 你可以继续添加其他块...
  24.         x = self.fc1(x)
  25.         x = self.fc2(x)
  26.         return x
  28. pytorch_model = CustomModel()

然后，你可以将Keras模型的权重加载到PyTorch模型中，但需要进行一些额外的处理，因为它们的结构不同。你可以参考此代码示例中的方法。你需要逐层复制权重。

请注意，这可能需要一些手动操作，因为Keras和PyTorch的模型结构不完全相同。你需要确保权重的顺序和形状正确匹配。希望这可以帮助你进行权重转换。

英文:

I am making a project for checking if anime faces is similar or not. I tried training a model by myself, but it doesn't make significant progress. So I decided to take already trained model and tune it to my needs. I found this model on github.

The model structure is:

  1. N_CATEGORIES = 203
  2. IMAGE_SIZE = 224
  3. input_tensor = Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3))
  4. base_model = VGG16(weights='imagenet', include_top=False,input_tensor=input_tensor)
  5. x = base_model.output
  6. x = GlobalAveragePooling2D()(x)
  7. x = Dense(1024, activation='relu')(x)
  8. predictions = Dense(N_CATEGORIES, activation='softmax')(x)
  9. model = Model(inputs=base_model.input, outputs=predictions)
  10. for layer in base_model.layers[:15]:
  11. layer.trainable = False

Keras model summary:

  1. Model: "model_1"
  2. Layer (type)                Output Shape              Param  
  3. input_1 (InputLayer)        [(None, 224, 224, 3)]     0         
  4. block1_conv1 (Conv2D)       (None, 224, 224, 64)      1792      
  5. block1_conv2 (Conv2D)       (None, 224, 224, 64)      36928     
  6. block1_pool (MaxPooling2D)  (None, 112, 112, 64)      0         
  7. block2_conv1 (Conv2D)       (None, 112, 112, 128)     73856     
  8. block2_conv2 (Conv2D)       (None, 112, 112, 128)     147584    
  9. block2_pool (MaxPooling2D)  (None, 56, 56, 128)       0         
  10. block3_conv1 (Conv2D)       (None, 56, 56, 256)       295168    
  11. block3_conv2 (Conv2D)       (None, 56, 56, 256)       590080    
  12. block3_conv3 (Conv2D)       (None, 56, 56, 256)       590080    
  13. block3_pool (MaxPooling2D)  (None, 28, 28, 256)       0         
  14. block4_conv1 (Conv2D)       (None, 28, 28, 512)       1180160   
  15. block4_conv2 (Conv2D)       (None, 28, 28, 512)       2359808   
  16. block4_conv3 (Conv2D)       (None, 28, 28, 512)       2359808   
  17. block4_pool (MaxPooling2D)  (None, 14, 14, 512)       0         
  18. block5_conv1 (Conv2D)       (None, 14, 14, 512)       2359808   
  19. block5_conv2 (Conv2D)       (None, 14, 14, 512)       2359808   
  20. block5_conv3 (Conv2D)       (None, 14, 14, 512)       2359808   
  21. block5_pool (MaxPooling2D)  (None, 7, 7, 512)         0         
  22. global_average_pooling2d_1  (None, 512)               0         
  23. (GlobalAveragePooling2D)                                       
  24. dense_1 (Dense)             (None, 1024)              525312    
  25. dense_2 (Dense)             (None, 203)               208075    
  26. =================================================================

I downloaded an .hdf5 file for this model. By I can't convert it to .pt format. I made similar to vgg16 structure in torch:

  1. import torch
  2. import torch.nn as nn
  3. class CustomModel(nn.Module):
  4. def __init__(self):
  5. super(CustomModel, self).__init__()
  6. self.block1 = nn.Sequential(
  7. nn.Conv2d(3, 64, kernel_size=3, padding=1),
  8. nn.ReLU(),
  9. nn.Conv2d(64, 64, kernel_size=3, padding=1),
  10. nn.ReLU(),
  11. nn.MaxPool2d(kernel_size=2, stride=2)
  12. )
  13. self.block2 = nn.Sequential(
  14. nn.Conv2d(64, 128, kernel_size=3, padding=1),
  15. nn.ReLU(),
  16. nn.Conv2d(128, 128, kernel_size=3, padding=1),
  17. nn.ReLU(),
  18. nn.MaxPool2d(kernel_size=2, stride=2)
  19. )
  20. self.block3 = nn.Sequential(
  21. nn.Conv2d(128, 256, kernel_size=3, padding=1),
  22. nn.ReLU(),
  23. nn.Conv2d(256, 256, kernel_size=3, padding=1),
  24. nn.ReLU(),
  25. nn.Conv2d(256, 256, kernel_size=3, padding=1),
  26. nn.ReLU(),
  27. nn.MaxPool2d(kernel_size=2, stride=2)
  28. )
  29. self.block4 = nn.Sequential(
  30. nn.Conv2d(256, 512, kernel_size=3, padding=1),
  31. nn.ReLU(),
  32. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  33. nn.ReLU(),
  34. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  35. nn.ReLU(),
  36. nn.MaxPool2d(kernel_size=2, stride=2)
  37. )
  38. self.block5 = nn.Sequential(
  39. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  40. nn.ReLU(),
  41. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  42. nn.ReLU(),
  43. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  44. nn.ReLU(),
  45. nn.MaxPool2d(kernel_size=2, stride=2)
  46. )
  47. self.global_avg_pool = nn.AdaptiveAvgPool2d((1, 1))
  48. self.fc1 = nn.Linear(512, 1024)
  49. self.fc2 = nn.Linear(1024, 203)
  50. def forward(self, x):
  51. x = self.block1(x)
  52. x = self.block2(x)
  53. x = self.block3(x)
  54. x = self.block4(x)
  55. x = self.block5(x)
  56. x = self.global_avg_pool(x)
  57. x = torch.flatten(x, 1)
  58. x = self.fc1(x)
  59. x = self.fc2(x)
  60. return x
  61. pytorch_model = CustomModel()

I imported weight to keras:

  1. from keras.models import load_model
  2. model = load_model('animeface_vgg16.hdf5')
  3. weights=model.get_weights()

Tried to follow method like in this code example. I understand how to assing keras weights to fc layers. But there wasn't any example with convolutional layers.

I found only examples of building model structure in other questions on stack overflow, but nobody showed how to assign weights to it.

Please explain me how to do it.

答案1

得分: 0

  1. 找到了答案。以下是模型结构：
  3. ```python
  4. import torch
  5. import torch.nn as nn
  7. class CustomModel(nn.Module):
  8.     def __init__(self, n_categories):
  9.         super(CustomModel, self).__init__()
  11.         self.block1 = nn.Sequential(
  12.             nn.Conv2d(3, 64, kernel_size=3, padding=1),
  13.             nn.ReLU(),
  14.             nn.Conv2d(64, 64, kernel_size=3, padding=1),
  15.             nn.ReLU(),
  16.             nn.MaxPool2d(kernel_size=2, stride=2)
  17.         )
  19.         self.block2 = nn.Sequential(
  20.             nn.Conv2d(64, 128, kernel_size=3, padding=1),
  21.             nn.ReLU(),
  22.             nn.Conv2d(128, 128, kernel_size=3, padding=1),
  23.             nn.ReLU(),
  24.             nn.MaxPool2d(kernel_size=2, stride=2)
  25.         )
  27.         self.block3 = nn.Sequential(
  28.             nn.Conv2d(128, 256, kernel_size=3, padding=1),
  29.             nn.ReLU(),
  30.             nn.Conv2d(256, 256, kernel_size=3, padding=1),
  31.             nn.ReLU(),
  32.             nn.Conv2d(256, 256, kernel_size=3, padding=1),
  33.             nn.ReLU(),
  34.             nn.MaxPool2d(kernel_size=2, stride=2)
  35.         )
  37.         self.block4 = nn.Sequential(
  38.             nn.Conv2d(256, 512, kernel_size=3, padding=1),
  39.             nn.ReLU(),
  40.             nn.Conv2d(512, 512, kernel_size=3, padding=1),
  41.             nn.ReLU(),
  42.             nn.Conv2d(512, 512, kernel_size=3, padding=1),
  43.             nn.ReLU(),
  44.             nn.MaxPool2d(kernel_size=2, stride=2)
  45.         )
  47.         self.block5 = nn.Sequential(
  48.             nn.Conv2d(512, 512, kernel_size=3, padding=1),
  49.             nn.ReLU(),
  50.             nn.Conv2d(512, 512, kernel_size=3, padding=1),
  51.             nn.ReLU(),
  52.             nn.Conv2d(512, 512, kernel_size=3, padding=1),
  53.             nn.ReLU(),
  54.             nn.MaxPool2d(kernel_size=2, stride=2),
  55.             nn.AdaptiveAvgPool2d((1, 1))
  56.         )
  58.         self.dense = nn.Sequential(
  59.             nn.Linear(512, 1024),
  60.             nn.ReLU(),
  61.             nn.Linear(1024, n_categories),
  62.             nn.Softmax(dim=0)
  63.         )
  65.     def forward(self, x):
  66.         x = self.block1(x)
  67.         x = self.block2(x)
  68.         x = self.block3(x)
  69.         x = self.block4(x)
  70.         x = self.block5(x)
  71.         x = torch.flatten(x)
  72.         x = self.dense(x)
  73.         return x
  75. N_CATEGORIES  = 203
  76. pytorch_model = CustomModel(N_CATEGORIES)

加载权重：

  1. pytorch_model.block1[0].weight.data = torch.from_numpy(np.transpose(weights[0]))
  2. pytorch_model.block1[0].bias.data = torch.from_numpy(weights[1])
  3. pytorch_model.block1[2].weight.data = torch.from_numpy(np.transpose(weights[2]))
  4. pytorch_model.block1[2].bias.data = torch.from_numpy(weights[3])
  6. pytorch_model.block2[0].weight.data = torch.from_numpy(np.transpose(weights[4]))
  7. pytorch_model.block2[0].bias.data = torch.from_numpy(weights[5])
  8. pytorch_model.block2[2].weight.data = torch.from_numpy(np.transpose(weights[6]))
  9. pytorch_model.block2[2].bias.data = torch.from_numpy(weights[7])
  11. pytorch_model.block3[0].weight.data = torch.from_numpy(np.transpose(weights[8]))
  12. pytorch_model.block3[0].bias.data = torch.from_numpy(weights[9])
  13. pytorch_model.block3[2].weight.data = torch.from_numpy(np.transpose(weights[10]))
  14. pytorch_model.block3[2].bias.data = torch.from_numpy(weights[11])
  15. pytorch_model.block3[4].weight.data = torch.from_numpy(np.transpose(weights[12]))
  16. pytorch_model.block3[4].bias.data = torch.from_numpy(weights[13])
  18. pytorch_model.block4[0].weight.data = torch.from_numpy(np.transpose(weights[14]))
  19. pytorch_model.block4[0].bias.data = torch.from_numpy(weights[15])
  20. pytorch_model.block4[2].weight.data = torch.from_numpy(np.transpose(weights[16]))
  21. pytorch_model.block4[2].bias.data = torch.from_numpy(weights[17])
  22. pytorch_model.block4[4].weight.data = torch.from_numpy(np.transpose(weights[18]))
  23. pytorch_model.block4[4].bias.data = torch.from_numpy(weights[19])
  25. pytorch_model.block5[0].weight.data = torch.from_numpy(np.transpose(weights[20]))
  26. pytorch_model.block5[0].bias.data = torch.from_numpy(weights[21])
  27. pytorch_model.block5[2].weight.data = torch.from_numpy(np.transpose(weights[22]))
  28. pytorch_model.block5[2].bias.data = torch.from_numpy(weights[23])
  29. pytorch_model.block5[4].weight.data = torch.from_numpy(np.transpose(weights[24]))
  30. pytorch_model.block5[4].bias.data = torch.from_numpy(weights[25])
  32. pytorch_model.dense[0].weight.data = torch.from_numpy(np.transpose(weights[26]))
  33. pytorch_model.dense[0].bias.data = torch.from_numpy(weights[27])
  35. pytorch_model.dense[2].weight.data = torch.from_numpy(np.transpose(weights[28]))
  36. pytorch_model.dense[2].bias.data = torch.from_numpy(weights[29])
英文:

Found an answer by myself. Here is a model structure:

  1. import torch
  2. import torch.nn as nn
  3. class CustomModel(nn.Module):
  4. def __init__(self, n_categories):
  5. super(CustomModel, self).__init__()
  6. self.block1 = nn.Sequential(
  7. nn.Conv2d(3, 64, kernel_size=3, padding=1),
  8. nn.ReLU(),
  9. nn.Conv2d(64, 64, kernel_size=3, padding=1),
  10. nn.ReLU(),
  11. nn.MaxPool2d(kernel_size=2, stride=2)
  12. )
  13. self.block2 = nn.Sequential(
  14. nn.Conv2d(64, 128, kernel_size=3, padding=1),
  15. nn.ReLU(),
  16. nn.Conv2d(128, 128, kernel_size=3, padding=1),
  17. nn.ReLU(),
  18. nn.MaxPool2d(kernel_size=2, stride=2)
  19. )
  20. self.block3 = nn.Sequential(
  21. nn.Conv2d(128, 256, kernel_size=3, padding=1),
  22. nn.ReLU(),
  23. nn.Conv2d(256, 256, kernel_size=3, padding=1),
  24. nn.ReLU(),
  25. nn.Conv2d(256, 256, kernel_size=3, padding=1),
  26. nn.ReLU(),
  27. nn.MaxPool2d(kernel_size=2, stride=2)
  28. )
  29. self.block4 = nn.Sequential(
  30. nn.Conv2d(256, 512, kernel_size=3, padding=1),
  31. nn.ReLU(),
  32. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  33. nn.ReLU(),
  34. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  35. nn.ReLU(),
  36. nn.MaxPool2d(kernel_size=2, stride=2)
  37. )
  38. self.block5 = nn.Sequential(
  39. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  40. nn.ReLU(),
  41. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  42. nn.ReLU(),
  43. nn.Conv2d(512, 512, kernel_size=3, padding=1),
  44. nn.ReLU(),
  45. nn.MaxPool2d(kernel_size=2, stride=2),
  46. nn.AdaptiveAvgPool2d((1, 1))
  47. )
  48. self.dense = nn.Sequential(
  49. nn.Linear(512, 1024),
  50. nn.ReLU(),
  51. nn.Linear(1024, n_categories),
  52. nn.Softmax(dim=0)
  53. )
  54. def forward(self, x):
  55. x = self.block1(x)
  56. x = self.block2(x)
  57. x = self.block3(x)
  58. x = self.block4(x)
  59. x = self.block5(x)
  60. x = torch.flatten(x)
  61. x = self.dense(x)
  62. return x
  63. N_CATEGORIES  = 203
  64. pytorch_model = CustomModel(N_CATEGORIES)

It's a loading of weights:

  1. pytorch_model.block1[0].weight.data = torch.from_numpy(np.transpose(weights[0]))
  2. pytorch_model.block1[0].bias.data = torch.from_numpy(weights[1])
  3. pytorch_model.block1[2].weight.data = torch.from_numpy(np.transpose(weights[2]))
  4. pytorch_model.block1[2].bias.data = torch.from_numpy(weights[3])
  5. pytorch_model.block2[0].weight.data = torch.from_numpy(np.transpose(weights[4]))
  6. pytorch_model.block2[0].bias.data = torch.from_numpy(weights[5])
  7. pytorch_model.block2[2].weight.data = torch.from_numpy(np.transpose(weights[6]))
  8. pytorch_model.block2[2].bias.data = torch.from_numpy(weights[7])
  9. pytorch_model.block3[0].weight.data = torch.from_numpy(np.transpose(weights[8]))
  10. pytorch_model.block3[0].bias.data = torch.from_numpy(weights[9])
  11. pytorch_model.block3[2].weight.data = torch.from_numpy(np.transpose(weights[10]))
  12. pytorch_model.block3[2].bias.data = torch.from_numpy(weights[11])
  13. pytorch_model.block3[4].weight.data = torch.from_numpy(np.transpose(weights[12]))
  14. pytorch_model.block3[4].bias.data = torch.from_numpy(weights[13])
  15. pytorch_model.block4[0].weight.data = torch.from_numpy(np.transpose(weights[14]))
  16. pytorch_model.block4[0].bias.data = torch.from_numpy(weights[15])
  17. pytorch_model.block4[2].weight.data = torch.from_numpy(np.transpose(weights[16]))
  18. pytorch_model.block4[2].bias.data = torch.from_numpy(weights[17])
  19. pytorch_model.block4[4].weight.data = torch.from_numpy(np.transpose(weights[18]))
  20. pytorch_model.block4[4].bias.data = torch.from_numpy(weights[19])
  21. pytorch_model.block5[0].weight.data = torch.from_numpy(np.transpose(weights[20]))
  22. pytorch_model.block5[0].bias.data = torch.from_numpy(weights[21])
  23. pytorch_model.block5[2].weight.data = torch.from_numpy(np.transpose(weights[22]))
  24. pytorch_model.block5[2].bias.data = torch.from_numpy(weights[23])
  25. pytorch_model.block5[4].weight.data = torch.from_numpy(np.transpose(weights[24]))
  26. pytorch_model.block5[4].bias.data = torch.from_numpy(weights[25])
  27. pytorch_model.dense[0].weight.data = torch.from_numpy(np.transpose(weights[26]))
  28. pytorch_model.dense[0].bias.data = torch.from_numpy(weights[27])
  29. pytorch_model.dense[2].weight.data = torch.from_numpy(np.transpose(weights[28]))
  30. pytorch_model.dense[2].bias.data = torch.from_numpy(weights[29])

答案2

得分: 0

一个简单的方法是直接将模型从Keras转换为PyTorch，使用ONNX作为中间模型。可以在几个步骤内轻松完成：

步骤1：将Keras模型保存为ONNX

  1. from tensorflow.python.keras import backend as K
  2. from tensorflow.python.keras.models import load_model
  3. import onnx
  4. import keras2onnx
  6. onnx_model_name = 'baselineonnxmodel.onnx'
  8. model = load_model('baselinekerasmodel.h5')
  9. onnx_model = keras2onnx.convert_keras(model, model.name)
  10. onnx.save_model(onnx_model, onnx_model_name)

步骤2：加载ONNX模型并转换为PyTorch。使用pip install onnx2pytorch安装onnx2pytorch。

  1. import onnx
  2. from onnx2pytorch import ConvertModel
  4. onnx_model = onnx.load('baselineonnxmodel.onnx')
  5. pytorch_model = ConvertModel(onnx_model)

现在，您可以使用pytorch_model进行预测。

英文:

A easy way to do it is to convert the model directly from keras to pytorch using onnx as the intermediate model. This can be easily done in a couple of steps:

Step 1: Save Keras model as ONNX

  1. from tensorflow.python.keras import backend as K
  2. from tensorflow.python.keras.models import load_model
  3. import onnx
  4. import keras2onnx
  6. onnx_model_name = 'baselineonnxmodel.onnx'
  8. model = load_model('baselinekerasmodel.h5')
  9. onnx_model = keras2onnx.convert_keras(model, model.name)
  10. onnx.save_model(onnx_model, onnx_model_name)

Step 2: Load ONNX model and convert to pytorch. Install onnx2pytorch using pip install onnx2pytorch

  1. import onnx
  2. from onnx2pytorch import ConvertModel
  4. onnx_model = onnx.load('baselineonnxmodel.onnx')
  5. pytorch_model = ConvertModel(onnx_model)

You can now use the pytorch_model for making predictions.

huangapple
  • 本文由 发表于 2023年7月31日 19:17:46
  • 转载请务必保留本文链接：https://go.coder-hub.com/76803096.html
  • deep-learning
  • keras
  • pytorch
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定