Tensorflow: Filter中的3D张量（图像的）形状有None。

huangapple
go评论104阅读模式
英文:

Tensorflow: Shape of 3D tensor (of an image) in Filter has None

问题

以下是您要翻译的部分：

I am following [this tutorial][1] on [tensorflow.org][2].

I have folder images with two folders cat and dog in it. Following above tutorial I am trying to convert .jpg and ..png images to features (NumPy array) for modeling.

Problem

After processing the images to tensors I found that some images were converted to tensor with shape (28, 28, 4). So I added condition to filter out such tensors. This logic works while explicitly looping each tensor, using for loop, after converting it to numpy array, but same logic does not work when used with filter.

Please help me fix this filter() I went through [filter()][3] documentation and could not find any solution.

Source code

  1. import tensorflow as tf
  2. import os
  4. print("TensorFlow version:", tf.__version__)
  6. def process_image(file_path_tensor):
  7.     parts = tf.strings.split(file_path_tensor, os.sep)
  8.     label = parts[-2]
  10.     image = tf.io.read_file(file_path_tensor)
  11.     image = tf.image.decode_jpeg(image)
  12.     image = tf.image.resize(image, [128, 128])
  13.     image = tf.image.convert_image_dtype(image, tf.float32)
  14.     image = image / 255
  16.     return image, label
  19. def check_shape(x, y):
  20.     print("\nShape received in filter():", x.shape)
  21.     d1, d2, d3 = x.shape
  22.     return d3 == 3
  25. images_ds = tf.data.Dataset.list_files("./images/*/*", shuffle=True)
  27. file_path = next(iter(images_ds))
  28. image, label = process_image(file_path)
  30. print("Shape:", image.shape)
  31. print("Class label:", label.numpy().decode())
  33. # ETL pipeline.
  34. X_y_tensors = (
  35.     images_ds
  36.     .map(process_image)   # Extra and Transform
  37.     .filter(check_shape)  # Filter
  38.     .as_numpy_iterator()  # Load
  39. )
  41. print("\nTechnique 1:")
  42. print("Final X count:", len(list(X_y_tensors)))
  45. X_y_tensors = images_ds.map(process_image)
  47. count = 0
  48. for x, y in X_y_tensors:
  49.     d1, d2, d3 = x.shape
  50.     if d3 > 3:
  51.         continue
  52.     count += 1
  54. print("\nTechnique 2:")
  55. print("Final X count:", count)

Output

  1. TensorFlow version: 2.6.0
  2. Shape: (128, 128, 3)
  3. Class label: cat
  5. Shape received in filter(): (128, 128, None)
  7. Technique 1:
  8. Final X count: 0
  10. Technique 2:
  11. Final X count: 123
  12. ```
  14. As it can be seen,
  16. 1. Count is 0 when _Technique 1_ is used to filter tensors, since the shape of the tensor received is `(128, 128, None)`.
  17. 1. Count is 123 (image count after filtering) when _Technique 2_ is used.
  19. I do not think [this][5] is an issue since I am **not using batches**.
  21. [Full code with dataset][4]
  23. [1]: https://www.tensorflow.org/guide/data#preprocessing_data
  24. [2]: https://www.tensorflow.org/
  25. [3]: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#filter
  26. [4]: https://github.com/DheemanthBhat/tensorflow-image-pipeline
  27. [5]: https://stackoverflow.com/questions/58331837/filter-data-in-tensorflow
  29. <details>
  30. <summary>英文:</summary>
  32. I am following [this tutorial][1] on [tensorflow.org][2].
  34. I have folder _images_ with two folders _cat_ and _dog_ in it. Following above tutorial I am trying to convert .jpg and .png images to features (NumPy array) for modeling.  
  36. ### Problem
  38. After processing the images to tensors I found that some images were converted to tensor with shape `(28, 28, 4)`. So I added condition to filter out such tensors. This logic works while explicitly looping each tensor, using `for` loop, after converting it to numpy array, but same logic does not work when used with `filter`.
  40. Please help me fix this `filter()` I went through [`filter()`][3] documentation and could not find any solution.
  42. ### Source code
  44. ```python
  45. import tensorflow as tf
  46. import os
  48. print(&quot;TensorFlow version:&quot;, tf.__version__)
  50. def process_image(file_path_tensor):
  51.     parts = tf.strings.split(file_path_tensor, os.sep)
  52.     label = parts[-2]
  54.     image = tf.io.read_file(file_path_tensor)
  55.     image = tf.image.decode_jpeg(image)
  56.     image = tf.image.resize(image, [128, 128])
  57.     image = tf.image.convert_image_dtype(image, tf.float32)
  58.     image = image / 255
  60.     return image, label
  63. def check_shape(x, y):
  64.     print(&quot;\nShape received in filter():&quot;, x.shape)
  65.     d1, d2, d3 = x.shape
  66.     return d3 == 3
  69. images_ds = tf.data.Dataset.list_files(&quot;./images/*/*&quot;, shuffle=True)
  71. file_path = next(iter(images_ds))
  72. image, label = process_image(file_path)
  74. print(&quot;Shape:&quot;, image.shape)
  75. print(&quot;Class label:&quot;, label.numpy().decode())
  77. # ETL pipeline.
  78. X_y_tensors = (
  79.     images_ds
  80.     .map(process_image)   # Extra and Transform
  81.     .filter(check_shape)  # Filter
  82.     .as_numpy_iterator()  # Load
  83. )
  85. print(&quot;\nTechnique 1:&quot;)
  86. print(&quot;Final X count:&quot;, len(list(X_y_tensors)))
  89. X_y_tensors = images_ds.map(process_image)
  91. count = 0
  92. for x, y in X_y_tensors:
  93.     d1, d2, d3 = x.shape
  94.     if d3 &gt; 3:
  95.         continue
  96.     count += 1
  98. print(&quot;\nTechnique 2:&quot;)
  99. print(&quot;Final X count:&quot;, count)
  100. ```
  102. ### Output
  103. ```
  104. TensorFlow version: 2.6.0
  105. Shape: (128, 128, 3)
  106. Class label: cat
  108. Shape received in filter(): (128, 128, None)
  110. Technique 1:
  111. Final X count: 0
  113. Technique 2:
  114. Final X count: 123
  115. ```
  117. As it can be seen,
  119. 1. Count is 0 when _Technique 1_ is used to filter tensors, since the shape of the tensor received is `(128, 128, None)`.
  120. 1. Count is 123 (image count after filtering) when _Technique 2_ is used.
  122. I do not think [this][5] is an issue since I am **not using batches**.
  124. [Full code with dataset][4]
  126. [1]: https://www.tensorflow.org/guide/data#preprocessing_data
  127. [2]: https://www.tensorflow.org/
  128. [3]: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#filter
  129. [4]: https://github.com/DheemanthBhat/tensorflow-image-pipeline
  130. [5]: https://stackoverflow.com/questions/58331837/filter-data-in-tensorflow
  132. </details>
  135. # 答案1
  136. **得分**: 0
  138. 问题与`filter()`无关，而是由`decode_jpeg()`引起的。将`tf.io.decode_jpeg()`的[`channels`][1]参数设置为3。
  140. 数字3表示图像的RGB颜色通道。
  142. 解决方案
  144. ```python
  145. import tensorflow as tf
  146. import os
  148. print("TensorFlow版本:", tf.__version__)
  150. def process_image(file_path_tensor):
  151.     parts = tf.strings.split(file_path_tensor, os.sep)
  152.     label = parts[-2]
  154.     image = tf.io.read_file(file_path_tensor)
  155.     image = tf.image.decode_jpeg(image, channels=3)
  156.     image = tf.image.resize(image, [128, 128])
  157.     image = tf.image.convert_image_dtype(image, tf.float32)
  158.     image = image / 255
  160.     return image, label
  163. def check_shape(x, y):
  164.     print("\n在filter()中接收到的形状:", x.shape)
  165.     d1, d2, d3 = x.shape
  166.     return d3 == 3
  169. images_ds = tf.data.Dataset.list_files("./images/*/*", shuffle=True)
  171. file_path = next(iter(images_ds))
  172. image, label = process_image(file_path)
  174. print("形状:", image.shape)
  175. print("类别标签:", label.numpy().decode())
  177. # ETL流水线。
  178. X_y_tensors = (
  179.     images_ds
  180.     .map(process_image)   # 提取和转换
  181.     .filter(check_shape)  # 过滤
  182.     .as_numpy_iterator()  # 载入
  183. )
  185. print("\n技巧 1:")
  186. print("最终 X 计数:", len(list(X_y_tensors)))
  189. X_y_tensors = images_ds.map(process_image)
  191. count = 0
  192. for x, y in X_y_tensors:
  193.     d1, d2, d3 = x.shape
  194.     if d3 > 3:
  195.         continue
  196.     count += 1
  198. print("\n技巧 2:")
  199. print("最终 X 计数:", count)
  200. ```
  202. 输出
  204. ```python
  205. TensorFlow版本: 2.6.0
  206. 形状: (128, 128, 3)
  207. 类别标签: 猫
  209. 在filter()中接收到的形状: (128, 128, 3)
  211. 技巧 1:
  212. 最终 X 计数: 129
  214. 技巧 2:
  215. 最终 X 计数: 129
  216. ```
  218. [1]: https://www.tensorflow.org/api_docs/python/tf/io/decode_jpeg#args
  221. <details>
  222. <summary>英文:</summary>
  224. Okay. So the problem is not related to `filter()` its caused by `decode_jpeg()`. Set [`channels`][1] parameter of `tf.io.decode_jpeg()` to value 3. 
  226. Three indicates RGB color channels of an image.
  228. ### Solution
  230. ```python
  231. import tensorflow as tf
  232. import os
  234. print(&quot;TensorFlow version:&quot;, tf.__version__)
  236. def process_image(file_path_tensor):
  237.     parts = tf.strings.split(file_path_tensor, os.sep)
  238.     label = parts[-2]
  240.     image = tf.io.read_file(file_path_tensor)
  241.     image = tf.image.decode_jpeg(image, channels=3)
  242.     image = tf.image.resize(image, [128, 128])
  243.     image = tf.image.convert_image_dtype(image, tf.float32)
  244.     image = image / 255
  246.     return image, label
  249. def check_shape(x, y):
  250.     print(&quot;\nShape received in filter():&quot;, x.shape)
  251.     d1, d2, d3 = x.shape
  252.     return d3 == 3
  255. images_ds = tf.data.Dataset.list_files(&quot;./images/*/*&quot;, shuffle=True)
  257. file_path = next(iter(images_ds))
  258. image, label = process_image(file_path)
  260. print(&quot;Shape:&quot;, image.shape)
  261. print(&quot;Class label:&quot;, label.numpy().decode())
  263. # ETL pipeline.
  264. X_y_tensors = (
  265.     images_ds
  266.     .map(process_image)   # Extra and Transform
  267.     .filter(check_shape)  # Filter
  268.     .as_numpy_iterator()  # Load
  269. )
  271. print(&quot;\nTechnique 1:&quot;)
  272. print(&quot;Final X count:&quot;, len(list(X_y_tensors)))
  275. X_y_tensors = images_ds.map(process_image)
  277. count = 0
  278. for x, y in X_y_tensors:
  279.     d1, d2, d3 = x.shape
  280.     if d3 &gt; 3:
  281.         continue
  282.     count += 1
  284. print(&quot;\nTechnique 2:&quot;)
  285. print(&quot;Final X count:&quot;, count)
  286. ```
  288. Output
  290. ```
  291. TensorFlow version: 2.6.0
  292. Shape: (128, 128, 3)
  293. Class label: cat
  295. Shape received in filter(): (128, 128, 3)
  297. Technique 1:
  298. Final X count: 129
  300. Technique 2:
  301. Final X count: 129
  302. ```
  304. [1]: https://www.tensorflow.org/api_docs/python/tf/io/decode_jpeg#args
  306. </details>

huangapple
  • 本文由 发表于 2023年2月18日 22:03:38
  • 转载请务必保留本文链接：https://go.coder-hub.com/75493853.html
  • deep-learning
  • machine-learning
  • numpy
  • python
  • tensorflow2.0
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定