无法将 JSON 转换为数据框，Polars 出现了错误。

huangapple
go评论94阅读模式
英文:

unable to convert json to dataframe, polars panicked

问题

Here's the translated code portion:

  1. use reqwest::blocking::get;
  2. use polars::prelude::*;
  3. use serde::{Deserialize, Serialize};
  4. use std::io::Cursor;
  6. #[derive(Serialize, Deserialize, Debug)]
  7. struct StockZhAHist {
  8.     date: String,
  9.     open: f64,
  10.     close: f64,
  11.     high: f64,
  12.     low: f64,
  13.     volume: f64,
  14.     turnover: f64,
  15.     amplitude: f64,
  16.     change_rate: f64,
  17.     change_amount: f64,
  18.     turnover_rate: f64,
  19. }
  21. fn main() -> Result<(), Box<dyn std::error::Error>> {
  22.     let url = "http://127.0.0.1:8080/api/public/stock_zh_a_hist";
  23.     let response = get(url)?.text()?;
  24.     // Deserialize response with serde
  25.     let stock_zh_a_hist: Vec<StockZhAHist> = serde_json::from_str(&response)?;
  26.     // Convert to polars dataframe
  27.     println!("{:?}", stock_zh_a_hist);
  28.     let df = JsonReader::new(Cursor::new(response))
  29.         .infer_schema_len(Some(1000))
  30.         .finish()?;
  31.     println!("{:?}", df);
  32.     Ok(())
  33. }

Regarding the error you're encountering with Polars, it seems like the Arrow datatype Struct is not supported by Polars. You may need to activate a specific data-type feature to resolve this issue. You mentioned adding the dtype-decimal feature, but it seems it didn't help. You might need to investigate further or seek assistance from the Polars community to determine the correct feature to activate or if there's an alternative approach to handle this specific data type in Polars.

英文: 
  1. use reqwest::blocking::get;
  2. use polars::prelude::*;
  3. use serde::{Deserialize, Serialize};
  4. use std::io::Cursor;
  7. #[derive(Serialize, Deserialize, Debug)]
  8. struct StockZhAHist {
  9.     date: String,
  10.     open: f64,
  11.     close: f64,
  12.     high: f64,
  13.     low: f64,
  14.     volume: f64,
  15.     turnover: f64,
  16.     amplitude: f64,
  17.     change_rate: f64,
  18.     change_amount: f64,
  19.     turnover_rate: f64,
  20. }
  22. fn main() -&gt; Result&lt;(), Box&lt;dyn std::error::Error&gt;&gt; {
  23.     let url = &quot;http://127.0.0.1:8080/api/public/stock_zh_a_hist&quot;;
  24.     let response = get(url)?.text()?;
  25.     // println!(&quot;{}&quot;, response);
  26.     // deserialize response with serde
  27.     let stock_zh_a_hist: Vec&lt;StockZhAHist&gt; = serde_json::from_str(&amp;response)?;
  28.     // convert to polars dataframe
  29.     println!(&quot;{:?}&quot;, stock_zh_a_hist);
  30.     let df = JsonReader::new(Cursor::new(response))
  31.         .infer_schema_len(Some(1000))
  32.         .finish()?;
  33.     println!(&quot;{:?}&quot;, df);
  34.     Ok(())
  35. }

i have written a script that retrieves raw data from my server and i was able to print out the JSON 'stock_zh_a_hist'. but somehow when i tried to print out the dataframe converted from this JSON, the programme panicked and returned :

> thread 'main' panicked at 'Arrow datatype Struct([Field { name:
> "date", data_type: LargeUtf8, is_nullable: true, metadata: {} }, Field
> { name: "open", data_type: Float64, is_nullable: true, metadata: {} },
> Field { name: "close", data_type: Float64, is_nullable: true,
> metadata: {} }, Field { name: "high", data_type: Float64, is_nullable:
> true, metadata: {} }, Field { name: "low", data_type: Float64,
> is_nullable: true, metadata: {} }, Field { name: "volume", data_type:
> Int64, is_nullable: true, metadata: {} }, Field { name: "turnover",
> data_type: Float64, is_nullable: true, metadata: {} }, Field { name:
> "amplitude", data_type: Float64, is_nullable: true, metadata: {} },
> Field { name: "change_rate", data_type: Float64, is_nullable: true,
> metadata: {} }, Field { name: "change_amount", data_type: Float64,
> is_nullable: true, metadata: {} }, Field { name: "turnover_rate",
> data_type: Float64, is_nullable: true, metadata: {} }]) not supported
> by Polars. You probably need to activate that data-type feature.',
> /home/arthur/.cargo/registry/src/index.crates.io-6f17d22bba15001f/polars-core-0.30.0/src/datatypes/field.rs:158:19.

i also read source code there which reads:

  1.  #[cfg(feature = &quot;dtype-decimal&quot;)]
  2.             ArrowDataType::Decimal(precision, scale) =&gt; DataType::Decimal(Some(*precision), Some(*scale)),
  3.             dt =&gt; panic!(&quot;Arrow datatype {dt:?} not supported by Polars. You probably need to activate that data-type feature.&quot;),

i am assuming polars is having some problem reading the json with correct datatypes, but how can i solve it?

update 1:
i have added the feature dtype-decimal in cargo.toml but the same error persists.

update 2:
the content of stock_zh_a_hist is as follows:

  1. [StockZhAHist { date: &quot;1996-12-16T00:00:00.000&quot;, open: 16.86, close: 16.86, high: 16.86, low: 16.86, volume: 62442.0, turnover: 105277000.0, amplitude: 0.0, change_rate: -10.22, change_amount: -1.92, turnover_rate: 0.87 },
  2. StockZhAHist { date: &quot;1996-12-17T00:00:00.000&quot;, open: 15.17, close: 15.17, high: 16.79, low: 15.17, volume: 463675.0, turnover: 718902016.0, amplitude: 9.61, change_rate: -10.02, change_amount: -1.69, turnover_rate: 6.49 },
  3. StockZhAHist { date: &quot;1996-12-18T00:00:00.000&quot;, open: 15.28, close: 16.69, high: 16.69, low: 15.18, volume: 445380.0, turnover: 719400000.0, amplitude: 9.95, change_rate: 10.02, change_amount: 1.52, turnover_rate: 6.24 },
  4. StockZhAHist { date: &quot;1996-12-19T00:00:00.000&quot;, open: 17.01, close: 16.4, high: 17.9, low: 15.99, volume: 572946.0, turnover: 970124992.0, amplitude: 11.44, change_rate: -1.74, change_amount: -0.29, turnover_rate: 8.03 },
  5. StockZhAHist { date: &quot;1996-12-20T00:00:00.000&quot;, open: 16.19, close: 16.39, high: 16.68, low: 15.9, volume: 277758.0, turnover: 451471008.0, amplitude: 4.76, change_rate: -0.06, change_amount: -0.01, turnover_rate: 3.89 },
  6. StockZhAHist { date: &quot;1996-12-23T00:00:00.000&quot;, open: 16.58, close: 16.57, high: 16.89, low: 16.4, volume: 155369.0, turnover: 258260992.0, amplitude: 2.99, change_rate: 1.1, change_amount: 0.18, turnover_rate: 2.18 },
  7. StockZhAHist { date: &quot;1996-12-24T00:00:00.000&quot;, open: 16.58, close: 15.95, high: 16.73, low: 15.93, volume: 153168.0, turnover: 249324000.0, amplitude: 4.83, change_rate: -3.74, change_amount: -0.62, turnover_rate: 2.15 },
  8. StockZhAHist { date: &quot;1996-12-25T00:00:00.000&quot;, open: 16.1, close: 16.47, high: 16.7, low: 15.86, volume: 175391.0, turnover: 285567008.0, amplitude: 5.27, change_rate: 3.26, change_amount: 0.52, turnover_rate: 2.46 },
  9. StockZhAHist { date: &quot;1996-12-26T00:00:00.000&quot;, open: 16.68, close: 16.47, high: 16.78, low: 16.3, volume: 51516.0, turnover: 85214000.0, amplitude: 2.91, change_rate: 0.0, change_amount: 0.0, turnover_rate: 0.72 }...]

答案1

得分: 2

似乎是一个错误 - 如果您启用特性 dtype-struct，这个问题会消失，但您绝对不应该为了从 JSON 中读取而这样做。

作为一种解决方法，即使没有 dtype-struct，您也可以通过将数据转换为 JSON Lines 来使其工作。

  1. use polars::prelude::{JsonReader, SerReader};
  2. use serde::{Deserialize, Serialize};
  3. use std::io::Cursor;
  5. #[derive(Serialize, Deserialize, Debug)]
  6. struct StockZhAHist {
  7.     date: String,
  8.     open: f64,
  9.     close: f64,
  10.     high: f64,
  11.     low: f64,
  12.     volume: f64,
  13.     turnover: f64,
  14.     amplitude: f64,
  15.     change_rate: f64,
  16.     change_amount: f64,
  17.     turnover_rate: f64,
  18. }
  20. fn main() -> Result<(), Box<dyn std::error::Error>> {
  21.     let data = vec![
  22.         StockZhAHist {
  23.             date: "1996-12-16T00:00:00.000".into(),
  24.             open: 16.86,
  25.             close: 16.86,
  26.             high: 16.86,
  27.             low: 16.86,
  28.             volume: 62442.0,
  29.             turnover: 105277000.0,
  30.             amplitude: 0.0,
  31.             change_rate: -10.22,
  32.             change_amount: -1.92,
  33.             turnover_rate: 0.87,
  34.         },
  35.         StockZhAHist {
  36.             date: "1996-12-17T00:00:00.000".into(),
  37.             open: 15.17,
  38.             close: 15.17,
  39.             high: 16.79,
  40.             low: 15.17,
  41.             volume: 463675.0,
  42.             turnover: 718902016.0,
  43.             amplitude: 9.61,
  44.             change_rate: -10.02,
  45.             change_amount: -1.69,
  46.             turnover_rate: 6.49,
  47.         },
  48.         StockZhAHist {
  49.             date: "1996-12-18T00:00:00.000".into(),
  50.             open: 15.28,
  51.             close: 16.69,
  52.             high: 16.69,
  53.             low: 15.18,
  54.             volume: 445380.0,
  55.             turnover: 719400000.0,
  56.             amplitude: 9.95,
  57.             change_rate: 10.02,
  58.             change_amount: 1.52,
  59.             turnover_rate: 6.24,
  60.         },
  61.     ];
  62.     // JSON Lines 数据
  63.     let response = data
  64.         .into_iter()
  65.         .map(|d| serde_json::to_string(&d))
  66.         .collect::<Result<Vec<_>, _>>()?
  67.         .join("\n");
  68.     println!("{}", response);
  69.     // 转换为 Polars 数据框
  70.     let df = JsonReader::new(Cursor::new(response))
  71.         .infer_schema_len(Some(1000))
  72.         .with_json_format(polars::prelude::JsonFormat::JsonLines)
  73.         .finish()?;
  74.     println!("{:?}", df);
  75.     Ok(())
  76. }
  1. {"date":"1996-12-16T00:00:00.000","open":16.86,"close":16.86,"high":16.86,"low":16.86,"volume":62442.0,"turnover":105277000.0,"amplitude":0.0,"change_rate":-10.22,"change_amount":-1.92,"turnover_rate":0.87}
  2. {"date":"1996-12-17T00:00:00.000","open":15.17,"close":15.17,"high":16.79,"low":15.17,"volume":463675.0,"turnover":718902016.0,"amplitude":9.61,"change_rate":-10.02,"change_amount":-1.69,"turnover_rate":6.49}
  3. {"date":"1996-12-18T00:00:00.000","open":15.28,"close":16.69,"high":16.69,"low":15.18,"volume":445380.0,"turnover":719400000.0,"amplitude":9.95,"change_rate":10.02,"change_amount":1.52,"turnover_rate":6.24}
  4. [/Users/ben/.cargo/registry/src/index.crates.io-6f17d22bba15001f/polars-io-0.30.0/src/ndjson/core.rs:162] &data_type = Struct(
  5.     [
  6.         Field {
  7.             name: "date",
  8.             data_type: LargeUtf8,
  9.             is_nullable: true,
  10.             metadata: {},
  11.         },
  12.         Field {
  13.             name: "open",
  14.             data_type: Float64,
  15.             is_nullable: true,
  16.             metadata: {},
  17.         },
  18.         Field {
  19.             name: "close",
  20.             data_type: Float64,
  21.             is_nullable: true,
  22.             metadata: {},
  23.         },
  24.         Field {
  25.             name: "high",
  26.             data_type: Float64,
  27.             is_nullable: true,
  28.             metadata: {},
  29.         },
  30.         Field {
  31.             name: "low",
  32.             data_type: Float64,
  33.             is_nullable: true,
  34.             metadata: {},
  35.         },
  36.         Field {
  37.             name: "volume",
  38.             data_type: Float64,
  39.             is_nullable: true,
  40.             metadata: {},
  41.         },
  42.         Field {
  43.             name: "turnover",
  44.             data_type: Float64,
  45.             is_nullable: true,
  46.             metadata: {},
  47.         },
  48.         Field {
  49.             name: "amplitude",
  50.             data_type: Float64,
  51.             is_nullable: true,
  52.             metadata: {},
  53.         },
  54.         Field {
  55.             name: "change_rate",
  56.             data_type: Float64,
  57.             is_nullable: true,
  58.             metadata: {},
  59.         },
  60.         Field {
  61.             name: "change_amount",
  62.             data_type: Float64,
  63.             is_nullable: true,
  64.             metadata: {},
  65.         },
  66.         Field {
  67.             name: "turnover_rate",
  68.             data_type: Float64,
  69.             is_nullable: true,
  70.             metadata: {},
  71.         },
  72.     ],
  73. )
  74. shape: (3, 11)
  75. ┌─────────────────────────┬───────┬───────┬───────┬───┬───────────┬─────────────┬───────────────┬───────────────┐
  76. │ date                    ┆ open  ┆ close ┆ high  ┆ … ┆ amplitude ┆ change_rate ┆ change_amount ┆ turnover_rate │
  77. │ ---                     ┆ ---   ┆ ---   ┆ ---   ┆   ┆ ---       ┆ ---         ┆ ---           ┆ ---           │
  78. │ str                     ┆ f64   ┆ f64   ┆ f64   ┆   ┆ f64       ┆ f64         ┆ f64           ┆ f64           │
  79. ╞═════════════════════════╪═══════╪═══════╪═══════╪═══╪════════
  81. <details>
  82. <summary>英文:</summary>
  84. Seems like a bug — this issue goes away if you enable feature `dtype-struct`, but you definitely shouldn&#39;t have to in order to read from json.
  86. As a workaround, even without `dtype-struct`, you can get this to work by converting your data to JSON Lines.
  88. ```rust
  89. use polars::prelude::{JsonReader, SerReader};
  90. use serde::{Deserialize, Serialize};
  91. use std::io::Cursor;
  93. #[derive(Serialize, Deserialize, Debug)]
  94. struct StockZhAHist {
  95. 	date: String,
  96. 	open: f64,
  97. 	close: f64,
  98. 	high: f64,
  99. 	low: f64,
  100. 	volume: f64,
  101. 	turnover: f64,
  102. 	amplitude: f64,
  103. 	change_rate: f64,
  104. 	change_amount: f64,
  105. 	turnover_rate: f64,
  106. }
  108. fn main() -&gt; Result&lt;(), Box&lt;dyn std::error::Error&gt;&gt; {
  109. 	let data = vec![
  110. 		StockZhAHist {
  111. 			date: &quot;1996-12-16T00:00:00.000&quot;.into(),
  112. 			open: 16.86,
  113. 			close: 16.86,
  114. 			high: 16.86,
  115. 			low: 16.86,
  116. 			volume: 62442.0,
  117. 			turnover: 105277000.0,
  118. 			amplitude: 0.0,
  119. 			change_rate: -10.22,
  120. 			change_amount: -1.92,
  121. 			turnover_rate: 0.87,
  122. 		},
  123. 		StockZhAHist {
  124. 			date: &quot;1996-12-17T00:00:00.000&quot;.into(),
  125. 			open: 15.17,
  126. 			close: 15.17,
  127. 			high: 16.79,
  128. 			low: 15.17,
  129. 			volume: 463675.0,
  130. 			turnover: 718902016.0,
  131. 			amplitude: 9.61,
  132. 			change_rate: -10.02,
  133. 			change_amount: -1.69,
  134. 			turnover_rate: 6.49,
  135. 		},
  136. 		StockZhAHist {
  137. 			date: &quot;1996-12-18T00:00:00.000&quot;.into(),
  138. 			open: 15.28,
  139. 			close: 16.69,
  140. 			high: 16.69,
  141. 			low: 15.18,
  142. 			volume: 445380.0,
  143. 			turnover: 719400000.0,
  144. 			amplitude: 9.95,
  145. 			change_rate: 10.02,
  146. 			change_amount: 1.52,
  147. 			turnover_rate: 6.24,
  148. 		},
  149. 	];
  150. 	// JSON Lines data
  151. 	let response = data
  152. 		.into_iter()
  153. 		.map(|d| serde_json::to_string(&amp;d))
  154. 		.collect::&lt;Result&lt;Vec&lt;_&gt;, _&gt;&gt;()?
  155. 		.join(&quot;\n&quot;);
  156. 	println!(&quot;{}&quot;, response);
  157. 	// convert to polars dataframe
  158. 	let df = JsonReader::new(Cursor::new(response))
  159. 		.infer_schema_len(Some(1000))
  160. 		.with_json_format(polars::prelude::JsonFormat::JsonLines)
  161. 		.finish()?;
  162. 	println!(&quot;{:?}&quot;, df);
  163. 	Ok(())
  164. }
  1. {&quot;date&quot;:&quot;1996-12-16T00:00:00.000&quot;,&quot;open&quot;:16.86,&quot;close&quot;:16.86,&quot;high&quot;:16.86,&quot;low&quot;:16.86,&quot;volume&quot;:62442.0,&quot;turnover&quot;:105277000.0,&quot;amplitude&quot;:0.0,&quot;change_rate&quot;:-10.22,&quot;change_amount&quot;:-1.92,&quot;turnover_rate&quot;:0.87}
  2. {&quot;date&quot;:&quot;1996-12-17T00:00:00.000&quot;,&quot;open&quot;:15.17,&quot;close&quot;:15.17,&quot;high&quot;:16.79,&quot;low&quot;:15.17,&quot;volume&quot;:463675.0,&quot;turnover&quot;:718902016.0,&quot;amplitude&quot;:9.61,&quot;change_rate&quot;:-10.02,&quot;change_amount&quot;:-1.69,&quot;turnover_rate&quot;:6.49}
  3. {&quot;date&quot;:&quot;1996-12-18T00:00:00.000&quot;,&quot;open&quot;:15.28,&quot;close&quot;:16.69,&quot;high&quot;:16.69,&quot;low&quot;:15.18,&quot;volume&quot;:445380.0,&quot;turnover&quot;:719400000.0,&quot;amplitude&quot;:9.95,&quot;change_rate&quot;:10.02,&quot;change_amount&quot;:1.52,&quot;turnover_rate&quot;:6.24}
  4. [/Users/ben/.cargo/registry/src/index.crates.io-6f17d22bba15001f/polars-io-0.30.0/src/ndjson/core.rs:162] &amp;data_type = Struct(
  5. [
  6. Field {
  7. name: &quot;date&quot;,
  8. data_type: LargeUtf8,
  9. is_nullable: true,
  10. metadata: {},
  11. },
  12. Field {
  13. name: &quot;open&quot;,
  14. data_type: Float64,
  15. is_nullable: true,
  16. metadata: {},
  17. },
  18. Field {
  19. name: &quot;close&quot;,
  20. data_type: Float64,
  21. is_nullable: true,
  22. metadata: {},
  23. },
  24. Field {
  25. name: &quot;high&quot;,
  26. data_type: Float64,
  27. is_nullable: true,
  28. metadata: {},
  29. },
  30. Field {
  31. name: &quot;low&quot;,
  32. data_type: Float64,
  33. is_nullable: true,
  34. metadata: {},
  35. },
  36. Field {
  37. name: &quot;volume&quot;,
  38. data_type: Float64,
  39. is_nullable: true,
  40. metadata: {},
  41. },
  42. Field {
  43. name: &quot;turnover&quot;,
  44. data_type: Float64,
  45. is_nullable: true,
  46. metadata: {},
  47. },
  48. Field {
  49. name: &quot;amplitude&quot;,
  50. data_type: Float64,
  51. is_nullable: true,
  52. metadata: {},
  53. },
  54. Field {
  55. name: &quot;change_rate&quot;,
  56. data_type: Float64,
  57. is_nullable: true,
  58. metadata: {},
  59. },
  60. Field {
  61. name: &quot;change_amount&quot;,
  62. data_type: Float64,
  63. is_nullable: true,
  64. metadata: {},
  65. },
  66. Field {
  67. name: &quot;turnover_rate&quot;,
  68. data_type: Float64,
  69. is_nullable: true,
  70. metadata: {},
  71. },
  72. ],
  73. )
  74. shape: (3, 11)
  75. ┌─────────────────────────┬───────┬───────┬───────┬───┬───────────┬─────────────┬───────────────┬───────────────┐
  76. │ date                    ┆ open  ┆ close ┆ high  ┆ … ┆ amplitude ┆ change_rate ┆ change_amount ┆ turnover_rate │
  77. │ ---                     ┆ ---   ┆ ---   ┆ ---   ┆   ┆ ---       ┆ ---         ┆ ---           ┆ ---           │
  78. │ str                     ┆ f64   ┆ f64   ┆ f64   ┆   ┆ f64       ┆ f64         ┆ f64           ┆ f64           │
  79. ╞═════════════════════════╪═══════╪═══════╪═══════╪═══╪═══════════╪═════════════╪═══════════════╪═══════════════╡
  80. │ 1996-12-16T00:00:00.000 ┆ 16.86 ┆ 16.86 ┆ 16.86 ┆ … ┆ 0.0       ┆ -10.22      ┆ -1.92         ┆ 0.87          │
  81. │ 1996-12-17T00:00:00.000 ┆ 15.17 ┆ 15.17 ┆ 16.79 ┆ … ┆ 9.61      ┆ -10.02      ┆ -1.69         ┆ 6.49          │
  82. │ 1996-12-18T00:00:00.000 ┆ 15.28 ┆ 16.69 ┆ 16.69 ┆ … ┆ 9.95      ┆ 10.02       ┆ 1.52          ┆ 6.24          │
  83. └─────────────────────────┴───────┴───────┴───────┴───┴───────────┴─────────────┴───────────────┴───────────────┘

huangapple
  • 本文由 发表于 2023年6月27日 17:26:55
  • 转载请务必保留本文链接：https://go.coder-hub.com/76563450.html
  • rust
  • rust-polars
  • serde
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定