Searching for hidden API to scrape data with Python

huangapple
go评论95阅读模式
英文:

Searching for hidden API to scrape data with Python

问题

I've successfully scraped data from this site:
https://football.goaloo2.com/subleague/2022-2023/40/261/?selectround=23

Now after some hints and some basic tutorials, I'm trying to scrape it using a hidden API... but on this site, I got something different from others I was able to scrape. This is my process: with Chrome, I do inspection/network and then Fetch/XHR. Reloading the page, I got what I thought would be the call (even if the preview is different from other scraped sites, there's no tree to navigate). Then I copy it as a cURL command, and after pasting it into a Python conversion site, I obtained this:

  1. import requests
  3. params = {
  4.     'sclassId': '40',
  5.     'subSclassId': '261',
  6.     'matchSeason': '2022-2023',
  7.     'round': '23',
  8.     'flesh': '0.6807624444063407',
  9. }
  11. response = requests.get('https://football.goaloo2.com/ajax/LeagueOddsAjax', params=params)
  13. odds = response.json()

I got this error: JSONDecodeError: Expecting value

I suspect the data is not embedded in a JSON object. Any idea on how to proceed if it is doable in Python?

英文:

i've succesfully scraped data from this site:
https://football.goaloo2.com/subleague/2022-2023/40/261/?selectround=23

Now after some hints and some basic tutorials i'm trying to scrape it using hidden API...but on this site i got something different from others i was able to scrape. This is my process: with chrome i do inspection/network and then Fetch/XHR, reloadong the page i got what i thought it would be the call (even if the preview is different from other scraped sites, there's no three to navigate); than i copy as curl(cmd) and after pasting in converting site to python i obtained this:

  1. import requests
  3. params = {
  4.     'sclassId': '40',
  5.     'subSclassId': '261',
  6.     'matchSeason': '2022-2023',
  7.     'round': '23',
  8.     'flesh': '0.6807624444063407',
  9. }
  10. response = requests.get('https://football.goaloo2.com/ajax/LeagueOddsAjax', params=params)
  13. odds = response.json()

I got this error: JSONDecodeError: Expecting value

I suspect data is not embedded in a json object, any idea on how to proceede if it is doable in python?

答案1

得分: 1

返回的对象不是可解码的JSON。您可以提取原始文本并将其转换为JSON格式，然后手动使用json.loads()函数加载它。请注意，网页抓取不总是标准化的，您应该检查操作的稳定性。一个非常快速而不太干净的转换如下：

  1. d = "{\"OddsData\": {" + response.text.replace('=', ':').replace(';', ',').replace('":]', '":"').replace('oddsData[', '').replace(',,\n', '}}')
  2. json.loads(d)

它返回如下：

  1. {'OddsData': {'O_2247562': [[4, 2.9, 2.95, 2.6],
  2. [16, 3, 2.95, 2.2],
  3. [18, 2.78, 3.1, 2.53],
  4. [60, 2.9, 2.9, 2.6],
  5. [70, 3.1, 3, 2.4],
  6. [81, 2.9, 3, 2.63],
  7. [82, 2.7, 2.9, 2.45],
  8. [88, 2.8, 3, 2.55],
  9. [90, 2.9, 3.3, 2.33],
  10. [104, 2.85, 2.9, 2.6],
  11. [110, 3.05, 3.15, 2.45],
  12. [115, 2.9, 2.9, 2.6],
  13. [158, 2.9, 3.1, 2.45],
  14. [173, 2.8, 2.8, 2.55],
  15. [177, 3.02, 3.25, 2.57],
  16. [255, 2.7, 3.25, 2.5],
  17. [281, 2.9, 3.1, 2.38],
  18. [370, 2.55, 2.6, 2.35],
  19. [422, 2.75, 3.2, 2.5],
  20. [450, 2.85, 2.95, 2.6],
  21. [474, 2.8, 3.1, 2.49],
  22. [499, 2.74, 3.25, 2.47],
  23. [517, 2.78, 3.1, 2.53],
  24. [545, 2.74, 3.25, 2.47],
  25. [659, 2.7, 3.07, 2.46],
  26. [976, 2.8, 3.15, 2.5]],
  27. 'L_2247562': [[3, 1.04, 0, 0.84],
  28. [8, 1.2, 0, 0.7],
  29. [12, 1.15, 0, 0.77],
  30. [14, 0.95, 0, 0.8],
  31. [17, 1.05, 0, 0.85],
  32. [19, 1.1, 0, 0.65],
  33. [22, 0.78, -0.25, 0.91],
  34. [23, 1.05, 0, 0.87],
  35. [24, 1.05, 0, 0.85],
  36. [31, 0.8, -0.25, 1.13],
  37. [35, 1.05, 0, 0.85],
  38. [42, 0.99, 0, 0.8]],
  39. 'T_2247562': [[3, 1.05, 2.25, 0.81],
  40. [4, 1.3, 2.5, 0.57],
  41. [8, 1, 2.25, 0.85],
  42. [9, 1.3, 2.5, 0.57],
  43. [12, 1.05, 2.25, 0.83],
  44. [14, 1, 2.25, 0.78],
  45. [17, 1.08, 2.25, 0.8],
  46. [19, 1.3, 2.5, 0.55],
  47. [22, 0.87, 2.25, 0.78],
  48. [23, 1.06, 2.25, 0.84],
  49. [24, 1.
  50. <details>
  51. <summary>英文:</summary>
  52. The returned object is not decodable JSON. You can take out the raw text and manipulate it into the JSON format. Then manually load it using json.loads() function. Note that, the web scrapping is not always standartized and you should check the stability of the manipulation. A very quick and dirty transformation as follows;
  53. d = &quot;{\&quot;OddsData\&quot;: {&quot; + response.text.replace(&#39;=&#39;, &#39;:&#39;).replace(&#39;;&#39;, &#39;,&#39;).replace(&#39;&quot;]:&#39;, &#39;&quot;:&#39;).replace(&#39;oddsData[&#39;, &#39;&#39;).replace(&#39;,,\n&#39;, &#39;}}&#39;)
  54. json.loads(d)
  55. It returns as follows;
  56. {&#39;OddsData&#39;: {&#39;O_2247562&#39;: [[4, 2.9, 2.95, 2.6],
  57. [16, 3, 2.95, 2.2],
  58. [18, 2.78, 3.1, 2.53],
  59. [60, 2.9, 2.9, 2.6],
  60. [70, 3.1, 3, 2.4],
  61. [81, 2.9, 3, 2.63],
  62. [82, 2.7, 2.9, 2.45],
  63. [88, 2.8, 3, 2.55],
  64. [90, 2.9, 3.3, 2.33],
  65. [104, 2.85, 2.9, 2.6],
  66. [110, 3.05, 3.15, 2.45],
  67. [115, 2.9, 2.9, 2.6],
  68. [158, 2.9, 3.1, 2.45],
  69. [173, 2.8, 2.8, 2.55],
  70. [177, 3.02, 3.25, 2.57],
  71. [255, 2.7, 3.25, 2.5],
  72. [281, 2.9, 3.1, 2.38],
  73. [370, 2.55, 2.6, 2.35],
  74. [422, 2.75, 3.2, 2.5],
  75. [450, 2.85, 2.95, 2.6],
  76. [474, 2.8, 3.1, 2.49],
  77. [499, 2.74, 3.25, 2.47],
  78. [517, 2.78, 3.1, 2.53],
  79. [545, 2.74, 3.25, 2.47],
  80. [659, 2.7, 3.07, 2.46],
  81. [976, 2.8, 3.15, 2.5]],
  82. &#39;L_2247562&#39;: [[3, 1.04, 0, 0.84],
  83. [8, 1.2, 0, 0.7],
  84. [12, 1.15, 0, 0.77],
  85. [14, 0.95, 0, 0.8],
  86. [17, 1.05, 0, 0.85],
  87. [19, 1.1, 0, 0.65],
  88. [22, 0.78, -0.25, 0.91],
  89. [23, 1.05, 0, 0.87],
  90. [24, 1.05, 0, 0.85],
  91. [31, 0.8, -0.25, 1.13],
  92. [35, 1.05, 0, 0.85],
  93. [42, 0.99, 0, 0.8]],
  94. &#39;T_2247562&#39;: [[3, 1.05, 2.25, 0.81],
  95. [4, 1.3, 2.5, 0.57],
  96. [8, 1, 2.25, 0.85],
  97. [9, 1.3, 2.5, 0.57],
  98. [12, 1.05, 2.25, 0.83],
  99. [14, 1, 2.25, 0.78],
  100. [17, 1.08, 2.25, 0.8],
  101. [19, 1.3, 2.5, 0.55],
  102. [22, 0.87, 2.25, 0.78],
  103. [23, 1.06, 2.25, 0.84],
  104. [24, 1.08, 2.25, 0.8],
  105. [31, 1.05, 2.25, 0.85],
  106. [35, 1.09, 2.25, 0.81],
  107. [42, 1.04, 2.25, 0.76],
  108. [49, 1.25, 2.5, 0.57]],
  109. &#39;O_2247563&#39;: [[4, 3, 3.1, 2.4],
  110. [16, 2.8, 2.7, 2.45],
  111. [18, 2.98, 3.05, 2.39],
  112. [60, 3, 3.05, 2.44],
  113. [70, 2.95, 2.85, 2.6],
  114. [81, 3.13, 3, 2.5],
  115. [82, 2.87, 2.9, 2.35],
  116. [88, 3, 3, 2.4],
  117. [90, 2.8, 3.1, 2.43],
  118. [104, 2.95, 3.05, 2.4],
  119. [110, 3, 3.05, 2.5],
  120. [115, 3, 3, 2.45],
  121. [158, 2.87, 3.1, 2.45],
  122. [173, 2.9, 2.9, 2.36],
  123. [177, 3.06, 2.98, 2.66],
  124. [255, 2.9, 3.2, 2.37],
  125. [281, 3, 3.1, 2.4],
  126. [370, 2.65, 2.75, 2.2],
  127. [422, 2.87, 3.1, 2.45],
  128. [450, 2.95, 3.05, 2.4],
  129. [474, 2.86, 2.95, 2.56],
  130. [499, 2.86, 3.2, 2.38],
  131. [517, 2.98, 3.05, 2.39],
  132. [545, 2.86, 3.2, 2.38],
  133. [659, 2.84, 3.08, 2.35],
  134. [976, 3, 3.15, 2.35]],
  135. &#39;L_2247563&#39;: [[3, 0.79, -0.25, 1.09],
  136. [8, 0.73, -0.25, 1.15],
  137. [12, 0.75, -0.25, 1.18],
  138. [14, 0.78, -0.25, 1.08],
  139. [17, 0.8, -0.25, 1.11],
  140. [19, 1.05, 0, 0.7],
  141. [22, 0.64, -0.25, 1.11],
  142. [23, 0.8, -0.25, 1.13],
  143. [24, 0.8, -0.25, 1.11],
  144. [31, 1.07, 0, 0.85],
  145. [35, 0.8, -0.25, 1.11],
  146. [42, 0.75, -0.25, 1.05]],
  147. &#39;T_2247563&#39;: [[3, 0.79, 2, 1.07],
  148. [4, 1.3, 2.5, 0.57],
  149. [8, 0.73, 2, 1.15],
  150. [9, 1.3, 2.5, 0.57],
  151. [12, 0.75, 2, 1.15],
  152. [14, 0.75, 2, 1.05],
  153. [17, 1.08, 2.25, 0.8],
  154. [19, 1.3, 2.5, 0.55],
  155. [22, 0.63, 2, 1.06],
  156. [23, 0.8, 2, 1.11],
  157. [24, 1.08, 2.25, 0.8],
  158. [31, 1.11, 2.25, 0.8],
  159. [35, 1.09, 2.25, 0.81],
  160. [42, 1.04, 2.25, 0.76],
  161. [49, 1.25, 2.5, 0.55]],
  162. &#39;O_2247564&#39;: [[4, 2.4, 3, 3.1],
  163. [16, 2.1, 2.9, 3.3],
  164. [18, 2.17, 3.15, 3.25],
  165. [60, 2.45, 3, 3.05],
  166. [70, 2.05, 3.15, 3.65],
  167. [81, 2.3, 3.1, 3.4],
  168. [82, 2.2, 3, 3.1],
  169. [88, 2.2, 3.1, 3.25],
  170. [90, 2.03, 3.3, 3.5],
  171. [104, 2.45, 3, 3],
  172. [110, 2.15, 3.2, 3.6],
  173. [115, 2.4, 3, 3.1],
  174. [158, 2.15, 3.2, 3.4],
  175. [173, 2.37, 2.85, 2.95],
  176. [177, 2.18, 3.28, 3.64],
  177. [255, 2.25, 3.3, 3.1],
  178. [281, 2.1, 3.2, 3.4],
  179. [370, 2.2, 2.7, 2.7],
  180. [422, 2.2, 3.2, 3.25],
  181. [450, 2.41, 3, 3],
  182. [474, 2.17, 3.15, 3.3],
  183. [499, 2.23, 3.3, 3.05],
  184. [517, 2.25, 3.15, 3.15],
  185. [545, 2.23, 3.3, 3.05],
  186. [659, 2.28, 3.15, 2.9],
  187. [976, 2.3, 3.2, 3.05]],
  188. &#39;L_2247564&#39;: [[3, 0.97, 0.25, 0.91],
  189. [8, 0.83, 0.25, 1.03],
  190. [12, 0.75, 0.25, 1.18],
  191. [14, 0.88, 0.25, 0.95],
  192. [17, 1.05, 0.25, 0.85],
  193. [19, 1.05, 0.5, 0.7],
  194. [22, 0.79, 0.25, 0.9],
  195. [23, 0.98, 0.25, 0.94],
  196. [24, 1.05, 0.25, 0.85],
  197. [31, 0.87, 0.25, 1.05],
  198. [35, 1.05, 0.25, 0.85],
  199. [42, 0.99, 0.25, 0.8]],
  200. &#39;T_2247564&#39;: [[3, 0.98, 2.25, 0.88],
  201. [4, 1.2, 2.5, 0.6],
  202. [8, 1.03, 2.25, 0.83],
  203. [9, 1.2, 2.5, 0.62],
  204. [12, 1.05, 2.25, 0.83],
  205. [14, 0.95, 2.25, 0.88],
  206. [17, 0.98, 2.25, 0.9],
  207. [19, 1.2, 2.5, 0.6],
  208. [22, 0.91, 2.25, 0.74],
  209. [23, 0.99, 2.25, 0.91],
  210. [24, 0.98, 2.25, 0.9],
  211. [31, 1.05, 2.25, 0.85],
  212. [35, 0.99, 2.25, 0.91],
  213. [42, 0.93, 2.25, 0.85],
  214. [49, 1.15, 2.5, 0.62]],
  215. &#39;O_2247565&#39;: [[4, 2.3, 3.1, 3.2],
  216. [16, 2.2, 2.9, 3],
  217. [18, 2.16, 3.2, 3.3],
  218. [60, 2.35, 3.05, 3.15],
  219. [70, 2.05, 3.15, 3.65],
  220. [81, 2.3, 3.1, 3.4],
  221. [82, 2.15, 3, 3.1],
  222. [88, 2.2, 3.1, 3.25],
  223. [90, 2.27, 3.2, 3.1],
  224. [104, 2.3, 3.1, 3.15],
  225. [110, 2.3, 3.15, 3.25],
  226. [115, 2.3, 3.1, 3.2],
  227. [158, 2.25, 3.3, 3.1],
  228. [173, 2.27, 2.9, 3.05],
  229. [177, 2.36, 3.07, 3.46],
  230. [255, 2.25, 3.3, 3.1],
  231. [281, 2.25, 3.2, 3.2],
  232. [370, 2.1, 2.75, 2.8],
  233. [422, 2.25, 3.3, 3.1],
  234. [450, 2.31, 3.05, 3.15],
  235. [474, 2.35, 3.15, 2.98],
  236. [499, 2.21, 3.3, 3.1],
  237. [517, 2.25, 3.2, 3.1],
  238. [545, 2.21, 3.3, 3.1],
  239. [659, 2.22, 3.15, 2.98],
  240. [976, 2.25, 3.2, 3.15]],
  241. &#39;L_2247565&#39;: [[3, 0.95, 0.25, 0.93],
  242. [8, 0.98, 0.25, 0.88],
  243. [12, 1.02, 0.25, 0.88],
  244. [14, 0.91, 0.25, 0.91],
  245. [17, 1, 0.25, 0.9],
  246. [19, 0.6, 0, 1.2],
  247. [22, 0.91, 0.25, 0.78],
  248. [23, 0.96, 0.25, 0.96],
  249. [24, 1, 0.25, 0.9],
  250. [31, 1.02, 0.25, 0.9],
  251. [35, 1, 0.25, 0.9],
  252. [42, 0.94, 0.25, 0.84]],
  253. &#39;T_2247565&#39;: [[3, 0.98, 2.25, 0.88],
  254. [4, 1.2, 2.5, 0.6],
  255. [8, 0.95, 2.25, 0.9],
  256. [9, 1.2, 2.5, 0.62],
  257. [12, 1.05, 2.25, 0.83],
  258. [14, 0.95, 2.25, 0.88],
  259. [17, 0.98, 2.25, 0.9],
  260. [19, 1.2, 2.5, 0.6],
  261. [22, 0.83, 2.25, 0.82],
  262. [23, 0.99, 2.25, 0.91],
  263. [24, 0.98, 2.25, 0.9],
  264. [31, 1, 2.25, 0.9],
  265. [35, 0.99, 2.25, 0.91],
  266. [42, 0.93, 2.25, 0.85],
  267. [49, 1.15, 2.5, 0.62]],
  268. &#39;O_2247566&#39;: [[4, 1.9, 3.25, 4.2],
  269. [16, 1.8, 3, 4.3],
  270. [18, 1.95, 3.25, 3.85],
  271. [60, 1.95, 3.2, 4.1],
  272. [70, 1.87, 3.1, 4.5],
  273. [81, 1.95, 3.2, 4.2],
  274. [82, 1.87, 3.1, 3.9],
  275. [88, 1.91, 3.2, 4],
  276. [90, 1.92, 3.4, 3.7],
  277. [104, 1.95, 3.2, 4],
  278. [110, 1.9, 3.3, 4.25],
  279. [115, 1.91, 3.2, 4.2],
  280. [158, 1.93, 3.3, 3.9],
  281. [173, 1.89, 3.05, 3.95],
  282. [177, 1.88, 3.38, 4.96],
  283. [255, 1.93, 3.3, 3.9],
  284. [281, 1.91, 3.4, 4],
  285. [370, 1.75, 2.85, 3.5],
  286. [422, 1.93, 3.3, 3.9],
  287. [450, 1.92, 3.2, 4],
  288. [474, 1.96, 3.25, 3.8],
  289. [499, 1.94, 3.35, 3.75],
  290. [517, 1.95, 3.25, 3.85],
  291. [545, 1.94, 3.35, 3.75],
  292. [659, 1.89, 3.26, 3.75],
  293. [976, 1.9, 3.35, 4]],
  294. &#39;L_2247566&#39;: [[3, 0.94, 0.5, 0.94],
  295. [8, 0.9, 0.5, 0.95],
  296. [12, 0.94, 0.5, 0.96],
  297. [14, 0.88, 0.5, 0.89],
  298. [17, 0.95, 0.5, 0.95],
  299. [19, 0.9, 0.5, 0.8],
  300. [22, 0.78, 0.5, 0.91],
  301. [23, 0.95, 0.5, 0.97],
  302. [24, 0.95, 0.5, 0.95],
  303. [31, 0.97, 0.5, 0.95],
  304. [35, 0.95, 0.5, 0.95],
  305. [42, 0.89, 0.5, 0.89]],
  306. &#39;T_2247566&#39;: [[3, 0.98, 2.25, 0.88],
  307. [4, 1.2, 2.5, 0.6],
  308. [8, 0.98, 2.25, 0.88],
  309. [9, 1.2, 2.5, 0.62],
  310. [12, 1, 2.25, 0.88],
  311. [14, 0.93, 2.25, 0.85],
  312. [17, 0.98, 2.25, 0.9],
  313. [19, 1.1, 2.5, 0.65],
  314. [22, 1.09, 2.5, 0.62],
  315. [23, 0.99, 2.25, 0.91],
  316. [24, 0.98, 2.25, 0.9],
  317. [31, 1, 2.25, 0.9],
  318. [35, 0.99, 2.25, 0.91],
  319. [42, 0.93, 2.25, 0.85],
  320. [49, 1.15, 2.5, 0.62]],
  321. &#39;O_2247567&#39;: [[4, 2.75, 2.95, 2.75],
  322. [16, 2.45, 3, 2.55],
  323. [18, 2.6, 3.2, 2.6],
  324. [60, 2.75, 2.9, 2.75],
  325. [70, 2.65, 3.05, 2.7],
  326. [81, 2.8, 3, 2.75],
  327. [82, 2.62, 2.9, 2.55],
  328. [88, 2.65, 3, 2.6],
  329. [90, 2.41, 3.4, 2.7],
  330. [104, 2.75, 2.9, 2.75],
  331. [110, 2.55, 3.15, 2.85],
  332. [115, 2.75, 2.9, 2.75],
  333. [158, 2.6, 3.2, 2.6],
  334. [173, 2.65, 2.8, 2.7],
  335. [177, 2.67, 3.3, 2.78],
  336. [255, 2.7, 2.9, 2.75],
  337. [281, 2.75, 3.1, 2.7],
  338. [370, 2.4, 2.65, 2.45],
  339. [422, 2.6, 3.2, 2.6],
  340. [450, 2.65, 3, 2.7],
  341. [474, 2.62, 3.1, 2.67],
  342. [499, 2.65, 3.25, 2.55],
  343. [517, 2.6, 3.2, 2.6],
  344. [545, 2.65, 3.25, 2.55],
  345. [659, 2.58, 3.07, 2.58],
  346. [976, 2.65, 3.1, 2.65]],
  347. &#39;L_2247567&#39;: [[3, 0.98, 0, 0.9],
  348. [8, 0.9, 0, 0.95],
  349. [12, 0.84, 0, 1.06],
  350. [14, 0.91, 0, 0.85],
  351. [17, 0.95, 0, 0.95],
  352. [19, 0.8, 0, 0.9],
  353. [22, 0.81, 0, 0.88],
  354. [23, 0.99, 0, 0.93],
  355. [24, 0.95, 0, 0.95],
  356. [31, 0.94, 0, 0.98],
  357. [35, 0.95, 0, 0.95],
  358. [42, 0.89, 0, 0.89]],
  359. &#39;T_2247567&#39;: [[3, 1.05, 2.25, 0.81],
  360. [4, 1.3, 2.5, 0.57],
  361. [8, 0.9, 2.25, 0.95],
  362. [9, 1.3, 2.5, 0.57],
  363. [12, 0.87, 2.25, 1.01],
  364. [14, 1, 2.25, 0.78],
  365. [17, 1.08, 2.25, 0.8],
  366. [19, 1.3, 2.5, 0.55],
  367. [22, 0.77, 2.25, 0.87],
  368. [23, 1.06, 2.25, 0.84],
  369. [24, 1.08, 2.25, 0.8],
  370. [31, 0.95, 2.25, 0.95],
  371. [35, 1.09, 2.25, 0.81],
  372. [42, 1.04, 2.25, 0.76],
  373. [49, 1.25, 2.5, 0.57]],
  374. &#39;O_2247568&#39;: [[4, 3.1, 3.25, 2.3],
  375. [16, 2.85, 2.9, 2.3],
  376. [18, 3.25, 3.15, 2.2],
  377. [60, 3.1, 3.15, 2.32],
  378. [70, 3, 3.15, 2.35],
  379. [81, 3.3, 3, 2.38],
  380. [82, 3.1, 2.9, 2.2],
  381. [88, 3.2, 3, 2.25],
  382. [90, 2.9, 3.3, 2.32],
  383. [104, 3.1, 3.25, 2.25],
  384. [110, 3, 3.2, 2.4],
  385. [115, 3.4, 3, 2.25],
  386. [158, 2.95, 3.2, 2.35],
  387. [173, 3, 3, 2.25],
  388. [177, 3.08, 3.2, 2.49],
  389. [255, 3.2, 3.2, 2.25],
  390. [281, 2.9, 3.2, 2.38],
  391. [370, 2.75, 2.85, 2.05],
  392. [422, 3.2, 3.2, 2.25],
  393. [450, 3.05, 3.15, 2.28],
  394. [474, 2.82, 3.1, 2.48],
  395. [499, 3.1, 3.25, 2.23],
  396. [517, 3.25, 3.15, 2.2],
  397. [545, 3.1, 3.25, 2.23],
  398. [659, 3.12, 3.1, 2.18],
  399. [976, 3.3, 3.15, 2.2]],
  400. &#39;L_2247568&#39;: [[3, 0.92, -0.25, 0.96],
  401. [8, 0.75, -0.25, 1.13],
  402. [12, 0.81, -0.25, 1.1],
  403. [14, 0.85, -0.25, 0.91],
  404. [17, 0.95, -0.25, 0.95],
  405. [19, 1.05, 0, 0.7],
  406. [22, 0.71, -0.25, 1],
  407. [23, 0.93, -0.25, 0.99],
  408. [24, 0.95, -0.25, 0.95],
  409. [31, 0.8, -0.25, 1.13],
  410. [35, 0.95, -0.25, 0.95],
  411. [42, 0.89, -0.25, 0.89]],
  412. &#39;T_2247568&#39;: [[3, 1.05, 2.25, 0.81],
  413. [4, 1.3, 2.5, 0.57],
  414. [8, 0.98, 2.25, 0.88],
  415. [9, 1.3, 2.5, 0.57],
  416. [12, 1.01, 2.25, 0.87],
  417. [14, 1, 2.25, 0.78],
  418. [17, 1.08, 2.25, 0.8],
  419. [19, 1.2, 2.5, 0.6],
  420. [22, 0.85, 2.25, 0.8],
  421. [23, 1.06, 2.25, 0.84],
  422. [24, 1.08, 2.25, 0.8],
  423. [31, 1, 2.25, 0.9],
  424. [35, 1.09, 2.25, 0.81],
  425. [42, 1.04, 2.25, 0.76],
  426. [49, 1.25, 2.5, 0.57]],
  427. &#39;O_2247569&#39;: [[4, 2.8, 2.95, 2.7],
  428. [16, 2.4, 2.8, 2.85],
  429. [18, 2.53, 3, 2.84],
  430. [60, 2.8, 2.9, 2.75],
  431. [70, 2.5, 2.95, 3],
  432. [81, 2.63, 3, 2.9],
  433. [82, 2.5, 2.87, 2.7],
  434. [88, 2.45, 2.9, 2.9],
  435. [90, 2.36, 3.1, 3],
  436. [104, 2.7, 3.05, 2.65],
  437. [110, 2.5, 3, 3.05],
  438. [115, 2.8, 2.9, 2.7],
  439. [158, 2.45, 3.1, 2.95],
  440. [173, 2.7, 2.75, 2.65],
  441. [177, 2.58, 3.12, 3.13],
  442. [255, 2.6, 3.2, 2.65],
  443. [281, 2.5, 3, 2.9],
  444. [370, 2.5, 2.6, 2.4],
  445. [422, 2.5, 3.1, 2.8],
  446. [450, 2.75, 2.9, 2.7],
  447. [474, 2.49, 2.94, 2.95],
  448. [499, 2.6, 3.15, 2.64],
  449. [517, 2.53, 3, 2.84],
  450. [545, 2.6, 3.15, 2.64],
  451. [659, 2.66, 3, 2.54],
  452. [976, 2.75, 3.05, 2.6]],
  453. &#39;L_2247569&#39;: [[3, 0.93, 0, 0.95],
  454. [8, 0.73, 0, 1.15],
  455. [12, 0.74, 0, 1.19],
  456. [14, 0.78, 0, 1.08],
  457. [17, 1, 0, 0.9],
  458. [19, 0.7, 0, 1.05],
  459. [22, 0.69, 0, 1.03],
  460. [23, 0.94, 0, 0.98],
  461. [24, 1, 0, 0.9],
  462. [31, 0.8, 0, 1.13],
  463. [35, 1, 0, 0.9],
  464. [42, 0.94, 0, 0.84]],
  465. &#39;T_2247569&#39;: [[3, 0.87, 2, 0.99],
  466. [4, 1.37, 2.5, 0.5],
  467. [8, 0.9, 2, 0.95],
  468. [9, 1.5, 2.5, 0.5],
  469. [12, 0.98, 2, 0.9],
  470. [14, 0.85, 2, 1],
  471. [17, 0.88, 2, 1],
  472. [19, 1.4, 2.5, 0.5],
  473. [22, 1.06, 2.25, 0.63],
  474. [23, 0.88, 2, 1.02],
  475. [24, 0.88, 2, 1],
  476. [31, 0.9, 2, 1],
  477. [35, 0.89, 2, 1.01],
  478. [42, 0.83, 2, 0.95],
  479. [49, 1.37, 2.5, 0.51]],
  480. &#39;O_2247570&#39;: [[4, 3.4, 3.1, 2.2],
  481. [16, 3, 2.75, 2.35],
  482. [18, 3.3, 3.15, 2.18],
  483. [60, 3.4, 3.1, 2.2],
  484. [70, 3, 3, 2.45],
  485. [81, 3.4, 3.1, 2.3],
  486. [82, 3.1, 2.9, 2.2],
  487. [88, 3.2, 3, 2.25],
  488. [90, 2.8, 3.3, 2.34],
  489. [104, 3.35, 3.15, 2.2],
  490. [110, 3, 3.15, 2.45],
  491. [115, 3.4, 3.1, 2.2],
  492. [158, 2.9, 3.2, 2.37],
  493. [173, 3.3, 2.95, 2.13],
  494. [177, 3.23, 3, 2.53],
  495. [255, 3.2, 3.25, 2.2],
  496. [281, 2.9, 3.2, 2.38],
  497. [370, 3, 2.8, 1.95],
  498. [422, 3.2, 3.25, 2.2],
  499. [450, 3.38, 3.15, 2.16],
  500. [474, 2.82, 3.1, 2.48],
  501. [499, 3.15, 3.25, 2.19],
  502. [517, 3.3, 3.15, 2.18],
  503. [545, 3.15, 3.25, 2.19],
  504. [659, 3.2, 3.14, 2.12],
  505. [976, 3.35, 3.2, 2.15]],
  506. &#39;L_2247570&#39;: [[3, 0.97, -0.25, 0.91],
  507. [8, 0.75, -0.25, 1.13],
  508. [12, 0.76, -0.25, 1.16],
  509. [14, 0.9, -0.25, 0.95],
  510. [17, 1, -0.25, 0.9],
  511. [19, 1.05, 0, 0.7],
  512. [22, 0.72, -0.25, 0.98],
  513. [23, 0.98, -0.25, 0.94],
  514. [24, 1, -0.25, 0.9],
  515. [31, 0.8, -0.25, 1.13],
  516. [35, 1, -0.25, 0.9],
  517. [42, 0.94, -0.25, 0.84]],
  518. &#39;T_2247570&#39;: [[3, 1, 2.25, 0.86],
  519. [4, 1.2, 2.5, 0.6],
  520. [8, 0.95, 2.25, 0.9],
  521. [9, 1.25, 2.5, 0.6],
  522. [12, 0.95, 2.25, 0.93],
  523. [14, 0.98, 2.25, 0.85],
  524. [17, 1.03, 2.25, 0.85],
  525. [19, 1.2, 2.5, 0.6],
  526. [22, 0.81, 2.25, 0.83],
  527. [23, 1.01, 2.25, 0.89],
  528. [24, 1.03, 2.25, 0.85],
  529. [31, 1, 2.25, 0.9],
  530. [35, 1.04, 2.25, 0.86],
  531. [42, 0.98, 2.25, 0.81],
  532. [49, 1.2, 2.5, 0.6]],
  533. &#39;O_2247571&#39;: [[4, 2.7, 3.1, 2.7],
  534. [16, 2.5, 2.75, 2.7],
  535. [18, 2.61, 3.15, 2.63],
  536. [60, 2.65, 3.15, 2.65],
  537. [70, 2.85, 2.88, 2.7],
  538. [81, 2.75, 3.1, 2.75],
  539. [82, 2.55, 3, 2.55],
  540. [88, 2.62, 3, 2.65],
  541. [90, 2.6, 3.2, 2.6],
  542. [104, 2.6, 3.3, 2.6],
  543. [110, 2.7, 3.1, 2.7],
  544. [115, 2.63, 3.2, 2.63],
  545. [158, 2.6, 3.1, 2.7],
  546. [173, 2.55, 3, 2.6],
  547. [177, 2.72, 2.99, 2.98],
  548. [255, 2.7, 2.95, 2.7],
  549. [281, 2.55, 3.2, 2.75],
  550. [370, 2.35, 2.85, 2.35],
  551. [422, 2.7, 2.95, 2.7],
  552. [450, 2.6, 3.15, 2.6],
  553. [474, 2.62, 3.1, 2.67],
  554. [499, 2.56, 3.35, 2.56],
  555. [517, 2.61, 3.15, 2.63],
  556. [545, 2.56, 3.35, 2.56],
  557. [659, 2.55, 3.16, 2.55],
  558. [976, 2.6, 3.2, 2.6]],
  559. &#39;L_2247571&#39;: [[3, 0.94, 0, 0.94],
  560. [8, 0.85, 0, 1],
  561. [12, 0.93, 0, 0.97],
  562. [14, 0.91, 0, 0.93],
  563. [17, 0.95, 0, 0.95],
  564. [19, 0.8, 0, 0.9],
  565. [22, 1.15, 0.25, 0.61],
  566. [23, 0.95, 0, 0.97],
  567. [24, 0.95, 0, 0.95],
  568. [31, 0.94, 0, 0.98],
  569. [35, 0.95, 0, 0.95],
  570. [42, 0.89, 0, 0.89]],
  571. &#39;T_2247571&#39;: [[3, 0.92, 2.25, 0.94],
  572. [4, 1.15, 2.5, 0.61],
  573. [8, 0.93, 2.25, 0.93],
  574. [9, 1.1, 2.5, 0.67],
  575. [12, 0.97, 2.25, 0.91],
  576. [14, 0.91, 2.25, 0.93],
  577. [17, 0.93, 2.25, 0.95],
  578. [19, 1.1, 2.5, 0.65],
  579. [22, 1.05, 2.5, 0.64],
  580. [23, 0.93, 2.25, 0.97],
  581. [24, 0.93, 2.25, 0.95],
  582. [31, 0.95, 2.25, 0.95],
  583. [35, 0.94, 2.25, 0.96],
  584. [42, 0.88, 2.25, 0.9],
  585. [49, 1.15, 2.5, 0.62]]}}
  586. </details>
  587. # 答案2
  588. **得分**: 1
  589. 响应不是`json`格式，这就是为什么`response.json()`失败
  590. 我们可以使用正则表达式来提取数据，然后使用`eval`将字符串转换为列表
  591. ```python3
  592. import re
  593. import requests
  594. params = {
  595. 'sclassId': '40',
  596. 'subSclassId': '261',
  597. 'matchSeason': '2022-2023',
  598. 'round': '23',
  599. 'flesh': '0.6807624444063407',
  600. }
  601. response = requests.get('https://football.goaloo2.com/ajax/LeagueOddsAjax', params=params)
  602. data = {}
  603. for id_, lst in re.findall('oddsData\["(.+?)"\]=(.+?);', response.text):
  604. data[id_] = eval(lst)
英文:

The response isn't in json format that's why the response.json() fails

we can use regular expressions to extract the data
and eval to convert string to list

  1. import re
  2. import requests
  4. params = {
  5.     &#39;sclassId&#39;: &#39;40&#39;,
  6.     &#39;subSclassId&#39;: &#39;261&#39;,
  7.     &#39;matchSeason&#39;: &#39;2022-2023&#39;,
  8.     &#39;round&#39;: &#39;23&#39;,
  9.     &#39;flesh&#39;: &#39;0.6807624444063407&#39;,
  10. }
  12. response = requests.get(&#39;https://football.goaloo2.com/ajax/LeagueOddsAjax&#39;, params=params)
  14. data = {}
  15. for id_, lst in re.findall(&#39;oddsData\[&quot;(.+?)&quot;\]=(.+?);&#39;, response.text):
  16.     data[id_] = eval(lst)

huangapple
  • 本文由 发表于 2023年2月8日 17:23:46
  • 转载请务必保留本文链接：https://go.coder-hub.com/75383609.html
  • api
  • json
  • python
  • web-scraping
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定