Converting aiohttp script to asyncio + requests (aiohttp not working on ubuntu while asyncio + requests works)

huangapple go评论79阅读模式
英文:

Converting aiohttp script to asyncio + requests (aiohttp not working on ubuntu while asyncio + requests works)

问题

以下是翻译后的代码和错误信息:

这是我的原始代码。

import aiohttp
import asyncio

async def get_data(session, x):
    while True:
        try:
            async with session.get(url=f'https://api.abc.com/{x}') as response:
                if response.status == 200:
                    data = await response.json()
                    try:
                        data = float(data)
                        return data
                    except ValueError:
                        print("数据不是有效的浮点数。重试...")
                else:
                    print("收到非200状态码。重试...")
            await asyncio.sleep(1)  # 在重试之前等待1秒
        except Exception as e:
            print("无法获取网址 {},原因是 {}. 重试...".format(x, e.__class__))
            await asyncio.sleep(1)  # 在重试之前等待1秒

async def main(datas):
    async with aiohttp.ClientSession() as session:
        ret = await asyncio.gather(*[get_data(session, data) for data in datas])
        return {datas[i]: ret[i] for i in range(len(datas))}  # 将结果作为字典返回

datas = ['x1', 'x2', 'x3', 'x4']
results = asyncio.run(main(datas))

这是我的代码

import asyncio
import requests

async def get_data(x):
    while True:
        try:
            response = requests.get(url=f'https://api.abc.com/{x}')
            if response.status_code == 200:
                try:
                    data = float(response.json())
                    return data
                except ValueError:
                    print("数据不是有效的浮点数。重试...")
            else:
                print("收到非200状态码。重试...")
            await asyncio.sleep(1)  # 在重试之前等待1秒
        except Exception as e:
            print("无法获取网址 {},原因是 {}. 重试...".format(x, e.__class__))
            await asyncio.sleep(1)  # 在重试之前等待1秒

async def main(datas):
    tasks = [get_data(data) for data in datas]
    ret = await asyncio.gather(*tasks)
    return {datas[i]: ret[i] for i in range(len(datas))}  # 将结果作为字典返回

datas = ['x1', 'x2', 'x3', 'x4']
results = asyncio.run(main(datas))
print(results)

到目前为止,我收到的错误信息如下:

无法获取 x1 的数据,原因是 <class 'TypeError'>。重试...
无法获取 x2 的数据,原因是 <class 'TypeError'>。重试...
无法获取 x3 的数据,原因是 <class 'TypeError'>。重试...
无法获取 x4 的数据,原因是 <class 'TypeError'>。重试...
无法获取 x1 的数据,原因是 <class 'TypeError'>。重试...
...

请注意,错误信息中的 <class 'TypeError'> 是占位符,实际错误信息可能会提供更多有关错误的详细信息。您可能需要进一步调试以查明问题的根本原因。

英文:

I am using the following script to do queries on a website.

It works on macos, however it does not work with ubuntu. I have tried requests and it works, I also tried a simple asyncio + requests with the website and it works.

I need help converting it to using requests instead of aiohttp. Any help would be appreciated.

This is my original code.

import aiohttp
import asyncio

async def get_data(session, x):
    while True:
        try:
            async with session.get(url=f&#39;https://api.abc.com/{x}&#39;) as response:
                if response.status == 200:
                    data = await response.json()
                    try:
                        data = float(data)
                        return data
                    except ValueError:
                        print(&quot;Data is not a valid float. Retrying...&quot;)
                else:
                    print(&quot;Received non-200 status code. Retrying...&quot;)
            await asyncio.sleep(1)  # Wait for 1 second before retrying
        except Exception as e:
            print(&quot;Unable to get url {} due to {}. Retrying...&quot;.format(x, e.__class__))
            await asyncio.sleep(1)  # Wait for 1 second before retrying

async def main(datas):
    async with aiohttp.ClientSession() as session:
        ret = await asyncio.gather(*[get_data(session, data) for data in datas])
        return {datas[i]: ret[i] for i in range(len(datas))}  # Return the results as a dictionary

datas = [&#39;x1&#39;, &#39;x2&#39;, &#39;x3&#39;, &#39;x4&#39;]
results = asyncio.run(main(datas))

Here is my code


import asyncio
import requests

async def get_data(x):
    while True:
        try:
            response = requests.get(url=f&#39;https://api.abc.com/{x}&#39;)
            if response.status_code == 200:
                try:
                    data = float(response.json())
                    return data
                except ValueError:
                    print(&quot;Data is not a valid float. Retrying...&quot;)
            else:
                print(&quot;Received non-200 status code. Retrying...&quot;)
            await asyncio.sleep(1)  # Wait for 1 second before retrying
        except Exception as e:
            print(&quot;Unable to get url {} due to {}. Retrying...&quot;.format(x, e.__class__))
            await asyncio.sleep(1)  # Wait for 1 second before retrying

async def main(datas):
    tasks = [get_data(data) for data in datas]
    ret = await asyncio.gather(*tasks)
    return {datas[i]: ret[i] for i in range(len(datas))}  # Return the results as a dictionary

datas = [&#39;x1&#39;, &#39;x2&#39;, &#39;x3&#39;, &#39;x4&#39;]
results = asyncio.run(main(datas))
print(results)

and the error I am getting so far,

Unable to get data for x1 due to &lt;class &#39;TypeError&#39;&gt;. Retrying...
Unable to get data for x2 due to &lt;class &#39;TypeError&#39;&gt;. Retrying...
Unable to get data for x3 due to &lt;class &#39;TypeError&#39;&gt;. Retrying...
Unable to get data for x4 due to &lt;class &#39;TypeError&#39;&gt;. Retrying...
Unable to get data for x1 due to &lt;class &#39;TypeError&#39;&gt;. Retrying...
...

答案1

得分: 2

Most likely Ubuntu doesn't work because of certificate problems.

certifi package can be used for providing fresh client certificates (please don't forget to keep the latest version of certifi in your setup).

Tuning aiohttp to use certifi instead of system storage is relatively simple:

import aiohttp
import asyncio
import certifi
import ssl

async def get_data(session, x):
    while True:
        try:
            ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS)
            ssl_context.load_verify_locations(capath=certifi.where())
            async with session.get(
                url=f'https://api.abc.com/{x}',
                connector=aiohttp.TCPConnector(ssl=ssl_context),
            ) as response:
                if response.status == 200:
                    data = await response.json()
                    try:
                        data = float(data)
                        return data
                    except ValueError:
                        print("Data is not a valid float. Retrying...")
                else:
                    print("Received non-200 status code. Retrying...")
            await asyncio.sleep(1)  # Wait for 1 second before retrying
        except Exception as e:
            print("Unable to get url {} due to {}. Retrying...".format(x, e.__class__))
            await asyncio.sleep(1)  # Wait for 1 second before retrying

async def main(datas):
    async with aiohttp.ClientSession() as session:
        ret = await asyncio.gather(*[get_data(session, data) for data in datas])
        return {datas[i]: ret[i] for i in range(len(datas))}  # Return the results as a dictionary

datas = ['x1', 'x2', 'x3', 'x4']
results = asyncio.run(main(datas))
英文:

Most likely Ubuntu doesn't work because of certificate problems.

certifi package can be used for providing fresh client certificates (please don't forget to keep the latest version of certifi in your setup).

Tuning aiohttp to use certifi instead of system storage is relatively simple:

import aiohttp
import asyncio
import certifi
import ssl

async def get_data(session, x):
    while True:
        try:
            ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS)
            ssl_context.load_verify_locations(capath=certifi.where())
            async with session.get(
                url=f&#39;https://api.abc.com/{x}&#39;,
                connector=aiohttp.TCPConnector(ssl=ssl_context),
            ) as response:
                if response.status == 200:
                    data = await response.json()
                    try:
                        data = float(data)
                        return data
                    except ValueError:
                        print(&quot;Data is not a valid float. Retrying...&quot;)
                else:
                    print(&quot;Received non-200 status code. Retrying...&quot;)
            await asyncio.sleep(1)  # Wait for 1 second before retrying
        except Exception as e:
            print(&quot;Unable to get url {} due to {}. Retrying...&quot;.format(x, e.__class__))
            await asyncio.sleep(1)  # Wait for 1 second before retrying

async def main(datas):
    async with aiohttp.ClientSession() as session:
        ret = await asyncio.gather(*[get_data(session, data) for data in datas])
        return {datas[i]: ret[i] for i in range(len(datas))}  # Return the results as a dictionary

datas = [&#39;x1&#39;, &#39;x2&#39;, &#39;x3&#39;, &#39;x4&#39;]
results = asyncio.run(main(datas))

答案2

得分: 1

通过捕获 ValueError 并打印一条消息,该函数不应再引发与解析 JSON 数据相关的任何错误。

async def get_data(x):
    while True:
        try:
            response = requests.get(url=f'https://api.abc.com/{x}')
            if response.status_code == 200:
                try:
                    data = float(response.json())
                    return data
                except ValueError:
                    print("数据不是有效的浮点数。重试中...")
            else:
                print(f"收到非 200 状态码 ({response.status_code}). 重试中...")
            await asyncio.sleep(1)  # 在重试之前等待 1 秒
        except Exception as e:
            print(f"无法获取 {x} 的数据,原因是 {type(e).__name__}。重试中...")
            await asyncio.sleep(1)  # 在重试之前等待 1 秒
英文:

By catching the ValueError and printing a message, the function should not raise any more errors related to parsing JSON data

async def get_data(x):
    while True:
        try:
            response = requests.get(url=f&#39;https://api.abc.com/{x}&#39;)
            if response.status_code == 200:
                try:
                    data = float(response.json())
                    return data
                except ValueError:
                    print(&quot;Data is not a valid float. Retrying...&quot;)
            else:
                print(f&quot;Received non-200 status code ({response.status_code}). Retrying...&quot;)
            await asyncio.sleep(1)  # Wait for 1 second before retrying
        except Exception as e:
            print(f&quot;Unable to get data for {x} due to {type(e).__name__}. Retrying...&quot;)
            await asyncio.sleep(1)  # Wait for 1 second before retrying

huangapple
  • 本文由 发表于 2023年7月27日 23:07:14
  • 转载请务必保留本文链接:https://go.coder-hub.com/76781109.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定