如何在Python中使用多线程进行更快的API调用,而不使用requests?

huangapple go评论70阅读模式
英文:

How do you make faster A.P.I calls using multithreading without using requests in Python?

问题

我正在尝试获取标普500指数中每家公司的历史股票数据。问题是获取数据花费了很长时间。

from ApiStuff import ApiStuff
import fundamentalanalysis as fa
import pickle

tickers = pickle.load(open('S&P500_TICKERS.dat','rb'))

api_key = ApiStuff.api_key
data_from_tickers = []

for ticker in tickers:
    balance_sheet_annually  = fa.balance_sheet_statement(ticker, api_key, period="annual")
    data_from_tickers.append(balance_sheet_annually)

我尝试在互联网上搜索如何加快速度,但它们使用其他模块(例如requests、aiohttp)来加快数据的检索,而我依赖于这个模块(fundamentalanalysis)来检索基本数据。

有没有办法让我继续使用这个模块,并通过描述的方法加快API请求的速度?

英文:

I'm trying to receive historical stock data for every company in the S&P 500. The problem is that it is taking a really longtime to get the data.

from ApiStuff import ApiStuff
import fundamentalanalysis as fa
import pickle

tickers = pickle.load(open('S&P500_TICKERS.dat','rb'))

api_key = ApiStuff.api_key
data_from_tickers = []

for ticker in tickers:
    balance_sheet_annually  = fa.balance_sheet_statement(ticker, api_key, period="annual")
    data_from_tickers.append(balance_sheet_annually)

I tried searching on the internet on how to speed it up but they use other modules (i.e requests, aiohttp) for making the retrieval of the data faster and I am dependent on this module (fundamentalanalysis) to retrieve fundamental data.

Is there a way for me to still use this module and make api requests faster via the methods described?

答案1

得分: 1

可以使用多个进程来完成这个任务;concurrent.futures 正是为这种需求而设计的。另一方面,这也是一个学习开源的绝佳机会。fundamentalanalysis 的源代码可以在 Github 上找到。你正在使用的函数,balance_sheet_statement,非常简单,基本上由一个 GET 请求、一些数据映射和构建 Pandas 数据帧组成。

使用 aiohttprequests 复制这个逻辑要比处理多进程模块容易得多!

英文:

You certainly can do this with multiple processes; concurrent.futures is made for this type of need. On the other hand, this is also a great learning opportunity for the use of open source. The source for fundamentalanalysis is available on Github. The function you're using, balance_sheet_statement, is very straightforward and basically consists of a GET request, a couple of data mappings, and the construction of a Pandas dataframe.

Replicating this logic using aiohttp or requests is going to be easier than wrangling the multiprocessing modules!

答案2

得分: 1

如果 fundamentalanalysis 支持多线程,可以通过以下方式替换 for 循环来实现:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=10) as e:
    data_from_tickers = list(e.map(lambda t: fa.balance_sheet_statement(t, api_key, period="annual"), tickers))

最大工作线程数可以进行调整。

英文:

If fundamentalanalysis supports multithreading, it can be done by replacing the for-loop with:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=10) as e:
    data_from_tickers = list(e.map(lambda t: fa.balance_sheet_statement(t, api_key, period="annual"), tickers))

The maximum number of workers can be adjusted.

huangapple
  • 本文由 发表于 2023年2月26日 21:02:19
  • 转载请务必保留本文链接:https://go.coder-hub.com/75572163.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定