如何在Python中使用多线程进行更快的API调用,而不使用requests?

huangapple go评论98阅读模式
英文:

How do you make faster A.P.I calls using multithreading without using requests in Python?

问题

我正在尝试获取标普500指数中每家公司的历史股票数据。问题是获取数据花费了很长时间。

  1. from ApiStuff import ApiStuff
  2. import fundamentalanalysis as fa
  3. import pickle
  4. tickers = pickle.load(open('S&P500_TICKERS.dat','rb'))
  5. api_key = ApiStuff.api_key
  6. data_from_tickers = []
  7. for ticker in tickers:
  8. balance_sheet_annually = fa.balance_sheet_statement(ticker, api_key, period="annual")
  9. data_from_tickers.append(balance_sheet_annually)

我尝试在互联网上搜索如何加快速度,但它们使用其他模块(例如requests、aiohttp)来加快数据的检索,而我依赖于这个模块(fundamentalanalysis)来检索基本数据。

有没有办法让我继续使用这个模块,并通过描述的方法加快API请求的速度?

英文:

I'm trying to receive historical stock data for every company in the S&P 500. The problem is that it is taking a really longtime to get the data.

  1. from ApiStuff import ApiStuff
  2. import fundamentalanalysis as fa
  3. import pickle
  4. tickers = pickle.load(open('S&P500_TICKERS.dat','rb'))
  5. api_key = ApiStuff.api_key
  6. data_from_tickers = []
  7. for ticker in tickers:
  8. balance_sheet_annually = fa.balance_sheet_statement(ticker, api_key, period="annual")
  9. data_from_tickers.append(balance_sheet_annually)

I tried searching on the internet on how to speed it up but they use other modules (i.e requests, aiohttp) for making the retrieval of the data faster and I am dependent on this module (fundamentalanalysis) to retrieve fundamental data.

Is there a way for me to still use this module and make api requests faster via the methods described?

答案1

得分: 1

可以使用多个进程来完成这个任务;concurrent.futures 正是为这种需求而设计的。另一方面,这也是一个学习开源的绝佳机会。fundamentalanalysis 的源代码可以在 Github 上找到。你正在使用的函数,balance_sheet_statement,非常简单,基本上由一个 GET 请求、一些数据映射和构建 Pandas 数据帧组成。

使用 aiohttprequests 复制这个逻辑要比处理多进程模块容易得多!

英文:

You certainly can do this with multiple processes; concurrent.futures is made for this type of need. On the other hand, this is also a great learning opportunity for the use of open source. The source for fundamentalanalysis is available on Github. The function you're using, balance_sheet_statement, is very straightforward and basically consists of a GET request, a couple of data mappings, and the construction of a Pandas dataframe.

Replicating this logic using aiohttp or requests is going to be easier than wrangling the multiprocessing modules!

答案2

得分: 1

如果 fundamentalanalysis 支持多线程,可以通过以下方式替换 for 循环来实现:

  1. from concurrent.futures import ThreadPoolExecutor
  2. with ThreadPoolExecutor(max_workers=10) as e:
  3. data_from_tickers = list(e.map(lambda t: fa.balance_sheet_statement(t, api_key, period="annual"), tickers))

最大工作线程数可以进行调整。

英文:

If fundamentalanalysis supports multithreading, it can be done by replacing the for-loop with:

  1. from concurrent.futures import ThreadPoolExecutor
  2. with ThreadPoolExecutor(max_workers=10) as e:
  3. data_from_tickers = list(e.map(lambda t: fa.balance_sheet_statement(t, api_key, period="annual"), tickers))

The maximum number of workers can be adjusted.

huangapple
  • 本文由 发表于 2023年2月26日 21:02:19
  • 转载请务必保留本文链接:https://go.coder-hub.com/75572163.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定