2020年1月7日 00:41:36go评论109阅读模式

英文:

How can I improve performance of nested comprehension?

问题

我尝试使用Python 3.x的推导式来创建一个嵌套字典结构。我的推导式语法有效，但对于大型数据集来说速度非常慢。我已经使用循环创建了我想要的数据结构，它运行得更快，但我想知道是否有方法可以改进这个推导式，使其更有效率，可能能够与我的循环代码一样快甚至更快地运行。

我的输入数据是一个包含字典的列表，每个字典都描述了业余无线电联系（日志条目）的具体信息。以下是我的数据的随机子集（限制为20个条目，并删除了字典中的非关键信息，以使其更清晰）：

我想创建一个字典，其中每个键都是一个波段（10M、20M等），值将是一个字典，列出该波段上联系的国家作为键，每个国家上的联系计数作为值。以下是我的输出示例：

这是我想出的用于创建输出的推导式。它有效，对于这里显示的有限数据集，运行速度很快，但对于包含几千个条目的输入列表，运行时间很长。

worked_dxcc_by_band = {
    z["BAND"]: {
        x["COUNTRY"]: len([y["COUNTRY"]
                           for y in log_entries
                           if y["COUNTRY"] == x["COUNTRY"] and y["BAND"] == z["BAND"]])
        for x in log_entries
        if x["BAND"] == z["BAND"]
    }
    for z in log_entries
}

由于这是一个三重嵌套的推导式，所有三个循环都遍历整个log_entries列表，我认为这就是为什么它变得非常慢的原因。

是否有更有效的方法可以使用推导式来完成这个任务？我可以使用循环处理数据，但我试图提高使用推导式的技能，所以我认为这将是一个很好的练习！

这是我在不使用推导式的情况下所做的：我有一个名为analyze_log_entry的函数，每次从文件中加载一个日志条目时都会调用它。

from collections import Counter

worked_dxcc_by_band = {}

def analyze_log_entry(entry):
    if "BAND" in entry:
        if "COUNTRY" in entry:
            if entry["BAND"] in worked_dxcc_by_band:
                worked_dxcc_by_band[entry["BAND"]][entry["COUNTRY"]] += 1
            else:
                worked_dxcc_by_band[entry["BAND"]] = Counter()
                worked_dxcc_by_band[entry["BAND"]][entry["COUNTRY"]] = 1

这本身可能不是非常高效，但我的完整代码在analyze_log_entry函数中有许多类似的块，用于构建多个字典。因为我只遍历我的数据一次，并在适当的地方构建字典，所以它可能比使用推导式更高效，后者本质上是多个循环。正如我所说，这更多地是一个练习，以学习如何使用不同的方法完成相同的任务。

英文:

I am trying to use python 3.x comprehension to create a nested dictionary structure. My comprehension syntax works, but it is very slow, especially with a large data set. I have also created my desired data structure using loops and it runs much faster, but I would like to know if there is a way to improve this comprehension to make it more efficient and potentially run as fast as, or faster than my loop code.

My input data is a list of dictionaries, each dictionary outlining the specifics of an amateur radio contact (log entry). Here is a random subset of my data (limited to 20 entries, and non-essential keys in the dictionary removed to make this more clear)

[{&#39;BAND&#39;: &#39;20M&#39;,
  &#39;CALL&#39;: &#39;AA9GL&#39;,
  &#39;COUNTRY&#39;: &#39;UNITED STATES OF AMERICA&#39;,
  &#39;QSO_DATE&#39;: &#39;20170528&#39;,
  &#39;TIME_ON&#39;: &#39;132100&#39;},
 {&#39;BAND&#39;: &#39;20M&#39;,
  &#39;CALL&#39;: &#39;KE4BFI&#39;,
  &#39;COUNTRY&#39;: &#39;UNITED STATES OF AMERICA&#39;,
  &#39;QSO_DATE&#39;: &#39;20150704&#39;,
  &#39;TIME_ON&#39;: &#39;034600&#39;},
 {&#39;BAND&#39;: &#39;20M&#39;,
  &#39;CALL&#39;: &#39;W8OTR&#39;,
  &#39;COUNTRY&#39;: &#39;UNITED STATES OF AMERICA&#39;,
  &#39;QSO_DATE&#39;: &#39;20190119&#39;,
  &#39;TIME_ON&#39;: &#39;194645&#39;},
 {&#39;BAND&#39;: &#39;10M&#39;,
  &#39;CALL&#39;: &#39;FY5FY&#39;,
  &#39;COUNTRY&#39;: &#39;FRENCH GUIANA&#39;,
  &#39;QSO_DATE&#39;: &#39;20150328&#39;,
  &#39;TIME_ON&#39;: &#39;161953&#39;},
 {&#39;BAND&#39;: &#39;17M&#39;,
  &#39;CALL&#39;: &#39;KD5FOY&#39;,
  &#39;COUNTRY&#39;: &#39;UNITED STATES OF AMERICA&#39;,
  &#39;QSO_DATE&#39;: &#39;20190121&#39;,
  &#39;TIME_ON&#39;: &#39;145630&#39;},
 {&#39;BAND&#39;: &#39;10M&#39;,
  &#39;CALL&#39;: &#39;K5GQ&#39;,
  &#39;COUNTRY&#39;: &#39;UNITED STATES OF AMERICA&#39;,
  &#39;QSO_DATE&#39;: &#39;20150110&#39;,
  &#39;TIME_ON&#39;: &#39;195326&#39;},
 {&#39;BAND&#39;: &#39;10M&#39;,
  &#39;CALL&#39;: &#39;CR5L&#39;,
  &#39;COUNTRY&#39;: &#39;PORTUGAL&#39;,
  &#39;QSO_DATE&#39;: &#39;20151025&#39;,
  &#39;TIME_ON&#39;: &#39;182351&#39;},
 {&#39;BAND&#39;: &#39;20M&#39;,
  &#39;CALL&#39;: &#39;AD4TR&#39;,
  &#39;COUNTRY&#39;: &#39;UNITED STATES OF AMERICA&#39;,
  &#39;QSO_DATE&#39;: &#39;20170325&#39;,
  &#39;TIME_ON&#39;: &#39;144606&#39;},
 {&#39;BAND&#39;: &#39;40M&#39;,
  &#39;CALL&#39;: &#39;EA8FJ&#39;,
  &#39;COUNTRY&#39;: &#39;CANARY ISLANDS&#39;,
  &#39;QSO_DATE&#39;: &#39;20170618&#39;,
  &#39;TIME_ON&#39;: &#39;020300&#39;},
 {&#39;BAND&#39;: &#39;10M&#39;,
  &#39;CALL&#39;: &#39;PY2DPM&#39;,
  &#39;COUNTRY&#39;: &#39;BRAZIL&#39;,
  &#39;QSO_DATE&#39;: &#39;20150104&#39;,
  &#39;TIME_ON&#39;: &#39;205900&#39;},
 {&#39;BAND&#39;: &#39;17M&#39;,
  &#39;CALL&#39;: &#39;MM0HVU&#39;,
  &#39;COUNTRY&#39;: &#39;SCOTLAND&#39;,
  &#39;QSO_DATE&#39;: &#39;20170416&#39;,
  &#39;TIME_ON&#39;: &#39;130200&#39;},
 {&#39;BAND&#39;: &#39;10M&#39;,
  &#39;CALL&#39;: &#39;LW3DG&#39;,
  &#39;COUNTRY&#39;: &#39;ARGENTINA&#39;,
  &#39;QSO_DATE&#39;: &#39;20161029&#39;,
  &#39;TIME_ON&#39;: &#39;210629&#39;},
 {&#39;BAND&#39;: &#39;10M&#39;,
  &#39;CALL&#39;: &#39;LW3DG&#39;,
  &#39;COUNTRY&#39;: &#39;ARGENTINA&#39;,
  &#39;QSO_DATE&#39;: &#39;20151025&#39;,
  &#39;TIME_ON&#39;: &#39;210714&#39;},
 {&#39;BAND&#39;: &#39;20M&#39;,
  &#39;CALL&#39;: &#39;EI7HDB&#39;,
  &#39;COUNTRY&#39;: &#39;IRELAND&#39;,
  &#39;QSO_DATE&#39;: &#39;20170423&#39;,
  &#39;TIME_ON&#39;: &#39;184000&#39;},
 {&#39;BAND&#39;: &#39;20M&#39;,
  &#39;CALL&#39;: &#39;KM0NAS&#39;,
  &#39;COUNTRY&#39;: &#39;UNITED STATES OF AMERICA&#39;,
  &#39;QSO_DATE&#39;: &#39;20180102&#39;,
  &#39;TIME_ON&#39;: &#39;142151&#39;},
 {&#39;BAND&#39;: &#39;10M&#39;,
  &#39;CALL&#39;: &#39;PY2TKB&#39;,
  &#39;COUNTRY&#39;: &#39;BRAZIL&#39;,
  &#39;QSO_DATE&#39;: &#39;20150328&#39;,
  &#39;TIME_ON&#39;: &#39;223535&#39;},
 {&#39;BAND&#39;: &#39;40M&#39;,
  &#39;CALL&#39;: &#39;EB1DJ&#39;,
  &#39;COUNTRY&#39;: &#39;SPAIN&#39;,
  &#39;QSO_DATE&#39;: &#39;20170326&#39;,
  &#39;TIME_ON&#39;: &#39;232430&#39;},
 {&#39;BAND&#39;: &#39;40M&#39;,
  &#39;CALL&#39;: &#39;LU6PCK&#39;,
  &#39;COUNTRY&#39;: &#39;ARGENTINA&#39;,
  &#39;QSO_DATE&#39;: &#39;20150615&#39;,
  &#39;TIME_ON&#39;: &#39;000200&#39;},
 {&#39;BAND&#39;: &#39;17M&#39;,
  &#39;CALL&#39;: &#39;G3RKF&#39;,
  &#39;COUNTRY&#39;: &#39;ENGLAND&#39;,
  &#39;QSO_DATE&#39;: &#39;20190121&#39;,
  &#39;TIME_ON&#39;: &#39;144315&#39;},
 {&#39;BAND&#39;: &#39;20M&#39;,
  &#39;CALL&#39;: &#39;UA1ZKI&#39;,
  &#39;COUNTRY&#39;: &#39;EUROPEAN RUSSIA&#39;,
  &#39;QSO_DATE&#39;: &#39;20170508&#39;,
  &#39;TIME_ON&#39;: &#39;141400&#39;}]

I want to create a dictionary where each key is a band (10M, 20M, etc) and the value will be a dictionary listing the counties contacted on that band as keys and a count of contacts for each country on that band as the values. Here is what my output looks like:

{&#39;10M&#39;: {&#39;ARGENTINA&#39;: 2,
         &#39;BRAZIL&#39;: 2,
         &#39;FRENCH GUIANA&#39;: 1,
         &#39;PORTUGAL&#39;: 1,
         &#39;UNITED STATES OF AMERICA&#39;: 1},
 &#39;17M&#39;: {&#39;ENGLAND&#39;: 1, &#39;SCOTLAND&#39;: 1, &#39;UNITED STATES OF AMERICA&#39;: 1},
 &#39;20M&#39;: {&#39;EUROPEAN RUSSIA&#39;: 1, &#39;IRELAND&#39;: 1, &#39;UNITED STATES OF AMERICA&#39;: 5},
 &#39;40M&#39;: {&#39;ARGENTINA&#39;: 1, &#39;CANARY ISLANDS&#39;: 1, &#39;SPAIN&#39;: 1}}

This is the comprehension that I came up with to create the output. It works, and with the limited data set shown here, it runs quickly, but with an input list of a couple thousand entries, it takes quite a long time to run.

worked_dxcc_by_band = {
    z[&quot;BAND&quot;]: {
        x[&quot;COUNTRY&quot;]: len([y[&quot;COUNTRY&quot;]
                           for y in log_entries
                           if y[&quot;COUNTRY&quot;] == x[&quot;COUNTRY&quot;] and y[&quot;BAND&quot;] == z[&quot;BAND&quot;]])
        for x in log_entries
        if x[&quot;BAND&quot;] == z[&quot;BAND&quot;]
    }
    for z in log_entries
}

Because this is a triple-nested comprehension, and all 3 loops run through the entire log_entries list, I am assuming that is why it gets very slow.

Is there a more efficient way to accomplish this with comprehension? I am fine using my loop to process the data but I am trying to enhance my skills regarding comprehensions so I thought this would be a good exercise!

This is what I am doing without using comprehension: I have a function analyize_log_entry which I call as I load each log entry in from a file.

from collections import Counter

worked_dxcc_by_band = {}


def analyze_log_entry(entry):
    if &quot;BAND&quot; in entry:
        if &quot;COUNTRY&quot; in entry:
            if entry[&quot;BAND&quot;] in worked_dxcc_by_band:
                worked_dxcc_by_band[entry[&quot;BAND&quot;]][entry[&quot;COUNTRY&quot;]] += 1
            else:
                worked_dxcc_by_band[entry[&quot;BAND&quot;]] = Counter()
                worked_dxcc_by_band[entry[&quot;BAND&quot;]][entry[&quot;COUNTRY&quot;]] = 1

This in itself may not be that efficient but my full code has many similar blocks within the analyze_log_entry function that build multiple dictionaries. Because I am only going through all of my data once, and building the dictionaries where appropriate, it is probably much more efficient than using comprehension, which is essentially multiple loops. As I said, this is more of an exercise to learn how to accomplish the same task with different methods.

答案1

得分: 3

以下是代码的中文翻译部分：

# 使用字典推导式版本：
out = {band: dict(Counter(v['COUNTRY'] for v in g)) for band, g in groupby(sorted(data, key=lambda k: k['BAND']), lambda k: k['BAND'])}

# 可以结合 itertools.groupby 和 collections.Counter：
from itertools import groupby
from collections import Counter

s = sorted(data, key=lambda k: k['BAND'])

out = {}
for band, g in groupby(s, lambda k: k['BAND']):
    c = Counter(v['COUNTRY'] for v in g)
    out[band] = dict(c)

# 不使用模块的版本：
out = {}
for i in data:
    out.setdefault(i['BAND'], {}).setdefault(i['COUNTRY'], 0)
    out[i['BAND']][i['COUNTRY']] += 1

# 基准测试：
from timeit import timeit

from itertools import groupby
from collections import Counter

def sol_orig():
    worked_dxcc_by_band = {z["BAND"]: {x["COUNTRY"] : len([y["COUNTRY"] for y in data if y["COUNTRY"] == x["COUNTRY"] and y["BAND"] == z["BAND"]]) for x in data if x["BAND"] == z["BAND"]} for z in data}
    return worked_dxcc_by_band

def solution():
    out = {band: dict(Counter(v['COUNTRY'] for v in g)) for band, g in groupby(sorted(data, key=lambda k: k['BAND']), lambda k: k['BAND'])}
    return out

def solution_2():
    out = {}
    for i in data:
        out.setdefault(i['BAND'], {}).setdefault(i['COUNTRY'], 0)
        out[i['BAND']][i['COUNTRY']] += 1
    return out

t1 = timeit(lambda: solution(), number=10000)
t2 = timeit(lambda: solution_2(), number=10000)
t3 = timeit(lambda: sol_orig(), number=10000)

print(t1)
print(t2)
print(t3)

请注意，这只是代码的翻译部分，不包括任何问题的回答。

英文:

EDIT: Dictionary comprehension version:

out = {band: dict(Counter(v[&#39;COUNTRY&#39;] for v in g)) for band, g in groupby(sorted(data, key=lambda k: k[&#39;BAND&#39;]), lambda k: k[&#39;BAND&#39;])}

You can combine itertools.groupby and collections.Counter:

from itertools import groupby
from collections import Counter

s = sorted(data, key=lambda k: k[&#39;BAND&#39;])

out = {}
for band, g in groupby(s, lambda k: k[&#39;BAND&#39;]):
    c = Counter(v[&#39;COUNTRY&#39;] for v in g)
    out[band] = dict(c)

from pprint import pprint
pprint(out)

Prints:

{&#39;10M&#39;: {&#39;ARGENTINA&#39;: 2,
         &#39;BRAZIL&#39;: 2,
         &#39;FRENCH GUIANA&#39;: 1,
         &#39;PORTUGAL&#39;: 1,
         &#39;UNITED STATES OF AMERICA&#39;: 1},
 &#39;17M&#39;: {&#39;ENGLAND&#39;: 1, &#39;SCOTLAND&#39;: 1, &#39;UNITED STATES OF AMERICA&#39;: 1},
 &#39;20M&#39;: {&#39;EUROPEAN RUSSIA&#39;: 1, &#39;IRELAND&#39;: 1, &#39;UNITED STATES OF AMERICA&#39;: 5},
 &#39;40M&#39;: {&#39;ARGENTINA&#39;: 1, &#39;CANARY ISLANDS&#39;: 1, &#39;SPAIN&#39;: 1}}

EDIT: Without modules:

out = {}
for i in data:
    out.setdefault(i[&#39;BAND&#39;], {}).setdefault(i[&#39;COUNTRY&#39;], 0)
    out[i[&#39;BAND&#39;]][i[&#39;COUNTRY&#39;]] += 1

from pprint import pprint
pprint(out)

Benchmark:

from timeit import timeit

from itertools import groupby
from collections import Counter

def sol_orig():
    worked_dxcc_by_band = {z[&quot;BAND&quot;]: {x[&quot;COUNTRY&quot;] : len([y[&quot;COUNTRY&quot;] for y in data if y[&quot;COUNTRY&quot;] == x[&quot;COUNTRY&quot;] and y[&quot;BAND&quot;] == z[&quot;BAND&quot;]]) for x in data if x[&quot;BAND&quot;] == z[&quot;BAND&quot;]} for z in data}
    return worked_dxcc_by_band

def solution():
    out = {band: dict(Counter(v[&#39;COUNTRY&#39;] for v in g)) for band, g in groupby(sorted(data, key=lambda k: k[&#39;BAND&#39;]), lambda k: k[&#39;BAND&#39;])}
    return out

def solution_2():
    out = {}
    for i in data:
        out.setdefault(i[&#39;BAND&#39;], {}).setdefault(i[&#39;COUNTRY&#39;], 0)
        out[i[&#39;BAND&#39;]][i[&#39;COUNTRY&#39;]] += 1
    return out

t1 = timeit(lambda: solution(), number=10000)
t2 = timeit(lambda: solution_2(), number=10000)
t3 = timeit(lambda: sol_orig(), number=10000)

print(t1)
print(t2)
print(t3)

Prints:

0.18113317096140236
0.08159565401729196
3.5367472909856588

通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。

如何改进嵌套理解的性能？

问题

答案1

设置Python中Excel文件的所有工作表的标题颜色和边框。

如何避免在pytest中出现”fixtures not found”错误？

TypeError导入hdbscan时出现问题。

Python不将str转换为int。

What's the correct way to type hint an empty list as a literal in python?

如何在Highcharts Gantt中更改本地化的星期名称

如何在同一个流中使用多个过滤器和映射函数？

如何使用Map/Set来将代码优化到O(n)？

.NET MAUI Android在GitHub Actions上构建失败，错误代码为1。

如何在Playwright视觉比较中屏蔽多个定位器？

在C++中，可以使用可变模板参数来检索类型的内部类型。

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found

Creating and opening a URL to log in to Website via Basic Auth with Robot Framework/Selenium (Python)

AG Grid 在上下文菜单中以大文本形式打开

发表评论