What is an example of a buggy function that would be hard to find to discover the bug without fuzz testing?

huangapple go评论56阅读模式
英文:

What is an example of a buggy function that would be hard to find to discover the bug without fuzz testing?

问题

I'd like to come up with a motivating example or code challenge for fuzz testing and/or property-based testing.

我想提出一个激励的示例或代码挑战,用于模糊测试和/或基于属性的测试。

What I'm looking for is a concise situation where such testing is maximally critical/necessary.

我正在寻找一个简洁的情况,其中这种测试是最关键/必要的。

For example, ideally it would take enough fuzz runs that a human would be unlikely to discover the bug by manually trying random unit tests or relying on intuition to come up with edge cases.

例如,理想情况下,它应该需要足够多的模糊运行,以至于人类不太可能通过手动尝试随机单元测试或依靠直觉提出边缘情况来发现错误。

Bonus if:

  • in TypeScript (but not a big deal; I can translate)

奖励如果:

  • 使用 TypeScript(但不是什么大问题;我可以翻译)

  • an example from real/historical software

来自真实/历史软件的示例

I tried asking ChatGPT but the bug was too obvious. I also tried a bit of Googling and found this, but it's still quite obvious and probably also reveals itself after a few unit tests. I also considered making some kind of broken lookup table (inspired by Pentium FDIV bug) but I couldn't figure out how to make it so that you can't trivially solve it by just computing the correct lookup table and comparing it.

我尝试向ChatGPT提问,但错误太明显了。我还试着搜索了一下,找到了这个,但它仍然相当明显,可能在几个单元测试后也会显现。我还考虑过制作某种破损的查找表(受到奔腾FDIV错误的启发),但我无法想出如何使它不容易通过计算正确的查找表并进行比较来解决。

英文:

I'd like to come up with a motivating example or code challenge for fuzz testing and/or property-based testing.

What I'm looking for is a concise situation where such testing is maximally critical/necessary.

For example, ideally it would take enough fuzz runs that a human would be unlikely to discover the bug by manually trying random unit tests or relying on intuition to come up with edge cases.

Bonus if:

  • in TypeScript (but not a big deal; I can translate)
  • an example from real/historical software

I tried asking ChatGPT but the bug was too obvious. I also tried a bit of Googling and found this, but it's still quite obvious and probably also reveals itself after a few unit tests. I also considered making some kind of broken lookup table (inspired by Pentium FDIV bug) but I couldn't figure out how to make it so that you can't trivially solve it by just computing the correct lookup table and comparing it.

答案1

得分: 1

以下是已翻译的内容:

在我看来,一个令人信服且仍然易于理解的例子是质因数分解。查看关于这种算法的博客文章

*全面披露:*我是这篇文章的作者,也是所使用的PBT库的主要贡献者。

考虑这个实现(抱歉,它是Java),它是在使用属性进行TDD的几个步骤之后得出的:

public static List<Integer> factorize(int number) {
    List<Integer> factors = new ArrayList<>();
    int candidate = 2;
    while (number >= candidate) {
        while (number % candidate != 0) {
            candidate++;
        }
        factors.add(candidate);
        number /= candidate;
    }
    return factors;
}

从算法角度看,factorize 运行良好。一些缺陷 - 例如,处理大数、潜在的整数溢出 - 仅在通过通用属性对其进行压力测试时才会发现:

@Property
void all_numbers_above_1_can_be_factorized(
        @ForAll @IntRange(min = 2, max = 10000) int number
) {
    List<Integer> factors = Primes.factorize(number);
    Integer product = factors.stream().reduce(1, (a, b) -> a * b);
    Assertions.assertThat(product).isEqualTo(number);
}

如果您将max增加到1_000_000以上,并朝着Integer.MAX_VALUE范围增加,那么该算法将运行很长时间,或者根本无法完成。这些失败的属性导致修改算法以处理整数的全范围因子分解。一个在最大整数值之前运行快速的实现示例如下:

public static List<Integer> factorize(int number) {
    List<Integer> factors = new ArrayList<>();
    int candidate = 2;
    while (number >= candidate) {
        while (number % candidate != 0) {
            if (Math.sqrt(number) < candidate) {
                candidate = number;
            } else {
                candidate++;
            }
        }
        factors.add(candidate);
        number /= candidate;
    }
    return factors;
}

在我了解PBT之前,我倾向于不测试这些事情;现在它自然而然地出现了。

英文:

One in my opinion convincing and still easy-to-understand example is prime factorization. Look at the blog article on property-driven development of such an algorithm.

Full disclosure: I'm the author of the article and main committer of the used PBT library.

Consider the implementation (sorry it's Java) as it resulted after a few steps of TDD using properties:

public static List&lt;Integer&gt; factorize(int number) {
    List&lt;Integer&gt; factors = new ArrayList&lt;&gt;();
    int candidate = 2;
    while (number &gt;= candidate) {
        while (number % candidate != 0) {
            candidate++;
        }
        factors.add(candidate);
        number /= candidate;
    }
    return factors;
}

From an algorithmic standpoint factorize works alright. Some of the failings - e.g. handling large numbers, potential integer overflows - are only discovered when you set it under stress with a generic property:

@Property
void all_numbers_above_1_can_be_factorized(
        @ForAll @IntRange(min = 2, max = 10000) int number
) {
    List&lt;Integer&gt; factors = Primes.factorize(number);
    Integer product = factors.stream().reduce(1, (a, b) -&gt; a * b);
    Assertions.assertThat(product).isEqualTo(number);
}

If you start increasing max beyond 1_000_000 and towards Integer.MAX_VALUE range the algorithm will run for a very long time or not finish at all. Those failing properties lead to changes of the algorithm to handle factorization for the full range of integers. An implementation running fast up to the maximum integer value is e.g.:

public static List&lt;Integer&gt; factorize(int number) {
    List&lt;Integer&gt; factors = new ArrayList&lt;&gt;();
    int candidate = 2;
    while (number &gt;= candidate) {
        while (number % candidate != 0) {
            if (Math.sqrt(number) &lt; candidate) {
                candidate = number;
            } else {
                candidate++;
            }
        }
        factors.add(candidate);
        number /= candidate;
    }
    return factors;
}

Before I learned about PBT, I tended to not test such things; now it comes naturally.

huangapple
  • 本文由 发表于 2023年6月29日 17:06:04
  • 转载请务必保留本文链接:https://go.coder-hub.com/76579617.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定