英文:

Selenium + Cucumber, parallel execution

问题

I think I have a silly question that I'm overthinking but looking for feedback. My team is using Selenium Java with Cucumber JUnit. I really feel like we aren't using it to its full capacity as far as execution goes, everything feels very outdated. We have our Scenarios in a variety of feature files based on functionality. A couple Scenarios for each feature are Tagged with @Sanity for quick Sanity testing, and full Regression (hundreds of test cases) are tagged with either @Regression1 or @Regression2. We have a couple other tags we use but those are the primary ones.

When we run Regression, we have two Runner classes, TestRunner1.java with the Cucumber options tag @Regression1 and TestRunner2.java with tag @Regression2. Then execution is done with the Maven Surefire plugin, set to classes with 2 threads. So hundreds of test cases only executing two at a time, kicked off in Jenkins with a simple "mvn test" command on a VM. I'm newer to the team so I'm not trying to step on toes or rock the boat too hard but this seems really bad.

What is a better way to parallelize the execution in Jenkins? Can obviously increase the number of TestRunner classes and split by class still, or switch to the method parameter of Surefire. But I've found our VMs really can't support more than like 4 threads at a time without crashing/erroring out. Otherwise, I was thinking like breaking down our Regression tags into @Regression1 thru @Regression10, and in a Jenkins grid execution run 10 different Maven commands "mvn test -Dcucumber.filter.tags=@Regression1", "mvn test -Dcucumber.filter.tags=@Regression2", "mvn test -Dcucumber.filter.tags=@Regression3", etc., with each getting assigned its own VM.

Thank you for any feedback before I dig more into this and present it to my team.

英文:

I think I have a silly question that I'm over thinking but looking for feedback. My team is using Selenium Java with Cucumber JUnit. I really feel like we aren't using it to its full capacity as far as execution goes, everything feels very outdated. We have our Scenarios in a variety of feature files based on functionality. A couple Scenarios for each feature are Tagged with @ Sanity for quick Sanity testing, and full Regression (hundreds of test cases) are tagged with either @ Regression1 or @ Regression2. We have a couple other tags we use but those are the primary ones.

When we run Regression, we have two Runner classes, TestRunner1.java with the cucumber options tag @ Regression1 and TestRunner2.java with tag @ Regression2. Then execution is done with the maven surefire plugin, is set to <parallel>classes</parallel> with 2 threads. So hundreds of test cases only executing two at time, kicked off in jenkins with a simple mvn test command on a vm. I'm newer to the team so I'm not trying to step on toes or rock the boat too hard but this seems really bad.

What is a better way to parallelize the execution in jenkins? Can obviously increase the number of TestRunner classes and split by class still, or switch to the method parameter of surefure. But I've found our vms really cant support more than like 4 threads at time without crashing/erroring out. Otherwise I was thinking like breaking down our Regression tags into @ Regression 1 thru 10, and in a Jenkins grid execution run 10 different mvn commands "mvn test -Dcucumber.filter.tags=@Regression1", "mvn test -Dcucumber.filter.tags=@Regression2", "mvn test -Dcucumber.filter.tags=@Regression3", etc.. with each getting assigned its own vm.

Thank you for any feedback before I dig more into this and present it to my team

答案1

得分: 0

I'm newer to the team so I'm not trying to step on toes or rock the boat too hard but this seems really bad.

我是团队中的新成员，所以我不想踩到任何人的脚趾或者搅乱局面，但这看起来真的很糟糕。

Cucumber initially did not support parallel execution. So there were quite a few tools available to generate runner classes for individual scenarios. These could then be executed in parallel. These would presumably have been a better solution then tagging all tests by hand.

起初，Cucumber不支持并行执行。因此，有很多工具可用来生成单独场景的运行类，然后可以并行执行它们。这可能本来是一个比手动为所有测试打标签更好的解决方案。

But none of that matters now. Since v4 of Cucumber parallel execution is supported natively.

但现在这些都不重要了。因为Cucumber的v4版本开始原生支持并行执行。

But I've found our vms really cant support more than like 4 threads at time without crashing/erroring out.

但我发现我们的虚拟机一次只能支持不超过4个线程，否则会崩溃或出错。

It would be interesting to investigate why.

探究一下为什么会这样会很有趣。

It could be that your VM can't handle more than 4 web drivers due to cpu/memory constraints. This cool be overcome by using larger instances and/or allocating more resources.

这可能是因为你的虚拟机由于CPU/内存限制而无法处理超过4个Web驱动程序。可以通过使用更大的实例和/或分配更多资源来解决这个问题。

It could also be that your tests expect certain elements to be present within a certain time frame (i.e. after Thread.sleep(10)) rather than explicitly wait for the element to be present.

也可能是因为你的测试期望某些元素在一定的时间范围内出现（例如，在 Thread.sleep(10) 之后），而不是显式等待元素出现。

Otherwise I was thinking like breaking down our Regression tags into @ Regression 1 thru 10, and in a Jenkins grid execution run 10 different mvn commands

否则，我考虑将我们的回归标签拆分成 @ Regression 1 到 10，并在Jenkins网格执行中运行10个不同的mvn命令。

This isn't a bad strategy.

这不是一个坏策略。

Though rather than explicitly tagging tests consider using Cucumber with the JUnit Platform Launcher API.

不过，不要直接为测试添加标签，考虑使用Cucumber与JUnit Platform Launcher API。

With JUnit 5 you get a really nice programmatic API to discover, select and execute tests. You could have one job to discover all tests and write them to file. The next 10 jobs could then each pick up this file and execute every entry where entry_index modulo total_test_runners == test_runner_index.

使用JUnit 5，您可以获得一个非常好的编程API来发现、选择和执行测试。您可以有一个任务来发现所有测试并将它们写入文件。接下来的10个任务可以分别接管这个文件，并执行其中每个满足 entry_index modulo total_test_runners == test_runner_index 的条目。

As a starting point see Cucumber JUnit Platform Engine docs. Best to read it from top to bottom as well as the linker resources.

作为一个起点，可以参考Cucumber JUnit Platform Engine文档。最好从上到下阅读它以及相关的资源链接。

英文:

> I'm newer to the team so I'm not trying to step on toes or rock the boat too hard but this seems really bad.

Cucumber initially did not support parallel execution. So there were quite a few tools available to generate runner classes for individual scenarios. These could then be executed in parallel. These would presumably have been a better solution then tagging all tests by hand.

But none of that matters now. Since v4 of Cucumber parallel execution is supported natively.

> But I've found our vms really cant support more than like 4 threads at time without crashing/erroring out.

It would be interesting to investigate why.

It could be that your VM can't handle more than 4 web drivers due to cpu/memory constraints. This cool be overcome by using larger instances and/ot allocating more resources.

It could also be that your tests expect certain elements to be present within a certain time frame (i.e. after Thread.sleep(10)) rather than explicitly wait for the element to be present.

> Otherwise I was thinking like breaking down our Regression tags into @ Regression 1 thru 10, and in a Jenkins grid execution run 10 different mvn commands

This isn't a bad strategy.

Though rather than explicitly tagging tests consider using Cucumber with the JUnit Platform Launcher API.

With JUnit 5 you get a really nice programmatic API to discover, select and execute tests. You could have one job to discover all tests and write them to file. The next 10 jobs could then each pick up this file and execute every entry where entry_index modulo total_test_runners == test_runner_index.

As a starting point see Cucumber JUnit Platform Engine docs. Best to read it from top to bottom as well as the linker resources.