[Bug] CTest unable to run large amount of tests in parallel

[theuke]

Brief Issue Summary

This issue is effectively a duplicate of #3804 (which has been closed). I am just creating this new, duplicate issue, because I already have a fix lined up for it in #4830 and want to get the fix merged in.

Here is a sample snippet of the offending ctest command whose input regex is too long

[proc] Executing command: /opt/cmake/bin/ctest -j4 -C Debug -T test --output-on-failure -R "^pre_ctest$|^post_ctest$|^HexaConvertorTest\.testConvertStringToHexa$|^HexaConvertorTest\.testConvertStringToHexaToString$|^HexaConvertorTest\.testConvertHexaToByte$|^HexaConvertorTest\.testConvertHexaToString$|^SerializerTest\.testConstructor$|^SerializerTest\.testSimpleSerialize$|^SerializerTest\.testSimpleDeserialize$|^SerializerTest\.testConstructJson$|^SerializerTest\. 
 ...snip... |^ResolverTest\.getSuccess$"
[ctest] CTest finished with return code -1

CMake Tools Diagnostics

None. Refer to #3804 if needed.

Debug Log

None. Refer to #3804 if needed.

Additional Information

No response

[theuke]

Tagging @gcampbell-msft since I believe he was the initial author of the parallel test running and was quite involved with the various issues relating to this subject (namely #3804).

[yanghhhhhhh]

Hi @theuke , thank you for reporting this issue. We tried to check this issue on Visual Studio Code v1.112.0 + CMake Tools v1.22.28. But we can't reproduce this issue. Could you help to take a look at it (Repro Video: repro.zip)? If we missed something, please let us know. In order for us to investigate this further, could you please provide a sample project to us that can reproduce this issue? Thanks.

[theuke]

@yanghhhhhhh Attached are screenshots of the bug in action, a ready-to-go sample code-workspace + CMakeLists.txt, and concrete repro steps. The reason you were unable reproduce is because the issue only occurs when the sum total length of the regex string passed into the ctest command exceeds some OS limit.

In my contrived example, this is 2000 tests with fairly long test names. Although this example is contrived, I run into it on the regular at work where we have many tests and many of which are parameterized (parameterized test names tend to get long).

But, please note that this is not the first ticket for this issue (as mentioned, #3804 and few others talked about it).

Repro Steps

• System Info: Visual Studio Code v1.111.0 + CMake Tools v1.22.28
  • Note that the VS Code version is not going to make a difference. The core issue here is the length of the regex string being passed into the ctest tool via command line.

1. Download test-parallel.zip, unzip it, and in vscode open the workspace it contains.
2. Run just the BigBoyTestSuite unit test suite. Note if you run ALL unit tests by clicking on the higher "level" of the hierarchy, it runs because the regex produced in that case is just "match all" ^..
   1. 
   2. 
3. We observe the ctest command fails with exit code -1 and looks exactly as it does in my initial description, just with a bigger regex.

[proc] Executing command: /cmake-3.20.1-linux-x86_64/bin/ctest -j16 -C Debug -T test --output-on-failure -R ^BigBoyTestSuite\.testWithLongName1_PretendThisWasAparameterizedTestWithAVeryLongName$|.<1999 MORE REGEXES - ONE PER UNIT TEST>
[ctest] CTest finished with return code -1

High Level Analysis

The core problem here is that the REGEX string can can very big since it is constructed by adding an "exact match" entry for every single unit test. Generally this is a safe approach since it handles all edge cases of unit tests being filtered / single selected.

The fix for this is relatively straight forward to understand: minimize the regex in such a way that all of the tests selected by the user are run WITHOUT running any extra tests. Because the cmake tools extension is passed in a list of test names to run AND the list of ALL test names in the project, we should be able to programatically devise such a regex.

As shown in this example, the user intent was to run all unit tests under BigBoyTestSuite well, that can be satisfied with a regex like ^BigBogTestSuite.*

[theuke]

@yanghhhhhhh For reference, the fix I propose in PR #4830, for this same test scenario (the BigBoyTestSuite) results in the below ctest command being issued. As you can see, in this case since we are running all BigBoyTestSuite tests, the regex of ^B is sufficient to run all of those tests WITHOUT running any non-selected tests (i.e. the ATestSuite tests).

The core reason we are able to get away with just ^B instead of say ^BigBoyTestSuite is because the extension is passed in the target test cases names and the list of ALL tests in the project which allows for a solution that greedily constructs the minimal matching regex that does not "hit" any of the forbidden tests ($forbiddenTests = allTests - targetTests$).

[proc] Executing command: /cmake-3.20.1-linux-x86_64/bin/ctest -j16 -C Debug -T test --output-on-failure -R ^B