initial support for automated testing with NTFC

[szafonimateusz-mi]

Summary

First step to solve #17717

This PR adds support for NTFC in CI and migrates sim/citest to use NTFC instead of tools/ci/testrun tool.
The rest of citest configs uses the old testing method.

635cb70 must be also ported to nuttx-apps.
EDIT: PR for nuttx-apps is here apache/nuttx-apps#3291 [DONE]

Impact

NTFC performs more tests so sim-01 will take longer but will provide better code coverage.
EDIT: With this change Linux sim-01 took 1h 13m 56s, before this change it took 58m 25s.

Testing

Tested in a private repository, let's see if it works with upstream.

Here is an example run test for sim/citest on my fork with all tests passed:
https://github.com/szafonimateusz-mi/nuttx/actions/runs/20580444513/job/59106747416#step:11:154

I will update this section when upstream CI is finished.

EDIT: successful run is here:
https://github.com/apache/nuttx/actions/runs/20582851657/job/59113926720?pr=17728#step:11:247

Test start:

Test end:

Linux sim-01 took 1h 13m 56s, before this change it took 58m 25s, so tests for this target take 15 minutes more (not bad):

Summary from pytest before this PR:

================ 1058 passed, 10 skipped in 2735.05s (0:45:35) =================

Summary from pytest for this PR:

================ 1224 passed, 548 skipped in 3625.97s (1:00:25) ================

[szafonimateusz-mi]

I have updated the description according to the CI results. Linux sim-01 took 1h 13m 56s, before this change it took 58m 25s, so tests take 15 minutes more. For comparison, the entire CI workflow took 3 hours.

Summary from pytest before this PR:

================ 1058 passed, 10 skipped in 2735.05s (0:45:35) =================

Summary from pytest for this PR:

================ 1224 passed, 548 skipped in 3625.97s (1:00:25) ================

[acassis]

@szafonimateusz-mi please include basic Documentation about this new test framework

[szafonimateusz-mi]

What do you want documented? The NTFC documentation is in the NTFC repository https://github.com/szafonimateusz-mi/nuttx-ntfc/

[acassis]

What do you want documented? The NTFC documentation is in the NTFC repository https://github.com/szafonimateusz-mi/nuttx-ntfc/

Sorry, I missed that Running CI Tests with tools/ci/testrun section. So I think it is fine

[szafonimateusz-mi]

@acassis No need to sorry, I updated this file after your comment :P I've also created a new top section in the docs: "Testing." Ultimately, CI's operation can be better described there.

[hartmannathan]

Hi all, as I said in my review comment, I hope the plan is to eventually move the framework and official test cases under the apache/nuttx repositories.

[simbit18]

Question:

Could this repository Apache NuttX Testing

be reused to contain both repositories NuttX Test Framework and NTFC test cases ?

[raiden00pl]

@simbit18 we need 2 separate repositories. NTFC is not dependent on NTFC test cases in any way, the user can use their own test cases. In addition, NTFC is organized as a Python package and has different requirements regarding maintainence and CI.

@hartmannathan yes, it will be moved to Apache, we just need one more repository for nuttx-ntfc. Do we need INFRA to create a new repository or can we do it ourselves?

[hartmannathan]

@simbit18 we need 2 separate repositories. NTFC is not dependent on NTFC test cases in any way, the user can use their own test cases. In addition, NTFC is organized as a Python package and has different requirements regarding maintainence and CI.

@hartmannathan yes, it will be moved to Apache, we just need one more repository for nuttx-ntfc. Do we need INFRA to create a new repository or can we do it ourselves?

@raiden00pl We can create repositories using INFRA's Self-Service tools, but I think that will create empty repositories. I think what we need in this case is to clone the repos; correct me if this isn't the case. I don't see a way to create a repo as a clone. We can ask INFRA to do it by opening a Jira ticket and telling them the URLs of the repos we want cloned... Thoughts?

[raiden00pl]

@hartmannathan We can clone the repo with INFRA, but we can also do it differently: create an empty repository and add all changes in one pull request. Both options are OK for me, the question is which one is faster :)

[raiden00pl]

Here's a PR for nuttx-testing adding test cases to repo: apache/nuttx-testing#102

[hartmannathan]

@hartmannathan We can clone the repo with INFRA, but we can also do it differently: create an empty repository and add all changes in one pull request. Both options are OK for me, the question is which one is faster :)

@raiden00pl I don't know, but let's try the way you suggested. Would you like to do the honors and setup the new repos? (If you'd like me to do it, I'll need to know what repository names you'd like...)