Discover tests using a custom test harness

[MichaReiser]

First, thank you for your hard work on rust-analyzer.

We consider migrating from dirtest to datatest-stable. dir-test exports a macro that generates a #[test] function for every file in a given directory. This is great because it plays really nicely with r-a: r-a detects the tests and we can run them by name. However, it has the downside that it requires a build.rs file that invalidates the macro whenever one of our spec files change, which leads to very slow iteration cycles.

That's why we consider migrating to datatest-stable. It uses a custom test harness that mimics cargo test. This is great because it removes the need for the build.rs file and the need to recompile the tests after making changes to the spec files. However, we lose the r-a integration because the tests are now no longer expanded in a macro.

Supporting this use case is tricky because r-a can't make any assumptions about a custom test harness. One possible solution would be to support fetching the known tests using cargo test -- --list (ideally with a custom filter). Obviously, it would fail to automatically detect new tests but that's already the case with our existing dirtest setup where it's necessary to manually clear the macro-cache.

[MichaReiser]

I just found #16662 which would suggest that this should "just work". I'll try to debug this a little further

[MichaReiser]

Okay. The issue is that libtest-mimic doesn't support the JSON output and the solution is to use https://docs.rs/libtest2-mimic/latest/libtest2_mimic/

[MichaReiser]

Reopening because my analysis above was incorrect. r-a identifies all functions with a #[test] attribute as tests, as can be seen here

rust-analyzer/crates/ide/src/test_explorer.rs

Lines 90 to 109 in eda27bb

	for def in module.declarations(db) {
	let ModuleDef::Function(f) = def else {
	continue;
	};
	if !f.is_test(db) {
	continue;
	}
	let nav = f.try_to_nav(&sema).map(|r| r.call_site);
	let fn_name = f.name(db).as_str().to_owned();
	r.push(TestItem {
	id: format!("{prefix_id}::{fn_name}"),
	kind: TestItemKind::Function,
	label: fn_name,
	parent: Some(prefix_id.clone()),
	file: nav.as_ref().map(|n| n.file_id),
	text_range: nav.as_ref().map(|n| n.focus_or_full_range()),
	runnable: runnable_fn(&sema, f),
	});
	}
	r

Reading this code, I wonder if r-a could call RUSTC_BOOTSTRAP=1 cargo test --no-fail-fast -- mdtest -Z unstable-options --list --format=json if it finds a test-target without a single #[test] function. Or do so if the test target uses harness=false

Note that libtest-mimic currently doesn't support --list --output=json but I've a PR adding support for it (see here)

[ChayimFriedman2]

Where will be the test shown? r-a currently only offers a codelen to run them above the test function/module.

[MichaReiser]

I would expect them to show in the test explorer view