fix: add Jest 30 support and fix time limit in loop-runner

[mohammedahmed18]

Summary

• Add Jest 30 compatibility to the custom loop-runner by detecting Jest version and using the appropriate API (TestRunner class for Jest 30, runTest function for Jest 29)
• Resolve jest-runner from the project's node_modules instead of codeflash's bundled version to ensure version compatibility
• Fix time limit enforcement by using local time tracking instead of trying to share state with capture.js (Jest runs tests in worker processes, so state isn't shared between runner and tests)
• Integrate stability-based early stopping into capturePerf by tracking runtimes per invocation
• Use plain object instead of Set for stableInvocations to survive Jest module resets

Test plan

• Verified Jest 30 project (express) benchmarking now works
• Verified time limit properly stops benchmark loops (tested with 2s and 5s limits)
• Verified timing markers are correctly emitted and collected

🤖 Generated with Claude Code

[claude]

✅ Fixed in latest commit - timing matching code has been removed/refactored

[claude]

Bug: Plain object won't survive Jest module resets

The comment mentions using "plain object instead of Set for stableInvocations to survive Jest module resets", but plain objects ({}) don't survive module resets either - they're reset along with the module.

The correct approach (which you're already using elsewhere) is to store this on process[PERF_STATE_KEY] which does persist across resets. However, this line is inside the initialization block that creates the shared state, so it should work correctly. The comment is misleading though - it's not the "plain object vs Set" that matters, it's storing on process that enables persistence.

[claude]

Question: Verify async iteration count

When an async function is detected on the first iteration (batchIndex=0), this returns immediately and delegates to _capturePerfAsync. That function will:

1. Await the first promise (already started)
2. Loop from startBatchIndex + 1 (which is 1) to batchSize

This should give us batchSize total iterations (1 already done + remaining iterations), which is correct.

However, if the async detection happens on a later iteration (e.g., batchIndex=5), we'd be starting fresh from iteration 6, potentially losing iterations 0-4. Is this scenario possible, or is async/sync status always consistent for a given function?

[claude]

PR Review Summary

Prek Checks ✅

All linting and formatting checks passed. No issues found.

Mypy Checks ⚠️

Mypy reported 160 errors across the changed files, but these are pre-existing errors that also exist on the main branch. No new type errors were introduced by this PR.

The PR includes fixes for several type issues:

• Fixed variable redefinition in PrComment.py (loop variable renamed)
• Fixed return type annotations for async throughput values
• Used proper method references instead of lambda wrappers

Code Review 📝

This is a significant PR that adds Jest 30 support and fixes multiple JavaScript testing infrastructure issues:

Key Changes:

1. Jest 30 Compatibility - Added support for Jest 30's TestRunner class architecture
2. Loop Runner Infrastructure - Implemented custom Jest runner for performance benchmarking (packages/codeflash/runtime/loop-runner.js)
3. Async Function Benchmarking - Fixed async function looping with proper iteration handling
4. Monorepo Support - Added hoisted dependency detection for workspace-based projects
5. Import Path Resolution - Fixed relative import paths for JavaScript/TypeScript in temp directories
6. Time Limit Enforcement - Fixed time limit tracking using local state instead of shared state

Existing Review Comments:

• ✅ Comment #2763041276: Fixed - timing matching logic was removed
• ℹ️ Comment #2763042198: Observation about plain objects in capture.js (comment is misleading but implementation is correct)
• ℹ️ Comment #2763043788: Question about async iteration count (implementation appears correct)

No Critical Issues Found - The PR includes extensive refactoring and new features, but no security vulnerabilities or logic errors were identified in this review.

Test Coverage 📊

Coverage analysis for changed Python files (PR branch):

File	Statements	Missed	Coverage
codeflash/github/PrComment.py	34	10	70.59%
codeflash/languages/javascript/parse.py	191	90	52.88%
codeflash/optimization/function_optimizer.py	1156	944	18.34%
codeflash/verification/verifier.py	66	41	37.88%
TOTAL	1447	1085	25.02%

Analysis:

• Most changed files have low to moderate test coverage
• function_optimizer.py has particularly low coverage (18.34%), but this is a large file with complex logic
• The new JavaScript infrastructure code (loop-runner.js, capture.js) cannot be measured by Python coverage tools
• Note: Main branch coverage comparison could not be completed due to timeout

Recommendation: While coverage is low for some files, these appear to be pre-existing conditions. The PR adds significant new functionality that may require additional integration tests, but this can be addressed in follow-up work.

---

Last updated: 2026-02-04 14:08 UTC

[codeflash-ai]

⚡️ Codeflash found optimizations for this PR

📄 329% (3.29x) speedup for PrComment.to_json in codeflash/github/PrComment.py

⏱️ Runtime : 1.61 milliseconds → 374 microseconds (best of 31 runs)

A dependent PR with the suggested changes has been created. Please review:

• ⚡️ Speed up method PrComment.to_json by 329% in PR #1318 (fix/js-jest30-loop-runner) #1383

If you approve, it will be merged into this PR (branch fix/js-jest30-loop-runner).

[codeflash-ai]

⚡️ Codeflash found optimizations for this PR

📄 22% (0.22x) speedup for humanize_runtime in codeflash/code_utils/time_utils.py

⏱️ Runtime : 324 microseconds → 266 microseconds (best of 250 runs)

A dependent PR with the suggested changes has been created. Please review:

• ⚡️ Speed up function humanize_runtime by 22% in PR #1318 (fix/js-jest30-loop-runner) #1388

If you approve, it will be merged into this PR (branch fix/js-jest30-loop-runner).

[codeflash-ai]

⚡️Codeflash found 67% (0.67x) speedup for TestType.to_name in codeflash/models/test_type.py

⏱️ Runtime : 290 microseconds → 173 microseconds (best of 250 runs)

📝 Explanation and details

The optimized code achieves a 67% runtime speedup (from 290μs to 173μs) by implementing lazy attribute caching to eliminate repeated dictionary lookups.

Key Optimization

What changed: The original code performed a dictionary lookup (_TO_NAME_MAP.get(self, "")) on every call to to_name(). The optimized version caches the result in self._display_name after the first lookup, so subsequent calls simply return the cached attribute.

Why it's faster:

• Dictionary lookups have O(1) average complexity but still involve hashing and collision resolution overhead
• Attribute access via self._display_name is faster than dictionary lookup because it's a direct attribute retrieval
• The line profiler shows the dictionary lookup took ~927ns per call (original), while cached attribute access takes only ~313ns per call (optimized)
• The try/except overhead is negligible (~232ns) and only occurs once per enum instance

Performance Impact by Test Pattern

The optimization shows different speedup patterns based on usage:

1. First call penalty: Initial calls are slightly slower (~350-370ns vs ~750-800ns) due to the try/except and cache setup, but this is a one-time cost per enum instance

2. Repeated calls benefit most: Subsequent calls show the biggest gains:

   • 2nd call: 52-120% faster (320ns → 200-210ns)
   • 3rd+ calls: 63-94% faster (260-330ns → 150-180ns)
   • Batch operations with 1000 calls: 63.5% faster overall

3. Idempotent workloads: The test_to_name_idempotent_on_repeated_calls test shows progressive speedup as the cache eliminates repeated lookups

4. Large-scale operations: Tests iterating over all enum members multiple times see 72-93% speedups, making this optimization particularly valuable when to_name() is called frequently in loops or batch processing scenarios

Real-World Context

Given that enum members are typically long-lived singleton objects, this caching strategy is ideal for workloads where:

• Display names are needed repeatedly for UI rendering or logging
• Enum values are processed in batches or iterations
• The same enum instances are used throughout application lifetime

The optimization maintains correctness (all 20+ test cases pass) while delivering substantial runtime improvements for repeated access patterns.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	🔘 None Found
🌀 Generated Regression Tests	✅ 1104 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	✅ 1 Passed
📊 Tests Coverage	100.0%

🌀 Click to see Generated Regression Tests

import itertools  # used to build large-scale test sequences

# imports
import pytest  # used for our unit tests
from codeflash.models.test_type import TestType

@pytest.mark.parametrize(
    "member, expected",
    [
        # Check that each mapped TestType returns the exact expected display name with emoji + text.
        (TestType.EXISTING_UNIT_TEST, "⚙️ Existing Unit Tests"),
        (TestType.INSPIRED_REGRESSION, "🎨 Inspired Regression Tests"),
        (TestType.GENERATED_REGRESSION, "🌀 Generated Regression Tests"),
        (TestType.REPLAY_TEST, "⏪ Replay Tests"),
        (TestType.CONCOLIC_COVERAGE_TEST, "🔎 Concolic Coverage Tests"),
    ],
)
def test_to_name_returns_expected_for_mapped_values(member, expected):
    # For mapped enum members, to_name should return the exact mapped string.
    codeflash_output = member.to_name(); result = codeflash_output # 3.87μs -> 1.95μs (97.9% faster)

def test_to_name_returns_empty_string_for_unmapped_member():
    # The enum has one member not present in the mapping: INIT_STATE_TEST.
    member = TestType.INIT_STATE_TEST
    # Call the method under test; it must not raise and must return an empty string.
    codeflash_output = member.to_name(); result = codeflash_output # 751ns -> 341ns (120% faster)

def test_to_name_idempotent_on_repeated_calls():
    # Calling to_name multiple times on the same member must yield the same result every time.
    member = TestType.GENERATED_REGRESSION
    codeflash_output = member.to_name(); first = codeflash_output # 781ns -> 331ns (136% faster)
    codeflash_output = member.to_name(); second = codeflash_output # 320ns -> 210ns (52.4% faster)
    codeflash_output = member.to_name(); third = codeflash_output # 261ns -> 160ns (63.1% faster)

def test_all_members_produce_strings_and_mapped_names_nonempty():
    # Iterating over all enum members, we expect to always get a string back.
    # For those members present in the mapping, the string must be non-empty.
    mapped_members = {
        TestType.EXISTING_UNIT_TEST,
        TestType.INSPIRED_REGRESSION,
        TestType.GENERATED_REGRESSION,
        TestType.REPLAY_TEST,
        TestType.CONCOLIC_COVERAGE_TEST,
    }
    for member in TestType:
        codeflash_output = member.to_name(); value = codeflash_output # 2.19μs -> 1.22μs (79.5% faster)
        # If this member is one of the known mapped members, the return must not be empty.
        if member in mapped_members:
            pass
        else:
            pass

def test_mapped_names_are_unique_among_mapped_members():
    # Ensure that all non-empty names are unique to avoid collisions.
    seen = set()
    for member in TestType:
        codeflash_output = member.to_name(); name = codeflash_output # 2.38μs -> 1.25μs (90.4% faster)
        if name:  # only consider non-empty names
            seen.add(name)

def test_to_name_does_not_raise_for_unmapped_member_and_is_strictly_empty():
    # Defensive check: ensure no exception and exact empty string for members not in the mapping.
    member = TestType.INIT_STATE_TEST
    # Use pytest.raises to assert no exception is raised during normal call (redundant but explicit).
    # Here, we just call and assert afterwards - Python would surface any exception as test failure.
    codeflash_output = member.to_name(); result = codeflash_output # 732ns -> 331ns (121% faster)

def test_large_scale_repeated_calls_over_many_members():
    # Build a large-ish sequence (under 1000 elements as requested) by cycling through all enum members.
    all_members = list(TestType)
    # Create a repeated sequence of length 500 (well under the 1000-step guidance).
    repeated = (all_members * ((500 // len(all_members)) + 1))[:500]

    # Call to_name for every element and collect results.
    results = [m.to_name() for m in repeated]

    # 2) Count of empty strings in results should equal number of times the unmapped member appears.
    unmapped_count_expected = repeated.count(TestType.INIT_STATE_TEST)
    unmapped_count_actual = sum(1 for r in results if r == "")

    # 3) All non-empty results must be among the known mapped strings (verifying no unexpected values).
    known_non_empty = {
        "⚙️ Existing Unit Tests",
        "🎨 Inspired Regression Tests",
        "🌀 Generated Regression Tests",
        "⏪ Replay Tests",
        "🔎 Concolic Coverage Tests",
    }
    for r in results:
        if r:
            pass

def test_special_characters_and_keywords_present_in_mapped_names():
    # Ensure specific keywords and emojis appear in their mapped names to catch accidental truncation or replacement.
    codeflash_output = TestType.EXISTING_UNIT_TEST.to_name() # 761ns -> 361ns (111% faster)
    codeflash_output = TestType.INSPIRED_REGRESSION.to_name() # 391ns -> 230ns (70.0% faster)
    codeflash_output = TestType.GENERATED_REGRESSION.to_name() # 300ns -> 170ns (76.5% faster)
    codeflash_output = TestType.REPLAY_TEST.to_name() # 330ns -> 180ns (83.3% faster)
    codeflash_output = TestType.CONCOLIC_COVERAGE_TEST.to_name() # 300ns -> 161ns (86.3% faster)

    # Emoji characters must be preserved. Check presence of at least one expected emoji per mapped member.
    codeflash_output = TestType.EXISTING_UNIT_TEST.to_name() # 280ns -> 150ns (86.7% faster)
    codeflash_output = TestType.INSPIRED_REGRESSION.to_name() # 270ns -> 141ns (91.5% faster)
    codeflash_output = TestType.GENERATED_REGRESSION.to_name() # 271ns -> 150ns (80.7% faster)
    codeflash_output = TestType.REPLAY_TEST.to_name() # 251ns -> 151ns (66.2% faster)
    codeflash_output = TestType.CONCOLIC_COVERAGE_TEST.to_name() # 270ns -> 150ns (80.0% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

import pytest
from codeflash.models.test_type import TestType

def test_existing_unit_test_to_name():
    """Test that EXISTING_UNIT_TEST enum value converts to the correct name."""
    codeflash_output = TestType.EXISTING_UNIT_TEST.to_name() # 761ns -> 350ns (117% faster)

def test_inspired_regression_to_name():
    """Test that INSPIRED_REGRESSION enum value converts to the correct name."""
    codeflash_output = TestType.INSPIRED_REGRESSION.to_name() # 781ns -> 361ns (116% faster)

def test_generated_regression_to_name():
    """Test that GENERATED_REGRESSION enum value converts to the correct name."""
    codeflash_output = TestType.GENERATED_REGRESSION.to_name() # 771ns -> 351ns (120% faster)

def test_replay_test_to_name():
    """Test that REPLAY_TEST enum value converts to the correct name."""
    codeflash_output = TestType.REPLAY_TEST.to_name() # 792ns -> 351ns (126% faster)

def test_concolic_coverage_test_to_name():
    """Test that CONCOLIC_COVERAGE_TEST enum value converts to the correct name."""
    codeflash_output = TestType.CONCOLIC_COVERAGE_TEST.to_name() # 772ns -> 340ns (127% faster)

def test_init_state_test_to_name():
    """Test that INIT_STATE_TEST enum value returns empty string (not in map)."""
    codeflash_output = TestType.INIT_STATE_TEST.to_name() # 771ns -> 311ns (148% faster)

def test_all_enum_members_have_to_name_method():
    """Test that all TestType enum members have the to_name method callable."""
    for test_type in TestType:
        # Verify it returns a string
        codeflash_output = test_type.to_name(); result = codeflash_output # 2.43μs -> 1.26μs (92.6% faster)

def test_to_name_returns_string_type():
    """Test that to_name always returns a string, even for unmapped values."""
    for test_type in TestType:
        codeflash_output = test_type.to_name(); result = codeflash_output # 2.43μs -> 1.28μs (89.8% faster)

def test_unmapped_enum_returns_empty_string():
    """Test that unmapped enum values return empty string rather than None or error."""
    # INIT_STATE_TEST is defined in the enum but not in _TO_NAME_MAP
    codeflash_output = TestType.INIT_STATE_TEST.to_name(); result = codeflash_output # 732ns -> 331ns (121% faster)

def test_to_name_with_emoji_preservation():
    """Test that emoji characters in names are preserved correctly."""
    # Test each mapped value contains its expected emoji
    codeflash_output = TestType.EXISTING_UNIT_TEST.to_name() # 772ns -> 370ns (109% faster)
    codeflash_output = TestType.INSPIRED_REGRESSION.to_name() # 401ns -> 220ns (82.3% faster)
    codeflash_output = TestType.GENERATED_REGRESSION.to_name() # 300ns -> 170ns (76.5% faster)
    codeflash_output = TestType.REPLAY_TEST.to_name() # 330ns -> 170ns (94.1% faster)
    codeflash_output = TestType.CONCOLIC_COVERAGE_TEST.to_name() # 320ns -> 170ns (88.2% faster)

def test_to_name_consistency_multiple_calls():
    """Test that calling to_name multiple times returns consistent results."""
    test_type = TestType.EXISTING_UNIT_TEST
    codeflash_output = test_type.to_name(); result1 = codeflash_output # 712ns -> 351ns (103% faster)
    codeflash_output = test_type.to_name(); result2 = codeflash_output # 360ns -> 200ns (80.0% faster)
    codeflash_output = test_type.to_name(); result3 = codeflash_output # 260ns -> 150ns (73.3% faster)

def test_to_name_no_side_effects():
    """Test that calling to_name does not modify the enum or its values."""
    original_enum = TestType.EXISTING_UNIT_TEST
    expected_name = "⚙️ Existing Unit Tests"
    
    # Call to_name multiple times
    for _ in range(10):
        codeflash_output = original_enum.to_name(); result = codeflash_output # 3.03μs -> 1.76μs (72.2% faster)

def test_to_name_case_sensitive():
    """Test that the returned names have correct case sensitivity."""
    # Verify that names match exactly (case-sensitive)
    codeflash_output = TestType.EXISTING_UNIT_TEST.to_name() # 752ns -> 350ns (115% faster)
    codeflash_output = TestType.EXISTING_UNIT_TEST.to_name() # 341ns -> 191ns (78.5% faster)
    codeflash_output = TestType.EXISTING_UNIT_TEST.to_name() # 260ns -> 160ns (62.5% faster)

def test_to_name_exact_string_match():
    """Test exact string matching for all mapped values."""
    expected_mappings = {
        TestType.EXISTING_UNIT_TEST: "⚙️ Existing Unit Tests",
        TestType.INSPIRED_REGRESSION: "🎨 Inspired Regression Tests",
        TestType.GENERATED_REGRESSION: "🌀 Generated Regression Tests",
        TestType.REPLAY_TEST: "⏪ Replay Tests",
        TestType.CONCOLIC_COVERAGE_TEST: "🔎 Concolic Coverage Tests",
    }
    
    for enum_member, expected_name in expected_mappings.items():
        codeflash_output = enum_member.to_name() # 1.85μs -> 1.04μs (77.5% faster)

def test_all_enum_members_to_name_in_loop():
    """Test to_name method for all enum members in a loop to check performance."""
    # Create a list of results for all enum members
    results = []
    for test_type in TestType:
        codeflash_output = test_type.to_name(); result = codeflash_output # 2.40μs -> 1.29μs (86.1% faster)
        results.append((test_type, result))
    
    # Verify all results are strings
    for enum_member, result in results:
        pass

def test_repeated_calls_performance():
    """Test that repeated calls to to_name maintain performance (no degradation)."""
    test_type = TestType.EXISTING_UNIT_TEST
    
    # Call to_name many times and collect results
    results = []
    for i in range(1000):
        codeflash_output = test_type.to_name(); result = codeflash_output # 245μs -> 149μs (63.5% faster)
        results.append(result)

def test_all_enum_members_in_large_batch():
    """Test all enum members processed in a batch to ensure consistency."""
    # Process each enum member 100 times
    batch_results = {}
    for test_type in TestType:
        batch_results[test_type] = [test_type.to_name() for _ in range(100)]
    
    # Verify consistency within each batch
    for test_type, results in batch_results.items():
        unique_results = set(results)

def test_enum_to_name_mapping_completeness():
    """Test that all enum members either have a mapping or return empty string."""
    mapped_count = 0
    unmapped_count = 0
    
    for test_type in TestType:
        codeflash_output = test_type.to_name(); result = codeflash_output # 2.48μs -> 1.35μs (83.7% faster)
        if result == "":
            unmapped_count += 1
        else:
            mapped_count += 1

def test_to_name_return_type_homogeneity():
    """Test that all enum members return the same type from to_name."""
    types_returned = set()
    for test_type in TestType:
        codeflash_output = test_type.to_name(); result = codeflash_output # 2.42μs -> 1.27μs (90.6% faster)
        types_returned.add(type(result))

def test_string_length_variation():
    """Test that returned strings have expected length variations."""
    name_lengths = {}
    for test_type in TestType:
        codeflash_output = test_type.to_name(); result = codeflash_output # 2.34μs -> 1.26μs (86.0% faster)
        name_lengths[test_type] = len(result)
    
    # All mapped values should have non-zero length
    for test_type in TestType:
        if test_type != TestType.INIT_STATE_TEST:
            pass

def test_enum_iteration_order_independence():
    """Test that the order of iteration doesn't affect results."""
    # Get all enum members as list
    all_members = list(TestType)
    
    # Create results dictionary
    results1 = {member: member.to_name() for member in all_members}
    
    # Reverse the list and create results again
    reversed_members = list(reversed(all_members))
    results2 = {member: member.to_name() for member in reversed_members}
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

from codeflash.models.test_type import TestType

def test_TestType_to_name():
    TestType.to_name(TestType.REPLAY_TEST)

🔎 Click to see Concolic Coverage Tests

To test or edit this optimization locally git merge codeflash/optimize-pr1318-2026-02-04T19.53.39

Suggested change

	return _TO_NAME_MAP.get(self, "")
	try:
	return self._display_name
	except AttributeError:
	self._display_name = _TO_NAME_MAP.get(self, "")
	return self._display_name