Skip to content

⚡️ Speed up function _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir by 11%#125

Open
codeflash-ai[bot] wants to merge 1 commit intomainfrom
codeflash/optimize-_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir-mlcm8ck0
Open

⚡️ Speed up function _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir by 11%#125
codeflash-ai[bot] wants to merge 1 commit intomainfrom
codeflash/optimize-_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir-mlcm8ck0

Conversation

@codeflash-ai
Copy link

@codeflash-ai codeflash-ai bot commented Feb 7, 2026

📄 11% (0.11x) speedup for _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir in src/datasets/data_files.py

⏱️ Runtime : 9.22 milliseconds 8.30 milliseconds (best of 6 runs)

📝 Explanation and details

This optimization achieves an 11% runtime improvement by eliminating unnecessary memory allocations in a path-filtering function. The key changes are:

What Changed:

  1. Replaced list comprehensions with counting loops: Instead of building two intermediate lists (hidden_directories_in_path and hidden_directories_in_pattern), the code now uses simple integer counters that increment as hidden parts are found.

  2. Eliminated set allocations: The original code used set(part) == {"."} to check if a part consists only of dots. The optimized version uses part.strip(".") != "" instead, avoiding the overhead of creating a set object for every path component.

Why It's Faster:

  • Reduced memory allocations: In Python, creating lists and sets has significant overhead. By counting directly, we avoid allocating memory for intermediate data structures that are only used to determine their length.
  • Lower Python interpreter overhead: Integer increments are much cheaper than list append operations, which require bounds checking and potential memory reallocation.
  • Faster character check: String operations like strip() are implemented in optimized C code and avoid the overhead of set creation, hashing, and comparison.

Performance Context:
Based on the function_references, this function is called from resolve_pattern() in a list comprehension that filters file paths during glob operations. This means it's invoked once per matched file when resolving data file patterns. In workflows that scan directories with many files (especially those with hidden files/directories like .git/ or .venv/), this optimization compounds:

  • For 1,000 files checked: saves ~0.92ms
  • The test results show consistent 10-25% speedups across various path structures, with the best gains (27-41%) on deeply nested hidden directories

Test Case Performance:
The optimization excels with:

  • Paths with multiple hidden parts (27-41% faster for 50+ nested hidden directories)
  • Alternating hidden/regular structures (16-17% faster)
  • Unicode hidden directories (11% faster)
  • All test patterns show improvement, indicating robust performance across diverse real-world scenarios

Correctness verification report:

Test Status
⚙️ Existing Unit Tests 39 Passed
🌀 Generated Regression Tests 1078 Passed
⏪ Replay Tests 🔘 None Found
🔎 Concolic Coverage Tests 🔘 None Found
📊 Tests Coverage 100.0%
⚙️ Click to see Existing Unit Tests
🌀 Click to see Generated Regression Tests
from pathlib import PurePath

# imports
import pytest  # used for our unit tests
from src.datasets.data_files import \
    _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir

def test_no_hidden_parts_returns_false():
    # Basic: when neither the path nor the pattern contain hidden parts, the function should return False
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("a/b.txt", "**") # 17.5μs -> 16.5μs (5.97% faster)
    # Also check a single-level non-hidden file
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("file.txt", "file.txt") # 7.75μs -> 7.12μs (8.83% faster)

def test_hidden_file_at_root_behavior():
    # Basic example from docstring: a hidden file at root should be considered unrequested when
    # the pattern doesn't explicitly mention the leading dot
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**") # 17.6μs -> 15.3μs (15.3% faster)
    # If the pattern explicitly includes a hidden-file segment, the counts match and result is False
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*") # 8.88μs -> 7.66μs (15.9% faster)

def test_hidden_directory_containing_file_variations():
    # Hidden directory containing a non-hidden file, as in the docstring.
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**") # 19.7μs -> 17.2μs (14.3% faster)
    # Pattern that explicitly requests a hidden directory then any file: counts match, expect False
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".*/*") # 10.5μs -> 9.08μs (16.2% faster)
    # Pattern that names the hidden directory explicitly also matches counts -> False
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*") # 8.76μs -> 7.48μs (17.2% faster)

def test_hidden_dir_and_hidden_file_combinations():
    # Hidden directory and hidden file nested - multiple variations from the docstring
    path = ".hidden_dir/.hidden_file.txt"
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, "**") # 20.1μs -> 17.1μs (17.7% faster)
    # Pattern only mentions the hidden directory (one hidden part) but not the hidden file (two hidden parts in path)
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, ".*/.*") # 11.3μs -> 9.24μs (22.0% faster)
    # Pattern mentions a hidden directory but not the hidden file explicitly:
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, ".hidden_dir/*") # 9.45μs -> 7.76μs (21.8% faster)
    # Pattern mentions both hidden directory and hidden filename explicitly -> counts equal -> False
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, ".hidden_dir/.*") # 8.61μs -> 7.44μs (15.8% faster)

def test_parts_composed_only_of_dots_are_ignored():
    # Edge: parts that are only '.' or '..' should not be treated as hidden parts
    # Path contains '..' followed by a hidden directory. The pattern mirrors that structure.
    path = "../.hidden_dir/file.txt"
    pattern_with_hidden = "../.hidden_dir/*"
    # Both path and pattern contain exactly one hidden part ('.hidden_dir'), so function returns False
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern_with_hidden) # 22.3μs -> 19.0μs (17.1% faster)

    # If the pattern does not include the hidden directory explicitly, it should be considered unrequested
    pattern_without_hidden = "../**"
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern_without_hidden) # 10.7μs -> 9.03μs (18.2% faster)

def test_dot_in_middle_of_name_not_hidden_and_only_leading_dot_counts():
    # Edge: a name that contains dots but does not start with a dot should NOT be counted as hidden
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("a/file.name", "**") # 17.3μs -> 16.2μs (6.50% faster)
    # Leading dot in middle of path should be counted
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("a/.hidden_sub/file", "**") # 12.1μs -> 9.55μs (26.3% faster)
    # If pattern explicitly contains that leading-dot directory, counts match -> False
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("a/.hidden_sub/file", "a/.*/*") # 9.66μs -> 8.50μs (13.7% faster)

def test_pattern_with_glob_suffix_and_dot_prefix_counts():
    # Edge: parts in the pattern that start with a dot but include other glob characters should still be counted
    path = ".hidden_dir_sub/a.txt"
    # Pattern with a dot-prefixed part that uses a glob suffix still startswith '.' -> counts included
    pattern = ".hidden_dir*/a.txt"
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern) # 21.6μs -> 17.6μs (22.2% faster)
    # If pattern omits the hidden prefix, it's considered unrequested
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, "**") # 9.20μs -> 8.09μs (13.8% faster)

def test_mismatch_both_ways():
    # Edge: ensure function returns True whenever the number of hidden parts differ in either direction
    # path has two hidden parts, pattern has one -> True
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".a/.b/file", ".a/*") # 21.3μs -> 18.5μs (15.0% faster)
    # path has one hidden part, pattern has two -> True
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".a/file", ".a/.*") # 9.96μs -> 8.75μs (13.7% faster)

def test_large_scale_various_hidden_counts():
    # Large-scale test (within the 1000-elements and loops constraint): generate 500 different paths
    # with 0, 1 or 2 hidden parts in a repeating pattern and verify behavior when the pattern
    # mirrors the hidden count (expect False) and when the pattern has no hidden parts (expect True for >0 hidden)
    total = 500  # well under the 1000-element restriction
    paths = []
    patterns_matching = []
    for i in range(total):
        # Create three classes cyclically: 0 hidden, 1 hidden, 2 hidden
        if i % 3 == 0:
            # 0 hidden parts
            p = f"dir{i}/file{i}.txt"
            pat = "dir*/file*"
        elif i % 3 == 1:
            # 1 hidden part
            p = f".hidden{i}/file{i}.txt"
            pat = f".hidden{i}/file*"
        else:
            # 2 hidden parts
            p = f".hiddenA{i}/.hiddenB{i}/file{i}.txt"
            pat = f".hiddenA{i}/.hiddenB{i}/file*"
        paths.append(p)
        patterns_matching.append(pat)

    # When pattern mirrors hidden parts count exactly, function should return False for all
    for p, pat in zip(paths, patterns_matching):
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(p, pat) # 4.02ms -> 3.62ms (10.9% faster)

    # When the pattern has zero hidden parts (e.g., "**"), any path that contains >0 hidden parts should return True
    for p in paths:
        has_hidden = any(part.startswith(".") and not set(part) == {"."} for part in PurePath(p).parts)
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(p, "**"); result = codeflash_output # 3.38ms -> 3.08ms (9.89% faster)

def test_explicit_and_implicit_hidden_with_similar_names():
    # Edge: ensure that names that start with a dot followed by only dots are ignored,
    # but names that start with dot and include other chars (including digits) count as hidden
    path_explicit = ".1hidden/.2hidden/file"
    # pattern that explicitly mentions both hidden parts: counts equal -> False
    pattern = ".1hidden/.2hidden/*"
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path_explicit, pattern) # 22.6μs -> 19.4μs (16.6% faster)

    # pattern that mentions only one of them -> True
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path_explicit, ".1hidden/*") # 10.5μs -> 9.07μs (15.4% faster)

    # A path part that is just '..' shouldn't be counted as hidden:
    path_with_dots_only = "../.. /file".replace(" ", "")  # builds "../../file"
    # pattern mirrors the dots-only parts; neither pattern nor path contribute hidden count -> False
    codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path_with_dots_only, "../..") # 9.41μs -> 8.61μs (9.31% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.
from pathlib import PurePath

# imports
import pytest
from src.datasets.data_files import \
    _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir

class TestBasicFunctionality:
    """Tests for basic functionality with simple, straightforward inputs."""

    def test_hidden_file_with_wildcard_pattern(self):
        """Test that a hidden file is flagged as unrequested when matched with '**' pattern."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**"); result = codeflash_output # 21.8μs -> 18.2μs (19.3% faster)

    def test_hidden_file_with_explicit_pattern(self):
        """Test that a hidden file is NOT flagged when explicitly requested with '.*' pattern."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*"); result = codeflash_output # 19.7μs -> 16.7μs (18.3% faster)

    def test_file_in_hidden_dir_with_wildcard(self):
        """Test that a file in a hidden directory is flagged as unrequested with '**' pattern."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**"); result = codeflash_output # 20.2μs -> 18.4μs (10.3% faster)

    def test_file_in_hidden_dir_with_explicit_hidden_dir_pattern(self):
        """Test that a file in a hidden directory is NOT flagged when hidden dir is explicit."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*"); result = codeflash_output # 20.9μs -> 18.3μs (14.5% faster)

    def test_regular_file_with_wildcard(self):
        """Test that a regular (non-hidden) file is not flagged with '**' pattern."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("regular_file.txt", "**"); result = codeflash_output # 15.6μs -> 15.0μs (3.76% faster)

    def test_regular_file_with_explicit_pattern(self):
        """Test that a regular file matches explicit pattern without flagging."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("file.txt", "file.txt"); result = codeflash_output # 16.3μs -> 14.6μs (12.0% faster)

    def test_nested_hidden_file_in_hidden_dir(self):
        """Test a hidden file inside a hidden directory with '**' pattern."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", "**"); result = codeflash_output # 20.1μs -> 17.4μs (15.0% faster)

    def test_nested_hidden_file_with_full_explicit_pattern(self):
        """Test hidden file in hidden dir with both parts explicitly in pattern."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/.*"); result = codeflash_output # 22.0μs -> 17.8μs (23.5% faster)

    def test_nested_hidden_file_with_partial_explicit_pattern(self):
        """Test hidden file in hidden dir with only directory part explicit."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/*"); result = codeflash_output # 21.3μs -> 17.6μs (21.1% faster)

class TestEdgeCases:
    """Tests for edge cases and boundary conditions."""

    def test_dot_only_directory_name(self):
        """Test that directories or files named only '.' are not considered hidden."""
        # A file named '.' shouldn't count as a hidden part
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("./file.txt", "**/file.txt"); result = codeflash_output # 17.6μs -> 16.6μs (6.25% faster)

    def test_double_dots_directory(self):
        """Test behavior with '..' (parent directory reference)."""
        # '..' starts with dot but isn't a hidden directory name in the traditional sense
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir("../file.txt", "**"); result = codeflash_output # 18.6μs -> 17.3μs (7.27% faster)

    def test_file_with_multiple_dots(self):
        """Test hidden file with multiple dots in name."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden.file.txt", "**"); result = codeflash_output # 17.3μs -> 14.8μs (16.5% faster)

    def test_file_with_multiple_dots_explicit(self):
        """Test hidden file with multiple dots explicitly requested."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden.file.txt", ".hidden.file.txt"); result = codeflash_output # 17.8μs -> 14.9μs (19.7% faster)

    def test_deeply_nested_hidden_dirs_all_explicit(self):
        """Test deeply nested structure with all hidden parts explicit."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".a/.b/.c/file.txt", ".a/.b/.c/*"
        ); result = codeflash_output # 22.4μs -> 21.1μs (5.80% faster)

    def test_deeply_nested_hidden_dirs_partial_explicit(self):
        """Test deeply nested structure with only top-level hidden dir explicit."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".a/.b/.c/file.txt", ".a/**"
        ); result = codeflash_output # 22.4μs -> 19.7μs (13.7% faster)

    def test_deeply_nested_hidden_dirs_none_explicit(self):
        """Test deeply nested hidden structure with no explicit hidden parts."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".a/.b/.c/file.txt", "**"
        ); result = codeflash_output # 20.9μs -> 18.7μs (11.6% faster)

    def test_mixed_hidden_and_regular_dirs_all_explicit(self):
        """Test mixed path with hidden and regular directories, all hidden parts explicit."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            "regular/.hidden/normal/file.txt", "regular/.hidden/normal/*"
        ); result = codeflash_output # 23.0μs -> 19.6μs (17.0% faster)

    def test_mixed_hidden_and_regular_dirs_partial_explicit(self):
        """Test mixed path with hidden parts partially explicit."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            "regular/.hidden/normal/file.txt", "regular/**"
        ); result = codeflash_output # 21.7μs -> 18.8μs (15.5% faster)

    def test_empty_pattern(self):
        """Test with empty pattern string."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden/file.txt", ""); result = codeflash_output # 18.1μs -> 16.0μs (12.8% faster)

    def test_single_character_hidden_file(self):
        """Test hidden file with single character name."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".x", "**"); result = codeflash_output # 16.5μs -> 15.3μs (8.27% faster)

    def test_single_character_hidden_file_explicit(self):
        """Test single character hidden file explicitly requested."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".x", ".x"); result = codeflash_output # 16.5μs -> 14.6μs (13.5% faster)

    def test_wildcard_in_hidden_filename(self):
        """Test pattern with wildcard matching hidden files."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*"); result = codeflash_output # 17.6μs -> 15.4μs (14.8% faster)

    def test_multiple_hidden_parts_in_pattern_vs_path(self):
        """Test when pattern has more hidden parts than path."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".hidden/file.txt", ".hidden/.other/*"
        ); result = codeflash_output # 21.5μs -> 18.8μs (14.3% faster)

    def test_special_characters_in_hidden_name(self):
        """Test hidden directory with special characters."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".hidden-dir_123/file.txt", "**"
        ); result = codeflash_output # 19.7μs -> 17.6μs (12.0% faster)

    def test_special_characters_in_hidden_name_explicit(self):
        """Test hidden directory with special characters explicitly requested."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".hidden-dir_123/file.txt", ".hidden-dir_123/*"
        ); result = codeflash_output # 20.8μs -> 17.4μs (19.3% faster)

class TestComplexScenarios:
    """Tests for more complex real-world scenarios."""

    def test_git_directory_not_requested(self):
        """Test that .git directory contents are flagged as unrequested by default."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".git/config", "**"); result = codeflash_output # 18.7μs -> 17.7μs (5.63% faster)

    def test_git_directory_explicitly_requested(self):
        """Test that .git directory contents are not flagged when explicitly requested."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".git/config", ".git/*"); result = codeflash_output # 19.4μs -> 18.2μs (6.87% faster)

    def test_nested_git_subdirs(self):
        """Test nested structure within .git directory."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".git/objects/ab/cdef123456", "**"
        ); result = codeflash_output # 21.4μs -> 19.3μs (10.8% faster)

    def test_nested_git_subdirs_explicit(self):
        """Test nested .git structure explicitly requested."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".git/objects/ab/cdef123456", ".git/**"
        ); result = codeflash_output # 21.6μs -> 19.7μs (9.37% faster)

    def test_venv_hidden_directory(self):
        """Test virtual environment hidden directory."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".venv/lib/python3.9/site-packages/module.py", "**"
        ); result = codeflash_output # 22.2μs -> 20.0μs (10.7% faster)

    def test_venv_directory_partially_explicit(self):
        """Test venv directory with partial explicit path."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".venv/lib/python3.9/site-packages/module.py", ".venv/**"
        ); result = codeflash_output # 21.7μs -> 20.1μs (7.91% faster)

    def test_multiple_consecutive_hidden_parts(self):
        """Test multiple consecutive hidden directories."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".a/.b/.c/.d/file.txt", "**"
        ); result = codeflash_output # 22.1μs -> 20.0μs (10.1% faster)

    def test_multiple_consecutive_hidden_parts_all_explicit(self):
        """Test multiple consecutive hidden directories all explicitly listed."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".a/.b/.c/.d/file.txt", ".a/.b/.c/.d/*"
        ); result = codeflash_output # 24.0μs -> 21.1μs (13.8% faster)

    def test_hidden_files_in_regular_dir(self):
        """Test hidden files in regular (non-hidden) directories."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            "regular_dir/.hidden_file.txt", "**"
        ); result = codeflash_output # 19.8μs -> 17.2μs (14.9% faster)

    def test_hidden_files_in_regular_dir_explicit(self):
        """Test hidden files in regular dirs explicitly requested."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            "regular_dir/.hidden_file.txt", "regular_dir/.*"
        ); result = codeflash_output # 20.8μs -> 17.7μs (17.6% faster)

    def test_multiple_hidden_files_same_level(self):
        """Test multiple hidden files at the same directory level."""
        # Test first hidden file
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".file1.txt", "**"
        ); result1 = codeflash_output # 17.2μs -> 15.0μs (14.6% faster)
        # Test second hidden file
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".file2.txt", "**"
        ); result2 = codeflash_output # 8.12μs -> 7.06μs (15.1% faster)

    def test_pattern_with_multiple_wildcard_levels(self):
        """Test pattern with multiple levels of wildcards."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".hidden/regular/file.txt", "*/*/*.txt"
        ); result = codeflash_output # 20.7μs -> 19.2μs (8.18% faster)

    def test_pattern_with_multiple_wildcard_levels_hidden_explicit(self):
        """Test multiple wildcard levels with hidden part explicit."""
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(
            ".hidden/regular/file.txt", ".hidden/*/*"
        ); result = codeflash_output # 21.0μs -> 18.6μs (12.9% faster)

class TestLargeScale:
    """Tests for performance and scalability with larger data samples."""

    def test_many_consecutive_hidden_directories(self):
        """Test deeply nested structure with many hidden directories."""
        # Create a path with 50 consecutive hidden directories
        path_parts = [f".hidden{i}" for i in range(50)]
        path = "/".join(path_parts) + "/file.txt"
        
        # Without explicit hidden parts in pattern, should be flagged
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, "**"); result = codeflash_output # 62.8μs -> 49.3μs (27.3% faster)

    def test_many_consecutive_hidden_directories_explicit(self):
        """Test deeply nested hidden structure with all parts explicit."""
        # Create a path with 50 consecutive hidden directories
        path_parts = [f".hidden{i}" for i in range(50)]
        path = "/".join(path_parts) + "/file.txt"
        pattern = "/".join(path_parts) + "/*"
        
        # With explicit hidden parts, should not be flagged
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern); result = codeflash_output # 89.5μs -> 63.5μs (41.1% faster)

    def test_large_path_with_many_regular_and_few_hidden(self):
        """Test very long path with mostly regular dirs and a few hidden ones."""
        # Create a complex path with multiple levels
        path_parts = []
        for i in range(40):
            if i % 10 == 0:
                path_parts.append(f".hidden{i}")
            else:
                path_parts.append(f"regular{i}")
        path = "/".join(path_parts) + "/file.txt"
        
        # Without explicit hidden parts, should be flagged
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, "**"); result = codeflash_output # 39.5μs -> 35.5μs (11.1% faster)

    def test_large_path_with_many_regular_and_few_hidden_partial_explicit(self):
        """Test large path with partial explicit hidden parts."""
        # Create a complex path with multiple levels
        path_parts = []
        for i in range(40):
            if i % 10 == 0:
                path_parts.append(f".hidden{i}")
            else:
                path_parts.append(f"regular{i}")
        path = "/".join(path_parts) + "/file.txt"
        
        # Only first hidden part explicit
        pattern = ".hidden0/**"
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern); result = codeflash_output # 40.2μs -> 40.3μs (0.283% slower)

    def test_performance_with_very_long_path(self):
        """Test performance with extremely long path strings."""
        # Create a very long path with 200 levels
        path_parts = [f"dir{i}" for i in range(200)]
        path = "/".join(path_parts) + "/file.txt"
        
        # Should handle without issues
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, "**"); result = codeflash_output # 114μs -> 110μs (4.27% faster)

    def test_performance_with_many_hidden_parts_in_pattern(self):
        """Test pattern with many hidden directory specifications."""
        # Create path with some hidden parts
        path = ".h1/.h2/.h3/regular/file.txt"
        
        # Create pattern with many more hidden parts
        pattern = ".h1/.h2/.h3/.h4/.h5/.h6/.h7/*"
        
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern); result = codeflash_output # 26.3μs -> 24.0μs (9.81% faster)

    def test_alternating_hidden_and_regular_structure(self):
        """Test alternating hidden and regular directory structure."""
        # Create alternating structure: .hidden -> regular -> .hidden -> regular...
        path_parts = []
        for i in range(100):
            if i % 2 == 0:
                path_parts.append(f".hidden{i}")
            else:
                path_parts.append(f"regular{i}")
        path = "/".join(path_parts) + "/file.txt"
        
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, "**"); result = codeflash_output # 77.9μs -> 66.8μs (16.6% faster)

    def test_all_hidden_parts_deep_structure(self):
        """Test structure where all parts except filename are hidden."""
        # Create 60 hidden directories
        path_parts = [f".h{i}" for i in range(60)]
        path = "/".join(path_parts) + "/file.txt"
        pattern = "/".join(path_parts) + "/*"
        
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern); result = codeflash_output # 88.1μs -> 74.9μs (17.7% faster)

    def test_large_path_with_wildcard_patterns(self):
        """Test large path with various wildcard patterns."""
        path = ".hidden/dir1/dir2/dir3/dir4/dir5/file.txt"
        
        # Test various wildcard patterns
        patterns = [
            "**",
            ".hidden/**",
            ".hidden/*/file.txt",
            ".hidden/dir1/dir2/**",
        ]
        
        expected_results = [True, False, False, False]
        
        for pattern, expected in zip(patterns, expected_results):
            codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern); result = codeflash_output # 56.4μs -> 49.6μs (13.8% faster)

    def test_unicode_hidden_directory_names(self):
        """Test hidden directories with unicode characters."""
        path = ".隐藏/normal/file.txt"
        
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, "**"); result = codeflash_output # 20.5μs -> 19.4μs (6.06% faster)

    def test_unicode_hidden_directory_names_explicit(self):
        """Test hidden directories with unicode characters explicitly requested."""
        path = ".隐藏/normal/file.txt"
        pattern = ".隐藏/*"
        
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern); result = codeflash_output # 21.9μs -> 19.7μs (11.3% faster)

    def test_stress_many_hidden_parts_comparison(self):
        """Stress test comparing paths and patterns with many hidden parts."""
        # Create matching path and pattern with 30 hidden parts
        hidden_parts = [f".h{i}" for i in range(30)]
        path = "/".join(hidden_parts) + "/file.txt"
        pattern = "/".join(hidden_parts) + "/*"
        
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern); result = codeflash_output # 53.0μs -> 46.8μs (13.3% faster)

    def test_stress_mismatch_many_hidden_parts(self):
        """Stress test with many hidden parts but intentional mismatch."""
        # Path has 30 hidden parts
        hidden_parts_path = [f".h{i}" for i in range(30)]
        path = "/".join(hidden_parts_path) + "/file.txt"
        
        # Pattern has only 15 hidden parts
        hidden_parts_pattern = [f".h{i}" for i in range(15)]
        pattern = "/".join(hidden_parts_pattern) + "/**"
        
        codeflash_output = _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(path, pattern); result = codeflash_output # 46.4μs -> 40.0μs (16.1% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

To edit these changes git checkout codeflash/optimize-_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir-mlcm8ck0 and push.

Codeflash Static Badge

This optimization achieves an **11% runtime improvement** by eliminating unnecessary memory allocations in a path-filtering function. The key changes are:

**What Changed:**
1. **Replaced list comprehensions with counting loops**: Instead of building two intermediate lists (`hidden_directories_in_path` and `hidden_directories_in_pattern`), the code now uses simple integer counters that increment as hidden parts are found.

2. **Eliminated set allocations**: The original code used `set(part) == {"."}` to check if a part consists only of dots. The optimized version uses `part.strip(".") != ""` instead, avoiding the overhead of creating a set object for every path component.

**Why It's Faster:**
- **Reduced memory allocations**: In Python, creating lists and sets has significant overhead. By counting directly, we avoid allocating memory for intermediate data structures that are only used to determine their length.
- **Lower Python interpreter overhead**: Integer increments are much cheaper than list append operations, which require bounds checking and potential memory reallocation.
- **Faster character check**: String operations like `strip()` are implemented in optimized C code and avoid the overhead of set creation, hashing, and comparison.

**Performance Context:**
Based on the `function_references`, this function is called from `resolve_pattern()` in a list comprehension that filters file paths during glob operations. This means it's invoked **once per matched file** when resolving data file patterns. In workflows that scan directories with many files (especially those with hidden files/directories like `.git/` or `.venv/`), this optimization compounds:
- For 1,000 files checked: saves ~0.92ms
- The test results show consistent 10-25% speedups across various path structures, with the best gains (27-41%) on deeply nested hidden directories

**Test Case Performance:**
The optimization excels with:
- Paths with multiple hidden parts (27-41% faster for 50+ nested hidden directories)
- Alternating hidden/regular structures (16-17% faster)
- Unicode hidden directories (11% faster)
- All test patterns show improvement, indicating robust performance across diverse real-world scenarios
@codeflash-ai codeflash-ai bot requested a review from aseembits93 February 7, 2026 17:56
@codeflash-ai codeflash-ai bot added ⚡️ codeflash Optimization PR opened by Codeflash AI 🎯 Quality: High Optimization Quality according to Codeflash labels Feb 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

⚡️ codeflash Optimization PR opened by Codeflash AI 🎯 Quality: High Optimization Quality according to Codeflash

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants