Skip to content

Commit 970a738

Browse files
[3.13] gh-118150: warn in doc about results asymmetry for difflib junk (GH-153892) (GH-154660)
(cherry picked from commit 59e67c2) Co-authored-by: Lenormand Julien <lenormand.julien0@gmail.com>
1 parent af04851 commit 970a738

2 files changed

Lines changed: 14 additions & 17 deletions

File tree

Doc/library/difflib.rst

Lines changed: 12 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -44,13 +44,18 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
4444
complicated way on how many elements the sequences have in common; best case
4545
time is linear.
4646

47-
**Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that
48-
automatically treats certain sequence items as junk. The heuristic counts how many
49-
times each individual item appears in the sequence. If an item's duplicates (after
50-
the first one) account for more than 1% of the sequence and the sequence is at least
51-
200 items long, this item is marked as "popular" and is treated as junk for
52-
the purpose of sequence matching. This heuristic can be turned off by setting
53-
the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`.
47+
**Junk**: :class:`SequenceMatcher` accepts an ``isjunk`` predicate and an
48+
``autojunk`` flag. Items that are considered as junk will not be considered
49+
to find similar content blocks. This can produce better results for humans
50+
(typically breaking on whitespace) and faster (because it reduces the number
51+
of possible combinations). But it can also cause pathological cases where
52+
too many items considered junk cause an unexpectedly large (but correct)
53+
diff result.
54+
You should consider tuning them or turning them off depending on your data.
55+
Moreover, only the second sequence is inspected for junk. This causes the diff
56+
output to not be symmetrical.
57+
When ``autojunk=True``, it will consider as junk the items that account for more
58+
than 1% of the sequence, if it is at least 200 items long.
5459

5560
.. versionchanged:: 3.2
5661
Added the *autojunk* parameter.
@@ -554,16 +559,6 @@ The :class:`SequenceMatcher` class has this constructor:
554559
to try :meth:`quick_ratio` or :meth:`real_quick_ratio` first to get an
555560
upper bound.
556561

557-
.. note::
558-
559-
Caution: The result of a :meth:`ratio` call may depend on the order of
560-
the arguments. For instance::
561-
562-
>>> SequenceMatcher(None, 'tide', 'diet').ratio()
563-
0.25
564-
>>> SequenceMatcher(None, 'diet', 'tide').ratio()
565-
0.5
566-
567562

568563
.. method:: quick_ratio()
569564

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Clarify in the :mod:`difflib` documentation what *junk* actually does, its
2+
drawbacks, and how to control it.

0 commit comments

Comments
 (0)