Skip to content

Commit 59e67c2

Browse files
authored
gh-118150: warn in doc about results asymmetry for difflib junk (GH-153892)
1 parent 9560bd8 commit 59e67c2

2 files changed

Lines changed: 14 additions & 17 deletions

File tree

Doc/library/difflib.rst

Lines changed: 12 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -40,13 +40,18 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
4040
complicated way on how many elements the sequences have in common; best case
4141
time is linear.
4242

43-
**Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that
44-
automatically treats certain sequence items as junk. The heuristic counts how many
45-
times each individual item appears in the sequence. If an item's duplicates (after
46-
the first one) account for more than 1% of the sequence and the sequence is at least
47-
200 items long, this item is marked as "popular" and is treated as junk for
48-
the purpose of sequence matching. This heuristic can be turned off by setting
49-
the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`.
43+
**Junk**: :class:`SequenceMatcher` accepts an ``isjunk`` predicate and an
44+
``autojunk`` flag. Items that are considered as junk will not be considered
45+
to find similar content blocks. This can produce better results for humans
46+
(typically breaking on whitespace) and faster (because it reduces the number
47+
of possible combinations). But it can also cause pathological cases where
48+
too many items considered junk cause an unexpectedly large (but correct)
49+
diff result.
50+
You should consider tuning them or turning them off depending on your data.
51+
Moreover, only the second sequence is inspected for junk. This causes the diff
52+
output to not be symmetrical.
53+
When ``autojunk=True``, it will consider as junk the items that account for more
54+
than 1% of the sequence, if it is at least 200 items long.
5055

5156
.. versionchanged:: 3.2
5257
Added the *autojunk* parameter.
@@ -558,16 +563,6 @@ The :class:`SequenceMatcher` class has this constructor:
558563
to try :meth:`quick_ratio` or :meth:`real_quick_ratio` first to get an
559564
upper bound.
560565

561-
.. note::
562-
563-
Caution: The result of a :meth:`ratio` call may depend on the order of
564-
the arguments. For instance::
565-
566-
>>> SequenceMatcher(None, 'tide', 'diet').ratio()
567-
0.25
568-
>>> SequenceMatcher(None, 'diet', 'tide').ratio()
569-
0.5
570-
571566

572567
.. method:: quick_ratio()
573568

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Clarify in the :mod:`difflib` documentation what *junk* actually does, its
2+
drawbacks, and how to control it.

0 commit comments

Comments
 (0)