Skip to content

Commit f6e54c2

Browse files
committed
docs: warn about results asymmetry for difflib junk
1 parent 1736526 commit f6e54c2

2 files changed

Lines changed: 73 additions & 7 deletions

File tree

Doc/library/difflib.rst

Lines changed: 70 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,16 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
4040
complicated way on how many elements the sequences have in common; best case
4141
time is linear.
4242

43+
**User-defined junk predicate**: :class:`SequenceMatcher` accepts an *isjunk*
44+
parameter to let users determine which lines should be ignored while computing
45+
the diff.
46+
47+
.. note::
48+
49+
This predicate only ever inspects the second sequence for junk. That has
50+
an unexpected consequence: swapping the inputs can produce a different
51+
diff than simply reversing the output, sometimes very large.
52+
4353
**Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that
4454
automatically treats certain sequence items as junk. The heuristic counts how many
4555
times each individual item appears in the sequence. If an item's duplicates (after
@@ -48,6 +58,12 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
4858
the purpose of sequence matching. This heuristic can be turned off by setting
4959
the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`.
5060

61+
.. note::
62+
63+
This heuristic only ever inspects the second sequence for junk. That has
64+
an unexpected consequence: swapping the inputs can produce a different
65+
diff than simply reversing the output, sometimes very large.
66+
5167
.. versionchanged:: 3.2
5268
Added the *autojunk* parameter.
5369

@@ -189,6 +205,11 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
189205

190206
See :ref:`difflib-interface` for a more detailed example.
191207

208+
.. note::
209+
210+
This uses a :class:`SequenceMatcher` internally, so swapping *a* and
211+
*b* can produce different results.
212+
192213

193214
.. function:: get_close_matches(word, possibilities, n=3, cutoff=0.6)
194215

@@ -250,6 +271,11 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
250271
+ tree
251272
+ emu
252273

274+
.. note::
275+
276+
This uses a :class:`SequenceMatcher` internally, so swapping *a* and
277+
*b* can produce different results.
278+
253279

254280
.. function:: restore(sequence, which)
255281

@@ -319,6 +345,11 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
319345

320346
See :ref:`difflib-interface` for a more detailed example.
321347

348+
.. note::
349+
350+
This uses a :class:`SequenceMatcher` internally, so swapping *a* and
351+
*b* can produce different results.
352+
322353
.. versionchanged:: 3.15
323354
Added the *color* parameter.
324355

@@ -397,6 +428,18 @@ The :class:`SequenceMatcher` class has this constructor:
397428
of positions where they occur. All three are reset whenever *b* is reset
398429
with :meth:`set_seqs` or :meth:`set_seq2`.
399430

431+
.. note::
432+
433+
Both *isjunk* and the *autojunk* heuristic only consider *b* for filtering
434+
out items. Consequently, swapping *a* and *b* can change which items are treated
435+
as junk, which can in turn change the matching blocks found by
436+
:meth:`get_matching_blocks` and :meth:`get_opcodes`, the value returned by
437+
:meth:`ratio`, and the diffs produced by :func:`unified_diff`,
438+
:func:`context_diff`, and :func:`ndiff`, all of which build a
439+
:class:`SequenceMatcher` internally.
440+
A custom *isjunk* has this effect at any sequence length, and the automatic
441+
*autojunk* heuristic has the same effect for sequences of 200 or more items.
442+
400443
.. versionadded:: 3.2
401444
The *bjunk* and *bpopular* attributes.
402445

@@ -486,6 +529,21 @@ The :class:`SequenceMatcher` class has this constructor:
486529
>>> s.get_matching_blocks()
487530
[Match(a=0, b=0, size=2), Match(a=3, b=2, size=2), Match(a=5, b=4, size=0)]
488531

532+
.. note::
533+
534+
Caution: the result may depend on the order of the arguments. For
535+
instance::
536+
537+
>>> SequenceMatcher(None, 'tide', 'diet').get_matching_blocks()
538+
[Match(a=0, b=3, size=1), Match(a=4, b=4, size=0)]
539+
>>> SequenceMatcher(None, 'diet', 'tide').get_matching_blocks()
540+
[Match(a=0, b=2, size=1), Match(a=2, b=3, size=1), Match(a=4, b=4, size=0)]
541+
542+
This comes from the matching algorithm itself. Another cause for order
543+
causing different outputs, sometimes wildly, is the junk filtering
544+
heuristic (a custom *isjunk* at any sequence length, or the
545+
automatic *autojunk* heuristic for sequences of 200 or more
546+
items). See the :class:`SequenceMatcher` for details.
489547

490548
.. method:: get_opcodes()
491549

@@ -527,6 +585,11 @@ The :class:`SequenceMatcher` class has this constructor:
527585
equal a[4:6] --> b[3:5] 'cd' --> 'cd'
528586
insert a[6:6] --> b[5:6] '' --> 'f'
529587

588+
.. note::
589+
590+
This is derived from :meth:`get_matching_blocks`, and so shares
591+
its order-dependence; see the note there.
592+
530593

531594
.. method:: get_grouped_opcodes(n=3)
532595

@@ -538,6 +601,11 @@ The :class:`SequenceMatcher` class has this constructor:
538601

539602
The groups are returned in the same format as :meth:`get_opcodes`.
540603

604+
.. note::
605+
606+
This is derived from :meth:`get_matching_blocks`, and so shares
607+
its order-dependence; see the note there.
608+
541609

542610
.. method:: ratio()
543611

@@ -555,13 +623,8 @@ The :class:`SequenceMatcher` class has this constructor:
555623

556624
.. note::
557625

558-
Caution: The result of a :meth:`ratio` call may depend on the order of
559-
the arguments. For instance::
560-
561-
>>> SequenceMatcher(None, 'tide', 'diet').ratio()
562-
0.25
563-
>>> SequenceMatcher(None, 'diet', 'tide').ratio()
564-
0.5
626+
This is derived from :meth:`get_matching_blocks`, and so shares
627+
its order-dependence; see the note there.
565628

566629

567630
.. method:: quick_ratio()
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Clarify in the :mod:`difflib` documentation that both a custom *isjunk*
2+
predicate and the *autojunk* heuristic of :class:`~difflib.SequenceMatcher`
3+
can treat ``(a, b)`` and ``(b, a)`` differently.

0 commit comments

Comments
 (0)