Add subpixel_match: sub-pixel template-match refinement

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 亚像素模板匹配精修
+
+把匹配中心精修到像素的分数位,供拖曳 / 滑块 / 高 DPI 精度。完整参考:[`docs/source/Zh/doc/new_features/v171_features_doc.rst`](../docs/source/Zh/doc/new_features/v171_features_doc.rst)。
+
+- **`match_subpixel` / `refine_peak`**(`AC_match_subpixel`):每个匹配器都从 `cv2.minMaxLoc` 返回*整数*坐标——对拖曳把手、细滑块或高 DPI 显示器,这种舍入是主要的点击落点误差。本功能以抛物线拟合峰值周围的 3×3 分数邻域(x/y 各自独立,标准 NCC 亚像素法),返回带浮点 `cx`/`cy` 与套用的 `offset_x`/`offset_y` 的 `SubPixelMatch`。重用 `visual_match._score_map`;`haystack` 可注入;不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 失败 / 无效果动作的修复策略引擎
 
 当动作没效果时选择下一个修复战术——并驱动重试循环。完整参考:[`docs/source/Zh/doc/new_features/v170_features_doc.rst`](../docs/source/Zh/doc/new_features/v170_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 次像素樣板比對精修
+
+把比對中心精修到像素的分數位,供拖曳 / 滑桿 / 高 DPI 精度。完整參考:[`docs/source/Zh/doc/new_features/v171_features_doc.rst`](../docs/source/Zh/doc/new_features/v171_features_doc.rst)。
+
+- **`match_subpixel` / `refine_peak`**(`AC_match_subpixel`):每個比對器都從 `cv2.minMaxLoc` 回傳*整數*座標——對拖曳把手、細滑桿或高 DPI 顯示器,這種捨入是主要的點擊落點誤差。本功能以拋物線擬合峰值周圍的 3×3 分數鄰域(x/y 各自獨立,標準 NCC 次像素法),回傳帶浮點 `cx`/`cy` 與套用的 `offset_x`/`offset_y` 的 `SubPixelMatch`。重用 `visual_match._score_map`;`haystack` 可注入;不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 失敗 / 無效果動作的修復策略引擎
 
 當動作沒效果時選擇下一個修復戰術——並驅動重試迴圈。完整參考:[`docs/source/Zh/doc/new_features/v170_features_doc.rst`](../docs/source/Zh/doc/new_features/v170_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Sub-Pixel Template-Match Refinement
+
+Refine a match's centre to a fraction of a pixel for drag / slider / high-DPI precision. Full reference: [`docs/source/Eng/doc/new_features/v171_features_doc.rst`](docs/source/Eng/doc/new_features/v171_features_doc.rst).
+
+- **`match_subpixel` / `refine_peak`** (`AC_match_subpixel`): every matcher returns *integer* coordinates from `cv2.minMaxLoc` — for a drag handle, fine slider or high-DPI display that rounding is the dominant click-placement error. This fits a parabola to the 3×3 score neighbourhood around the peak (independently on x/y, the standard NCC sub-pixel method) and returns a `SubPixelMatch` with float `cx`/`cy` + the applied `offset_x`/`offset_y`. Reuses `visual_match._score_map`; injectable `haystack`; no `PySide6`.
+
 ## What's new (2026-06-24) — Repair-Tactic Policy for Failed / No-Effect Actions
 
 Pick the next repair tactic when an action does nothing — and drive the retry loop. Full reference: [`docs/source/Eng/doc/new_features/v170_features_doc.rst`](docs/source/Eng/doc/new_features/v170_features_doc.rst).

docs/source/Eng/doc/new_features/v171_features_doc.rst

@@ -0,0 +1,40 @@
+Sub-Pixel Template-Match Refinement
+===================================
+
+Every matcher (``match_template`` / ``match_rotated`` / ``match_masked``) returns *integer*
+pixel coordinates straight from ``cv2.minMaxLoc``. For a drag handle, a fine slider, a
+sub-pixel-rendered target, or a high-DPI display, that integer rounding is the dominant
+click-placement error. ``subpixel_match`` refines the peak to a fraction of a pixel by fitting
+a parabola to the 3x3 score neighbourhood around the integer maximum — independently on x and y
+(the standard NCC sub-pixel method).
+
+It reuses ``visual_match._score_map`` (the full ``matchTemplate`` surface the public matchers
+discard); no matching code is duplicated. The ``haystack`` is injectable; the analysis is
+unit-testable on synthetic arrays. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import match_subpixel, refine_peak
+
+    hit = match_subpixel("slider_thumb.png", min_score=0.8)
+    if hit:
+        # cx / cy are floats — pass to a sub-pixel-aware drag / move
+        move_to(hit.cx, hit.cy)
+
+    # the fitting primitive over any score surface
+    offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y))
+
+``match_subpixel`` returns a ``SubPixelMatch`` (integer ``x`` / ``y`` / ``width`` / ``height`` /
+``score`` plus float ``cx`` / ``cy`` and the ``offset_x`` / ``offset_y`` applied), or ``None``
+below ``min_score``. ``refine_peak`` returns the ``[-0.5, 0.5]`` quadratic-fit offset of a peak
+from its neighbours — usable on any correlation surface.
+
+Executor command
+----------------
+
+``AC_match_subpixel`` (``template`` / ``min_score`` / ``region`` / ``method`` →
+``{found, match}``) is exposed as the MCP tool ``ac_match_subpixel`` (read-only) and as the
+Script Builder command **Match Template (sub-pixel)** under **Image**.

docs/source/Eng/eng_index.rst

@@ -193,6 +193,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v168_features_doc
    doc/new_features/v169_features_doc
    doc/new_features/v170_features_doc
+   doc/new_features/v171_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v171_features_doc.rst

@@ -0,0 +1,37 @@
+次像素樣板比對精修
+==================
+
+每個比對器(``match_template`` / ``match_rotated`` / ``match_masked``)都直接從
+``cv2.minMaxLoc`` 回傳*整數*像素座標。對拖曳把手、細滑桿、次像素渲染的目標或高 DPI 顯示器
+而言,這種整數捨入是主要的點擊落點誤差來源。``subpixel_match`` 以拋物線擬合整數極大值周圍的
+3x3 分數鄰域(x 與 y 各自獨立,即標準 NCC 次像素法),把峰值精修到像素的分數位。
+
+本功能重用 ``visual_match._score_map``(公開比對器丟棄的完整 ``matchTemplate`` 曲面);
+不重複任何比對程式。``haystack`` 可注入;分析可在合成陣列上單元測試。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import match_subpixel, refine_peak
+
+    hit = match_subpixel("slider_thumb.png", min_score=0.8)
+    if hit:
+        # cx / cy 為浮點數——傳給支援次像素的拖曳 / 移動
+        move_to(hit.cx, hit.cy)
+
+    # 可用於任何分數曲面的擬合原語
+    offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y))
+
+``match_subpixel`` 回傳 ``SubPixelMatch``(整數 ``x`` / ``y`` / ``width`` / ``height`` /
+``score`` 加上浮點 ``cx`` / ``cy`` 與套用的 ``offset_x`` / ``offset_y``),或在低於
+``min_score`` 時回傳 ``None``。``refine_peak`` 回傳峰值相對鄰點的 ``[-0.5, 0.5]`` 拋物線擬合
+偏移——可用於任何相關性曲面。
+
+執行器指令
+----------
+
+``AC_match_subpixel``(``template`` / ``min_score`` / ``region`` / ``method`` →
+``{found, match}``)以 MCP 工具 ``ac_match_subpixel``(唯讀)及 Script Builder 指令
+**Match Template (sub-pixel)**(位於 **Image** 分類下)形式提供。

docs/source/Zh/zh_index.rst

@@ -193,6 +193,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v168_features_doc
    doc/new_features/v169_features_doc
    doc/new_features/v170_features_doc
+   doc/new_features/v171_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -295,6 +295,10 @@
 from je_auto_control.utils.match_autothresh import (
     auto_threshold, match_auto,
 )
+# Sub-pixel refinement of a template match (quadratic peak fit)
+from je_auto_control.utils.subpixel_match import (
+    SubPixelMatch, match_subpixel, refine_peak,
+)
 # Coarse labelled cell grid for VLM grounding (point <-> cell mapping)
 from je_auto_control.utils.screen_grid import (
     GridCell, cell_for_point, grid_cells, point_for_cell,
@@ -1246,6 +1250,9 @@ def start_autocontrol_gui(*args, **kwargs):
     "edge_match",
     "edge_match_all",
     "chamfer_distance",
+    "SubPixelMatch",
+    "match_subpixel",
+    "refine_peak",
     "GridCell",
     "grid_cells",
     "cell_for_point",

je_auto_control/gui/script_builder/command_schema.py

@@ -396,6 +396,17 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Find every edge-shape (Chamfer) match (NMS-deduped).",
     ))
+    specs.append(CommandSpec(
+        "AC_match_subpixel", "Image", "Match Template (sub-pixel)",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.0,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Match with a sub-pixel-refined centre (drag / slider precision).",
+    ))
     specs.append(CommandSpec(
         "AC_grid_cells", "Image", "Grid Cells (coarse grounding)",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -3387,6 +3387,19 @@ def _edge_match_all(template: str, min_score: Any = 0.7, max_results: Any = 20,
     return {"count": len(matches), "matches": [m.to_dict() for m in matches]}
 
 
+def _match_subpixel(template: str, min_score: Any = 0.0, region: Any = None,
+                    method: str = "ccoeff_normed") -> Dict[str, Any]:
+    """Adapter: best template match with a sub-pixel-refined centre."""
+    import json
+    from je_auto_control.utils.subpixel_match import match_subpixel
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    match = match_subpixel(template, region=region, method=method,
+                           min_score=float(min_score))
+    return {"found": match is not None,
+            "match": match.to_dict() if match else None}
+
+
 def _region_arg(value: Any) -> Optional[List[int]]:
     """Coerce a JSON-string / list region arg into a list of ints, or None."""
     import json
@@ -5986,6 +5999,7 @@ def __init__(self):
             "AC_match_auto": _match_auto,
             "AC_edge_match": _edge_match,
             "AC_edge_match_all": _edge_match_all,
+            "AC_match_subpixel": _match_subpixel,
             "AC_grid_cells": _grid_cells,
             "AC_cell_for_point": _cell_for_point,
             "AC_point_for_cell": _point_for_cell,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -3759,6 +3759,22 @@ def rotated_match_tools() -> List[MCPTool]:
             handler=h.edge_match_all,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_match_subpixel",
+            description=("Find 'template' and refine the centre to SUB-PIXEL precision "
+                         "(quadratic fit on the score peak). Returns {found, match:"
+                         "{x,y,width,height,score,cx,cy,offset_x,offset_y,center}} — "
+                         "cx/cy are float for drag / slider / high-DPI accuracy. "
+                         "'min_score', 'region', 'method'."),
+            input_schema=schema({
+                "template": {"type": "string"},
+                "min_score": {"type": "number"},
+                "region": {"type": "array", "items": {"type": "integer"}},
+                "method": {"type": "string"}},
+                required=["template"]),
+            handler=h.match_subpixel,
+            annotations=READ_ONLY,
+        ),
     ]
 
 

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2137,6 +2137,11 @@ def edge_match_all(template, min_score=0.7, max_results=20, nms_iou=0.3,
     return _edge_match_all(template, min_score, max_results, nms_iou, region)
 
 
+def match_subpixel(template, min_score=0.0, region=None, method="ccoeff_normed"):
+    from je_auto_control.utils.executor.action_executor import _match_subpixel
+    return _match_subpixel(template, min_score, region, method)
+
+
 def grid_cells(rows, cols, region=None):
     from je_auto_control.utils.executor.action_executor import _grid_cells
     return _grid_cells(rows, cols, region)

je_auto_control/utils/subpixel_match/__init__.py

@@ -0,0 +1,6 @@
+"""Sub-pixel refinement of a template match by quadratic peak fitting."""
+from je_auto_control.utils.subpixel_match.subpixel_match import (
+    SubPixelMatch, match_subpixel, refine_peak,
+)
+
+__all__ = ["SubPixelMatch", "match_subpixel", "refine_peak"]

je_auto_control/utils/subpixel_match/subpixel_match.py

@@ -0,0 +1,95 @@
+"""Sub-pixel refinement of a template match by quadratic peak fitting.
+
+Every matcher (``match_template`` / ``match_rotated`` / ``match_masked``) returns *integer*
+pixel coordinates straight from ``cv2.minMaxLoc``. For a drag handle, a fine slider, a
+sub-pixel-rendered target, or a high-DPI display, that integer rounding is the dominant
+click-placement error. ``subpixel_match`` refines the peak to a fraction of a pixel by fitting
+a parabola to the 3x3 score neighbourhood around the integer maximum (the standard NCC
+sub-pixel method) — independently on x and y.
+
+It reuses ``visual_match._score_map`` (the full ``matchTemplate`` surface the public matchers
+discard); no matching code is duplicated. The ``haystack`` is injectable; the analysis is
+unit-testable on synthetic arrays. Imports no ``PySide6``.
+"""
+from dataclasses import asdict, dataclass
+from typing import Any, Dict, List, Optional, Sequence, Tuple
+
+from je_auto_control.utils.visual_match.visual_match import _score_map
+
+ImageSource = Any
+
+
+@dataclass(frozen=True)
+class SubPixelMatch:
+    """A match whose centre is refined to sub-pixel precision."""
+
+    x: int
+    y: int
+    width: int
+    height: int
+    score: float
+    cx: float
+    cy: float
+    offset_x: float
+    offset_y: float
+
+    @property
+    def center(self) -> List[int]:
+        """The integer centre point ``[x, y]`` (top-left + half size)."""
+        return [self.x + self.width // 2, self.y + self.height // 2]
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the match as a plain dict including the integer centre point."""
+        data = asdict(self)
+        data["center"] = self.center
+        return data
+
+
+def _parabola_offset(low: float, mid: float, high: float) -> float:
+    """Sub-sample offset in [-0.5, 0.5] of a peak from its two neighbours' scores."""
+    denominator = low - 2.0 * mid + high
+    if abs(denominator) < 1e-12:
+        return 0.0
+    offset = 0.5 * (low - high) / denominator
+    return max(-0.5, min(0.5, offset))
+
+
+def refine_peak(score_map, peak_xy: Tuple[int, int]) -> Tuple[float, float]:
+    """Return the sub-pixel ``(offset_x, offset_y)`` of a peak via 3x3 quadratic fit."""
+    peak_x, peak_y = int(peak_xy[0]), int(peak_xy[1])
+    height, width = score_map.shape
+    offset_x = offset_y = 0.0
+    if 0 < peak_x < width - 1:
+        offset_x = _parabola_offset(float(score_map[peak_y, peak_x - 1]),
+                                    float(score_map[peak_y, peak_x]),
+                                    float(score_map[peak_y, peak_x + 1]))
+    if 0 < peak_y < height - 1:
+        offset_y = _parabola_offset(float(score_map[peak_y - 1, peak_x]),
+                                    float(score_map[peak_y, peak_x]),
+                                    float(score_map[peak_y + 1, peak_x]))
+    return offset_x, offset_y
+
+
+def match_subpixel(template: ImageSource, *, haystack: Optional[ImageSource] = None,
+                   region: Optional[Sequence[int]] = None,
+                   method: str = "ccoeff_normed",
+                   min_score: float = 0.0) -> Optional[SubPixelMatch]:
+    """Return the best match with a sub-pixel-refined centre, or ``None``.
+
+    The integer top-left and score come from the correlation peak; ``cx`` / ``cy`` add the
+    quadratic-fit offset to the integer centre for fractional-pixel click placement.
+    """
+    import cv2
+    score_map, tmpl = _score_map(template, haystack, region=region, method=method)
+    if score_map is None:
+        return None
+    _, max_val, _, max_loc = cv2.minMaxLoc(score_map)
+    if max_val < min_score:
+        return None
+    peak_x, peak_y = int(max_loc[0]), int(max_loc[1])
+    offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y))
+    height, width = tmpl.shape[:2]
+    return SubPixelMatch(peak_x, peak_y, width, height, round(float(max_val), 4),
+                         round(peak_x + width / 2.0 + offset_x, 3),
+                         round(peak_y + height / 2.0 + offset_y, 3),
+                         round(offset_x, 4), round(offset_y, 4))