diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index d5fabbb6..90eca1a6 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-24) — 亚像素模板匹配精修 + +把匹配中心精修到像素的分数位,供拖曳 / 滑块 / 高 DPI 精度。完整参考:[`docs/source/Zh/doc/new_features/v171_features_doc.rst`](../docs/source/Zh/doc/new_features/v171_features_doc.rst)。 + +- **`match_subpixel` / `refine_peak`**(`AC_match_subpixel`):每个匹配器都从 `cv2.minMaxLoc` 返回*整数*坐标——对拖曳把手、细滑块或高 DPI 显示器,这种舍入是主要的点击落点误差。本功能以抛物线拟合峰值周围的 3×3 分数邻域(x/y 各自独立,标准 NCC 亚像素法),返回带浮点 `cx`/`cy` 与套用的 `offset_x`/`offset_y` 的 `SubPixelMatch`。重用 `visual_match._score_map`;`haystack` 可注入;不导入 `PySide6`。 + ## 本次更新 (2026-06-24) — 失败 / 无效果动作的修复策略引擎 当动作没效果时选择下一个修复战术——并驱动重试循环。完整参考:[`docs/source/Zh/doc/new_features/v170_features_doc.rst`](../docs/source/Zh/doc/new_features/v170_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 4fe1e509..902e379d 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-24) — 次像素樣板比對精修 + +把比對中心精修到像素的分數位,供拖曳 / 滑桿 / 高 DPI 精度。完整參考:[`docs/source/Zh/doc/new_features/v171_features_doc.rst`](../docs/source/Zh/doc/new_features/v171_features_doc.rst)。 + +- **`match_subpixel` / `refine_peak`**(`AC_match_subpixel`):每個比對器都從 `cv2.minMaxLoc` 回傳*整數*座標——對拖曳把手、細滑桿或高 DPI 顯示器,這種捨入是主要的點擊落點誤差。本功能以拋物線擬合峰值周圍的 3×3 分數鄰域(x/y 各自獨立,標準 NCC 次像素法),回傳帶浮點 `cx`/`cy` 與套用的 `offset_x`/`offset_y` 的 `SubPixelMatch`。重用 `visual_match._score_map`;`haystack` 可注入;不匯入 `PySide6`。 + ## 本次更新 (2026-06-24) — 失敗 / 無效果動作的修復策略引擎 當動作沒效果時選擇下一個修復戰術——並驅動重試迴圈。完整參考:[`docs/source/Zh/doc/new_features/v170_features_doc.rst`](../docs/source/Zh/doc/new_features/v170_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index d1b4ebd6..e9387c30 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-24) — Sub-Pixel Template-Match Refinement + +Refine a match's centre to a fraction of a pixel for drag / slider / high-DPI precision. Full reference: [`docs/source/Eng/doc/new_features/v171_features_doc.rst`](docs/source/Eng/doc/new_features/v171_features_doc.rst). + +- **`match_subpixel` / `refine_peak`** (`AC_match_subpixel`): every matcher returns *integer* coordinates from `cv2.minMaxLoc` — for a drag handle, fine slider or high-DPI display that rounding is the dominant click-placement error. This fits a parabola to the 3×3 score neighbourhood around the peak (independently on x/y, the standard NCC sub-pixel method) and returns a `SubPixelMatch` with float `cx`/`cy` + the applied `offset_x`/`offset_y`. Reuses `visual_match._score_map`; injectable `haystack`; no `PySide6`. + ## What's new (2026-06-24) — Repair-Tactic Policy for Failed / No-Effect Actions Pick the next repair tactic when an action does nothing — and drive the retry loop. Full reference: [`docs/source/Eng/doc/new_features/v170_features_doc.rst`](docs/source/Eng/doc/new_features/v170_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v171_features_doc.rst b/docs/source/Eng/doc/new_features/v171_features_doc.rst new file mode 100644 index 00000000..f22e58bd --- /dev/null +++ b/docs/source/Eng/doc/new_features/v171_features_doc.rst @@ -0,0 +1,40 @@ +Sub-Pixel Template-Match Refinement +=================================== + +Every matcher (``match_template`` / ``match_rotated`` / ``match_masked``) returns *integer* +pixel coordinates straight from ``cv2.minMaxLoc``. For a drag handle, a fine slider, a +sub-pixel-rendered target, or a high-DPI display, that integer rounding is the dominant +click-placement error. ``subpixel_match`` refines the peak to a fraction of a pixel by fitting +a parabola to the 3x3 score neighbourhood around the integer maximum — independently on x and y +(the standard NCC sub-pixel method). + +It reuses ``visual_match._score_map`` (the full ``matchTemplate`` surface the public matchers +discard); no matching code is duplicated. The ``haystack`` is injectable; the analysis is +unit-testable on synthetic arrays. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import match_subpixel, refine_peak + + hit = match_subpixel("slider_thumb.png", min_score=0.8) + if hit: + # cx / cy are floats — pass to a sub-pixel-aware drag / move + move_to(hit.cx, hit.cy) + + # the fitting primitive over any score surface + offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y)) + +``match_subpixel`` returns a ``SubPixelMatch`` (integer ``x`` / ``y`` / ``width`` / ``height`` / +``score`` plus float ``cx`` / ``cy`` and the ``offset_x`` / ``offset_y`` applied), or ``None`` +below ``min_score``. ``refine_peak`` returns the ``[-0.5, 0.5]`` quadratic-fit offset of a peak +from its neighbours — usable on any correlation surface. + +Executor command +---------------- + +``AC_match_subpixel`` (``template`` / ``min_score`` / ``region`` / ``method`` → +``{found, match}``) is exposed as the MCP tool ``ac_match_subpixel`` (read-only) and as the +Script Builder command **Match Template (sub-pixel)** under **Image**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index e46aa880..1118119c 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -193,6 +193,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v168_features_doc doc/new_features/v169_features_doc doc/new_features/v170_features_doc + doc/new_features/v171_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v171_features_doc.rst b/docs/source/Zh/doc/new_features/v171_features_doc.rst new file mode 100644 index 00000000..3eb34d6d --- /dev/null +++ b/docs/source/Zh/doc/new_features/v171_features_doc.rst @@ -0,0 +1,37 @@ +次像素樣板比對精修 +================== + +每個比對器(``match_template`` / ``match_rotated`` / ``match_masked``)都直接從 +``cv2.minMaxLoc`` 回傳*整數*像素座標。對拖曳把手、細滑桿、次像素渲染的目標或高 DPI 顯示器 +而言,這種整數捨入是主要的點擊落點誤差來源。``subpixel_match`` 以拋物線擬合整數極大值周圍的 +3x3 分數鄰域(x 與 y 各自獨立,即標準 NCC 次像素法),把峰值精修到像素的分數位。 + +本功能重用 ``visual_match._score_map``(公開比對器丟棄的完整 ``matchTemplate`` 曲面); +不重複任何比對程式。``haystack`` 可注入;分析可在合成陣列上單元測試。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import match_subpixel, refine_peak + + hit = match_subpixel("slider_thumb.png", min_score=0.8) + if hit: + # cx / cy 為浮點數——傳給支援次像素的拖曳 / 移動 + move_to(hit.cx, hit.cy) + + # 可用於任何分數曲面的擬合原語 + offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y)) + +``match_subpixel`` 回傳 ``SubPixelMatch``(整數 ``x`` / ``y`` / ``width`` / ``height`` / +``score`` 加上浮點 ``cx`` / ``cy`` 與套用的 ``offset_x`` / ``offset_y``),或在低於 +``min_score`` 時回傳 ``None``。``refine_peak`` 回傳峰值相對鄰點的 ``[-0.5, 0.5]`` 拋物線擬合 +偏移——可用於任何相關性曲面。 + +執行器指令 +---------- + +``AC_match_subpixel``(``template`` / ``min_score`` / ``region`` / ``method`` → +``{found, match}``)以 MCP 工具 ``ac_match_subpixel``(唯讀)及 Script Builder 指令 +**Match Template (sub-pixel)**(位於 **Image** 分類下)形式提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index a8364cd9..696c5fb7 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -193,6 +193,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v168_features_doc doc/new_features/v169_features_doc doc/new_features/v170_features_doc + doc/new_features/v171_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index c01e1065..2d62e639 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -295,6 +295,10 @@ from je_auto_control.utils.match_autothresh import ( auto_threshold, match_auto, ) +# Sub-pixel refinement of a template match (quadratic peak fit) +from je_auto_control.utils.subpixel_match import ( + SubPixelMatch, match_subpixel, refine_peak, +) # Coarse labelled cell grid for VLM grounding (point <-> cell mapping) from je_auto_control.utils.screen_grid import ( GridCell, cell_for_point, grid_cells, point_for_cell, @@ -1246,6 +1250,9 @@ def start_autocontrol_gui(*args, **kwargs): "edge_match", "edge_match_all", "chamfer_distance", + "SubPixelMatch", + "match_subpixel", + "refine_peak", "GridCell", "grid_cells", "cell_for_point", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index ba6bb37a..6087e1c2 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -396,6 +396,17 @@ def _add_image_specs(specs: List[CommandSpec]) -> None: ), description="Find every edge-shape (Chamfer) match (NMS-deduped).", )) + specs.append(CommandSpec( + "AC_match_subpixel", "Image", "Match Template (sub-pixel)", + fields=( + FieldSpec("template", FieldType.FILE_PATH), + FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.0, + min_value=0.0, max_value=1.0), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder=_REGION_PLACEHOLDER), + ), + description="Match with a sub-pixel-refined centre (drag / slider precision).", + )) specs.append(CommandSpec( "AC_grid_cells", "Image", "Grid Cells (coarse grounding)", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index b49081c6..bbf3e28a 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3387,6 +3387,19 @@ def _edge_match_all(template: str, min_score: Any = 0.7, max_results: Any = 20, return {"count": len(matches), "matches": [m.to_dict() for m in matches]} +def _match_subpixel(template: str, min_score: Any = 0.0, region: Any = None, + method: str = "ccoeff_normed") -> Dict[str, Any]: + """Adapter: best template match with a sub-pixel-refined centre.""" + import json + from je_auto_control.utils.subpixel_match import match_subpixel + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + match = match_subpixel(template, region=region, method=method, + min_score=float(min_score)) + return {"found": match is not None, + "match": match.to_dict() if match else None} + + def _region_arg(value: Any) -> Optional[List[int]]: """Coerce a JSON-string / list region arg into a list of ints, or None.""" import json @@ -5986,6 +5999,7 @@ def __init__(self): "AC_match_auto": _match_auto, "AC_edge_match": _edge_match, "AC_edge_match_all": _edge_match_all, + "AC_match_subpixel": _match_subpixel, "AC_grid_cells": _grid_cells, "AC_cell_for_point": _cell_for_point, "AC_point_for_cell": _point_for_cell, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index e76c5451..07becced 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -3759,6 +3759,22 @@ def rotated_match_tools() -> List[MCPTool]: handler=h.edge_match_all, annotations=READ_ONLY, ), + MCPTool( + name="ac_match_subpixel", + description=("Find 'template' and refine the centre to SUB-PIXEL precision " + "(quadratic fit on the score peak). Returns {found, match:" + "{x,y,width,height,score,cx,cy,offset_x,offset_y,center}} — " + "cx/cy are float for drag / slider / high-DPI accuracy. " + "'min_score', 'region', 'method'."), + input_schema=schema({ + "template": {"type": "string"}, + "min_score": {"type": "number"}, + "region": {"type": "array", "items": {"type": "integer"}}, + "method": {"type": "string"}}, + required=["template"]), + handler=h.match_subpixel, + annotations=READ_ONLY, + ), ] diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index bec7c7fd..de873e05 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2137,6 +2137,11 @@ def edge_match_all(template, min_score=0.7, max_results=20, nms_iou=0.3, return _edge_match_all(template, min_score, max_results, nms_iou, region) +def match_subpixel(template, min_score=0.0, region=None, method="ccoeff_normed"): + from je_auto_control.utils.executor.action_executor import _match_subpixel + return _match_subpixel(template, min_score, region, method) + + def grid_cells(rows, cols, region=None): from je_auto_control.utils.executor.action_executor import _grid_cells return _grid_cells(rows, cols, region) diff --git a/je_auto_control/utils/subpixel_match/__init__.py b/je_auto_control/utils/subpixel_match/__init__.py new file mode 100644 index 00000000..8cec6e09 --- /dev/null +++ b/je_auto_control/utils/subpixel_match/__init__.py @@ -0,0 +1,6 @@ +"""Sub-pixel refinement of a template match by quadratic peak fitting.""" +from je_auto_control.utils.subpixel_match.subpixel_match import ( + SubPixelMatch, match_subpixel, refine_peak, +) + +__all__ = ["SubPixelMatch", "match_subpixel", "refine_peak"] diff --git a/je_auto_control/utils/subpixel_match/subpixel_match.py b/je_auto_control/utils/subpixel_match/subpixel_match.py new file mode 100644 index 00000000..25299bef --- /dev/null +++ b/je_auto_control/utils/subpixel_match/subpixel_match.py @@ -0,0 +1,95 @@ +"""Sub-pixel refinement of a template match by quadratic peak fitting. + +Every matcher (``match_template`` / ``match_rotated`` / ``match_masked``) returns *integer* +pixel coordinates straight from ``cv2.minMaxLoc``. For a drag handle, a fine slider, a +sub-pixel-rendered target, or a high-DPI display, that integer rounding is the dominant +click-placement error. ``subpixel_match`` refines the peak to a fraction of a pixel by fitting +a parabola to the 3x3 score neighbourhood around the integer maximum (the standard NCC +sub-pixel method) — independently on x and y. + +It reuses ``visual_match._score_map`` (the full ``matchTemplate`` surface the public matchers +discard); no matching code is duplicated. The ``haystack`` is injectable; the analysis is +unit-testable on synthetic arrays. Imports no ``PySide6``. +""" +from dataclasses import asdict, dataclass +from typing import Any, Dict, List, Optional, Sequence, Tuple + +from je_auto_control.utils.visual_match.visual_match import _score_map + +ImageSource = Any + + +@dataclass(frozen=True) +class SubPixelMatch: + """A match whose centre is refined to sub-pixel precision.""" + + x: int + y: int + width: int + height: int + score: float + cx: float + cy: float + offset_x: float + offset_y: float + + @property + def center(self) -> List[int]: + """The integer centre point ``[x, y]`` (top-left + half size).""" + return [self.x + self.width // 2, self.y + self.height // 2] + + def to_dict(self) -> Dict[str, Any]: + """Return the match as a plain dict including the integer centre point.""" + data = asdict(self) + data["center"] = self.center + return data + + +def _parabola_offset(low: float, mid: float, high: float) -> float: + """Sub-sample offset in [-0.5, 0.5] of a peak from its two neighbours' scores.""" + denominator = low - 2.0 * mid + high + if abs(denominator) < 1e-12: + return 0.0 + offset = 0.5 * (low - high) / denominator + return max(-0.5, min(0.5, offset)) + + +def refine_peak(score_map, peak_xy: Tuple[int, int]) -> Tuple[float, float]: + """Return the sub-pixel ``(offset_x, offset_y)`` of a peak via 3x3 quadratic fit.""" + peak_x, peak_y = int(peak_xy[0]), int(peak_xy[1]) + height, width = score_map.shape + offset_x = offset_y = 0.0 + if 0 < peak_x < width - 1: + offset_x = _parabola_offset(float(score_map[peak_y, peak_x - 1]), + float(score_map[peak_y, peak_x]), + float(score_map[peak_y, peak_x + 1])) + if 0 < peak_y < height - 1: + offset_y = _parabola_offset(float(score_map[peak_y - 1, peak_x]), + float(score_map[peak_y, peak_x]), + float(score_map[peak_y + 1, peak_x])) + return offset_x, offset_y + + +def match_subpixel(template: ImageSource, *, haystack: Optional[ImageSource] = None, + region: Optional[Sequence[int]] = None, + method: str = "ccoeff_normed", + min_score: float = 0.0) -> Optional[SubPixelMatch]: + """Return the best match with a sub-pixel-refined centre, or ``None``. + + The integer top-left and score come from the correlation peak; ``cx`` / ``cy`` add the + quadratic-fit offset to the integer centre for fractional-pixel click placement. + """ + import cv2 + score_map, tmpl = _score_map(template, haystack, region=region, method=method) + if score_map is None: + return None + _, max_val, _, max_loc = cv2.minMaxLoc(score_map) + if max_val < min_score: + return None + peak_x, peak_y = int(max_loc[0]), int(max_loc[1]) + offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y)) + height, width = tmpl.shape[:2] + return SubPixelMatch(peak_x, peak_y, width, height, round(float(max_val), 4), + round(peak_x + width / 2.0 + offset_x, 3), + round(peak_y + height / 2.0 + offset_y, 3), + round(offset_x, 4), round(offset_y, 4)) diff --git a/test/unit_test/headless/test_subpixel_match_batch.py b/test/unit_test/headless/test_subpixel_match_batch.py new file mode 100644 index 00000000..9ce3da2e --- /dev/null +++ b/test/unit_test/headless/test_subpixel_match_batch.py @@ -0,0 +1,69 @@ +"""Headless tests for sub-pixel template-match refinement.""" +import pytest + +import je_auto_control as ac + +np = pytest.importorskip("numpy") +pytest.importorskip("cv2") + +from je_auto_control.utils.subpixel_match import ( # noqa: E402 + match_subpixel, refine_peak, +) + + +def _template(): + tmpl = np.zeros((24, 24), dtype=np.uint8) + tmpl[:, :12] = 200 + return tmpl + + +def _haystack(top, left): + hay = np.zeros((120, 160), dtype=np.uint8) + hay[top:top + 24, left:left + 24] = _template() + return hay + + +def test_refine_peak_symmetric_is_zero(): + smap = np.array([[0.0, 0.0, 0.0], + [0.0, 1.0, 0.0], + [0.0, 0.0, 0.0]], dtype=np.float32) + offset_x, offset_y = refine_peak(smap, (1, 1)) + assert abs(offset_x) < 1e-9 and abs(offset_y) < 1e-9 + + +def test_refine_peak_biased_offset_sign(): + # right neighbour higher than left → peak nudged toward +x + row = np.array([[0.2, 1.0, 0.6]], dtype=np.float32) + offset_x, offset_y = refine_peak(row, (1, 0)) + assert offset_x > 0.0 and abs(offset_y) < 1e-9 + + +def test_match_subpixel_locates_and_centres(): + match = match_subpixel(_template(), haystack=_haystack(20, 30), min_score=0.8) + assert match is not None + assert match.x == 30 and match.y == 20 + # exact alignment → offset ~0, cx ≈ integer centre + assert abs(match.cx - (30 + 12)) < 0.6 + assert abs(match.cy - (20 + 12)) < 0.6 + + +def test_match_subpixel_min_score_filters(): + blank = np.zeros((120, 160), dtype=np.uint8) + assert match_subpixel(_template(), haystack=blank, min_score=0.95) is None + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + assert "AC_match_subpixel" in set(ac.executor.known_commands()) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert "ac_match_subpixel" in names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert "AC_match_subpixel" in specs + + +def test_facade_exports(): + for name in ("match_subpixel", "refine_peak", "SubPixelMatch"): + assert hasattr(ac, name) and name in ac.__all__