Skip to content

Commit 19707e0

Browse files
committed
Document the 2026-06-17 automation toolkit additions
Add a "What's new (2026-06-17)" section to README.md and the zh-TW / zh-CN READMEs covering this round's 30+ primitives (human-like input, vision, flow/variable commands, composite + cron triggers, window capture/layout/snap, action-file signing + encryption, recoverable deletion, annotation, notifications, Recording-Editor undo) plus the fixes. Add the v4 reference page and wire it into the Eng/Zh toctrees.
1 parent 25b5275 commit 19707e0

6 files changed

Lines changed: 324 additions & 0 deletions

File tree

README.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-17)](#whats-new-2026-06-17)
1617
- [What's new (2026-06)](#whats-new-2026-06)
1718
- [What's new (2026-05)](#whats-new-2026-05)
1819
- [Features](#features)
@@ -56,6 +57,61 @@
5657

5758
---
5859

60+
## What's new (2026-06-17)
61+
62+
Thirty-plus automation primitives across input realism, vision, flow
63+
control, triggers, window management, and file security — plus recoverable
64+
deletion and an editor undo. Each ships with a headless API, an `AC_*`
65+
executor command, and a visual Script Builder entry; vision and window
66+
features keep their geometry / IO operations injectable so the logic is
67+
fully unit-tested. Full reference page:
68+
[`docs/source/Eng/doc/new_features/v4_features_doc.rst`](docs/source/Eng/doc/new_features/v4_features_doc.rst).
69+
70+
**Human-like input**
71+
- **Human-like mouse motion**`move_mouse_humanized` walks an eased, bowed cubic-Bezier path with optional overshoot + jitter, deterministic by `seed` (`AC_human_move`).
72+
- **Human-like typing**`type_text_humanized` types character by character with a jittered per-key delay and optional "thinking" pauses, seedable (`AC_human_type`).
73+
74+
**Vision**
75+
- **VLM natural-language assertion**`assert_by_description` asks a vision-language model whether the screen matches a description; the `verify()` companion to `locate_by_description` (`AC_assert_vlm`).
76+
- **Scroll-to-find**`scroll_until_visible` scrolls a direction until a template image or OCR text appears, or the budget runs out (`AC_scroll_to_find`).
77+
- **Region colour stats**`region_color_stats` reports a region's average + dominant colour and that colour's pixel fraction (`AC_region_color_stats`).
78+
- **QR reading**`read_qr_codes` decodes QR codes in a screen region via OpenCV's `QRCodeDetector` (no new dependency) (`AC_read_qr`).
79+
80+
**Flow control & variables**
81+
- **Reusable macros**`AC_define_macro` / `AC_call_macro`: define a named, parameterised action sub-routine once and call it with `${arg}` bindings.
82+
- **In-process parallel**`AC_parallel` runs branch action lists concurrently, each on an isolated executor so branches never race on shared variables.
83+
- **Performance-budget assertion**`assert_duration` / `AC_assert_duration` fails a block that takes longer than a millisecond budget.
84+
- **Read into a variable**`AC_ocr_to_var`, `AC_shell_to_var`, `AC_read_file_to_var`, `AC_http_to_var` (body or dotted JSON path), `AC_now_to_var` (strftime), `AC_random_to_var` (seeded int / float / choice).
85+
- **Transform a variable**`AC_transform_var`: upper / lower / strip / title / replace / regex-extract / slice, in place or into a new variable.
86+
- **Assert a variable**`assert_variable` / `AC_assert_var`: eq / ne / lt / gt / contains / regex through the assertion DSL.
87+
88+
**Triggers & smart waits**
89+
- **Composite triggers**`AllOfTrigger` / `AnyOfTrigger` / `SequenceTrigger` combine any existing trigger by boolean AND / OR / ordered sequence.
90+
- **Cron trigger**`CronTrigger` fires on a five-field cron expression, composing with the boolean triggers (e.g. "at 09:00 *and* only if the image is on screen").
91+
- **More smart waits**`wait_until_clipboard_changes` (`AC_wait_clipboard_change`) and `wait_until_window_closed` (`AC_wait_window_closed`).
92+
93+
**Window management**
94+
- **Per-window capture**`capture_window` screenshots exactly a window's bounds by title (`AC_capture_window`).
95+
- **Layout save / restore**`save_window_layout` / `restore_window_layout` snapshot every window's position to JSON and move them all back later (`AC_save_window_layout` / `AC_restore_window_layout`).
96+
- **Snap / tile**`snap_window` moves a window to a screen half, quarter, or maximize (`AC_snap_window`).
97+
98+
**File security & safety**
99+
- **Action-file signing**`sign_action_file` / `verify_action_file` (HMAC-SHA256 sidecar); `execute_files` can require signatures via `JE_AUTOCONTROL_REQUIRE_SIGNED_ACTIONS` (`AC_sign_action_file` / `AC_verify_action_file`).
100+
- **Action-file encryption**`encrypt_action_file` / `decrypt_action_file` (Fernet, AES-128-CBC + HMAC) (`AC_encrypt_action_file` / `AC_decrypt_action_file`).
101+
- **Recoverable deletion**`move_to_trash` sends a file to the OS recycle bin (Win32 `SHFileOperation` undo flag / macOS Trash / Linux XDG trash, preferring `send2trash`) (`AC_move_to_trash`).
102+
103+
**Reporting & notifications**
104+
- **Screenshot annotation**`annotate_screenshot` draws labelled boxes / translucent highlights / arrows / text onto a capture (`AC_annotate_screenshot`).
105+
- **Desktop notifications**`notify` shows a cross-platform toast (notify-send / osascript / PowerShell), injection-safe (`AC_notify`).
106+
107+
**GUI**
108+
- **Recording Editor undo** — every edit is snapshotted; **Ctrl+Z** (and an Undo button) restore the prior state.
109+
- **Triggers tab** — "Combine selected" wraps chosen triggers into a composite; new **Cron** trigger type.
110+
- **Assertions tab** — new **VLM** ("screen matches description") assertion kind.
111+
- Every new `AC_*` command appears in the visual **Script Builder**.
112+
113+
**Fixes** — repaired the USB-passthrough approval-prompt crash on PySide6 6.11.1 (`Q_ARG(object)` → a Qt signal), eight stale / broken GUI + USB tests, two lost exception chains, and brought thirteen functions back under the cyclomatic-complexity gate.
114+
59115
## What's new (2026-06)
60116

61117
Nine additions that turn the automation primitives into a full **QA / test
@@ -142,6 +198,7 @@ sense) a Qt GUI tab. Full reference page:
142198
## Features
143199

144200
- **QA / Test Framework** — assertion DSL (`assert_text` / `_image` / `_pixel` / `_window` + audio/video assertions), data-driven execution (CSV / JSON / SQLite / Excel → `AC_for_each_row`), a scored `run_suite` with setup/teardown/tags, JUnit + Allure report output, flaky-test detection with auto-quarantine, accessibility / i18n auditing (missing labels, WCAG contrast, truncation), and a parallel mobile device matrix. See [What's new (2026-06)](#whats-new-2026-06)
201+
- **Automation toolkit** — human-like mouse motion + typing, VLM / variable / duration assertions, reusable macros + in-process parallel blocks, composite + cron triggers, read-into-a-variable commands (OCR / shell / file / HTTP / time / random), variable transforms, scroll-to-find, region colour stats, QR reading, per-window capture / layout save-restore / snap, screenshot annotation, desktop notifications, action-file signing + encryption, recoverable (recycle-bin) deletion, and Recording-Editor undo. See [What's new (2026-06-17)](#whats-new-2026-06-17)
145202
- **Mouse Automation** — move, click, press, release, drag, and scroll with precise coordinate control
146203
- **Keyboard Automation** — press/release individual keys, type strings, hotkey combinations, key state detection
147204
- **Image Recognition** — locate UI elements on screen using OpenCV template matching with configurable threshold

README/README_zh-CN.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-17)](#本次更新-2026-06-17)
1516
- [本次更新 (2026-06)](#本次更新-2026-06)
1617
- [本次更新 (2026-05)](#本次更新-2026-05)
1718
- [功能特性](#功能特性)
@@ -55,6 +56,59 @@
5556

5657
---
5758

59+
## 本次更新 (2026-06-17)
60+
61+
新增 30+ 个自动化原语,涵盖输入拟真、视觉、流程控制、触发器、窗口管理与文件安全,
62+
另加“可还原删除(回收站)”与“编辑器 Undo”。每个都附带 headless API、`AC_*` 执行器
63+
指令,以及可视化脚本构建器项;视觉与窗口功能的 geometry / IO 操作皆可注入,逻辑完全
64+
单元测试。完整参考页:
65+
[`docs/source/Eng/doc/new_features/v4_features_doc.rst`](../docs/source/Eng/doc/new_features/v4_features_doc.rst)
66+
67+
**拟人化输入**
68+
- **拟人化鼠标移动**`move_mouse_humanized`:eased bezier 曲线 + overshoot + jitter,seed 可重现(`AC_human_move`)。
69+
- **拟人化打字**`type_text_humanized`:每字随机微延迟 + 偶尔停顿,seed 可重现(`AC_human_type`)。
70+
71+
**视觉**
72+
- **VLM 自然语言断言**`assert_by_description`:用 VLM 判断画面是否符合描述(`AC_assert_vlm`)。
73+
- **滚动找元素**`scroll_until_visible`:往某方向滚动直到图/文字出现(`AC_scroll_to_find`)。
74+
- **区域颜色统计**`region_color_stats`:平均色 + 主色 + 占比(`AC_region_color_stats`)。
75+
- **读 QR code**`read_qr_codes`:OpenCV QRCodeDetector 从屏幕区域解 QR(`AC_read_qr`)。
76+
77+
**流程控制与变量**
78+
- **可重用宏**`AC_define_macro` / `AC_call_macro`:具名、带参数的动作子程序,`${arg}` 绑定。
79+
- **同进程并行**`AC_parallel`:多分支并行,各自独立 executor,变量不互相 race。
80+
- **性能预算断言**`assert_duration` / `AC_assert_duration`:超过毫秒预算就判失败。
81+
- **读进变量**`AC_ocr_to_var``AC_shell_to_var``AC_read_file_to_var``AC_http_to_var`(body 或 dotted JSON path)、`AC_now_to_var`(strftime)、`AC_random_to_var`(seeded)。
82+
- **变量转换**`AC_transform_var`:upper/lower/strip/title/replace/regex 取出/slice。
83+
- **断言变量**`assert_variable` / `AC_assert_var`:eq/ne/lt/gt/contains/regex。
84+
85+
**触发器与智能等待**
86+
- **复合触发器**`AllOfTrigger` / `AnyOfTrigger` / `SequenceTrigger`:布尔 AND/OR/顺序组合任何现有触发器。
87+
- **Cron 触发器**`CronTrigger`:五字段 cron 排程,每分钟最多一次,可与布尔触发器组合。
88+
- **更多智能等待**`wait_until_clipboard_changes``AC_wait_clipboard_change`)、`wait_until_window_closed``AC_wait_window_closed`)。
89+
90+
**窗口管理**
91+
- **单一窗口截图**`capture_window`:依标题截出该窗口(`AC_capture_window`)。
92+
- **布局存/还原**`save_window_layout` / `restore_window_layout`:快照所有窗口位置 → JSON → 一键还原。
93+
- **贴齐/分割**`snap_window`:左/右半、四角、最大化(`AC_snap_window`)。
94+
95+
**文件安全**
96+
- **动作文件签名**`sign_action_file` / `verify_action_file`(HMAC-SHA256);`execute_files` 可在 `JE_AUTOCONTROL_REQUIRE_SIGNED_ACTIONS` 下强制验签。
97+
- **动作文件加密**`encrypt_action_file` / `decrypt_action_file`(Fernet)。
98+
- **可还原删除**`move_to_trash`:送进操作系统回收站(`AC_move_to_trash`)。
99+
100+
**报告与通知**
101+
- **截图标注**`annotate_screenshot`:画带标签方框/高亮/箭头/文字(`AC_annotate_screenshot`)。
102+
- **桌面通知**`notify`:跨平台 toast,injection-safe(`AC_notify`)。
103+
104+
**GUI**
105+
- **录制编辑器 Undo** — 每个编辑都快照;**Ctrl+Z** 与 Undo 按钮还原。
106+
- **触发器页** — “Combine selected”把选中的触发器组成复合;新增 **Cron** 类型。
107+
- **断言页** — 新增 **VLM** 断言类型。
108+
- 所有新 `AC_*` 指令都在可视化 **脚本构建器** 可用。
109+
110+
**修复** — 修了 PySide6 6.11.1 上 USB 授权弹窗的 `Q_ARG(object)` crash、8 个 stale/损坏的测试、2 个丢失异常链,并把 13 个函数拉回 CC≤10。
111+
58112
## 本次更新 (2026-06)
59113

60114
新增 9 个功能,把自动化原语升级成一套完整的 **QA / 测试框架**:验证画面状态、

README/README_zh-TW.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-17)](#本次更新-2026-06-17)
1516
- [本次更新 (2026-06)](#本次更新-2026-06)
1617
- [本次更新 (2026-05)](#本次更新-2026-05)
1718
- [功能特色](#功能特色)
@@ -55,6 +56,59 @@
5556

5657
---
5758

59+
## 本次更新 (2026-06-17)
60+
61+
新增 30+ 個自動化原語,涵蓋輸入擬真、視覺、流程控制、觸發器、視窗管理與檔案安全,
62+
另加「可還原刪除(資源回收桶)」與「編輯器 Undo」。每個都附帶 headless API、`AC_*`
63+
執行器指令,以及視覺化腳本建構器項目;視覺與視窗功能的 geometry / IO 操作皆可注入,
64+
邏輯完全單元測試。完整參考頁:
65+
[`docs/source/Eng/doc/new_features/v4_features_doc.rst`](../docs/source/Eng/doc/new_features/v4_features_doc.rst)
66+
67+
**擬人化輸入**
68+
- **擬人化滑鼠移動**`move_mouse_humanized`:eased bezier 曲線 + overshoot + jitter,seed 可重現(`AC_human_move`)。
69+
- **擬人化打字**`type_text_humanized`:每字隨機微延遲 + 偶爾停頓,seed 可重現(`AC_human_type`)。
70+
71+
**視覺**
72+
- **VLM 自然語言斷言**`assert_by_description`:用 VLM 判斷畫面是否符合描述(`AC_assert_vlm`)。
73+
- **捲動找元素**`scroll_until_visible`:往某方向捲動直到圖/文字出現(`AC_scroll_to_find`)。
74+
- **區域顏色統計**`region_color_stats`:平均色 + 主色 + 占比(`AC_region_color_stats`)。
75+
- **讀 QR code**`read_qr_codes`:OpenCV QRCodeDetector 從螢幕區域解 QR(`AC_read_qr`)。
76+
77+
**流程控制與變數**
78+
- **可重用巨集**`AC_define_macro` / `AC_call_macro`:具名、帶參數的動作子程序,`${arg}` 綁定。
79+
- **同進程平行**`AC_parallel`:多分支並行,各自獨立 executor,變數不互相 race。
80+
- **效能預算斷言**`assert_duration` / `AC_assert_duration`:超過毫秒預算就判失敗。
81+
- **讀進變數**`AC_ocr_to_var``AC_shell_to_var``AC_read_file_to_var``AC_http_to_var`(body 或 dotted JSON path)、`AC_now_to_var`(strftime)、`AC_random_to_var`(seeded)。
82+
- **變數轉換**`AC_transform_var`:upper/lower/strip/title/replace/regex 取出/slice。
83+
- **斷言變數**`assert_variable` / `AC_assert_var`:eq/ne/lt/gt/contains/regex。
84+
85+
**觸發器與智慧等待**
86+
- **複合觸發器**`AllOfTrigger` / `AnyOfTrigger` / `SequenceTrigger`:布林 AND/OR/順序組合任何現有觸發器。
87+
- **Cron 觸發器**`CronTrigger`:五欄 cron 排程,每分鐘最多一次,可與布林觸發器組合。
88+
- **更多智慧等待**`wait_until_clipboard_changes``AC_wait_clipboard_change`)、`wait_until_window_closed``AC_wait_window_closed`)。
89+
90+
**視窗管理**
91+
- **單一視窗截圖**`capture_window`:依標題截出該視窗(`AC_capture_window`)。
92+
- **版面存/還原**`save_window_layout` / `restore_window_layout`:快照所有視窗位置 → JSON → 一鍵還原。
93+
- **貼齊/分割**`snap_window`:左/右半、四角、最大化(`AC_snap_window`)。
94+
95+
**檔案安全**
96+
- **動作檔簽章**`sign_action_file` / `verify_action_file`(HMAC-SHA256);`execute_files` 可在 `JE_AUTOCONTROL_REQUIRE_SIGNED_ACTIONS` 下強制驗章。
97+
- **動作檔加密**`encrypt_action_file` / `decrypt_action_file`(Fernet)。
98+
- **可還原刪除**`move_to_trash`:送進作業系統資源回收桶(`AC_move_to_trash`)。
99+
100+
**報告與通知**
101+
- **截圖標註**`annotate_screenshot`:畫帶標籤方框/高亮/箭頭/文字(`AC_annotate_screenshot`)。
102+
- **桌面通知**`notify`:跨平台 toast,injection-safe(`AC_notify`)。
103+
104+
**GUI**
105+
- **錄製編輯器 Undo** — 每個編輯都快照;**Ctrl+Z** 與 Undo 按鈕還原。
106+
- **觸發器頁** — 「Combine selected」把選取的觸發器組成複合;新增 **Cron** 型別。
107+
- **斷言頁** — 新增 **VLM** 斷言型別。
108+
- 所有新 `AC_*` 指令都在視覺化 **腳本建構器** 可用。
109+
110+
**修正** — 修了 PySide6 6.11.1 上 USB 授權彈窗的 `Q_ARG(object)` crash、8 個 stale/壞掉的測試、2 個遺失例外鏈,並把 13 個函式拉回 CC≤10。
111+
58112
## 本次更新 (2026-06)
59113

60114
新增 9 個功能,把自動化原語升級成一套完整的 **QA / 測試框架**:驗證畫面狀態、

0 commit comments

Comments
 (0)