Skip to content

feat: add drag command and remote WS host binding support - #78

Open
icefairy wants to merge 4 commits into
Tencent:mainfrom
icefairy:pr-to-upstream
Open

feat: add drag command and remote WS host binding support#78
icefairy wants to merge 4 commits into
Tencent:mainfrom
icefairy:pr-to-upstream

Conversation

@icefairy

Copy link
Copy Markdown
   ## Summary

   This PR adds two major features to BrowserSkill:

   ### 1. bsk drag command for sliders & drag-and-drop
   - New `bsk drag` command supporting three modes: element + delta, coordinate + delta, and absolute path
 trajectory
   - Human-like trajectory generation with randomized grab point for realistic interactions
   - Useful for slider controls, cross-origin iframe captchas (e.g. Aliyun nc_1_nocaptcha), and precise multi-stop
 paths
   - Includes unit tests and full protocol/CLI implementation

   ### 2. Remote WS host binding support (`0.0.0.0`)
   - New `--ws-host` flag and `BSK_DAEMON_WS_HOST` env var for binding the daemon WebSocket to a custom host
   - Enables remote access from outside the container/host machine
   - Updated daemon startup logic and IPC communication

   ### Files changed
   - **Extension:** `interaction.ts` (new drag handler), `dispatcher.ts`, `background.ts`, `types.ts`, tests
   - **CLI:** `interaction.rs` (new drag subcommand), `daemon.rs` (ws-host flag), `mod.rs`, `start.rs`
   - **Protocol:** `method.rs`, `tools/interaction.rs`
   - **Docs:** `SKILL.md` updated with drag command documentation

icefairy added 4 commits July 31, 2026 16:12
Allows the WebSocket server to bind to 0.0.0.0 so the browser
extension can connect from another machine. Default remains
127.0.0.1 for local-only setups.
- New tool.drag RPC (DragParams/DragResult): three modes
  * element/ref + dx/dy delta
  * viewport coordinate + delta (works in cross-origin iframes, e.g. Aliyun nc_1_nocaptcha slider)
  * absolute path via repeatable --point
- Dispatches trusted CDP mouse events (mousePressed -> interpolated mouseMoved xN -> mouseReleased)
  with ease-in/out + jitter so slider CAPTCHA backends can't fingerprint a linear drag
- handleDrag mirrors handleClick's Agent Window overlay bypass
- BSK_NO_OVERLAY build flag: automation-only browsers can hide the on-page control mask
  (its capture layer swallows CDP mouse events)
- CLI: bsk drag [TARGET] --dx/--dy | --from-x/--from-y | --point
- SKILL.md docs for the three modes
- buildHumanDragPath: smooth bounded perpendicular random-walk
  (low-pass filtered, ±3px) + ease-in-out, so the mouse path
  wobbles like a hand instead of a perfect straight line
- element drags now grab a random point in the middle 40-60% of
  the element box instead of the exact centroid (CAPTCHA bots
  press dead-centre)
- 3 unit tests for trajectory shape (bounded wobble, axis
  independence, endpoint accuracy)
# Conflicts:
#	apps/extension/src/tools/dispatcher.ts
#	crates/bsk-cli/src/cli/interaction.rs
#	crates/bsk-cli/src/cli/mod.rs
@BB-fat

BB-fat commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Thanks @icefairy for this substantial contribution — the drag command design (three modes, human-like trajectories) covers slider/captcha scenarios nicely, and the full-chain wiring is thorough. Before we can merge, there are 4 blockers to fix:

🔴 1. Flaky tests: buildHumanDragPath leaves an unconverged ±3px random offset at the path end, but the test asserts the endpoint within ±1px — we reproduced ~55% failure rate locally (10k Monte Carlo runs), so frontend CI will go red. Same root cause lets the release point drift up to 3px from the target (slider captchas are sensitive to that) and makes DragResult.to_x/to_y ≠ from+dx/dy. Fix: zero/taper the offsets at both ends of the trajectory and land the release exactly on the target.

🔴 2. Conflicts with main (mergeable: dirty): crates/bsk-cli/src/daemon/start.rs collides with the merged #77 (daemon auto-update). Needs a rebase.

🔴 3. SKILL.md out of sync: only crates/bsk-cli/skill/SKILL.md is updated; the root skill/SKILL.md is not (they are byte-identical on main). Also the PR's SKILL.md has a duplicated ### Observation heading (lines ~167-168).

🔴 4. Dangling docs: SKILL.md documents bsk screenshot --full-page and types.ts adds ScreenshotParams.full_page, but nothing implements it (CLI/extension/protocol untouched) — looks like residue from another branch.

Smaller notes (🟡): --ws-host 0.0.0.0 binds without auth and without warning (Origin check only accepts chrome-extension:// shapes) — please add a prominent warning in help/docs, and fail loudly instead of silently falling back to 127.0.0.1 when the host can't resolve; CLI uses || for coordinate-mode detection while the extension requires pairs, so a lone --from-x silently drops the target; [v2-DRAG] debug marker left in dispatcher error message; element-mode without dx/dy should be invalid_params; dump-schema.rs registration missing for DragParams/DragResult; consider splitting or documenting the BSK_NO_OVERLAY build flag.

Happy to re-review once these are addressed! 功能方向很好,修掉这 4 个卡点(尤其 flaky 测试和 rebase)后欢迎继续推进。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants