web infra dev/midscene

English | 简体中文

<strong>Official Website</strong>: <a href="https://midscenejs.com/">https://midscenejs.com/</a>

<a href="https://trendshift.io/repositories/12524" target="_blank"><img src="https://trendshift.io/api/badge/repositories/12524" alt="web-infra-dev%2Fmidscene | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>

📣 Midscene Skills is here!

Use Midscene Skills to control any platform with OpenClaw

Showcases

• Web Automation - Automatically register the GitHub form in a web browser and pass all field validations
• iOS Automation - Meituan coffee order
• iOS Automation - Auto-like the first @midscene_ai tweet
• Android Automation - DCar: Xiaomi SU7 specs
• Android Automation - Booking a hotel for Christmas
• robotic arm + vision + voice for in-vehicle testing

💡 Why Midscene

Most UI automation — including AI tools that read the DOM or the accessibility tree — depends on page structure. That structure is fragile and incomplete: selectors break on every refactor, elements without semantic markup (icon-only buttons, custom controls, <canvas>) are invisible to it, native apps and cross-origin iframes are out of reach, and it cannot tell whether something actually looks right. Midscene works from the screenshot alone, and you describe each step in natural language:

• Less maintenance — no selectors to chase when the UI changes.
• Reach every element and surface — if a human can see it, Midscene can target it, even with no semantic annotations, on <canvas>, native apps, and cross-origin iframes.
• Assert what users actually see — verify colors, highlights, layout, and rendered state, not just whether a DOM node exists.
• Two ways to test — add Midscene to your Playwright / Vitest suite, or let an AI agent test autonomously via Skills.

Midscene is built for UI testing first, but the same vision-driven engine handles any UI automation task.

💡 What you can automate

Midscene works anywhere you can take a screenshot — web browsers, Android, iOS, HarmonyOS, desktop apps, and any custom interface — all through one API. Write automation with the JavaScript SDK or in YAML, hand it to AI agents via Skills, and look up every method (aiAct, aiQuery, aiAssert, and more) in the API reference.

🚀 Get started

• Write your first script in a few minutes — Quick start.
• No code? Try Midscene on any web page with the Chrome extension.
• Other platforms — getting-started guides for Android, iOS, HarmonyOS, and desktop.

✨ Driven by Multimodal Models

Midscene is all-in on pure vision for UI actions: element localization is based on screenshots only. It runs on multimodal models with strong UI localization, such as Qwen3.x, Doubao-Seed-2.1, GLM-4.6V, gemini-3.5-flash, and UI-TARS, including open-source options you can self-host. For data extraction and page understanding, you can still opt in to include DOM when needed.

Read more about Model Strategy.

📄 Resources

• Documentation: https://midscenejs.com
• Sample projects: midscene-example
• API reference: https://midscenejs.com/api

🤝 Community

• Discord
• Follow us on X
• Lark Group(飞书交流群)

🌟 Awesome Midscene

Community projects that extend Midscene.js capabilities:

• midscene-ios - iOS Mirror automation support for Midscene
• midscene-pc - PC operation device for Windows, macOS, and Linux
• midscene-pc-docker - Docker image with Midscene-PC server pre-installed
• Midscene-Python - Python SDK for Midscene automation
• midscene-java by @Master-Frank - Java SDK for Midscene automation
• midscene-java by @alstafeev - Java SDK for Midscene automation

📝 Credits

We would like to thank the following projects:

• Rsbuild and Rslib for the build tool.
• UI-TARS for the open-source agent model UI-TARS.
• Qwen-VL for the open-source multimodal model Qwen-VL.
• scrcpy and yume-chan allow us to control Android devices with browser.
• appium-adb for the javascript bridge of adb.
• appium-webdriveragent for the javascript operate XCTest。
• YADB for the yadb tool which improves the performance of text input.
• libnut-core for the cross-platform native keyboard and mouse control.
• Puppeteer for browser automation and control.
• Playwright for browser automation and control and testing.

📖 Citation

If you use Midscene.js in your research or project, please cite:

@software{Midscene.js,
  author = {Xiao Zhou, Tao Yu, YiBing Lin},
  title = {Midscene.js: Your AI Operator for Web, Android, iOS, Automation & Testing.},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/web-infra-dev/midscene}
}

✨ Star History

📝 License

Midscene.js is MIT licensed.