Modern enterprises are drowning in repetitive desktop interactions that drain productivity and stifle innovation. While robotic process automation promised relief, brittle scripts break at the slightest UI change, forcing teams into endless maintenance cycles. OpenAdapt emerges as a fresh paradigm that fuses human demonstration with multimodal learning to create resilient automation agents. By recording a workflow once and distilling it into a self‑healing replay, the tool sidesteps the fragility of traditional selectors and XPath hacks. The result is a locally executed agent that adapts to minor layout shifts without redeployment, delivering near‑zero operational cost after the initial capture. This approach is especially compelling for knowledge‑worker tasks that involve navigating system settings, legacy internal portals, or web‑based dashboards where API access is unavailable or prohibitively expensive. In a market saturated with point‑solutions that either require deep coding expertise or lock users into proprietary clouds, OpenAdapt offers an open‑source, CLI‑driven alternative that puts control back into the hands of developers and automation enthusiasts. The following sections unpack how the project achieves this balance of simplicity, power, and transparency, and why it merits a closer look from anyone tasked with scaling desktop automation.
At its core, OpenAdapt is structured as a modular meta‑package that decouples concerns while presenting a unified command‑line interface. Installing the base package via pip install openadapt immediately provides the flagship flow compiler, enabling users to turn a recorded session into a deterministic bundle with a single command. Additional capabilities—such as low‑level capture, machine‑learning policy training, evaluation harnesses, and privacy filters—are shipped as optional sub‑packages that can be added on demand. This design mirrors the microservices trend in cloud native architectures, allowing teams to adopt only the pieces that solve their immediate problem and expand later as requirements evolve. Because each sub‑package follows semantic versioning and publishes independently on PyPI, dependency conflicts are minimized and upgrades can be performed piecemeal. The modularity also encourages community contributions: developers can experiment with novel capture backends or alternative reinforcement‑learning algorithms withoutforking the entire codebase. In practice, a data‑science team might start with just the capture and flow modules to prototype a proof‑of‑concept, then later integrate the evals package to benchmark accuracy across OS updates. By exposing a clear contract between components, OpenAdapt reduces the cognitive overhead of assembling a custom automation stack while preserving the flexibility to swap out implementations as better techniques emerge.
The creators of OpenAdapt distilled the automation lifecycle into three intuitive stages: Demonstrate, Learn, and Execute. During Demonstration, a human operator performs the target task once while the observability layer records screen frames, input events, and accessibility metadata. This raw trace becomes the ground truth that subsequent stages will emulate. In the Learn phase, the recorded trace is transformed into a policy that maps visual cues and UI element states to the appropriate actions. Rather than relying solely on prompt‑engineered large multimodal models, OpenAdapt conditions those models on the demonstrated trace, effectively teaching them “what to do next” by example. Finally, in the Execute phase, the learned policy is packaged into a self‑contained binary that can be run on any compatible machine without internet access or external API calls. The replay engine watches the live desktop, matches the current scene to the stored demonstration, and issues the corresponding clicks, keystrokes, or scrolls. Because the policy is grounded in concrete examples, it exhibits a degree of generalization that pure prompting lacks—when a button shifts a few pixels, the agent can still locate it via learned visual similarity rather than a brittle coordinate. This closed loop mirrors how humans acquire procedural knowledge: observe, abstract, then apply.
Capturing a reliable demonstration is the foundation of any learning‑based automation system, and OpenAdapt treats this step with the rigor of a scientific experiment. On macOS, the tool leverages the system’s accessibility APIs to obtain a pixel‑accurate rendering of each frame while simultaneously logging keyboard and mouse events through the Quartz event tap mechanism. Users must grant three specific permissions—Accessibility, Screen Recording, and Input Monitoring—to enable this depth of observation; a concise permissions guide walks newcomers through the required System Settings toggles. On Windows, the capture mechanism falls back to the UI Automation framework and requires the process to run with elevated privileges to intercept low‑level input signals. Linux support is under active development, utilizing the accessibility bus and X11/Wayland compositors to achieve comparable fidelity. Beyond merely recording what happens, OpenAdapt enriches the trace with semantic tags derived from the accessibility tree, labeling buttons, text fields, and menus with stable identifiers. This enriched trace becomes invaluable during the Learn phase, where the model can associate visual appearance with functional role, reducing ambiguity when the same visual pattern appears in different contexts. The result is a demonstration that is not just a video replay but a structured dataset ready for supervised learning.
Why do zero‑shot vision‑language models stumble on GUI tasks despite their impressive general‑purpose reasoning? The answer lies in the ambiguity of UI affordances: identical icons can represent different actions depending on the surrounding workflow, and textual labels are often truncated or localized. OpenAdapt confronts this challenge by conditioning the model on the specific demonstration rather than asking it to infer intent from a bare prompt. In effect, the system says “here is exactly how a human solved this problem; now replicate it under similar conditions.” This demo‑conditioned prompting has been validated on a controlled macOS benchmark comprising forty‑five System Settings tasks that share a common navigation entry point. When presented with only a textual description, a vanilla VLM achieved a first‑action accuracy of 46.7 %, barely better than random guessing for many of the multi‑step procedures. After augmenting the prompt with the demonstrated trajectory, accuracy jumped to 100 % across the board. A length‑matched control experiment showed that the gain could not be explained merely by providing more tokens; the improvement was semantic, rooted in the model’s ability to align its internal representation with the demonstrated policy. These findings confirm that the bottleneck in GUI automation is not model capacity but the grounding of abstract instructions in concrete, observable behavior.
Once the policy has been learned, OpenAdapt compiles it into a deterministic bundle that can be executed with negligible overhead. The bundle contains a lightweight inference engine, a cached feature extractor for the visual encoder, and a lookup table that maps observed states to the demonstrated actions. At runtime, the agent continuously samples the desktop, computes a similarity score between each cached demonstration frame and the live view, and selects the action associated with the highest‑scoring match. Because the similarity metric operates in a learned embedding space rather than raw pixel space, the agent tolerates modest variations in theme, font scaling, or window decoration without failing. Moreover, the replay mechanism incorporates simple heuristics—such as re‑attempting a click if an expected UI change does not appear within a timeout—to achieve self‑healing behavior. If a dialog appears unexpectedly, the agent can fall back to a generic recovery sequence defined in the policy, rather than aborting outright. All of this runs locally on the user’s machine, eliminating latency, data‑transfer costs, and dependence on third‑party APIs. For organizations that handle sensitive data or operate in air‑gapped environments, this on‑premise execution model is a decisive advantage over cloud‑based RPA platforms that require constant connectivity.
Trust but verify is a guiding principle for any automation that will be deployed at scale, and OpenAdapt equips teams with tools to inspect and gate compiled bundles before they reach production. After a flow is generated, the CLI offers a subcommand to render the bundle as a step‑by‑step storyboard, complete with screenshots, annotated action points, and confidence scores for each transition. Stakeholders can walk through this visual script to ensure that the agent behaves as intended and that no undesirable side effects—such as inadvertent data deletion or permission escalation—are lurking in the policy. Additionally, the evals sub‑package provides a harness for running the bundle against a suite of test scenarios, measuring metrics like success rate, mean time to completion, and robustness to UI perturbations. By integrating these checks into a continuous‑integration pipeline, teams can automatically reject builds that fall below a predefined quality threshold. Privacy‑conscious users can leverage the privacy sub‑package to blur or redact sensitive regions from the captured frames before they ever reach the training pipeline, ensuring that shared automation artifacts do not leak proprietary UI structures or confidential information.
The empirical evidence backing OpenAdapt’s approach is both striking and reproducible. In the aforementioned macOS System Settings benchmark, the baseline zero‑shot VLM struggled with tasks that required navigating nested menus or toggling obscure switches, often misclicking the first element and derailing the entire sequence. After incorporating demonstration‑conditioned prompting, the agent cleared every task with perfect first‑action accuracy, a feat that held across multiple runs and different hardware configurations. To rule out the possibility that the improvement was merely due to providing a longer prompt, researchers constructed a length‑matched control that appended irrelevant descriptive text to match the token count of the demonstration‑enhanced prompt. The control yielded only an 11.1 percentage‑point increase, confirming that the bulk of the gain stemmed from the semantic richness of the demonstration, not from sheer token volume. These results have been documented in a publicly available research thesis, which outlines the experimental protocol, statistical significance testing, and limitations such as the dependence on consistent application versions. The publication roadmap indicates upcoming work on cross‑task transfer learning and methods to reduce the amount of demonstration data needed for high accuracy, suggesting that the performance ceiling is still rising.
OpenAdapt’s architectural novelty becomes even clearer when contrasted with related efforts in the computer‑use agent (CUA) space. Notably, the OpenCUA framework, which garnered a NeurIPS 2025 Spotlight award from the XLANG Lab, repurposed OpenAdapt’s macOS accessibility capture code for its own data collection pipeline. However, OpenCUA uses the gathered demonstrations exclusively to train a model in an offline fashion and then relies on pure prompting at inference time, foregoing the demo‑conditioned runtime adaptation that defines OpenAdapt. As of the latest survey, no open‑source CUA framework publicly implements demo‑conditioned inference, making this capability a distinctive moat for OpenAdapt. This distinction translates into tangible operational benefits: agents built with OpenAdapt retain the ability to adapt to minor UI drift without retraining, whereas OpenCUA‑based agents would require a fresh training cycle whenever the target application undergoes a visual redesign. For enterprises that manage a portfolio of legacy applications with infrequent but inevitable UI updates, this reduces the total cost of ownership dramatically. Moreover, the modular nature of OpenAdapt permits teams to swap in alternative perception backends—such as a transformer‑based visual encoder or a hybrid symbolic‑neural matcher—while preserving the demo‑conditioned inference contract, something that monolithic competitors struggle to offer.
The modular meta‑package philosophy of OpenAdapt delivers practical advantages that extend beyond theoretical elegance. By installing only the core package, a developer can immediately experiment with the flow command to compile a demonstration into a runnable binary, making the entry point exceptionally low‑friction. If the project later demands rigorous evaluation, adding openadapt‑evals via pip provides a ready‑made benchmarking suite without disturbing the existing workflow. Likewise, organizations concerned about GDPR or internal data‑handling policies can incorporate openadapt‑privacy to automatically blur or redact sensitive regions from the captured frames before they ever reach the training pipeline. Because each sub‑package adheres to a well‑defined interface—typically accepting and returning standardized protobuf or JSON structures—advanced users can even replace a module with a custom implementation that satisfies the same contract, enabling research‑grade experimentation. This à la carte approach mirrors the success of ecosystems like Node.js/npm and Python’s scientific stack, where developers compose solutions from tightly scoped libraries rather than wrestling with monolithic distributions. For teams that anticipate evolving needs—starting with a simple proof‑of‑concept, scaling to a production‑grade fleet, and eventually exploring novel learning algorithms—OpenAdapt’s modularity provides a clear upgrade path that avoids disruptive rewrites.
Getting started with OpenAdapt is deliberately streamlined to encourage adoption across skill levels. After ensuring Python 3.10 or newer is installed, a single pip install openadapt pulls in the base CLI and the flow compiler. To record a demonstration, users invoke openadapt capture –output demo.trace and then perform the desired actions; the tool will prompt for the necessary OS‑level permissions the first time it runs. Once the trace is saved, the command openadapt flow build –trace demo.trace –bundle my_automation produces a self‑executing artifact that can be distributed like any other binary. Advanced users can enhance the pipeline by installing supplemental packages—for instance, pip install openadapt‑ml opens access to the policy‑training module that fine‑tunes a vision‑language model on the collected trace, while openadapt‑evals enables side‑by‑side comparison of multiple policy variants. All source code is released under the permissive MIT license, encouraging commercial use and modification without legal encumbrance. The vibrant Discord community and comprehensive documentation provide troubleshooting tips, example workflows for popular applications ranging from Adobe Creative Suite to internal ERP systems, and guidance on extending the capture backend to support virtualized environments or remote desktop protocols.
Looking ahead, organizations should view OpenAdapt not as a one‑off tool but as a strategic component of a broader automation roadmap. Begin by identifying high‑frequency, UI‑heavy tasks that lack API access—such as periodic report generation in legacy software, compliance checklist completion, or data migration between incompatible systems. Run a pilot demonstration, evaluate the resulting bundle with the built‑in evals harness, and iterate until the success rate meets your threshold for production deployment. Keep the captured traces under version control alongside the policy bundles; this makes it trivial to regenerate the agent when a target application updates its UI, ensuring that your automation investment remains future‑proof. Participate in the open‑source discussion channels to stay abreast of new sub‑packages, contribute improvements that address domain‑specific challenges, and benefit from the collective wisdom of a growing practitioner base. As the line between conversational agents and desktop automation continues to blur, tools that ground AI in demonstrable, observable behavior—like OpenAdapt—will become indispensable for building trustworthy, cost‑effective, and scalable workflows. Now is the moment to experiment, to measure, and to integrate a solution that turns human expertise into lasting, self‑healing software automation.