Patent Number: 8,793,578

Title: Automating execution of arbitrary graphical interface applications

Abstract: Interaction with an application interface is monitored and an image of the interface prior to a user interaction is compared with an image after the user interaction to identify a region of interaction. The comparison may be carried out by detecting that a threshold change has occurred using a computationally inexpensive algorithm and then drawing boundaries for the region using a more computationally intensive algorithm. A marker in a UI evolution timeline is associated with an image of the region and coordinates of the interaction normalized and relative to the region. An interaction interval and a profile of change are also stored in association with the marker. For playback, a script can be constructed from the timeline to control automated execution of the application by matching a current interval and profile to the interval and profile for a recorded interaction, and automatically applying that interaction to the current user interface.

Inventors: Mounty; Daniel G. (Sunnyvale, CA), O'Connor; Michael B. (San Francisco, CA)

Assignee: International Business Machines Corporation

International Classification: G06F 3/00 (20060101); G06F 9/44 (20060101)

Expiration Date: 7/29/12018