This video introduces Sikuli, an interactive technique
that uses screenshots to search and automate
graphical user interfaces.
Sikuli Search allows users to search a documentation database
using screenshots of interface elements, like this dialog box.
User draws a rectangle around the dialog box
to capture it's screenshot and submit it to Sikuli.
Sikuli currently searches a database
of more than 50,000 screenshots from over 100 computer books.
It then returns a list of pages containing the screenshots
of the same dialog box.
User can browse and select a page
to read the details about this dialog box.
Sikuli Script allows users to write simple scripts
to automate GUI interaction by screenshots.
Consider this bus tracking interface.
With Sikuli Script, a user can draw a rectangle around the bus
symbol to take it screenshot and select a neighborhood
on the map to watch for the bus arrival,
also by taking a screenshot.
After taking the screenshots, the user
can use them directly in a statement asking Sikuli
to find the neighborhood on the map,
look inside, and find the bus symbol.
If such a symbol is found, a pop-up window
can be displayed to notify the user the bus has arrived.
Sikuli Script supports many visual commands.
It can be used to click on visual targets,
to drag a visual target onto another visual target, to type
a string in a visual target, in this case, the Google Search
box in the web browser.
With Sikuli, what you see is what you search
and what you see is what you script.