Testing Graphical User Interfaces¶
In this chapter, we explore how to generate tests for Graphical User Interfaces (GUIs), abstracting from our previous examples on Web testing. Building on general means to extract user interface elements and activate them, our techniques generalize to arbitrary graphical user interfaces, from rich Web applications to mobile apps, and systematically explore user interfaces through forms and navigation elements.
Prerequisites
- We build on the Web server introduced in the chapter on Web testing.
Synopsis¶
To use the code provided in this chapter, write
>>> from fuzzingbook.GUIFuzzer import <identifier>
and then make use of the following features.
Note: The examples in this section only work after the rest of the cells have been executed.
The function start_webdriver() starts a headless Web browser in the background and returns a GUI driver as handle for further communication.
gui_driver = start_webdriver()
The geckodriver version (0.34.0) detected in PATH at /Users/zeller/bin/geckodriver might not be compatible with the detected firefox version (144.09); currently, geckodriver 0.36.0 is recommended for firefox 144.*, so it is advised to delete the driver in PATH and retry
We let the browser open the URL of the server we want to investigate (in this case, the vulnerable server from the chapter on Web fuzzing) and obtain a screenshot.
gui_driver.get(httpd_url)
Image(gui_driver.get_screenshot_as_png())
The GUICoverageFuzzer class explores the user interface and builds a grammar that encodes all states as well as the user interactions required to move from one state to the next. It is paired with a GUIRunner which interacts with the GUI driver.
gui_fuzzer = GUICoverageFuzzer(gui_driver)
gui_runner = GUIRunner(gui_driver)
The explore_all() method extracts all states and all transitions from a Web user interface.
gui_fuzzer.explore_all(gui_runner)
The grammar embeds a finite state automation and is best visualized as such.
fsm_diagram(gui_fuzzer.grammar)
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:76, in run_check(cmd, input_lines, encoding, quiet, **kwargs) 75 kwargs['stdout'] = kwargs['stderr'] = subprocess.PIPE ---> 76 proc = _run_input_lines(cmd, input_lines, kwargs=kwargs) 77 else: File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:96, in _run_input_lines(cmd, input_lines, kwargs) 95 def _run_input_lines(cmd, input_lines, *, kwargs): ---> 96 popen = subprocess.Popen(cmd, stdin=subprocess.PIPE, **kwargs) 98 stdin_write = popen.stdin.write File /opt/homebrew/Cellar/python@3.13/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:1039, in Popen.__init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask, pipesize, process_group) 1036 self.stderr = io.TextIOWrapper(self.stderr, 1037 encoding=encoding, errors=errors) -> 1039 self._execute_child(args, executable, preexec_fn, close_fds, 1040 pass_fds, cwd, env, 1041 startupinfo, creationflags, shell, 1042 p2cread, p2cwrite, 1043 c2pread, c2pwrite, 1044 errread, errwrite, 1045 restore_signals, 1046 gid, gids, uid, umask, 1047 start_new_session, process_group) 1048 except: 1049 # Cleanup if the child failed starting. File /opt/homebrew/Cellar/python@3.13/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:1972, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session, process_group) 1971 if err_filename is not None: -> 1972 raise child_exception_type(errno_num, err_msg, err_filename) 1973 else: FileNotFoundError: [Errno 2] No such file or directory: PosixPath('dot') The above exception was the direct cause of the following exception: ExecutableNotFound Traceback (most recent call last) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/IPython/core/formatters.py:1036, in MimeBundleFormatter.__call__(self, obj, include, exclude) 1033 method = get_real_method(obj, self.print_method) 1035 if method is not None: -> 1036 return method(include=include, exclude=exclude) 1037 return None 1038 else: File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/jupyter_integration.py:98, in JupyterIntegration._repr_mimebundle_(self, include, exclude, **_) 96 include = set(include) if include is not None else {self._jupyter_mimetype} 97 include -= set(exclude or []) ---> 98 return {mimetype: getattr(self, method_name)() 99 for mimetype, method_name in MIME_TYPES.items() 100 if mimetype in include} File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/jupyter_integration.py:112, in JupyterIntegration._repr_image_svg_xml(self) 110 def _repr_image_svg_xml(self) -> str: 111 """Return the rendered graph as SVG string.""" --> 112 return self.pipe(format='svg', encoding=SVG_ENCODING) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:104, in Pipe.pipe(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 55 def pipe(self, 56 format: typing.Optional[str] = None, 57 renderer: typing.Optional[str] = None, (...) 61 engine: typing.Optional[str] = None, 62 encoding: typing.Optional[str] = None) -> typing.Union[bytes, str]: 63 """Return the source piped through the Graphviz layout command. 64 65 Args: (...) 102 '<?xml version=' 103 """ --> 104 return self._pipe_legacy(format, 105 renderer=renderer, 106 formatter=formatter, 107 neato_no_op=neato_no_op, 108 quiet=quiet, 109 engine=engine, 110 encoding=encoding) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/_tools.py:185, in deprecate_positional_args.<locals>.decorator.<locals>.wrapper(*args, **kwargs) 177 wanted = ', '.join(f'{name}={value!r}' 178 for name, value in deprecated.items()) 179 warnings.warn(f'The signature of {func_name} will be reduced' 180 f' to {supported_number} positional arg{s_}{qualification}' 181 f' {list(supported)}: pass {wanted} as keyword arg{s_}', 182 stacklevel=stacklevel, 183 category=category) --> 185 return func(*args, **kwargs) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:121, in Pipe._pipe_legacy(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 112 @_tools.deprecate_positional_args(supported_number=1, ignore_arg='self') 113 def _pipe_legacy(self, 114 format: typing.Optional[str] = None, (...) 119 engine: typing.Optional[str] = None, 120 encoding: typing.Optional[str] = None) -> typing.Union[bytes, str]: --> 121 return self._pipe_future(format, 122 renderer=renderer, 123 formatter=formatter, 124 neato_no_op=neato_no_op, 125 quiet=quiet, 126 engine=engine, 127 encoding=encoding) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:149, in Pipe._pipe_future(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 146 if encoding is not None: 147 if codecs.lookup(encoding) is codecs.lookup(self.encoding): 148 # common case: both stdin and stdout need the same encoding --> 149 return self._pipe_lines_string(*args, encoding=encoding, **kwargs) 150 try: 151 raw = self._pipe_lines(*args, input_encoding=self.encoding, **kwargs) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/piping.py:212, in pipe_lines_string(engine, format, input_lines, encoding, renderer, formatter, neato_no_op, quiet) 206 cmd = dot_command.command(engine, format, 207 renderer=renderer, 208 formatter=formatter, 209 neato_no_op=neato_no_op) 210 kwargs = {'input_lines': input_lines, 'encoding': encoding} --> 212 proc = execute.run_check(cmd, capture_output=True, quiet=quiet, **kwargs) 213 return proc.stdout File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:81, in run_check(cmd, input_lines, encoding, quiet, **kwargs) 79 except OSError as e: 80 if e.errno == errno.ENOENT: ---> 81 raise ExecutableNotFound(cmd) from e 82 raise 84 if not quiet and proc.stderr: ExecutableNotFound: failed to execute PosixPath('dot'), make sure the Graphviz executables are on your systems' PATH
<graphviz.graphs.Digraph at 0x1132cb570>
The GUI Fuzzer fuzz() method produces sequences of interactions that follow paths through the finite state machine. Since GUICoverageFuzzer is derived from CoverageFuzzer (see the chapter on coverage-based grammar fuzzing), it automatically covers (a) as many transitions between states as well as (b) as many form elements as possible. In our case, the first set of actions explores the transition via the "order form" link; the second set then goes until the "
gui_driver.get(httpd_url)
actions = gui_fuzzer.fuzz()
print(actions)
click('terms and conditions')
These actions can be fed into the GUI runner, which will execute them on the given GUI driver.
gui_driver.get(httpd_url)
result, outcome = gui_runner.run(actions)
Image(gui_driver.get_screenshot_as_png())
Further invocations of fuzz() will further cover the model – for instance, exploring the terms and conditions.
Internally, GUIFuzzer and GUICoverageFuzzer use a subclass GUIGrammarMiner which implements the analysis of the GUI and all its states. Subclassing GUIGrammarMiner allows extending the interpretation of GUIs; the GUIFuzzer constructor allows passing a miner via the miner keyword parameter.
A tool like GUICoverageFuzzer will provide "deep" exploration of user interfaces, even filling out forms to explore what is behind them. Keep in mind, though, that GUICoverageFuzzer is experimental: It only supports a subset of HTML form and link features, and does not take JavaScript into account.
# ignore
from ClassDiagram import display_class_hierarchy
from Fuzzer import Fuzzer, Runner
from Grammars import Grammar, Expansion
from GrammarFuzzer import GrammarFuzzer, DerivationTree
# ignore
display_class_hierarchy([GUIFuzzer, GUICoverageFuzzer,
GUIRunner, GUIGrammarMiner],
public_methods=[
Fuzzer.__init__,
Fuzzer.fuzz,
Fuzzer.run,
Fuzzer.runs,
Runner.__init__,
Runner.run,
GUIRunner.__init__,
GUIRunner.run,
GrammarFuzzer.__init__,
GrammarFuzzer.fuzz,
GrammarFuzzer.fuzz_tree,
GUIFuzzer.__init__,
GUIFuzzer.restart,
GUIFuzzer.run,
GUIGrammarMiner.__init__,
GrammarCoverageFuzzer.__init__,
GUICoverageFuzzer.__init__,
GUICoverageFuzzer.explore_all,
],
types={
'DerivationTree': DerivationTree,
'Expansion': Expansion,
'Grammar': Grammar
},
project='fuzzingbook')
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:76, in run_check(cmd, input_lines, encoding, quiet, **kwargs) 75 kwargs['stdout'] = kwargs['stderr'] = subprocess.PIPE ---> 76 proc = _run_input_lines(cmd, input_lines, kwargs=kwargs) 77 else: File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:96, in _run_input_lines(cmd, input_lines, kwargs) 95 def _run_input_lines(cmd, input_lines, *, kwargs): ---> 96 popen = subprocess.Popen(cmd, stdin=subprocess.PIPE, **kwargs) 98 stdin_write = popen.stdin.write File /opt/homebrew/Cellar/python@3.13/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:1039, in Popen.__init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask, pipesize, process_group) 1036 self.stderr = io.TextIOWrapper(self.stderr, 1037 encoding=encoding, errors=errors) -> 1039 self._execute_child(args, executable, preexec_fn, close_fds, 1040 pass_fds, cwd, env, 1041 startupinfo, creationflags, shell, 1042 p2cread, p2cwrite, 1043 c2pread, c2pwrite, 1044 errread, errwrite, 1045 restore_signals, 1046 gid, gids, uid, umask, 1047 start_new_session, process_group) 1048 except: 1049 # Cleanup if the child failed starting. File /opt/homebrew/Cellar/python@3.13/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:1972, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session, process_group) 1971 if err_filename is not None: -> 1972 raise child_exception_type(errno_num, err_msg, err_filename) 1973 else: FileNotFoundError: [Errno 2] No such file or directory: PosixPath('dot') The above exception was the direct cause of the following exception: ExecutableNotFound Traceback (most recent call last) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/IPython/core/formatters.py:1036, in MimeBundleFormatter.__call__(self, obj, include, exclude) 1033 method = get_real_method(obj, self.print_method) 1035 if method is not None: -> 1036 return method(include=include, exclude=exclude) 1037 return None 1038 else: File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/jupyter_integration.py:98, in JupyterIntegration._repr_mimebundle_(self, include, exclude, **_) 96 include = set(include) if include is not None else {self._jupyter_mimetype} 97 include -= set(exclude or []) ---> 98 return {mimetype: getattr(self, method_name)() 99 for mimetype, method_name in MIME_TYPES.items() 100 if mimetype in include} File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/jupyter_integration.py:112, in JupyterIntegration._repr_image_svg_xml(self) 110 def _repr_image_svg_xml(self) -> str: 111 """Return the rendered graph as SVG string.""" --> 112 return self.pipe(format='svg', encoding=SVG_ENCODING) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:104, in Pipe.pipe(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 55 def pipe(self, 56 format: typing.Optional[str] = None, 57 renderer: typing.Optional[str] = None, (...) 61 engine: typing.Optional[str] = None, 62 encoding: typing.Optional[str] = None) -> typing.Union[bytes, str]: 63 """Return the source piped through the Graphviz layout command. 64 65 Args: (...) 102 '<?xml version=' 103 """ --> 104 return self._pipe_legacy(format, 105 renderer=renderer, 106 formatter=formatter, 107 neato_no_op=neato_no_op, 108 quiet=quiet, 109 engine=engine, 110 encoding=encoding) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/_tools.py:185, in deprecate_positional_args.<locals>.decorator.<locals>.wrapper(*args, **kwargs) 177 wanted = ', '.join(f'{name}={value!r}' 178 for name, value in deprecated.items()) 179 warnings.warn(f'The signature of {func_name} will be reduced' 180 f' to {supported_number} positional arg{s_}{qualification}' 181 f' {list(supported)}: pass {wanted} as keyword arg{s_}', 182 stacklevel=stacklevel, 183 category=category) --> 185 return func(*args, **kwargs) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:121, in Pipe._pipe_legacy(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 112 @_tools.deprecate_positional_args(supported_number=1, ignore_arg='self') 113 def _pipe_legacy(self, 114 format: typing.Optional[str] = None, (...) 119 engine: typing.Optional[str] = None, 120 encoding: typing.Optional[str] = None) -> typing.Union[bytes, str]: --> 121 return self._pipe_future(format, 122 renderer=renderer, 123 formatter=formatter, 124 neato_no_op=neato_no_op, 125 quiet=quiet, 126 engine=engine, 127 encoding=encoding) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:149, in Pipe._pipe_future(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 146 if encoding is not None: 147 if codecs.lookup(encoding) is codecs.lookup(self.encoding): 148 # common case: both stdin and stdout need the same encoding --> 149 return self._pipe_lines_string(*args, encoding=encoding, **kwargs) 150 try: 151 raw = self._pipe_lines(*args, input_encoding=self.encoding, **kwargs) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/piping.py:212, in pipe_lines_string(engine, format, input_lines, encoding, renderer, formatter, neato_no_op, quiet) 206 cmd = dot_command.command(engine, format, 207 renderer=renderer, 208 formatter=formatter, 209 neato_no_op=neato_no_op) 210 kwargs = {'input_lines': input_lines, 'encoding': encoding} --> 212 proc = execute.run_check(cmd, capture_output=True, quiet=quiet, **kwargs) 213 return proc.stdout File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:81, in run_check(cmd, input_lines, encoding, quiet, **kwargs) 79 except OSError as e: 80 if e.errno == errno.ENOENT: ---> 81 raise ExecutableNotFound(cmd) from e 82 raise 84 if not quiet and proc.stderr: ExecutableNotFound: failed to execute PosixPath('dot'), make sure the Graphviz executables are on your systems' PATH
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:76, in run_check(cmd, input_lines, encoding, quiet, **kwargs) 75 kwargs['stdout'] = kwargs['stderr'] = subprocess.PIPE ---> 76 proc = _run_input_lines(cmd, input_lines, kwargs=kwargs) 77 else: File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:96, in _run_input_lines(cmd, input_lines, kwargs) 95 def _run_input_lines(cmd, input_lines, *, kwargs): ---> 96 popen = subprocess.Popen(cmd, stdin=subprocess.PIPE, **kwargs) 98 stdin_write = popen.stdin.write File /opt/homebrew/Cellar/python@3.13/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:1039, in Popen.__init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask, pipesize, process_group) 1036 self.stderr = io.TextIOWrapper(self.stderr, 1037 encoding=encoding, errors=errors) -> 1039 self._execute_child(args, executable, preexec_fn, close_fds, 1040 pass_fds, cwd, env, 1041 startupinfo, creationflags, shell, 1042 p2cread, p2cwrite, 1043 c2pread, c2pwrite, 1044 errread, errwrite, 1045 restore_signals, 1046 gid, gids, uid, umask, 1047 start_new_session, process_group) 1048 except: 1049 # Cleanup if the child failed starting. File /opt/homebrew/Cellar/python@3.13/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:1972, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session, process_group) 1971 if err_filename is not None: -> 1972 raise child_exception_type(errno_num, err_msg, err_filename) 1973 else: FileNotFoundError: [Errno 2] No such file or directory: PosixPath('dot') The above exception was the direct cause of the following exception: ExecutableNotFound Traceback (most recent call last) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/IPython/core/formatters.py:406, in BaseFormatter.__call__(self, obj) 404 method = get_real_method(obj, self.print_method) 405 if method is not None: --> 406 return method() 407 return None 408 else: File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/jupyter_integration.py:112, in JupyterIntegration._repr_image_svg_xml(self) 110 def _repr_image_svg_xml(self) -> str: 111 """Return the rendered graph as SVG string.""" --> 112 return self.pipe(format='svg', encoding=SVG_ENCODING) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:104, in Pipe.pipe(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 55 def pipe(self, 56 format: typing.Optional[str] = None, 57 renderer: typing.Optional[str] = None, (...) 61 engine: typing.Optional[str] = None, 62 encoding: typing.Optional[str] = None) -> typing.Union[bytes, str]: 63 """Return the source piped through the Graphviz layout command. 64 65 Args: (...) 102 '<?xml version=' 103 """ --> 104 return self._pipe_legacy(format, 105 renderer=renderer, 106 formatter=formatter, 107 neato_no_op=neato_no_op, 108 quiet=quiet, 109 engine=engine, 110 encoding=encoding) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/_tools.py:185, in deprecate_positional_args.<locals>.decorator.<locals>.wrapper(*args, **kwargs) 177 wanted = ', '.join(f'{name}={value!r}' 178 for name, value in deprecated.items()) 179 warnings.warn(f'The signature of {func_name} will be reduced' 180 f' to {supported_number} positional arg{s_}{qualification}' 181 f' {list(supported)}: pass {wanted} as keyword arg{s_}', 182 stacklevel=stacklevel, 183 category=category) --> 185 return func(*args, **kwargs) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:121, in Pipe._pipe_legacy(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 112 @_tools.deprecate_positional_args(supported_number=1, ignore_arg='self') 113 def _pipe_legacy(self, 114 format: typing.Optional[str] = None, (...) 119 engine: typing.Optional[str] = None, 120 encoding: typing.Optional[str] = None) -> typing.Union[bytes, str]: --> 121 return self._pipe_future(format, 122 renderer=renderer, 123 formatter=formatter, 124 neato_no_op=neato_no_op, 125 quiet=quiet, 126 engine=engine, 127 encoding=encoding) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:149, in Pipe._pipe_future(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 146 if encoding is not None: 147 if codecs.lookup(encoding) is codecs.lookup(self.encoding): 148 # common case: both stdin and stdout need the same encoding --> 149 return self._pipe_lines_string(*args, encoding=encoding, **kwargs) 150 try: 151 raw = self._pipe_lines(*args, input_encoding=self.encoding, **kwargs) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/piping.py:212, in pipe_lines_string(engine, format, input_lines, encoding, renderer, formatter, neato_no_op, quiet) 206 cmd = dot_command.command(engine, format, 207 renderer=renderer, 208 formatter=formatter, 209 neato_no_op=neato_no_op) 210 kwargs = {'input_lines': input_lines, 'encoding': encoding} --> 212 proc = execute.run_check(cmd, capture_output=True, quiet=quiet, **kwargs) 213 return proc.stdout File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:81, in run_check(cmd, input_lines, encoding, quiet, **kwargs) 79 except OSError as e: 80 if e.errno == errno.ENOENT: ---> 81 raise ExecutableNotFound(cmd) from e 82 raise 84 if not quiet and proc.stderr: ExecutableNotFound: failed to execute PosixPath('dot'), make sure the Graphviz executables are on your systems' PATH
<graphviz.graphs.Digraph at 0x113e80950>
Automated GUI Interaction¶
In the chapter on Web testing, we have shown how to test Web-based interfaces by directly interacting with a Web server using the HTTP protocol, and processing the retrieved HTML pages to identify user interface elements. While these techniques work well for user interfaces that are based on HTML only, they fail as soon as there are interactive elements that use JavaScript to execute code within the browser, and generate and change the user interface without having to interact with the browser.
In this chapter, we therefore take a different approach to user interface testing. Rather than using HTTP and HTML as the mechanisms for interaction, we leverage a dedicated UI testing framework, which allows us to
- query the program under test for available user interface elements, and
- query the UI elements for how they can be interacted with.
Although we will again illustrate our approach using a Web server, the approach easily generalizes to arbitrary user interfaces. In fact, the UI testing framework we use, Selenium, also comes in variants that run for Android apps.
Our Web Server, Again¶
As in the chapter on Web testing, we run a Web server that allows us to order products.
# ignore
if 'CI' in os.environ:
# Can't run this in our continuous environment,
# since it can't run a headless Web browser
sys.exit(0)
db = init_db()
This is the address of our web server:
httpd_process, httpd_url = start_httpd()
print_url(httpd_url)
Using webbrowser(), we can retrieve the HTML of the home page, and use HTML() to render it.
HTML(webbrowser(httpd_url))
127.0.0.1 - - [26/Oct/2025 14:35:26] "GET / HTTP/1.1" 200 -
Remote Control with Selenium¶
Let us take a look at the GUI above. In contrast to the chapter on Web testing, we do not assume we can access the HTML source of the current page. All we assume is that there is a set of user interface elements we can interact with.
Selenium is a framework for testing Web applications by automating interaction in the browser. Selenium provides an API that allows one to launch a Web browser, query the state of the user interface, and interact with individual user interface elements. The Selenium API is available in a number of languages; we use the Selenium API for Python.
A Selenium web driver is the interface between a program and a browser controlled by the program. The following code starts a Web browser in the background, which we then control through the web driver.
We support both Firefox and Google Chrome.
BROWSER = 'firefox' # Set to 'chrome' if you prefer Chrome
Setting up Firefox¶
For Firefox, you have to make sure the geckodriver program is in your path.
if BROWSER == 'firefox':
assert shutil.which('geckodriver') is not None, \
"Please install the 'geckodriver' executable " \
"from https://github.com/mozilla/geckodriver/releases"
Setting up Chrome¶
For Chrome, you may have to make sure the chromedriver program is in your path.
if BROWSER == 'chrome':
assert shutil.which('chromedriver') is not None, \
"Please install the 'chromedriver' executable " \
"from https://chromedriver.chromium.org"
Running a Headless Browser¶
The browser is headless, meaning that it does not show on the screen.
HEADLESS = True
Note: If the notebook server runs locally (i.e. on the same machine on which you are seeing this), you can also set HEADLESS to False and see what happens right on the screen as you execute the notebook cells. This is very much recommended for interactive sessions.
Starting the Web driver¶
This code starts the Selenium web driver.
def start_webdriver(browser=BROWSER, headless=HEADLESS, zoom=1.4):
# Set headless option
if browser == 'firefox':
options = webdriver.FirefoxOptions()
if headless:
# See https://www.browserstack.com/guide/firefox-headless
options.add_argument("--headless")
elif browser == 'chrome':
options = webdriver.ChromeOptions()
if headless:
# See https://www.selenium.dev/blog/2023/headless-is-going-away/
options.add_argument("--headless=new")
else:
assert False, "Select 'firefox' or 'chrome' as browser"
# Start the browser, and obtain a _web driver_ object such that we can interact with it.
if browser == 'firefox':
# For firefox, set a higher resolution for our screenshots
options.set_preference("layout.css.devPixelsPerPx", repr(zoom))
gui_driver = webdriver.Firefox(options=options)
# We set the window size such that it fits our order form exactly;
# this is useful for not wasting too much space when taking screen shots.
gui_driver.set_window_size(700, 300)
elif browser == 'chrome':
gui_driver = webdriver.Chrome(options=options)
gui_driver.set_window_size(700, 210 if headless else 340)
return gui_driver
gui_driver = start_webdriver(browser=BROWSER, headless=HEADLESS)
The geckodriver version (0.34.0) detected in PATH at /Users/zeller/bin/geckodriver might not be compatible with the detected firefox version (144.09); currently, geckodriver 0.36.0 is recommended for firefox 144.*, so it is advised to delete the driver in PATH and retry
We can now interact with the browser programmatically. First, we have it navigate to the URL of our Web server:
gui_driver.get(httpd_url)
We see that the home page is actually accessed, together with a (failing) request to get a page icon:
print_httpd_messages()
127.0.0.1 - - [26/Oct/2025 14:36:01] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [26/Oct/2025 14:36:01] "GET /favicon.ico HTTP/1.1" 404 -
To see what the "headless" browser displays, we can obtain a screenshot. We see that it actually displays the home page.
Image(gui_driver.get_screenshot_as_png())
Filling out Forms¶
To interact with the Web page through Selenium and the browser, we can query Selenium for individual elements. For instance, we can access the UI element whose name attribute (as defined in HTML) is "name".
name = gui_driver.find_element(By.NAME, "name")
Once we have an element, we can interact with it. Since name is a text field, we can send it a string using the send_keys() method; the string will be translated into appropriate keystrokes.
name.send_keys("Jane Doe")
In the screenshot, we can see that the name field is now filled:
Image(gui_driver.get_screenshot_as_png())
Similarly, we can fill out the email, city, and ZIP fields:
email = gui_driver.find_element(By.NAME, "email")
email.send_keys("j.doe@example.com")
city = gui_driver.find_element(By.NAME, 'city')
city.send_keys("Seattle")
zip = gui_driver.find_element(By.NAME, 'zip')
zip.send_keys("98104")
Image(gui_driver.get_screenshot_as_png())
The check box for terms and conditions is not filled out, but clicked instead using the click() method.
terms = gui_driver.find_element(By.NAME, 'terms')
terms.click()
Image(gui_driver.get_screenshot_as_png())
The form is now fully filled out. By clicking on the submit button, we can place the order:
submit = gui_driver.find_element(By.NAME, 'submit')
submit.click()
We see that the order is being processed, and that the Web browser has switched to the confirmation page.
print_httpd_messages()
127.0.0.1 - - [26/Oct/2025 14:36:02] INSERT INTO orders VALUES ('tshirt', 'Jane Doe', 'j.doe@example.com', 'Seattle', '98104')
127.0.0.1 - - [26/Oct/2025 14:36:02] "GET /order?item=tshirt&name=Jane+Doe&email=j.doe%40example.com&city=Seattle&zip=98104&terms=on&submit=Place+order HTTP/1.1" 200 -
Image(gui_driver.get_screenshot_as_png())
Navigating¶
Just as we fill out forms, we can also navigate through a website by clicking on links. Let us go back to the home page:
gui_driver.back()
Image(gui_driver.get_screenshot_as_png())
We can query the web driver for all elements of a particular type. Querying for HTML anchor elements (<a>) for instance, gives us all links on a page.
links = gui_driver.find_elements(By.TAG_NAME, "a")
We can query the attributes of UI elements – for instance, the URL the first anchor on the page links to:
links[0].get_attribute('href')
'http://127.0.0.1:8800/terms'
What happens if we click on it? Very simple: We switch to the Web page being referenced.
links[0].click()
print_httpd_messages()
127.0.0.1 - - [26/Oct/2025 14:36:02] "GET /terms HTTP/1.1" 200 -
Image(gui_driver.get_screenshot_as_png())
Okay. Let's get back to our home page again.
gui_driver.back()
print_httpd_messages()
Image(gui_driver.get_screenshot_as_png())
Writing Test Cases¶
The above calls, interacting with a user interface automatically, are typically used in Selenium tests – that is, code snippets that interact with a website, occasionally checking whether everything works as expected. The following code, for instance, places an order just as above. It then retrieves the title element and checks whether the title contains a "Thank you" message, indicating success.
def test_successful_order(driver, url):
name = "Walter White"
email = "white@jpwynne.edu"
city = "Albuquerque"
zip_code = "87101"
driver.get(url)
driver.find_element(By.NAME, "name").send_keys(name)
driver.find_element(By.NAME, "email").send_keys(email)
driver.find_element(By.NAME, 'city').send_keys(city)
driver.find_element(By.NAME, 'zip').send_keys(zip_code)
driver.find_element(By.NAME, 'terms').click()
driver.find_element(By.NAME, 'submit').click()
title = driver.find_element(By.ID, 'title')
assert title is not None
assert title.text.find("Thank you") >= 0
confirmation = driver.find_element(By.ID, "confirmation")
assert confirmation is not None
assert confirmation.text.find(name) >= 0
assert confirmation.text.find(email) >= 0
assert confirmation.text.find(city) >= 0
assert confirmation.text.find(zip_code) >= 0
return True
test_successful_order(gui_driver, httpd_url)
True
In a similar vein, we can set up automated test cases for unsuccessful orders, canceling orders, changing orders, and many more. All these test cases would be automatically run after any change to the program code, ensuring the Web application still works.
Of course, writing such tests is quite some effort. Hence, in the remainder of this chapter, we will again explore how to automatically generate them.
Retrieving User Interface Actions¶
To automatically interact with a user interface, we first need to find out which elements there are, and which user interactions (or short actions) they support.
User Interface Elements¶
We start with finding available user elements. Let us get back to the order form.
gui_driver.get(httpd_url)
Image(gui_driver.get_screenshot_as_png())
Using find_elements(By.TAG_NAME, ) (and other similar find_elements_...() functions), we can retrieve all elements of a particular type, such as HTML input elements.
ui_elements = gui_driver.find_elements(By.TAG_NAME, "input")
For each element, we can retrieve its HTML attributes, using get_attribute(). We can thus retrieve the name and type of each input element (if defined).
for element in ui_elements:
print("Name: %-10s | Type: %-10s | Text: %s" %
(element.get_attribute('name'),
element.get_attribute('type'),
element.text))
Name: name | Type: text | Text: Name: email | Type: email | Text: Name: city | Type: text | Text: Name: zip | Type: number | Text: Name: terms | Type: checkbox | Text: Name: submit | Type: submit | Text:
ui_elements = gui_driver.find_elements(By.TAG_NAME, "a")
for element in ui_elements:
print("Name: %-10s | Type: %-10s | Text: %s" %
(element.get_attribute('name'),
element.get_attribute('type'),
element.text))
Name: | Type: | Text: terms and conditions
User Interface Actions¶
Similarly to what we did in the chapter on Web fuzzing, our idea is now to mine a grammar for the user interface – first for an individual user interface page (i.e., a single Web page), later for all pages offered by the application. The idea is that a grammar defines legal sequences of actions – clicks and keystrokes – that can be applied on the application.
We assume the following actions:
fill(<name>, <text>)– fill the UI input element named<name>with the text<text>.check(<name>, <value>)– set the UI checkbox<name>to the given value<value>(True or False)submit(<name>)– submit the form by clicking on the UI element<name>.click(<name>)– click on the UI element<name>, typically for following a link.
This sequence of actions, for instance would fill out the order form:
fill('name', "Walter White")
fill('email', "white@jpwynne.edu")
fill('city', "Albuquerque")
fill('zip', "87101")
check('terms', True)
submit('submit')
Our set of actions is deliberately defined to be small – for real user interfaces, one would also have to define interactions such as swipes, double clicks, long clicks, right button clicks, modifier keys, and more. Selenium supports all of this; but in the interest of simplicity, we focus on the most important set of interactions.
Retrieving Actions¶
As a first step in mining an action grammar, we need to be able to retrieve possible interactions. We introduce a class GUIGrammarMiner, which is set to do precisely that.
class GUIGrammarMiner:
"""Retrieve a grammar of possible GUI interaction sequences"""
def __init__(self, driver, stay_on_host: bool = True) -> None:
"""Constructor.
`driver` - a web driver as produced by Selenium.
`stay_on_host` - if True (default), no not follow links to other hosts.
"""
self.driver = driver
self.stay_on_host = stay_on_host
self.grammar: Grammar = {}
Let us show GUIGrammarMiner in action, using its mine_state_actions() method to retrieve all elements from our current page. We see that we obtain input element actions, button element actions, and link element actions.
gui_grammar_miner = GUIGrammarMiner(gui_driver)
gui_grammar_miner.mine_state_actions()
frozenset({"check('terms', <boolean>)",
"click('terms and conditions')",
"fill('city', '<text>')",
"fill('email', '<email>')",
"fill('name', '<text>')",
"fill('zip', '<number>')",
"submit('submit')"})
We assume that we can identify a user interface state from the set of interactive elements it contains – that is, the current Web page is identified by the set above. This is in contrast to Web fuzzing, where we assumed the URL to uniquely characterize a page – but with JavaScript, the URL can stay unchanged although the page contents change, and UIs other than the Web may have no concept of unique URLs. Therefore, we say that the way a UI can be interacted with uniquely defines its state.
Models for User Interfaces¶
User Interfaces as Finite State Machines¶
Now that we can retrieve UI elements from a page, let us go and systematically explore a user interface. The idea is to represent the user interface as a finite state machine – that is, a sequence of states that can be reached by interacting with the individual user interface elements.
Let us illustrate such a finite state machine by looking at our Web server. The following diagram shows the states our server can be in:
# ignore
from graphviz import Digraph
# ignore
from GrammarFuzzer import dot_escape
# ignore
dot = Digraph(comment="Finite State Machine")
dot.node(dot_escape('<start>'))
dot.edge(dot_escape('<start>'),
dot_escape('<Order Form>'))
dot.edge(dot_escape('<Order Form>'),
dot_escape('<Terms and Conditions>'), "click('Terms and conditions')")
dot.edge(dot_escape('<Order Form>'),
dot_escape('<Thank You>'), r"fill(...)\lsubmit('submit')")
dot.edge(dot_escape('<Terms and Conditions>'),
dot_escape('<Order Form>'), "click('order form')")
dot.edge(dot_escape('<Thank You>'),
dot_escape('<Order Form>'), "click('order form')")
display(dot)
Initially, we are in the <Order Form> state. From here, we can click on Terms and Conditions, and we'll be in the Terms and Conditions state, showing the page with the same title. We can also fill out the form and place the order, having us end in the Thank You state (again showing the page with the same title). From both <Terms and Conditions> and <Thank You>, we can return to the order form by clicking on the order form link.
State Machines as Grammars¶
To systematically explore a user interface, we must retrieve its finite state machine, and eventually cover all states and transitions. In the presence of forms, such an exploration is difficult, as we need a special mechanism to fill out forms and submit the values to get to the next state. There is a trick, though, which allows us to have a single representation for both states and (form) values. We can embed the finite state machine into a grammar, which is then used for both states and form values.
To embed a finite state machine into a grammar, we proceed as follows:
- Every state $\langle s \rangle$ in the finite state machine becomes a symbol $\langle s \rangle$ in the grammar.
- Every transition in the finite state machine from $\langle s \rangle$ to $\langle t \rangle$ and actions $a_1, a_2, \dots$ becomes an alternative of $\langle s \rangle$ in the form $a_1, a_2, dots$ $\langle t \rangle$ in the grammar.
The above finite state machine thus gets encoded into the grammar
<start> ::= <Order Form>
<Order Form> ::= click('Terms and Conditions') <Terms and Conditions> |
fill(...) submit('submit') <Thank You>
<Terms and Conditions> ::= click('order form') <Order Form>
<Thank You> ::= click('order form') <Order Form>
Expanding this grammar gets us a stream of actions, navigating through the user interface:
fill(...) submit('submit') click('order form') click('Terms and Conditions') click('order form') ...
This stream is actually infinite (as one can interact with the UI forever); to have it end, one can introduce an alternative <end> that simply expands to the empty string, without having any expansion (state) follow.
Retrieving State Grammars¶
Let us extend GUIGrammarMiner such that it retrieves a grammar from the user interface in its current state.
Let us show GUIGrammarMiner() in action. Its method mine_state_grammar() extracts the grammar for the current Web page:
gui_grammar_miner = GUIGrammarMiner(gui_driver)
state_grammar = gui_grammar_miner.mine_state_grammar()
state_grammar
{'<start>': ['<state>'],
'<unexplored>': [''],
'<end>': [''],
'<text>': ['<string>'],
'<string>': ['<character>', '<string><character>'],
'<character>': ['<letter>', '<digit>', '<special>'],
'<letter>': ['a',
'b',
'c',
'd',
'e',
'f',
'g',
'h',
'i',
'j',
'k',
'l',
'm',
'n',
'o',
'p',
'q',
'r',
's',
't',
'u',
'v',
'w',
'x',
'y',
'z',
'A',
'B',
'C',
'D',
'E',
'F',
'G',
'H',
'I',
'J',
'K',
'L',
'M',
'N',
'O',
'P',
'Q',
'R',
'S',
'T',
'U',
'V',
'W',
'X',
'Y',
'Z'],
'<number>': ['<digits>'],
'<digits>': ['<digit>', '<digits><digit>'],
'<digit>': ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'],
'<special>': ['.', ' ', '!'],
'<email>': ['<letters>@<letters>'],
'<letters>': ['<letter>', '<letters><letter>'],
'<boolean>': ['True', 'False'],
'<state>': ["click('terms and conditions')\n<state-1>",
"fill('email', '<email>')\ncheck('terms', <boolean>)\nfill('zip', '<number>')\nfill('name', '<text>')\nfill('city', '<text>')\nsubmit('submit')\n<state-2>",
'<end>'],
'<state-1>': ['<unexplored>'],
'<state-2>': ['<unexplored>']}
To better see the structure of the state grammar, we can visualize it as a state machine. We see that it nicely reflects what we can see from our Web server's home page:
fsm_diagram(state_grammar)
From the start state (<state>), we can go and either click on "terms and conditions", ending in <state-1>, or fill out the form, ending in <state-2>.
state_grammar[GUIGrammarMiner.START_STATE]
["click('terms and conditions')\n<state-1>",
"fill('email', '<email>')\ncheck('terms', <boolean>)\nfill('zip', '<number>')\nfill('name', '<text>')\nfill('city', '<text>')\nsubmit('submit')\n<state-2>",
'<end>']
Both these states are yet unexplored:
state_grammar['<state-1>']
['<unexplored>']
state_grammar['<state-2>']
['<unexplored>']
state_grammar['<unexplored>']
['']
Given the grammar, we can use any of our grammar fuzzers to create valid input sequences:
gui_fuzzer = GrammarFuzzer(state_grammar)
while True:
action = gui_fuzzer.fuzz()
if action.find('submit(') > 0:
break
print(action)
fill('email', 'G@C')
check('terms', True)
fill('zip', '342')
fill('name', '.')
fill('city', '6')
submit('submit')
These actions, however, must also be executed such that we can explore the user interface. This is what we do in the next section.
Executing User Interface Actions¶
To execute actions, we introduce a Runner class, conveniently named GUIRunner. Its run() method executes the actions as given in an action string.
class GUIRunner(Runner):
"""Execute the actions in a given action string"""
def __init__(self, driver) -> None:
"""Constructor. `driver` is a Selenium Web driver"""
self.driver = driver
Let us try out GUIRunner and its run() method. We create a runner on our Web server, and let it execute a fill() action:
gui_driver.get(httpd_url)
gui_runner = GUIRunner(gui_driver)
gui_runner.run("fill('name', 'Walter White')")
("fill('name', 'Walter White')", 'PASS')
Image(gui_driver.get_screenshot_as_png())
A submit() action submits the order. (Note that our Web server does no effort whatsoever to validate the form.)
gui_runner.run("submit('submit')")
("submit('submit')", 'PASS')
Image(gui_driver.get_screenshot_as_png())
Of course, we can also execute action sequences generated from the grammar. This allows us to fill the form again and again, using values matching the type given in the form.
gui_driver.get(httpd_url)
gui_fuzzer = GrammarFuzzer(state_grammar)
while True:
action = gui_fuzzer.fuzz()
if action.find('submit(') > 0:
break
print(action)
fill('email', 'HTrX@b')
check('terms', False)
fill('zip', '54')
fill('name', '.')
fill('city', '!')
submit('submit')
gui_runner.run(action)
("fill('email', 'HTrX@b')\ncheck('terms', False)\nfill('zip', '54')\nfill('name', '.')\nfill('city', '!')\nsubmit('submit')\n",
'PASS')
Image(gui_driver.get_screenshot_as_png())
Exploring User Interfaces¶
So far, our grammar retrieval and execution of actions is limited to the current user interface state (i.e., the current page shown). To systematically explore a user interface, we must explore all states, notably those ending in <unexplored> – and whenever we reach a new state, again retrieve its grammar such that we may be able to reach other states. Since some states can only be reached by generating inputs, test generation and user interface exploration take place at the same time.
Consequently, we introduce a GUIFuzzer class, which generates inputs for all forms and follows all links, and which updates its grammar (i.e., its user interface model as a finite state machine) every time it encounters a new state.
Let us put GUIFuzzer to use, enabling its logging mechanisms to see what it is doing.
gui_driver.get(httpd_url)
gui_fuzzer = GUIFuzzer(gui_driver, log_gui_exploration=True, disp_gui_exploration=True)
Running it the first time yields a new state:
gui_fuzzer.run(gui_runner)
Action fill('email', 'BU@L')
check('terms', False)
fill('zip', '1')
fill('name', '. 1')
fill('city', '1')
submit('submit') -> <state-2>
In new state <state-2> frozenset({"click('order form')"})
None
('<state-2>', 'PASS')
The next actions fill out the order form.
gui_fuzzer.run(gui_runner)
Action click('terms and conditions') -> <state-1>
In new state <state-1> frozenset({"click('order form')", "ignore('Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.')"})
None
('<state-1>', 'PASS')
gui_fuzzer.run(gui_runner)
Action click('terms and conditions') -> <end>
('<end>', 'PASS')
At this point, our GUI model is fairly complete already. In order to systematically cover all states, random exploration is not efficient enough, though.
Covering States¶
During exploration as well as during testing, we want to cover all states and transitions between states. How can we achieve this?
It turns out that we already have this. Our GrammarCoverageFuzzer from the chapter on coverage-based grammar testing strives to systematically cover all expansion alternatives in a grammar. In the finite state model, these expansion alternatives translate into transitions between states. Hence, applying the coverage strategy from GrammarCoverageFuzzer to our state grammars would automatically cover one transition after another.
How do we get these features into GUIFuzzer? Using multiple inheritance, we can create a class GUICoverageFuzzer which combines the run() method from GUIFuzzer with the coverage choices from GrammarCoverageFuzzer.
Since the __init__() constructor is defined in both superclasses, we need to define our own constructor that serves both:
inheritance_conflicts(GUIFuzzer, GrammarCoverageFuzzer)
['__firstlineno__', '__init__']
class GUICoverageFuzzer(GUIFuzzer, GrammarCoverageFuzzer):
"""Systematically explore all states of the current Web page"""
def __init__(self, *args, **kwargs):
"""Constructor. All args are passed to the `GUIFuzzer` superclass."""
GUIFuzzer.__init__(self, *args, **kwargs)
self.reset_coverage()
With GUICoverageFuzzer, we can set up a method explore_all() that keeps on running the fuzzer until there are no unexplored states anymore:
class GUICoverageFuzzer(GUICoverageFuzzer):
def explore_all(self, runner: GUIRunner, max_actions=100) -> None:
"""Explore all states of the GUI, up to `max_actions` (default 100)."""
actions = 0
while (self.miner.UNEXPLORED_STATE in self.grammar and
actions < max_actions):
actions += 1
if self.log_gui_exploration:
print("Run #" + repr(actions))
try:
self.run(runner)
except ElementClickInterceptedException:
pass
except ElementNotInteractableException:
pass
except NoSuchElementException:
pass
Let us use this to fully explore our Web server:
gui_driver.get(httpd_url)
gui_fuzzer = GUICoverageFuzzer(gui_driver)
gui_fuzzer.explore_all(gui_runner)
Success! We have covered all states:
fsm_diagram(gui_fuzzer.grammar)
We can retrieve the expansions covered so far, which of course cover all states.
gui_fuzzer.covered_expansions
{'<boolean> -> False',
'<boolean> -> True',
'<character> -> <digit>',
'<character> -> <letter>',
'<character> -> <special>',
'<digit> -> 0',
'<digit> -> 2',
'<digit> -> 3',
'<digit> -> 4',
'<digit> -> 6',
'<digit> -> 7',
'<digit> -> 8',
'<digit> -> 9',
'<digits> -> <digit>',
'<digits> -> <digits><digit>',
'<email> -> <letters>@<letters>',
'<end> -> ',
'<letter> -> E',
'<letter> -> G',
'<letter> -> J',
'<letter> -> K',
'<letter> -> M',
'<letter> -> O',
'<letter> -> P',
'<letter> -> T',
'<letter> -> V',
'<letter> -> W',
'<letter> -> X',
'<letter> -> b',
'<letter> -> c',
'<letter> -> h',
'<letter> -> i',
'<letter> -> j',
'<letter> -> k',
'<letter> -> l',
'<letter> -> p',
'<letter> -> q',
'<letter> -> r',
'<letter> -> u',
'<letter> -> v',
'<letters> -> <letter>',
'<letters> -> <letters><letter>',
'<number> -> <digits>',
'<special> -> ',
'<start> -> <state>',
'<state-1> -> <end>',
'<state-1> -> <unexplored>',
"<state-1> -> click('order form')\n<state-3>",
'<state-2> -> <end>',
'<state-2> -> <unexplored>',
"<state-2> -> click('order form')\n<state-4>",
"<state-2> -> click('order form')\n<state>",
'<state-3> -> <unexplored>',
'<state-4> -> <unexplored>',
'<state> -> <end>',
"<state> -> click('terms and conditions')\n<state-1>",
"<state> -> fill('email', '<email>')\ncheck('terms', <boolean>)\nfill('zip', '<number>')\nfill('name', '<text>')\nfill('city', '<text>')\nsubmit('submit')\n<state-2>",
'<string> -> <character>',
'<string> -> <string><character>',
'<text> -> <string>',
'<unexplored> -> '}
Still, we haven't seen all expansions covered. A few digits and letters remain to be used.
gui_fuzzer.missing_expansion_coverage()
{'<digit> -> 1',
'<digit> -> 5',
'<letter> -> A',
'<letter> -> B',
'<letter> -> C',
'<letter> -> D',
'<letter> -> F',
'<letter> -> H',
'<letter> -> I',
'<letter> -> L',
'<letter> -> N',
'<letter> -> Q',
'<letter> -> R',
'<letter> -> S',
'<letter> -> U',
'<letter> -> Y',
'<letter> -> Z',
'<letter> -> a',
'<letter> -> d',
'<letter> -> e',
'<letter> -> f',
'<letter> -> g',
'<letter> -> m',
'<letter> -> n',
'<letter> -> o',
'<letter> -> s',
'<letter> -> t',
'<letter> -> w',
'<letter> -> x',
'<letter> -> y',
'<letter> -> z',
'<special> -> !',
'<special> -> .',
"<state-1> -> click('order form')\n<state>"}
Running the fuzzer again and again will eventually cover these expansions too, leading to letter and digit coverage within the order form.
Exploring Large Sites¶
Our GUI fuzzer is robust enough to handle exploration even on nontrivial sites such as fuzzingbook.org. Let us demonstrate this:
gui_driver.get("https://www.fuzzingbook.org/html/Fuzzer.html")
Image(gui_driver.get_screenshot_as_png())
book_runner = GUIRunner(gui_driver)
book_fuzzer = GUICoverageFuzzer(gui_driver, log_gui_exploration=True) # , disp_gui_exploration=True)
We explore the first few states of the site, defined in ACTIONS:
ACTIONS = 5
book_fuzzer.explore_all(book_runner, max_actions=ACTIONS)
Run #1
Action click('use the code provided in this chapter') -> <state-6>
In new state <state-6> frozenset({"click('the chapter on fuzzers')", "ignore('Pipenv')", "click('Fuzzer')", "ignore('requirements.txt file within the project root folder')", "ignore('official instructions')", "ignore('Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License')", "ignore('MIT License')", "ignore('')", "ignore('Imprint')", "click('The Fuzzing Book')", "click('Cite')", "ignore('Last change: 2023-11-11 18:18:06+01:00')", "ignore('bookutils.setup')", "ignore('installation instructions')", "ignore('apt.txt file in the binder/ folder')", "ignore('bookutils')", "click('fuzzingbook.Fuzzer')", "ignore('the project page')", "click('')", "click('fuzzingbook.')", "ignore('pyenv-win')"})
Run #2
Action click('use grammars to specify the input format and thus get many more valid inputs') -> <state-1>
In new state <state-1> frozenset({"click('Fuzzer')", "click('grammar toolbox')", "click('coverage-based')", "click('probabilities')", "click('create an efficient grammar fuzzer')", "click('The Fuzzing Book')", "click('fuzz configurations')", "ignore('CSmith')", "click('use the code provided in this chapter')", "ignore('Backus-Naur form')", "ignore('JSON specification')", "submit('')", "ignore('Last change: 2024-06-30 18:31:28+02:00')", "ignore('copy')", "ignore('Hanford et al, 1970')", "click('fuzzingbook.Grammars')", "ignore('Chomsky et al, 1956')", "ignore('Purdom et al, 1972')", "click('generator-based')", "ignore('Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License')", "ignore('typing')", "ignore('LangFuzz')", "check('e1505106-d3ed-11ef-9634-6298cf1a5790', <boolean>)", "click('MutationFuzzer')", "ignore('bookutils')", "click('constraints')", "ignore('Yang et al, 2011')", "ignore('string')", "click('')", "ignore('re')", "ignore('Hodov\xc3\xa1n et al, 2018')", "ignore('Le et al, 2014')", "ignore('ast')", "ignore('Dak\xe1\xb9\xa3iputra P\xc4\x81\xe1\xb9\x87ini, 350 BCE')", "click('the GrammarFuzzer class')", "ignore('Use the notebook')", "click('Chapter introducing fuzzing')", "ignore('MIT License')", "click('"Mutation-Based Fuzzing"')", "ignore('')", "ignore('EMI Project')", "click('Cite')", "ignore('bookutils.setup')", "ignore('Burkhardt et al, 1967')", "click('probabilistic-based')", "check('e1424af2-d3ed-11ef-9634-6298cf1a5790', <boolean>)", "click('chapter on coverage')", "ignore('Holler et al, 2012')", "click('coverage')", "click('mutation-based fuzzing')", "ignore('inspect')", "ignore('Imprint')", "ignore('Grammarinator')", "ignore('random')", "click('probabilistic grammar fuzzing')", "click('fuzzing functions and APIs')", "click('next chapter')", "click('later in this book')", "ignore('Domato')", "click('basic fuzzing')", "check('e1433cfa-d3ed-11ef-9634-6298cf1a5790', <boolean>)", "click('our chapter on coverage-based fuzzing')", "ignore('Wikipedia page on file formats')", "click('fuzzing graphical user interfaces')"})
Run #3
Action click('chapter on mining function specifications') -> <state-14>
In new state <state-14> frozenset({"click('introduction to testing')", "ignore('DAIKON dynamic invariant detector')", "ignore('Use the notebook')", "ignore('code snippet from StackOverflow')", "ignore('Ammons et al, 2002')", "ignore('MIT License')", "ignore('')", "click('The Fuzzing Book')", "click('fuzzingbook.DynamicInvariants')", "click('Cite')", "click('use the code provided in this chapter')", "click('ExpectError')", "ignore('bookutils.setup')", "click('symbolic fuzzing')", "click('the next part')", "click('our chapter with the same name')", "ignore('Last change: 2024-11-09 17:07:29+01:00')", "click('GrammarFuzzer')", "click('chapter on testing')", "click('concolic fuzzer')", "click('concolic')", "ignore('Ernst et al, 2001')", "click('chapter on coverage')", "ignore('subprocess')", "click('symbolic interpretation')", "ignore('showast')", "click('part on semantic fuzzing techniques')", "ignore('functools')", "ignore('Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License')", "click('Grammars')", "ignore('inspect')", "ignore('MonkeyType')", "ignore('Imprint')", "ignore('Mypy')", "ignore('typing')", "ignore('itertools')", "ignore('sys')", "ignore('Pacheco et al, 2005')", "click('chapter on information flow')", "ignore('tempfile')", "ignore('"The state of type hints in Python"')", "ignore('bookutils')", "ignore('curated list')", "click('symbolic')", "click('domain-specific fuzzing techniques')", "click('')", "click('Intro_Testing')", "ignore('PyAnnotate')", "click('Coverage')", "ignore('ast')"})
Run #4
Action click('Introduction to Testing') -> <state-16>
In new state <state-16> frozenset({"ignore('Use the notebook')", "ignore('Myers et al, 2004')", "ignore('"Effective Software Testing: A Developer's Guide"')", "ignore('MIT License')", "ignore('')", "click('The Fuzzing Book')", "click('Background')", "click('Cite')", "ignore('bookutils.setup')", "click('ExpectError')", "ignore('Maur\xc3\xadcio Aniche, 2022')", "ignore('Last change: 2023-11-11 18:18:06+01:00')", "submit('')", "check('7a77e688-d3ed-11ef-a281-6298cf1a5790', <boolean>)", "click('Web Page')", "click('Timer')", "ignore('Shellsort')", "click('Guide for Authors')", "ignore('Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License')", "check('79d7b92e-d3ed-11ef-a281-6298cf1a5790', <boolean>)", "ignore('Newton\xe2\x80\x93Raphson method')", "click('use fuzzing to test programs with random inputs')", "ignore('Pezz\xc3\xa8 et al, 2008')", "click('00_Table_of_Contents.ipynb')", "ignore('random')", "ignore('Beizer et al, 1990')", "ignore('Imprint')", "click('Timer module')", "ignore('bookutils')", "click('import it')", "check('79a472e4-d3ed-11ef-a281-6298cf1a5790', <boolean>)", "click('')", "ignore('Python tutorial')", "ignore('math.isclose()')"})
Run #5
Action click('ExpectError') -> <state-8>
In existing state <state> Replacing expected state <state-8> by <state>
After the first ACTIONS actions already, we can see that the finite state model is quite complex, with dozens of transitions still left to explore. Most of the yet unexplored states will eventually merge with existing states, yielding one state per chapter. Still, following all links on all pages will take quite some time.
# Inspect this graph in the notebook to see it in full glory
fsm_diagram(book_fuzzer.grammar)
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:76, in run_check(cmd, input_lines, encoding, quiet, **kwargs) 75 kwargs['stdout'] = kwargs['stderr'] = subprocess.PIPE ---> 76 proc = _run_input_lines(cmd, input_lines, kwargs=kwargs) 77 else: File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:96, in _run_input_lines(cmd, input_lines, kwargs) 95 def _run_input_lines(cmd, input_lines, *, kwargs): ---> 96 popen = subprocess.Popen(cmd, stdin=subprocess.PIPE, **kwargs) 98 stdin_write = popen.stdin.write File /opt/homebrew/Cellar/python@3.13/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:1039, in Popen.__init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask, pipesize, process_group) 1036 self.stderr = io.TextIOWrapper(self.stderr, 1037 encoding=encoding, errors=errors) -> 1039 self._execute_child(args, executable, preexec_fn, close_fds, 1040 pass_fds, cwd, env, 1041 startupinfo, creationflags, shell, 1042 p2cread, p2cwrite, 1043 c2pread, c2pwrite, 1044 errread, errwrite, 1045 restore_signals, 1046 gid, gids, uid, umask, 1047 start_new_session, process_group) 1048 except: 1049 # Cleanup if the child failed starting. File /opt/homebrew/Cellar/python@3.13/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:1972, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session, process_group) 1971 if err_filename is not None: -> 1972 raise child_exception_type(errno_num, err_msg, err_filename) 1973 else: FileNotFoundError: [Errno 2] No such file or directory: PosixPath('dot') The above exception was the direct cause of the following exception: ExecutableNotFound Traceback (most recent call last) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/IPython/core/formatters.py:1036, in MimeBundleFormatter.__call__(self, obj, include, exclude) 1033 method = get_real_method(obj, self.print_method) 1035 if method is not None: -> 1036 return method(include=include, exclude=exclude) 1037 return None 1038 else: File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/jupyter_integration.py:98, in JupyterIntegration._repr_mimebundle_(self, include, exclude, **_) 96 include = set(include) if include is not None else {self._jupyter_mimetype} 97 include -= set(exclude or []) ---> 98 return {mimetype: getattr(self, method_name)() 99 for mimetype, method_name in MIME_TYPES.items() 100 if mimetype in include} File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/jupyter_integration.py:112, in JupyterIntegration._repr_image_svg_xml(self) 110 def _repr_image_svg_xml(self) -> str: 111 """Return the rendered graph as SVG string.""" --> 112 return self.pipe(format='svg', encoding=SVG_ENCODING) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:104, in Pipe.pipe(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 55 def pipe(self, 56 format: typing.Optional[str] = None, 57 renderer: typing.Optional[str] = None, (...) 61 engine: typing.Optional[str] = None, 62 encoding: typing.Optional[str] = None) -> typing.Union[bytes, str]: 63 """Return the source piped through the Graphviz layout command. 64 65 Args: (...) 102 '<?xml version=' 103 """ --> 104 return self._pipe_legacy(format, 105 renderer=renderer, 106 formatter=formatter, 107 neato_no_op=neato_no_op, 108 quiet=quiet, 109 engine=engine, 110 encoding=encoding) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/_tools.py:185, in deprecate_positional_args.<locals>.decorator.<locals>.wrapper(*args, **kwargs) 177 wanted = ', '.join(f'{name}={value!r}' 178 for name, value in deprecated.items()) 179 warnings.warn(f'The signature of {func_name} will be reduced' 180 f' to {supported_number} positional arg{s_}{qualification}' 181 f' {list(supported)}: pass {wanted} as keyword arg{s_}', 182 stacklevel=stacklevel, 183 category=category) --> 185 return func(*args, **kwargs) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:121, in Pipe._pipe_legacy(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 112 @_tools.deprecate_positional_args(supported_number=1, ignore_arg='self') 113 def _pipe_legacy(self, 114 format: typing.Optional[str] = None, (...) 119 engine: typing.Optional[str] = None, 120 encoding: typing.Optional[str] = None) -> typing.Union[bytes, str]: --> 121 return self._pipe_future(format, 122 renderer=renderer, 123 formatter=formatter, 124 neato_no_op=neato_no_op, 125 quiet=quiet, 126 engine=engine, 127 encoding=encoding) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/piping.py:149, in Pipe._pipe_future(self, format, renderer, formatter, neato_no_op, quiet, engine, encoding) 146 if encoding is not None: 147 if codecs.lookup(encoding) is codecs.lookup(self.encoding): 148 # common case: both stdin and stdout need the same encoding --> 149 return self._pipe_lines_string(*args, encoding=encoding, **kwargs) 150 try: 151 raw = self._pipe_lines(*args, input_encoding=self.encoding, **kwargs) File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/piping.py:212, in pipe_lines_string(engine, format, input_lines, encoding, renderer, formatter, neato_no_op, quiet) 206 cmd = dot_command.command(engine, format, 207 renderer=renderer, 208 formatter=formatter, 209 neato_no_op=neato_no_op) 210 kwargs = {'input_lines': input_lines, 'encoding': encoding} --> 212 proc = execute.run_check(cmd, capture_output=True, quiet=quiet, **kwargs) 213 return proc.stdout File ~/.virtualenvs/python3.13/lib/python3.13/site-packages/graphviz/backend/execute.py:81, in run_check(cmd, input_lines, encoding, quiet, **kwargs) 79 except OSError as e: 80 if e.errno == errno.ENOENT: ---> 81 raise ExecutableNotFound(cmd) from e 82 raise 84 if not quiet and proc.stderr: ExecutableNotFound: failed to execute PosixPath('dot'), make sure the Graphviz executables are on your systems' PATH
<graphviz.graphs.Digraph at 0x10f736360>
We now have all the basic capabilities we need: We can automatically explore large websites; we can explore "deep" functionality by filling out forms; and we can have our coverage-based fuzzer automatically focus on yet unexplored states. Still, there is a lot more one can do; the exercises will give you some ideas.
gui_driver.quit()
Lessons Learned¶
- Selenium is a powerful framework for interacting with user interfaces, especially Web-based user interfaces.
- A finite state model can encode user interface states and transitions.
- Encoding user interface models into a grammar integrates generating text (for forms) and generating user interactions (for navigating)
- To systematically explore a user interface, cover all state transitions, which is equivalent to covering all expansion alternatives in the equivalent grammar.
We are done, so we clean up. We shut down our Web server, quit the Web driver (and the associated browser), and finally clean up temporary files left by Selenium.
httpd_process.terminate()
gui_driver.quit()
for temp_file in [ORDERS_DB, "geckodriver.log", "ghostdriver.log"]:
if os.path.exists(temp_file):
os.remove(temp_file)
Next Steps¶
From here, you can learn how to
- fuzz in the large. running a myriad of fuzzers on the same system
Background¶
Automatic testing of graphical user interfaces is a rich field – in research as in practice.
Coverage criteria for GUIs as well as how to achieve them were first discussed in \cite{Memon2001}. Memon also introduced the concept of GUI Ripping \cite{Memon2003} – the process in which the software's GUI is automatically traversed by interacting with all its user interface elements.
The CrawlJax tool \cite{Mesbah2012} uses dynamic state changes in Web user interfaces to identify candidate elements to interact with. As our approach above, it uses the set of interactable user interface elements as a state in a finite-state model.
The Alex framework uses a similar approach to learn automata for web applications. Starting from a set of test inputs, it produces a mixed-mode behavioral model of the application.
Exercises¶
As powerful as our GUI fuzzer is at this point, there are still several possibilities left for further optimization and extension. Here are some ideas to get you started. Enjoy user interface fuzzing!
Exercise 1: Stay in Local State¶
Rather than having each run() start at the very beginning, have the miner start from the current state and explore states reachable from there.
Exercise 2: Going Back¶
Make use of the web driver back() method and go back to an earlier state, from which we could again start exploration. (Note that a "back" functionality may not be available on non-Web user interfaces.)
Exercise 3: Avoiding Bad Form Values¶
Detect that some form values are invalid, such that the miner does not produce them again.
Exercise 4: Saving Form Values¶
Save successful form values, such that the tester does not have to infer them again and again.
Exercise 5: Same Names, Same States¶
When the miner finds a link with a name it has already seen, it is likely to lead to a state already seen, too; therefore, one could give its exploration a lower priority.
Exercise 6: Combinatorial Coverage¶
Extend the grammar miner such that for every boolean value, there is a separate value to be covered.
Exercise 7: Implicit Delays¶
Rather than using explicit (given) delays, use implicit delays and wait for specific elements to appear. these elements could stem from previous explorations of the state.
Exercise 8: Oracles¶
Extend the grammar miner such that it also produces oracles – for instance, checking for the presence of specific UI elements.
Exercise 9: More UI Elements¶
Run the miner on a website of your choice. Find out which other types of user interface elements and actions need to be supported.