Mining Input Grammars

So far, the grammars we have seen have been mostly specified manually – that is, you (or the person knowing the input format) had to design and write a grammar in the first place. While the grammars we have seen so far have been rather simple, creating a grammar for complex inputs can involve quite some effort. In this chapter, we therefore introduce techniques that automatically mine grammars from programs – by executing the programs and observing how they process which parts of the input. In conjunction with a grammar fuzzer, this allows us to

  1. take a program,
  2. extract its input grammar, and
  3. fuzz it with high efficiency and effectiveness, using the concepts in this book.

Prerequisites

Synopsis

To use the code provided in this chapter, write

>>> from fuzzingbook.GrammarMiner import <identifier>

and then make use of the following features.

This chapter provides a number of classes to mine input grammars from existing programs. The function recover_grammar() could be the easiest to use. It takes a function and a set of inputs, and returns a grammar that describes its input language.

We apply recover_grammar() on a url_parse() function that takes and decomposes URLs:

>>> url_parse('https://www.fuzzingbook.org/')
>>> URLS
['http://user:pass@www.google.com:80/?q=path#ref',
 'https://www.cispa.saarland:80/',
 'http://www.fuzzingbook.org/#News']

We extract the input grammar for url_parse() using recover_grammar():

>>> grammar = recover_grammar(url_parse, URLS)
>>> grammar
{'<start>': ['<urlsplit@394:url>'],
 '<urlsplit@394:url>': ['<__new__@12:scheme>:<_splitnetloc@386:url>'],
 '<__new__@12:scheme>': ['https', '<__new__@12:scheme>', 'http'],
 '<_splitnetloc@386:url>': ['//<__new__@12:netloc><urlsplit@415:url>',
  '//<__new__@12:netloc>/'],
 '<__new__@12:netloc>': ['www.cispa.saarland:80',
  '<__new__@12:netloc>',
  'user:pass@www.google.com:80',
  'www.fuzzingbook.org'],
 '<urlsplit@415:url>': ['/#<__new__@12:fragment>',
  '<urlsplit@420:url>#<__new__@12:fragment>'],
 '<urlsplit@420:url>': ['/?<__new__@12:query>'],
 '<__new__@12:query>': ['q=path', '<__new__@12:query>'],
 '<__new__@12:fragment>': ['News', '<__new__@12:fragment>', 'ref']}

The names of nonterminals are a bit technical; but the grammar nicely represents the structure of the input; for instance, the different schemes ("http", "https") are all identified. The grammar can be immediately used for fuzzing, producing arbitrary combinations of input elements, which are all syntactically valid.

>>> from GrammarCoverageFuzzer import GrammarCoverageFuzzer
>>> fuzzer = GrammarCoverageFuzzer(grammar)
>>> [fuzzer.fuzz() for i in range(5)]
['https://www.cispa.saarland:80/?q=path#News',
 'http://www.fuzzingbook.org/',
 'http://user:pass@www.google.com:80/#ref',
 'http://www.fuzzingbook.org/#News',
 'https://user:pass@www.google.com:80/?q=path#ref']

Being able to automatically extract a grammar and to use this grammar for fuzzing makes for very effective test generation with a minimum of manual work.

A Grammar Challenge

Consider the process_inventory() method from the chapter on parsers:

It takes inputs of the following form.

In [3]:
INVENTORY = """\
1997,van,Ford,E350
2000,car,Mercury,Cougar
1999,car,Chevy,Venture\
"""
In [4]:
print(process_inventory(INVENTORY))
We have a Ford E350 van from 1997 vintage.
It is an old but reliable model!
We have a Mercury Cougar car from 2000 vintage.
It is an old but reliable model!
We have a Chevy Venture car from 1999 vintage.
It is an old but reliable model!

We found from the chapter on parsers that coarse grammars do not work well for fuzzing when the input format includes details expressed only in code. That is, even though we have the formal specification of CSV files (RFC 4180), the inventory system includes further rules as to what is expected at each index of the CSV file. The solution of simply recombining existing inputs, while practical, is incomplete. In particular, it relies on a formal input specification being available in the first place. However, we have no assurance that the program obeys the input specification given.

One of the ways out of this predicament is to interrogate the program under test as to what its input specification is. That is, if the program under test is written in a style such that specific methods are responsible for handling specific parts of the input, one can recover the parse tree by observing the process of parsing. Further, one can recover a reasonable approximation of the grammar by abstraction from multiple input trees.

We start with the assumption (1) that the program is written in such a fashion that specific methods are responsible for parsing specific fragments of the program -- This includes almost all ad hoc parsers.

The idea is as follows:

  • Hook into the Python execution and observe the fragments of input string as they are produced and named in different methods.
  • Stitch the input fragments together in a tree structure to retrieve the Parse Tree.
  • Abstract common elements from multiple parse trees to produce the Context Free Grammar of the input.

A Simple Grammar Miner

Say we want to obtain the input grammar for the function process_vehicle(). We first collect the sample inputs for this function.

In [5]:
VEHICLES = INVENTORY.split('\n')

The set of methods responsible for processing inventory are the following.

In [6]:
INVENTORY_METHODS = {
    'process_inventory',
    'process_vehicle',
    'process_van',
    'process_car'}

We have seen from the chapter on configuration fuzzing that one can hook into the Python runtime to observe the arguments to a function and any local variables created. We have also seen that one can obtain the context of execution by inspecting the frame argument. Here is a simple tracer that can return the local variables and other contextual information in a traced function. We reuse the Coverage tracing class.

Tracer

In [9]:
class Tracer(Coverage):
    def traceit(self, frame, event, arg):
        method_name = inspect.getframeinfo(frame).function
        if method_name not in INVENTORY_METHODS:
            return
        file_name = inspect.getframeinfo(frame).filename

        param_names = inspect.getargvalues(frame).args
        lineno = inspect.getframeinfo(frame).lineno
        local_vars = inspect.getargvalues(frame).locals
        print(event, file_name, lineno, method_name, param_names, local_vars)
        return self.traceit

We run the code under trace context.

In [10]:
with Tracer() as tracer:
    process_vehicle(VEHICLES[0])
call <string> 1 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350'}
line <string> 2 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350'}
line <string> 3 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', '_': [], 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}
line <string> 4 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', '_': [], 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}
call <string> 1 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line <string> 2 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line <string> 3 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.']}
line <string> 4 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.'], 'iyear': 1997}
line <string> 7 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.'], 'iyear': 1997}
line <string> 8 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], 'iyear': 1997}
return <string> 8 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], 'iyear': 1997}
return <string> 4 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', '_': [], 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}

The main thing that we want out of tracing is a list of assignments of input fragments to different variables. We can use the tracing facility settrace() to get that as we showed above.

However, the settrace() function hooks into the Python debugging facility. When it is in operation, no debugger can hook into the program. That is, if there is a problem with our grammar miner, we will not be able to attach a debugger to it to understand what is happening. This is not ideal. Hence, we limit the tracer to the simplest implementation possible, and implement the core of grammar mining in later stages.

The traceit() function relies on information from the frame variable which exposes Python internals. We define a context class that encapsulates the information that we need from the frame.

Context

The Context class provides easy access to the information such as the current module, and parameter names.

In [11]:
class Context:
    def __init__(self, frame, track_caller=True):
        self.method = inspect.getframeinfo(frame).function
        self.parameter_names = inspect.getargvalues(frame).args
        self.file_name = inspect.getframeinfo(frame).filename
        self.line_no = inspect.getframeinfo(frame).lineno

    def _t(self):
        return (self.file_name, self.line_no, self.method,
                ','.join(self.parameter_names))

    def __repr__(self):
        return "%s:%d:%s(%s)" % self._t()

Here are a few convenience method that operate on the frame to Context.

In [12]:
class Context(Context):
    def extract_vars(self, frame):
        return inspect.getargvalues(frame).locals

    def parameters(self, all_vars):
        return {k: v for k, v in all_vars.items() if k in self.parameter_names}

    def qualified(self, all_vars):
        return {"%s:%s" % (self.method, k): v for k, v in all_vars.items()}

We hook printing the context to our traceit() to see it in action. First we define an log_event() for displaying events.

In [13]:
def log_event(event, var):
    print({'call': '->', 'return': '<-'}.get(event, '  '), var)

And use the log_event() in the traceit() function.

In [14]:
class Tracer(Tracer):
    def traceit(self, frame, event, arg):
        log_event(event, Context(frame))
        return self.traceit

Running process_vehicle() under trace prints the contexts encountered.

In [15]:
with Tracer() as tracer:
    process_vehicle(VEHICLES[0])
-> <string>:1:process_vehicle(vehicle)
   <string>:2:process_vehicle(vehicle)
   <string>:3:process_vehicle(vehicle)
   <string>:4:process_vehicle(vehicle)
-> <string>:1:process_van(year,company,model)
   <string>:2:process_van(year,company,model)
   <string>:3:process_van(year,company,model)
   <string>:4:process_van(year,company,model)
   <string>:7:process_van(year,company,model)
   <string>:8:process_van(year,company,model)
<- <string>:8:process_van(year,company,model)
<- <string>:4:process_vehicle(vehicle)
-> <string>:24:__exit__(self,exc_type,exc_value,tb)
   <string>:25:__exit__(self,exc_type,exc_value,tb)

The trace produced by executing any function can get overwhelmingly large. Hence, we need restrict our attention to specific modules. Further, we also restrict our attention exclusively to str variables since these variables are more likely to contain input fragments. (We will show how to deal with complex objects later in exercises.)

The Context class we developed earlier is used to decide which modules to monitor, and which variables to trace.

We store the current input string so that it can be used to determine if any particular string fragments came from the current input string. Any optional arguments are processed separately.

In [16]:
class Tracer(Tracer):
    def __init__(self, my_input, **kwargs):
        self.options(kwargs)
        self.my_input, self.trace = my_input, []

We use an optional argument files to indicate the specific source files we are interested in, and methods to indicate which specific methods that are of interest. Further, we also use log to specify whether verbose logging should be enabled during trace. We use the log_event() method we defined earlier for logging.

The options processing is as below.

In [17]:
class Tracer(Tracer):
    def options(self, kwargs):
        self.files = kwargs.get('files', [])
        self.methods = kwargs.get('methods', [])
        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None

The files and methods are checked to determine if a particular event should be traced or not

In [18]:
class Tracer(Tracer):
    def tracing_context(self, cxt, event, arg):
        fres = not self.files or any(
            cxt.file_name.endswith(f) for f in self.files)
        mres = not self.methods or any(cxt.method == m for m in self.methods)
        return fres and mres

Similar to the context of events, we also want to restrict our attention to specific variables. For now, we want to focus only on strings. (See the Exercises at the end of the chapter on how to extend it to other kinds of objects).

In [19]:
class Tracer(Tracer):
    def tracing_var(self, k, v):
        return isinstance(v, str)

We modify the traceit() to call an on_event() function with the context information only on the specific events we are interested in.

In [20]:
class Tracer(Tracer):
    def on_event(self, event, arg, cxt, my_vars):
        self.trace.append((event, arg, cxt, my_vars))

    def traceit(self, frame, event, arg):
        cxt = Context(frame)
        if not self.tracing_context(cxt, event, arg):
            return self.traceit
        self.log(event, cxt)

        my_vars = {
            k: v
            for k, v in cxt.extract_vars(frame).items()
            if self.tracing_var(k, v)
        }
        self.on_event(event, arg, cxt, my_vars)
        return self.traceit

The Tracer class can now focus on specific kinds of events on specific files. Further, it provides a first level filter for variables that we find interesting. For example, we want to focus specifically on variables from process_* methods that contain input fragments. Here is how our updated Tracer can be used

In [21]:
with Tracer(VEHICLES[0], methods=INVENTORY_METHODS, log=True) as tracer:
    process_vehicle(VEHICLES[0])
-> <string>:1:process_vehicle(vehicle)
   <string>:2:process_vehicle(vehicle)
   <string>:3:process_vehicle(vehicle)
   <string>:4:process_vehicle(vehicle)
-> <string>:1:process_van(year,company,model)
   <string>:2:process_van(year,company,model)
   <string>:3:process_van(year,company,model)
   <string>:4:process_van(year,company,model)
   <string>:7:process_van(year,company,model)
   <string>:8:process_van(year,company,model)
<- <string>:8:process_van(year,company,model)
<- <string>:4:process_vehicle(vehicle)

The execution produced the following trace.

In [22]:
for t in tracer.trace:
    print(t[0], t[2].method, dict(t[3]))
call process_vehicle {'vehicle': '1997,van,Ford,E350'}
line process_vehicle {'vehicle': '1997,van,Ford,E350'}
line process_vehicle {'vehicle': '1997,van,Ford,E350', 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}
line process_vehicle {'vehicle': '1997,van,Ford,E350', 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}
call process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
return process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
return process_vehicle {'vehicle': '1997,van,Ford,E350', 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}

Since we are saving the input already in Tracer, it is redundant to specify it separately again as an argument.

In [23]:
with Tracer(VEHICLES[0], methods=INVENTORY_METHODS, log=True) as tracer:
    process_vehicle(tracer.my_input)
-> <string>:1:process_vehicle(vehicle)
   <string>:2:process_vehicle(vehicle)
   <string>:3:process_vehicle(vehicle)
   <string>:4:process_vehicle(vehicle)
-> <string>:1:process_van(year,company,model)
   <string>:2:process_van(year,company,model)
   <string>:3:process_van(year,company,model)
   <string>:4:process_van(year,company,model)
   <string>:7:process_van(year,company,model)
   <string>:8:process_van(year,company,model)
<- <string>:8:process_van(year,company,model)
<- <string>:4:process_vehicle(vehicle)

DefineTracker

We define a DefineTracker class that processes the trace from the Tracer. The idea is to store different variable definitions which are input fragments.

The tracker identifies string fragments that are part of the input string, and stores them in a dictionary my_assignments. It saves the trace, and the corresponding input for processing. Finally it calls process() to process the trace it was given. We will start with a simple tracker that relies on certain assumptions, and later see how these assumptions can be relaxed.

In [24]:
class DefineTracker:
    def __init__(self, my_input, trace, **kwargs):
        self.options(kwargs)
        self.my_input = my_input
        self.trace = trace
        self.my_assignments = {}
        self.process()

One of the problems of using substring search is that short string sequences tend to be included in other string sequences even though they may not have come from the original string. That is, say the input fragment is v. It could have equally come from either van or chevy. We rely on being able to predict the exact place input where a given fragment occurred. Hence, we define a constant FRAGMENT_LEN such that we ignore strings up to that length. We also incorporate a logging facility as before.

In [25]:
FRAGMENT_LEN = 3
In [26]:
class DefineTracker(DefineTracker):
    def options(self, kwargs):
        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None
        self.fragment_len = kwargs.get('fragment_len', FRAGMENT_LEN)

Our tracer simply records the variable values as they occur. We next need to check if the variables contain values from the input string. Common ways to do this is to rely on symbolic execution or at least dynamic tainting, which are powerful, but also complex. However, one can obtain a reasonable approximation by simply relying on substring search. That is, we consider any value produced that is a substring of the original input string to have come from the original input.

We define is_input_fragment() method that relies on string inclusion to detect if the string came from the input.

In [27]:
class DefineTracker(DefineTracker):
    def is_input_fragment(self, var, value):
        return len(value) >= self.fragment_len and value in self.my_input

We can use is_input_fragment() to select only a subset of variables defined, as implemented below in fragments().

In [28]:
class DefineTracker(DefineTracker):
    def fragments(self, variables):
        return {k: v for k, v in variables.items(
        ) if self.is_input_fragment(k, v)}

The tracker processes each event, and at each event, it updates the dictionary my_assignments with the current local variables that contain strings that are part of the input. Note that there is a choice here with respect to what happens during reassignment. We can either discard all the reassignments, or keep only the last assignment. Here, we choose the later. If you want the former behavior, check whether the value exists in my_assignments before storing a fragment.

In [29]:
class DefineTracker(DefineTracker):
    def track_event(self, event, arg, cxt, my_vars):
        self.log(event, (cxt.method, my_vars))
        self.my_assignments.update(self.fragments(my_vars))

    def process(self):
        for event, arg, cxt, my_vars in self.trace:
            self.track_event(event, arg, cxt, my_vars)

Using the tracker, we can obtain the input fragments. For example, say we are only interested in strings that are at least 5 characters long.

In [30]:
tracker = DefineTracker(tracer.my_input, tracer.trace, fragment_len=5)
for k, v in tracker.my_assignments.items():
    print(k, '=', repr(v))
vehicle = '1997,van,Ford,E350'

Or strings that are 2 characters long (the default).

In [31]:
tracker = DefineTracker(tracer.my_input, tracer.trace)
for k, v in tracker.my_assignments.items():
    print(k, '=', repr(v))
vehicle = '1997,van,Ford,E350'
model = 'E350'
company = 'Ford'
kind = 'van'
year = '1997'
In [32]:
class DefineTracker(DefineTracker):
    def assignments(self):
        return self.my_assignments.items()

Assembling a Derivation Tree

The input fragments from the DefineTracker only tell half the story. The fragments may be created at different stages of parsing. Hence, we need to assemble the fragments to a derivation tree of the input. The basic idea is as follows:

Our input from the previous step was:

"1997,van,Ford,E350"

We start a derivation tree, and associate it with the start symbol in the grammar.

In [35]:
derivation_tree = (START_SYMBOL, [("1997,van,Ford,E350", [])])
In [36]:
display_tree(derivation_tree)
Out[36]:
%3 0 <start> 1 1997,van,Ford,E350 0->1

The next input was:

vehicle = "1997,van,Ford,E350"

Since vehicle covers the <start> node's value completely, we replace the value with the vehicle node.

In [37]:
derivation_tree = (START_SYMBOL, [('<vehicle>', [("1997,van,Ford,E350", [])],
                                   [])])
In [38]:
display_tree(derivation_tree)
Out[38]:
%3 0 <start> 1 <vehicle> 0->1 2 1997,van,Ford,E350 1->2

The next input was:

model = 'E350'

Traversing the derivation tree from <start>, we see that it replaces a portion of the <vehicle> node's value. Hence we split the <vehicle> node's value to two children, where one corresponds to the value "1997" and the other to ",van,Ford,E350", and replace the first one with the node <model>.

In [39]:
derivation_tree = (START_SYMBOL, [('<vehicle>', [('<model>', [('1997', [])]),
                                                 (",van,Ford,E350", [])], [])])
In [40]:
display_tree(derivation_tree)
Out[40]:
%3 0 <start> 1 <vehicle> 0->1 2 <model> 1->2 4 ,van,Ford,E350 1->4 3 1997 2->3

We perform similar operations for

company = 'Ford'
In [41]:
derivation_tree = (START_SYMBOL, [('<vehicle>', [('<model>', [('1997', [])]),
                                                 (",van,", []),
                                                 ('<company>', [('Ford', [])]),
                                                 (",E350", [])], [])])
In [42]:
display_tree(derivation_tree)
Out[42]:
%3 0 <start> 1 <vehicle> 0->1 2 <model> 1->2 4 ,van, 1->4 5 <company> 1->5 7 ,E350 1->7 3 1997 2->3 6 Ford 5->6

Similarly for

kind = 'van'

and

model = 'E350'
In [43]:
derivation_tree = (START_SYMBOL, [('<vehicle>', [('<model>', [('1997', [])]),
                                                 (",", []),
                                                 ("<kind>", [('van', [])]),
                                                 (",", []),
                                                 ('<company>', [('Ford', [])]),
                                                 (",", []),
                                                 ("<model>", [('E350', [])])
                                                 ], [])])
In [44]:
display_tree(derivation_tree)
Out[44]:
%3 0 <start> 1 <vehicle> 0->1 2 <model> 1->2 4 , 1->4 5 <kind> 1->5 7 , 1->7 8 <company> 1->8 10 , 1->10 11 <model> 1->11 3 1997 2->3 6 van 5->6 9 Ford 8->9 12 E350 11->12

We now develop the complete algorithm with the above described steps. The derivation tree TreeMiner is initialized with the input string, and the variable assignments, and it converts the assignments to the corresponding derivation tree.

In [45]:
class TreeMiner:
    def __init__(self, my_input, my_assignments, **kwargs):
        self.options(kwargs)
        self.my_input = my_input
        self.my_assignments = my_assignments
        self.tree = self.get_derivation_tree()

    def options(self, kwargs):
        self.log = log_call if kwargs.get('log') else lambda _i, _v: None

    def get_derivation_tree(self):
        return (START_SYMBOL, [])

the log_call() is as follows.

In [46]:
def log_call(indent, var):
    print('\t' * indent, var)

The basic idea is as follows:

  • For now, we assume that the value assigned to a variable is stable. That is, it is never reassigned. In particular, there are no recursive calls, or multiple calls to the same function from different parts. (We will show how to overcome this limitation later).
  • For each pair var, value found in my_assignments:
    1. We search for occurrences of value val in the derivation tree recursively.
    2. If an occurrence was found as a value V1 of a node P1, we partition the value of the node P1 into three parts, with the central part matching the value val, and the first and last part, the corresponding prefix and suffix in V1.
    3. Reconstitute the node P1 with three children, where prefix and suffix mentioned earlier are string values, and the matching value val is replaced by a node var with a single value val.

First, we define a wrapper to generate a nonterminal from a variable name.

In [47]:
def to_nonterminal(var):
    return "<" + var.lower() + ">"

The insert_into_tree method accepts a given tree tree and a (k,v) pair. It recursively checks whether the given pair can be applied. If the pair can be applied, it applies the pair and returns True.

In [48]:
class TreeMiner(TreeMiner):
    def insert_into_tree(self, my_tree, pair):
        var, values = my_tree
        k, v = pair
        self.log(1, "- Node: %s\t\t? (%s:%s)" % (var, k, repr(v)))
        applied = False
        for i, value_ in enumerate(values):
            value, arr = value_
            self.log(2, "-> [%d] %s" % (i, repr(value)))
            if is_nonterminal(value):
                applied = self.insert_into_tree(value_, pair)
                if applied:
                    break
            elif v in value:
                prefix_k_suffix = [
                    (k, [[v, []]]) if i == 1 else (e, [])
                    for i, e in enumerate(value.partition(v))
                    if e
                ]
                del values[i]
                for j, rep in enumerate(prefix_k_suffix):
                    values.insert(j + i, rep)
                applied = True

                self.log(2, " > %s" % (repr([i[0] for i in prefix_k_suffix])))
                break
            else:
                continue
        return applied

Here is how insert_into_tree() is used.

In [49]:
tree = (START_SYMBOL, [("1997,van,Ford,E350", [])])
m = TreeMiner('', {}, log=True)

First, we have our input string as the only node.

In [50]:
display_tree(tree)
Out[50]:
%3 0 <start> 1 1997,van,Ford,E350 0->1

Inserting the <vehicle> node.

In [51]:
v = m.insert_into_tree(tree, ('<vehicle>', "1997,van,Ford,E350"))
	 - Node: <start>		? (<vehicle>:'1997,van,Ford,E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['<vehicle>']
In [52]:
display_tree(tree)
Out[52]:
%3 0 <start> 1 <vehicle> 0->1 2 1997,van,Ford,E350 1->2

Inserting <model> node.

In [53]:
v = m.insert_into_tree(tree, ('<model>', 'E350'))
	 - Node: <start>		? (<model>:'E350')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<model>:'E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['1997,van,Ford,', '<model>']
In [54]:
display_tree((tree))
Out[54]:
%3 0 <start> 1 <vehicle> 0->1 2 1997,van,Ford, 1->2 3 <model> 1->3 4 E350 3->4

Inserting <company>.

In [55]:
v = m.insert_into_tree(tree, ('<company>', 'Ford'))
	 - Node: <start>		? (<company>:'Ford')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<company>:'Ford')
		 -> [0] '1997,van,Ford,'
		  > ['1997,van,', '<company>', ',']
In [56]:
display_tree(tree)
Out[56]:
%3 0 <start> 1 <vehicle> 0->1 2 1997,van, 1->2 3 <company> 1->3 5 , 1->5 6 <model> 1->6 4 Ford 3->4 7 E350 6->7

Inserting <kind>.

In [57]:
v = m.insert_into_tree(tree, ('<kind>', 'van'))
	 - Node: <start>		? (<kind>:'van')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<kind>:'van')
		 -> [0] '1997,van,'
		  > ['1997,', '<kind>', ',']
In [58]:
display_tree(tree)
Out[58]:
%3 0 <start> 1 <vehicle> 0->1 2 1997, 1->2 3 <kind> 1->3 5 , 1->5 6 <company> 1->6 8 , 1->8 9 <model> 1->9 4 van 3->4 7 Ford 6->7 10 E350 9->10

Inserting <year>.

In [59]:
v = m.insert_into_tree(tree, ('<year>', '1997'))
	 - Node: <start>		? (<year>:'1997')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<year>:'1997')
		 -> [0] '1997,'
		  > ['<year>', ',']
In [60]:
display_tree(tree)
Out[60]:
%3 0 <start> 1 <vehicle> 0->1 2 <year> 1->2 4 , 1->4 5 <kind> 1->5 7 , 1->7 8 <company> 1->8 10 , 1->10 11 <model> 1->11 3 1997 2->3 6 van 5->6 9 Ford 8->9 12 E350 11->12

To make life simple, we define a wrapper function nt_var() that will convert a token to its corresponding nonterminal symbol.

In [61]:
class TreeMiner(TreeMiner):
    def nt_var(self, var):
        return var if is_nonterminal(var) else to_nonterminal(var)

Now, we need to apply a new definition to an entire grammar.

In [62]:
class TreeMiner(TreeMiner):
    def apply_new_definition(self, tree, var, value):
        nt_var = self.nt_var(var)
        return self.insert_into_tree(tree, (nt_var, value))

This algorithm is implemented as get_derivation_tree().

In [63]:
class TreeMiner(TreeMiner):
    def get_derivation_tree(self):
        tree = (START_SYMBOL, [(self.my_input, [])])

        for var, value in self.my_assignments:
            self.log(0, "%s=%s" % (var, repr(value)))
            self.apply_new_definition(tree, var, value)
        return tree

The TreeMiner is used as follows:

In [64]:
with Tracer(VEHICLES[0]) as tracer:
    process_vehicle(tracer.my_input)
assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()
dt = TreeMiner(tracer.my_input, assignments, log=True)
dt.tree
 vehicle='1997,van,Ford,E350'
	 - Node: <start>		? (<vehicle>:'1997,van,Ford,E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['<vehicle>']
 model='E350'
	 - Node: <start>		? (<model>:'E350')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<model>:'E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['1997,van,Ford,', '<model>']
 company='Ford'
	 - Node: <start>		? (<company>:'Ford')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<company>:'Ford')
		 -> [0] '1997,van,Ford,'
		  > ['1997,van,', '<company>', ',']
 kind='van'
	 - Node: <start>		? (<kind>:'van')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<kind>:'van')
		 -> [0] '1997,van,'
		  > ['1997,', '<kind>', ',']
 year='1997'
	 - Node: <start>		? (<year>:'1997')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<year>:'1997')
		 -> [0] '1997,'
		  > ['<year>', ',']
Out[64]:
('<start>',
 [('<vehicle>',
   [('<year>', [['1997', []]]),
    (',', []),
    ('<kind>', [['van', []]]),
    (',', []),
    ('<company>', [['Ford', []]]),
    (',', []),
    ('<model>', [['E350', []]])])])

The obtained derivation tree is as below.

In [65]:
display_tree(TreeMiner(tracer.my_input, assignments).tree)
Out[65]:
%3 0 <start> 1 <vehicle> 0->1 2 <year> 1->2 4 , 1->4 5 <kind> 1->5 7 , 1->7 8 <company> 1->8 10 , 1->10 11 <model> 1->11 3 1997 2->3 6 van 5->6 9 Ford 8->9 12 E350 11->12

Combining all the pieces:

In [66]:
trees = []
for vehicle in VEHICLES:
    print(vehicle)
    with Tracer(vehicle) as tracer:
        process_vehicle(tracer.my_input)
    assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()
    trees.append((tracer.my_input, assignments))
    for var, val in assignments:
        print(var + " = " + repr(val))
    print()
1997,van,Ford,E350
vehicle = '1997,van,Ford,E350'
model = 'E350'
company = 'Ford'
kind = 'van'
year = '1997'

2000,car,Mercury,Cougar
vehicle = '2000,car,Mercury,Cougar'
model = 'Cougar'
company = 'Mercury'
kind = 'car'
year = '2000'

1999,car,Chevy,Venture
vehicle = '1999,car,Chevy,Venture'
model = 'Venture'
company = 'Chevy'
kind = 'car'
year = '1999'

The corresponding derivation trees are below.

In [67]:
csv_dt = []
for inputstr, assignments in trees:
    print(inputstr)
    dt = TreeMiner(inputstr, assignments)
    csv_dt.append(dt)
    display_tree(dt.tree)
1997,van,Ford,E350
2000,car,Mercury,Cougar
1999,car,Chevy,Venture

Recovering Grammars from Derivation Trees

We define a class Miner that can combine multiple derivation trees to produce the grammar. The initial grammar is empty.

In [68]:
class GrammarMiner:
    def __init__(self):
        self.grammar = {}

The tree_to_grammar() method converts our derivation tree to a grammar by picking one node at a time, and adding it to a grammar. The node name becomes the key, and any list of children it has becomes another alternative for that key.

In [69]:
class GrammarMiner(GrammarMiner):
    def tree_to_grammar(self, tree):
        node, children = tree
        one_alt = [ck for ck, gc in children]
        hsh = {node: [one_alt] if one_alt else []}
        for child in children:
            if not is_nonterminal(child[0]):
                continue
            chsh = self.tree_to_grammar(child)
            for k in chsh:
                if k not in hsh:
                    hsh[k] = chsh[k]
                else:
                    hsh[k].extend(chsh[k])
        return hsh
In [70]:
gm = GrammarMiner()
gm.tree_to_grammar(csv_dt[0].tree)
Out[70]:
{'<start>': [['<vehicle>']],
 '<vehicle>': [['<year>', ',', '<kind>', ',', '<company>', ',', '<model>']],
 '<year>': [['1997']],
 '<kind>': [['van']],
 '<company>': [['Ford']],
 '<model>': [['E350']]}

The grammar being generated here is canonical. We define a function readable() that takes in a canonical grammar and returns it in a readable form.

In [71]:
def readable(grammar):
    def readable_rule(rule):
        return ''.join(rule)

    return {k: list(set(readable_rule(a) for a in grammar[k]))
            for k in grammar}
In [72]:
syntax_diagram(readable(gm.tree_to_grammar(csv_dt[0].tree)))
start
vehicle
vehicle
year , kind , company , model
year
1997
kind
van
company
Ford
model
E350

The add_tree() method gets a combined list of non-terminals from current grammar, and the tree to be added to the grammar, and updates the definitions of each non-terminal.

In [74]:
class GrammarMiner(GrammarMiner):
    def add_tree(self, t):
        t_grammar = self.tree_to_grammar(t.tree)
        self.grammar = {
            key: self.grammar.get(key, []) + t_grammar.get(key, [])
            for key in itertools.chain(self.grammar.keys(), t_grammar.keys())
        }

The add_tree() is used as follows:

In [75]:
inventory_grammar = GrammarMiner()
for dt in csv_dt:
    inventory_grammar.add_tree(dt)
In [76]:
syntax_diagram(readable(inventory_grammar.grammar))
start
vehicle
vehicle
year , kind , company , model
year
2000 1999 1997
kind
van car
company
Ford Chevy Mercury
model
Venture E350 Cougar

Given execution traces from various inputs, one can define update_grammar() to obtain the complete grammar from the traces.

In [77]:
class GrammarMiner(GrammarMiner):
    def update_grammar(self, inputstr, trace):
        at = self.create_tracker(inputstr, trace)
        dt = self.create_tree_miner(inputstr, at.assignments())
        self.add_tree(dt)
        return self.grammar

    def create_tracker(self, *args):
        return DefineTracker(*args)

    def create_tree_miner(self, *args):
        return TreeMiner(*args)

The complete grammar recovery is implemented in recover_grammar().

In [78]:
def recover_grammar(fn, inputs, **kwargs):
    miner = GrammarMiner()
    for inputstr in inputs:
        with Tracer(inputstr, **kwargs) as tracer:
            fn(tracer.my_input)
        miner.update_grammar(tracer.my_input, tracer.trace)
    return readable(miner.grammar)

Note that the grammar could have been retrieved directly from the tracker, without the intermediate derivation tree stage. However, going through the derivation tree allows one to inspect the inputs being fragmented and verify that it happens correctly.

Example 1. Recovering the Inventory Grammar

In [79]:
inventory_grammar = recover_grammar(process_vehicle, VEHICLES)
In [80]:
inventory_grammar
Out[80]:
{'<start>': ['<vehicle>'],
 '<vehicle>': ['<year>,<kind>,<company>,<model>'],
 '<year>': ['2000', '1999', '1997'],
 '<kind>': ['van', 'car'],
 '<company>': ['Ford', 'Chevy', 'Mercury'],
 '<model>': ['Venture', 'E350', 'Cougar']}

Example 2. Recovering URL Grammar

Our algorithm is robust enough to recover grammar from real world programs. For example, the urlparse function in the Python urlib module accepts the following sample URLs.

In [81]:
URLS = [
    'http://user:pass@www.google.com:80/?q=path#ref',
    'https://www.cispa.saarland:80/',
    'http://www.fuzzingbook.org/#News',
]

The urllib caches its intermediate results for faster access. Hence, we need to disable it using clear_cache() after every invocation.

We use the sample URLs to recover grammar as follows. The urlparse function tends to cache its previous parsing results. Hence, we define a new method url_parse() that clears the cache before each call.

In [83]:
def url_parse(url):
    clear_cache()
    urlparse(url)
In [84]:
trees = []
for url in URLS:
    print(url)
    with Tracer(url) as tracer:
        url_parse(tracer.my_input)
    assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()
    trees.append((tracer.my_input, assignments))
    for var, val in assignments:
        print(var + " = " + repr(val))
    print()


url_dt = []
for inputstr, assignments in trees:
    print(inputstr)
    dt = TreeMiner(inputstr, assignments)
    url_dt.append(dt)
    display_tree(dt.tree)
http://user:pass@www.google.com:80/?q=path#ref
url = 'http://user:pass@www.google.com:80/?q=path#ref'
scheme = 'http'
netloc = 'user:pass@www.google.com:80'
fragment = 'ref'
query = 'q=path'

https://www.cispa.saarland:80/
url = 'https://www.cispa.saarland:80/'
rest = '//www.cispa.saarland:80/'
scheme = 'https'
netloc = 'www.cispa.saarland:80'

http://www.fuzzingbook.org/#News
url = 'http://www.fuzzingbook.org/#News'
scheme = 'http'
netloc = 'www.fuzzingbook.org'
fragment = 'News'

http://user:pass@www.google.com:80/?q=path#ref
https://www.cispa.saarland:80/
http://www.fuzzingbook.org/#News

Using url_parse() to recover grammar.

In [85]:
url_grammar = recover_grammar(url_parse, URLS, files=['urllib/parse.py'])
In [86]:
syntax_diagram(url_grammar)
start
url
url
scheme : rest scheme :// netloc /# fragment scheme :// netloc /? query # fragment
scheme
https http
netloc
www.cispa.saarland:80 user:pass@www.google.com:80 www.fuzzingbook.org
query
q=path
fragment
News ref
rest
// netloc /

The recovered grammar describes the URL format reasonably well.

Fuzzing

We can now use our recovered grammar for fuzzing as follows.

First, the inventory grammar.

In [87]:
f = GrammarFuzzer(inventory_grammar)
for _ in range(10):
    print(f.fuzz())
1997,van,Ford,Venture
1999,van,Chevy,E350
1997,car,Ford,Cougar
2000,van,Mercury,Cougar
1999,van,Ford,Venture
1999,van,Ford,E350
1997,van,Chevy,Venture
1997,van,Chevy,Venture
1997,van,Ford,E350
2000,van,Chevy,Venture

Next, the URL grammar.

In [88]:
f = GrammarFuzzer(url_grammar)
for _ in range(10):
    print(f.fuzz())
http://www.cispa.saarland:80/#ref
http://www.fuzzingbook.org/?q=path#ref
http://user:pass@www.google.com:80/?q=path#ref
https://www.fuzzingbook.org/?q=path#News
https://www.fuzzingbook.org/#ref
http://www.cispa.saarland:80/
https://www.cispa.saarland:80/
http://www.cispa.saarland:80/
https://www.cispa.saarland:80/#ref
http://user:pass@www.google.com:80/

What this means is that we can now take a program and a few samples, extract its grammar, and then use this very grammar for fuzzing. Now that's quite an opportunity!

Problems with the Simple Miner

One of the problems with our simple grammar miner is the assumption that the values assigned to variables are stable. Unfortunately, that may not hold true in all cases. For example, here is a URL with a slightly different format.

In [89]:
URLS_X = URLS + ['ftp://freebsd.org/releases/5.8']

The grammar generated from this set of samples is not as nice as what we got earlier

In [90]:
url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])
In [91]:
syntax_diagram(url_grammar)
start
url scheme :// netloc url
url
scheme :// netloc /# fragment scheme : rest scheme :// netloc /? query # fragment /releases/5.8
scheme
ftp https http
netloc
www.cispa.saarland:80 freebsd.org user:pass@www.google.com:80 www.fuzzingbook.org
query
q=path
fragment
News ref
rest
// netloc /

Clearly, something has gone wrong.

To investigate why the url definition has gone wrong, let us inspect the trace for the URL.

In [92]:
clear_cache()
with Tracer(URLS_X[0]) as tracer:
    urlparse(tracer.my_input)
for i, t in enumerate(tracer.trace):
    if t[0] in {'call', 'line'} and 'parse.py' in str(t[2]) and t[3]:
        print(i, t[2]._t()[1], t[3:])
0 361 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
1 367 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
5 119 ({'arg': ''},)
6 116 ({'arg': ''},)
7 121 ({'arg': ''},)
8 122 ({'arg': ''},)
10 368 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
11 394 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
12 400 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
16 119 ({'arg': ''},)
17 116 ({'arg': ''},)
18 121 ({'arg': ''},)
19 122 ({'arg': ''},)
21 401 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
22 402 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
23 403 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
24 404 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
25 406 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
26 408 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
27 409 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
28 410 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
29 411 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
30 412 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
31 413 ({'scheme': 'http', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
32 414 ({'scheme': 'http', 'url': '//user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
33 415 ({'scheme': 'http', 'url': '//user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
34 386 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)
35 387 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)
36 388 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)
37 389 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)
38 390 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)
39 391 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)
40 388 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)
41 389 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)
42 390 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)
43 391 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)
44 388 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)
45 389 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
46 390 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
47 391 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
48 388 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
49 392 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
51 416 ({'scheme': 'http', 'url': '/?q=path#ref', 'fragment': '', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
52 417 ({'scheme': 'http', 'url': '/?q=path#ref', 'fragment': '', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
53 419 ({'scheme': 'http', 'url': '/?q=path#ref', 'fragment': '', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
54 420 ({'scheme': 'http', 'url': '/?q=path#ref', 'fragment': '', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
55 421 ({'scheme': 'http', 'url': '/?q=path', 'fragment': 'ref', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
56 422 ({'scheme': 'http', 'url': '/?q=path', 'fragment': 'ref', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
57 423 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
61 424 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
62 425 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
67 369 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
68 370 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
69 373 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
70 374 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80', 'params': ''},)
74 375 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80', 'params': ''},)

Notice how the value of url changes as the parsing progresses? This violates our assumption that the value assigned to a variable is stable. We next look at how this limitation can be removed.

Grammar Miner with Reassignment

One way to uniquely identify different variables is to annotate them with line numbers both when they are defined and also when their value changes. Consider the code fragment below

Tracking variable assignment locations

In [93]:
def C(cp_1):
    c_2 = cp_1 + '@2'
    c_3 = c_2 + '@3'
    return c_3


def B(bp_7):
    b_8 = bp_7 + '@8'
    return C(b_8)


def A(ap_12):
    a_13 = ap_12 + '@13'
    a_14 = B(a_13) + '@14'
    a_14 = a_14 + '@15'
    a_13 = a_14 + '@16'
    a_14 = B(a_13) + '@17'
    a_14 = B(a_13) + '@18'

Notice how all variables are either named corresponding to either where they are defined, or the value is annotated to indicate that it was changed.

Let us run this under the trace.

In [94]:
with Tracer('____') as tracer:
    A(tracer.my_input)

for t in tracer.trace:
    print(t[0], "%d:%s" % (t[2].line_no, t[2].method), t[3])
call 12:A {'ap_12': '____'}
line 13:A {'ap_12': '____'}
line 14:A {'ap_12': '____', 'a_13': '____@13'}
call 7:B {'bp_7': '____@13'}
line 8:B {'bp_7': '____@13'}
line 9:B {'bp_7': '____@13', 'b_8': '____@13@8'}
call 1:C {'cp_1': '____@13@8'}
line 2:C {'cp_1': '____@13@8'}
line 3:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2'}
line 4:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2', 'c_3': '____@13@8@2@3'}
return 4:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2', 'c_3': '____@13@8@2@3'}
return 9:B {'bp_7': '____@13', 'b_8': '____@13@8'}
line 15:A {'ap_12': '____', 'a_13': '____@13', 'a_14': '____@13@8@2@3@14'}
line 16:A {'ap_12': '____', 'a_13': '____@13', 'a_14': '____@13@8@2@3@14@15'}
line 17:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15'}
call 7:B {'bp_7': '____@13@8@2@3@14@15@16'}
line 8:B {'bp_7': '____@13@8@2@3@14@15@16'}
line 9:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}
call 1:C {'cp_1': '____@13@8@2@3@14@15@16@8'}
line 2:C {'cp_1': '____@13@8@2@3@14@15@16@8'}
line 3:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2'}
line 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}
return 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}
return 9:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}
line 18:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15@16@8@2@3@17'}
call 7:B {'bp_7': '____@13@8@2@3@14@15@16'}
line 8:B {'bp_7': '____@13@8@2@3@14@15@16'}
line 9:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}
call 1:C {'cp_1': '____@13@8@2@3@14@15@16@8'}
line 2:C {'cp_1': '____@13@8@2@3@14@15@16@8'}
line 3:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2'}
line 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}
return 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}
return 9:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}
return 18:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15@16@8@2@3@18'}
call 24:__exit__ {}
line 25:__exit__ {}

Each variables were referenced first as follows:

  • cp_1 -- call 1:C
  • c_2 -- line 3:C (but the previous event was line 2:C)
  • c_3 -- line 4:C (but the previous event was line 3:C)
  • bp_7 -- call 7:B
  • b_8 -- line 9:B (but the previous event was line 8:B)
  • ap_12 -- call 12:A
  • a_13 -- line 14:A (but the previous event was line 13:A)
  • a_14 -- line 15:A (the previous event was return 9:B. However, the previous event in A was line 14:A)
  • reassign a_14 at 15 -- line 16:A (the previous event was line 15:A)
  • reassign a_13 at 16 -- line 17:A (the previous event was line 16:A)
  • reassign a_14 at 17 -- return 17:A (the previous event in A was line 17:A)
  • reassign a_14 at 18 -- return 18:A (the previous event in A was line 18:A)

So, our observations are that, if it is a call, the current location is the right one for any new variables being defined. On the other hand, if the variable being referenced for the first time (or reassigned a new value), then the right location to consider is the previous location in the same method invocation. Next, let us see how we can incorporate this information into variable naming.

Next, we need a way to track the individual method calls as they are being made. For this we define the class CallStack. Each method invocation gets a separate identifier, and when the method call is over, the identifier is reset.

CallStack

In [95]:
class CallStack:
    def __init__(self, **kwargs):
        self.options(kwargs)
        self.method_id = (START_SYMBOL, 0)
        self.method_register = 0
        self.mstack = [self.method_id]

    def enter(self, method):
        self.method_register += 1
        self.method_id = (method, self.method_register)
        self.log('call', "%s%s" % (self.indent(), str(self)))
        self.mstack.append(self.method_id)

    def leave(self):
        self.mstack.pop()
        self.log('return', "%s%s" % (self.indent(), str(self)))
        self.method_id = self.mstack[-1]

A few extra functions to make life simpler.

In [96]:
class CallStack(CallStack):
    def options(self, kwargs):
        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None

    def indent(self):
        return len(self.mstack) * "\t"

    def at(self, n):
        return self.mstack[n]

    def __len__(self):
        return len(mstack) - 1

    def __str__(self):
        return "%s:%d" % self.method_id

    def __repr__(self):
        return repr(self.method_id)

We also define a convenience method to display a given stack.

In [97]:
def display_stack(istack):
    def stack_to_tree(stack):
        current, *rest = stack
        if not rest:
            return (repr(current), [])
        return (repr(current), [stack_to_tree(rest)])
    display_tree(stack_to_tree(istack.mstack), graph_attr=lr_graph)

Here is how we can use the CallStack.

In [98]:
cs = CallStack()
display_stack(cs)
cs
Out[98]:
('<start>', 0)
In [99]:
cs.enter('hello')
display_stack(cs)
cs
Out[99]:
('hello', 1)
In [100]:
cs.enter('world')
display_stack(cs)
cs
Out[100]:
('world', 2)
In [101]:
cs.leave()
display_stack(cs)
cs
Out[101]:
('hello', 1)
In [102]:
cs.enter('world')
display_stack(cs)
cs
Out[102]:
('world', 3)
In [103]:
cs.leave()
display_stack(cs)
cs
Out[103]:
('hello', 1)

In order to account for variable reassignments, we need to have a more intelligent data structure than a dictionary for storing variables. We first define a simple interface Vars. It acts as a container for variables, and is instantiated at my_assignments.

Vars

The Vars stores references to variables as they occur during parsing in its internal dictionary defs. We initialize the dictionary with the original string.

In [104]:
class Vars:
    def __init__(self, original):
        self.defs = {}
        self.my_input = original

The dictionary needs two methods: update() that takes a set of key-value pairs to update itself, and _set_kv() that updates a particular key-value pair.

In [105]:
class Vars(Vars):
    def _set_kv(self, k, v):
        self.defs[k] = v

    def __setitem__(self, k, v):
        self._set_kv(k, v)

    def update(self, v):
        for k, v in v.items():
            self._set_kv(k, v)

The vars is a proxy for the internal dictionary. For example, here is how one can use it.

In [106]:
v = Vars('')
v.defs
Out[106]:
{}
In [107]:
v['x'] = 'X'
v.defs
Out[107]:
{'x': 'X'}
In [108]:
v.update({'x': 'x', 'y': 'y'})
v.defs
Out[108]:
{'x': 'x', 'y': 'y'}

AssignmentVars

We now extend the simple Vars to account for variable reassignments. For this, we define AssignmentVars.

The idea for detecting reassignments and renaming variables is as follows: We keep track of the previous reassignments to particular variables using accessed_seq_var. It contains the last rename of any particular variable as its corresponding value. The new_vars contains a list of all new variables that were added on this iteration.

In [109]:
class AssignmentVars(Vars):
    def __init__(self, original):
        super().__init__(original)
        self.accessed_seq_var = {}
        self.var_def_lines = {}
        self.current_event = None
        self.new_vars = set()
        self.method_init()

The method_init() method takes care of keeping track of method invocations using records saved in the call_stack. event_locations is for keeping track of the locations accessed within this method. This is used for line number tracking of variable definitions.

In [110]:
class AssignmentVars(AssignmentVars):
    def method_init(self):
        self.call_stack = CallStack()
        self.event_locations = {self.call_stack.method_id: []}

The update() is now modified to track the changed line numbers if any, using var_location_register(). We reinitialize the new_vars after use for the next event.

In [111]:
class AssignmentVars(AssignmentVars):
    def update(self, v):
        for k, v in v.items():
            self._set_kv(k, v)
        self.var_location_register(self.new_vars)
        self.new_vars = set()

The variable name now incorporates an index of how many reassignments it has gone through, effectively making each reassignment a unique variable.

In [112]:
class AssignmentVars(AssignmentVars):
    def var_name(self, var):
        return (var, self.accessed_seq_var[var])

While storing variables, we need to first check whether it was previously known. If it is not, we need to initialize the rename count. This is accomplished by var_access.

In [113]:
class AssignmentVars(AssignmentVars):
    def var_access(self, var):
        if var not in self.accessed_seq_var:
            self.accessed_seq_var[var] = 0
        return self.var_name(var)

During a variable reassignment, we update the accessed_seq_var to reflect the new count.

In [114]:
class AssignmentVars(AssignmentVars):
    def var_assign(self, var):
        self.accessed_seq_var[var] += 1
        self.new_vars.add(self.var_name(var))
        return self.var_name(var)

These methods can be used as follows

In [115]:
sav = AssignmentVars('')
sav.defs
Out[115]:
{}
In [116]:
sav.var_access('v1')
Out[116]:
('v1', 0)
In [117]:
sav.var_assign('v1')
Out[117]:
('v1', 1)

Assigning to it again increments the counter.

In [118]:
sav.var_assign('v1')
Out[118]:
('v1', 2)

The core of the logic is in _set_kv(). When a variable is being assigned, we get the sequenced variable name s_var. If the sequenced variable name was previously unknown in defs, then we have no further concerns. We add the sequenced variable to defs.

If the variable is previously known, then it is an indication of a possible reassignment. In this case, we look at the value the variable is holding. We check if the value changed. If it has not, then it is not.

If the value has changed, it is a reassignment. We first increment the variable usage sequence using var_assign, retrieve the new name, update the new name in defs.

In [119]:
class AssignmentVars(AssignmentVars):
    def _set_kv(self, var, val):
        s_var = self.var_access(var)
        if s_var in self.defs and self.defs[s_var] == val:
            return
        self.defs[self.var_assign(var)] = val

Here is how it can be used. Assigning a variable the first time initializes its counter.

In [120]:
sav = AssignmentVars('')
sav['x'] = 'X'
sav.defs
Out[120]:
{('x', 1): 'X'}

If the variable is assigned again with the same value, it is probably not a reassignment.

In [121]:
sav['x'] = 'X'
sav.defs
Out[121]:
{('x', 1): 'X'}

However, if the value changed, it is a reassignment.

In [122]:
sav['x'] = 'Y'
sav.defs
Out[122]:
{('x', 1): 'X', ('x', 2): 'Y'}

There is a subtlety here. It is possible for a child method to be called from the middle of a parent method, and for both to use the same variable name with different values. In this case, when the child returns, parent will have the old variable with old value in context. With our implementation, we consider this as a reassignment. However, this is OK because adding a new reassignment is harmless, but missing one is not. Further, we will discuss later how this can be avoided.

We also define book keeping codes for register_event() method_enter() and method_exit() which are the methods responsible for keeping track of the method stack. The basic idea is that, each method_enter() represents a new method invocation. Hence it merits a new method id, which is generated from the method_register, and saved in the method_id. Since this is a new method, the method stack is extended by one element with this id. In the case of method_exit(), we pop the method stack, and reset the current method_id to what was below the current one.

In [123]:
class AssignmentVars(AssignmentVars):
    def method_enter(self, cxt, my_vars):
        self.current_event = 'call'
        self.call_stack.enter(cxt.method)
        self.event_locations[self.call_stack.method_id] = []
        self.register_event(cxt)
        self.update(my_vars)

    def method_exit(self, cxt, my_vars):
        self.current_event = 'return'
        self.register_event(cxt)
        self.update(my_vars)
        self.call_stack.leave()

    def method_statement(self, cxt, my_vars):
        self.current_event = 'line'
        self.register_event(cxt)
        self.update(my_vars)

For each of the method events, we also register the event using register_event() which keeps track of the line numbers that were referenced in this method.

In [124]:
class AssignmentVars(AssignmentVars):
    def register_event(self, cxt):
        self.event_locations[self.call_stack.method_id].append(cxt.line_no)

The var_location_register() keeps the locations of newly added variables. The definition location of variables in a call is the current location. However, for a line, it would be the previous event in the current method.

In [125]:
class AssignmentVars(AssignmentVars):
    def var_location_register(self, my_vars):
        def loc(mid):
            if self.current_event == 'call':
                return self.event_locations[mid][-1]
            elif self.current_event == 'line':
                return self.event_locations[mid][-2]
            elif self.current_event == 'return':
                return self.event_locations[mid][-2]
            else:
                assert False

        my_loc = loc(self.call_stack.method_id)
        for var in my_vars:
            self.var_def_lines[var] = my_loc

We define defined_vars() which returns the names of variables annotated with the line numbers as below.

In [126]:
class AssignmentVars(AssignmentVars):
    def defined_vars(self, formatted=True):
        def fmt(k):
            v = (k[0], self.var_def_lines[k])
            return "%s@%s" % v if formatted else v

        return [(fmt(k), v) for k, v in self.defs.items()]

Similar to defined_vars() we define seq_vars() which annotates different variables with the number of times it was used.

In [127]:
class AssignmentVars(AssignmentVars):
    def seq_vars(self, formatted=True):
        def fmt(k):
            v = (k[0], self.var_def_lines[k], k[1])
            return "%s@%s:%s" % v if formatted else v

        return {fmt(k): v for k, v in self.defs.items()}

AssignmentTracker

The AssignmentTracker keeps the assignment definitions using the AssignmentVars we defined previously.

In [128]:
class AssignmentTracker(DefineTracker):
    def __init__(self, my_input, trace, **kwargs):
        self.options(kwargs)
        self.my_input = my_input

        self.my_assignments = self.create_assignments(my_input)

        self.trace = trace
        self.process()

    def create_assignments(self, *args):
        return AssignmentVars(*args)

To fine-tune the process, we define an optional parameter called track_return. During tracing a method return, Python produces a virtual variable that contains the result of the returned value. If the track_return is set, we capture this value as a variable.

  • track_return -- if true, add a virtual variable to the Vars representing the return value
In [129]:
class AssignmentTracker(AssignmentTracker):
    def options(self, kwargs):
        self.track_return = kwargs.get('track_return', False)
        super().options(kwargs)

There can be different kinds of events during a trace, which includes call when a function is entered, return when the function returns, exception when an exception is thrown and line when a statement is executed.

The previous Tracker was too simplistic in that it did not distinguish between the different events. We rectify that and define on_call(), on_return(), and on_line() respectively that gets called on their corresponding events.

Note that on_line() is called also for on_return(). The reason is that, Python invokes the trace function before the corresponding line is executed. Hence, effectively, the on_return() is called with the binding produced by the execution of the previous statement in the environment. Our processing in effect is done on values that were bound by the previous statement. Hence, calling on_line() here is appropriate as it provides the event handler a chance to work on the previous binding.

In [130]:
class AssignmentTracker(AssignmentTracker):
    def on_call(self, arg, cxt, my_vars):
        my_vars = cxt.parameters(my_vars)
        self.my_assignments.method_enter(cxt, self.fragments(my_vars))

    def on_line(self, arg, cxt, my_vars):
        self.my_assignments.method_statement(cxt, self.fragments(my_vars))

    def on_return(self, arg, cxt, my_vars):
        self.on_line(arg, cxt, my_vars)
        my_vars = {'<-%s' % cxt.method: arg} if self.track_return else {}
        self.my_assignments.method_exit(cxt, my_vars)

    def on_exception(self, arg, cxt, my_vara):
        return

    def track_event(self, event, arg, cxt, my_vars):
        self.current_event = event
        dispatch = {
            'call': self.on_call,
            'return': self.on_return,
            'line': self.on_line,
            'exception': self.on_exception
        }
        dispatch[event](arg, cxt, my_vars)

We can now use AssignmentTracker to track the different variables. To verify that our variable line number inference works, we recover definitions from the functions A, B and C (with data annotations removed so that the input fragments are correctly identified).

In [131]:
def C(cp_1):
    c_2 = cp_1
    c_3 = c_2
    return c_3


def B(bp_7):
    b_8 = bp_7
    return C(b_8)


def A(ap_12):
    a_13 = ap_12
    a_14 = B(a_13)
    a_14 = a_14
    a_13 = a_14
    a_14 = B(a_13)
    a_14 = B(a_14)[3:]

Running A() with sufficient input.

In [132]:
with Tracer('---xxx') as tracer:
    A(tracer.my_input)
tracker = AssignmentTracker(tracer.my_input, tracer.trace, log=True)
for k, v in tracker.my_assignments.seq_vars().items():
    print(k, '=', repr(v))
print()
for k, v in tracker.my_assignments.defined_vars(formatted=True):
    print(k, '=', repr(v))
ap_12@12:1 = '---xxx'
a_13@13:1 = '---xxx'
bp_7@7:1 = '---xxx'
b_8@8:1 = '---xxx'
cp_1@1:1 = '---xxx'
c_2@2:1 = '---xxx'
c_3@3:1 = '---xxx'
a_14@14:1 = '---xxx'
a_14@18:2 = 'xxx'

ap_12@12 = '---xxx'
a_13@13 = '---xxx'
bp_7@7 = '---xxx'
b_8@8 = '---xxx'
cp_1@1 = '---xxx'
c_2@2 = '---xxx'
c_3@3 = '---xxx'
a_14@14 = '---xxx'
a_14@18 = 'xxx'

As can be seen, the line numbers are now correctly identified for each variables.

Let us try retrieving the assignments for a real world example.

In [133]:
traces = []
for inputstr in URLS_X:
    clear_cache()
    with Tracer(inputstr, files=['urllib/parse.py']) as tracer:
        urlparse(tracer.my_input)
    traces.append((tracer.my_input, tracer.trace))

    tracker = AssignmentTracker(tracer.my_input, tracer.trace, log=True)
    for k, v in tracker.my_assignments.defined_vars():
        print(k, '=', repr(v))
    print()
url@361 = 'http://user:pass@www.google.com:80/?q=path#ref'
scheme@412 = 'http'
url@413 = '//user:pass@www.google.com:80/?q=path#ref'
url@415 = '/?q=path#ref'
netloc@415 = 'user:pass@www.google.com:80'
url@420 = '/?q=path'
fragment@420 = 'ref'
query@422 = 'q=path'
url@368 = 'http://user:pass@www.google.com:80/?q=path#ref'

url@361 = 'https://www.cispa.saarland:80/'
rest@432 = '//www.cispa.saarland:80/'
scheme@435 = 'https'
url@435 = '//www.cispa.saarland:80/'
netloc@438 = 'www.cispa.saarland:80'
url@368 = 'https://www.cispa.saarland:80/'

url@361 = 'http://www.fuzzingbook.org/#News'
scheme@412 = 'http'
url@413 = '//www.fuzzingbook.org/#News'
url@415 = '/#News'
netloc@415 = 'www.fuzzingbook.org'
fragment@420 = 'News'
url@368 = 'http://www.fuzzingbook.org/#News'

url@361 = 'ftp://freebsd.org/releases/5.8'
rest@432 = '//freebsd.org/releases/5.8'
scheme@435 = 'ftp'
url@435 = '//freebsd.org/releases/5.8'
url@438 = '/releases/5.8'
netloc@438 = 'freebsd.org'
url@368 = 'ftp://freebsd.org/releases/5.8'
url@369 = '/releases/5.8'

The line numbers of variables can be verified from the source code of urllib/parse.py.

Recovering a Derivation Tree

Does handling variable reassignments help with our URL examples? We look at these next.

In [134]:
class TreeMiner(TreeMiner):
    def get_derivation_tree(self):
        tree = (START_SYMBOL, [(self.my_input, [])])
        for var, value in self.my_assignments:
            self.log(0, "%s=%s" % (var, repr(value)))
            self.apply_new_definition(tree, var, value)
        return tree

Example 1: Recovering URL Derivation Tree

First we obtain the derivation tree of the URL 1

URL 1 derivation tree
In [135]:
clear_cache()
with Tracer(URLS_X[0], files=['urllib/parse.py']) as tracer:
    urlparse(tracer.my_input)
sm = AssignmentTracker(tracer.my_input, tracer.trace)
dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())
display_tree(dt.tree)
Out[135]:
%3 0 <start> 1 <url@361> 0->1 2 <scheme@412> 1->2 4 : 1->4 5 <url@413> 1->5 3 http 2->3 6 // 5->6 7 <netloc@415> 5->7 9 <url@415> 5->9 8 user:pass@www.google.com:80 7->8 10 <url@420> 9->10 14 # 9->14 15 <fragment@420> 9->15 11 /? 10->11 12 <query@422> 10->12 13 q=path 12->13 16 ref 15->16

Next, we obtain the derivation tree of URL 4

URL 4 derivation tree
In [136]:
clear_cache()
with Tracer(URLS_X[-1], files=['urllib/parse.py']) as tracer:
    urlparse(tracer.my_input)
sm = AssignmentTracker(tracer.my_input, tracer.trace)
dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())
display_tree(dt.tree)
Out[136]:
%3 0 <start> 1 <url@361> 0->1 2 <scheme@435> 1->2 4 : 1->4 5 <rest@432> 1->5 3 ftp 2->3 6 <url@435> 5->6 7 // 6->7 8 <netloc@438> 6->8 10 <url@438> 6->10 9 freebsd.org 8->9 11 <url@369> 10->11 12 /releases/5.8 11->12

The derivation trees seem to belong to the same grammar. Hence, we obtain the grammar for the complete set. First, we update the recover_grammar() to use AssignTracker.

Recover Grammar

In [137]:
class GrammarMiner(GrammarMiner):
    def update_grammar(self, inputstr, trace):
        at = self.create_tracker(inputstr, trace)
        dt = self.create_tree_miner(inputstr, at.my_assignments.defined_vars())
        self.add_tree(dt)
        return self.grammar

    def create_tracker(self, *args):
        return AssignmentTracker(*args)

    def create_tree_miner(self, *args):
        return TreeMiner(*args)

Next, we use the modified recover_grammar() on derivation trees obtained from URLs.

In [138]:
url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])

The recovered grammar is below.

In [139]:
syntax_diagram(url_grammar)
start
url@361
url@361
scheme@412 : url@413 scheme@435 : rest@432
scheme@412
http
url@413
// netloc@415 url@415
netloc@415
www.fuzzingbook.org user:pass@www.google.com:80
url@415
/# fragment@420 url@420 # fragment@420
url@420
/? query@422
query@422
q=path
fragment@420
News ref
scheme@435
ftp https
rest@432
url@435
url@435
// netloc@438 url@438 // netloc@438 /
netloc@438
freebsd.org www.cispa.saarland:80
url@438
url@369
url@369
/releases/5.8

Let us fuzz a little to see if the produced values are sane.

In [140]:
f = GrammarFuzzer(url_grammar)
for _ in range(10):
    print(f.fuzz())
https://freebsd.org/releases/5.8
http://www.fuzzingbook.org/#News
http://user:pass@www.google.com:80/#News
ftp://www.cispa.saarland:80/
http://user:pass@www.google.com:80/#News
http://user:pass@www.google.com:80/#News
https://www.cispa.saarland:80/releases/5.8
http://user:pass@www.google.com:80/?q=path#ref
https://www.cispa.saarland:80/releases/5.8
http://www.fuzzingbook.org/#ref

Our modifications does seem to help. Next, we check whether we can still retrieve the grammar for inventory.

Example 2: Recovering Inventory Grammar

In [141]:
inventory_grammar = recover_grammar(process_vehicle, VEHICLES)
In [142]:
syntax_diagram(inventory_grammar)
start
vehicle@1
vehicle@1
year@2 , kind@2 , company@2 , model@2
year@2
2000 1999 1997
kind@2
van car
company@2
Ford Chevy Mercury
model@2
Venture E350 Cougar

Using fuzzing to produce values from the grammar.

In [143]:
f = GrammarFuzzer(inventory_grammar)
for _ in range(10):
    print(f.fuzz())
1997,van,Mercury,Cougar
2000,van,Mercury,E350
2000,van,Chevy,E350
1997,car,Mercury,Venture
1999,van,Mercury,Venture
1999,van,Mercury,Venture
1999,car,Mercury,Cougar
2000,van,Mercury,E350
1999,car,Chevy,Cougar
1999,van,Mercury,Venture

Problems with the Grammar Miner with Reassignment

One of the problems with our grammar miner is that it doesn't yet account for the current context. That is, when replacing, a variable can replace tokens that it does not have access to (and hence, it is not a fragment of). Consider this example.

In [144]:
with Tracer(INVENTORY) as tracer:
    process_inventory(tracer.my_input)
sm = AssignmentTracker(tracer.my_input, tracer.trace)
dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())
display_tree(dt.tree, graph_attr=lr_graph)
Out[144]:
%3 0 <start> 1 <inventory@1> 0->1 2 <vehicle@3> 1->2 14 1->14 15 <vehicle@3> 1->15 27 1->27 28 <vehicle@3> 1->28 3 <year@2> 2->3 5 , 2->5 6 <kind@2> 2->6 8 , 2->8 9 <company@2> 2->9 11 , 2->11 12 <model@2> 2->12 4 1997 3->4 7 van 6->7 10 Ford 9->10 13 E350 12->13 16 <year@2> 15->16 18 , 15->18 19 <kind@2> 15->19 21 , 15->21 22 <company@2> 15->22 24 , 15->24 25 <model@2> 15->25 17 2000 16->17 20 car 19->20 23 Mercury 22->23 26 Cougar 25->26 29 <year@2> 28->29 31 ,car, 28->31 32 <company@2> 28->32 34 , 28->34 35 <model@2> 28->35 30 1999 29->30 33 Chevy 32->33 36 Venture 35->36

As can be seen, the derivation tree obtained is not quite what we expected. The issue is easily seen if we enable logging in the TreeMiner.

In [145]:
dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars(), log=True)
 inventory@1='1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture'
	 - Node: <start>		? (<inventory@1>:'1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture')
		 -> [0] '1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture'
		  > ['<inventory@1>']
 vehicle@3='1997,van,Ford,E350'
	 - Node: <start>		? (<vehicle@3>:'1997,van,Ford,E350')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<vehicle@3>:'1997,van,Ford,E350')
		 -> [0] '1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture'
		  > ['<vehicle@3>', '\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture']
 model@2='E350'
	 - Node: <start>		? (<model@2>:'E350')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<model@2>:'E350')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['1997,van,Ford,', '<model@2>']
 company@2='Ford'
	 - Node: <start>		? (<company@2>:'Ford')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<company@2>:'Ford')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Ford')
		 -> [0] '1997,van,Ford,'
		  > ['1997,van,', '<company@2>', ',']
 kind@2='van'
	 - Node: <start>		? (<kind@2>:'van')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<kind@2>:'van')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<kind@2>:'van')
		 -> [0] '1997,van,'
		  > ['1997,', '<kind@2>', ',']
 year@2='1997'
	 - Node: <start>		? (<year@2>:'1997')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<year@2>:'1997')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'1997')
		 -> [0] '1997,'
		  > ['<year@2>', ',']
 vehicle@3='2000,car,Mercury,Cougar'
	 - Node: <start>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] 'E350'
		 -> [1] '\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture'
		  > ['\n', '<vehicle@3>', '\n1999,car,Chevy,Venture']
 model@2='Cougar'
	 - Node: <start>		? (<model@2>:'Cougar')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<model@2>:'Cougar')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Cougar')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<model@2>:'Cougar')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<model@2>:'Cougar')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<model@2>:'Cougar')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<model@2>:'Cougar')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Cougar')
		 -> [0] '2000,car,Mercury,Cougar'
		  > ['2000,car,Mercury,', '<model@2>']
 company@2='Mercury'
	 - Node: <start>		? (<company@2>:'Mercury')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<company@2>:'Mercury')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Mercury')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<company@2>:'Mercury')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<company@2>:'Mercury')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<company@2>:'Mercury')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<company@2>:'Mercury')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Mercury')
		 -> [0] '2000,car,Mercury,'
		  > ['2000,car,', '<company@2>', ',']
 kind@2='car'
	 - Node: <start>		? (<kind@2>:'car')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<kind@2>:'car')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<kind@2>:'car')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<kind@2>:'car')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<kind@2>:'car')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<kind@2>:'car')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<kind@2>:'car')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<kind@2>:'car')
		 -> [0] '2000,car,'
		  > ['2000,', '<kind@2>', ',']
 year@2='2000'
	 - Node: <start>		? (<year@2>:'2000')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<year@2>:'2000')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'2000')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<year@2>:'2000')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<year@2>:'2000')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<year@2>:'2000')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<year@2>:'2000')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'2000')
		 -> [0] '2000,'
		  > ['<year@2>', ',']
 vehicle@3='1999,car,Chevy,Venture'
	 - Node: <start>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '2000'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'car'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'Mercury'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'Cougar'
		 -> [3] '\n1999,car,Chevy,Venture'
		  > ['\n', '<vehicle@3>']
 model@2='Venture'
	 - Node: <start>		? (<model@2>:'Venture')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<model@2>:'Venture')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Venture')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<model@2>:'Venture')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<model@2>:'Venture')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<model@2>:'Venture')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<model@2>:'Venture')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Venture')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<model@2>:'Venture')
		 -> [0] '2000'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<model@2>:'Venture')
		 -> [0] 'car'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<model@2>:'Venture')
		 -> [0] 'Mercury'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<model@2>:'Venture')
		 -> [0] 'Cougar'
		 -> [3] '\n'
		 -> [4] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Venture')
		 -> [0] '1999,car,Chevy,Venture'
		  > ['1999,car,Chevy,', '<model@2>']
 company@2='Chevy'
	 - Node: <start>		? (<company@2>:'Chevy')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<company@2>:'Chevy')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Chevy')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<company@2>:'Chevy')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<company@2>:'Chevy')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<company@2>:'Chevy')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<company@2>:'Chevy')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Chevy')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<company@2>:'Chevy')
		 -> [0] '2000'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<company@2>:'Chevy')
		 -> [0] 'car'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<company@2>:'Chevy')
		 -> [0] 'Mercury'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<company@2>:'Chevy')
		 -> [0] 'Cougar'
		 -> [3] '\n'
		 -> [4] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Chevy')
		 -> [0] '1999,car,Chevy,'
		  > ['1999,car,', '<company@2>', ',']
 year@2='1999'
	 - Node: <start>		? (<year@2>:'1999')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<year@2>:'1999')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'1999')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<year@2>:'1999')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<year@2>:'1999')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<year@2>:'1999')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<year@2>:'1999')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'1999')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<year@2>:'1999')
		 -> [0] '2000'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<year@2>:'1999')
		 -> [0] 'car'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<year@2>:'1999')
		 -> [0] 'Mercury'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<year@2>:'1999')
		 -> [0] 'Cougar'
		 -> [3] '\n'
		 -> [4] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'1999')
		 -> [0] '1999,car,'
		  > ['<year@2>', ',car,']

Look at the last statement. We have a value 1999,car, where only the year got replaced. We no longer have a 'car' variable to continue the replacement here. This happens because the 'car' value in '1999,car,Chevy,Venture' is not treated as a new value because the value 'car' had occurred for 'vehicle' variable in the exact same location for a different method call (for '2000,car,Mercury,Cougar').

A Grammar Miner with Scope

We need to incorporate inspection of the variables in the current context. We already have a stack of method calls so that we can obtain the current method at any point. We need to do the same for variables.

For that, we extend the CallStack to a new class InputStack which holds the method invoked as well as the parameters observed. It is essentially the record of activation of the method. We start with the original input at the base of the stack, and for each new method-call, we push the parameters of that call into the stack as a new record.

Input Stack

In [146]:
class InputStack(CallStack):
    def __init__(self, i, fragment_len=FRAGMENT_LEN):
        self.inputs = [{START_SYMBOL: i}]
        self.fragment_len = fragment_len
        super().__init__()

In order to check if a particular variable be saved, we define in_current_record() which checks only the variables in the current scope for inclusion (rather than the original input string).

In [147]:
class InputStack(InputStack):
    def in_current_record(self, val):
        return any(val in var for var in self.inputs[-1].values())
In [148]:
my_istack = InputStack('hello my world')
In [149]:
my_istack.in_current_record('hello')
Out[149]:
True
In [150]:
my_istack.in_current_record('bye')
Out[150]:
False
In [151]:
my_istack.inputs.append({'greeting': 'hello', 'location': 'world'})
In [152]:
my_istack.in_current_record('hello')
Out[152]:
True
In [153]:
my_istack.in_current_record('my')
Out[153]:
False

We define the method ignored() that returns true if either the variable is not a string, or the variable length is less than the defined fragment_len.

In [154]:
class InputStack(InputStack):
    def ignored(self, val):
        return not (isinstance(val, str) and len(val) >= self.fragment_len)
In [155]:
my_istack = InputStack('hello world')
my_istack.ignored(1)
Out[155]:
True
In [156]:
my_istack.ignored('a')
Out[156]:
True
In [157]:
my_istack.ignored('help')
Out[157]:
False

We can now define the in_scope() method that checks whether the variable needs to be ignored, and if it is not to be ignored, whether the variable value is present in the current scope.

In [158]:
class InputStack(InputStack):
    def in_scope(self, k, val):
        if self.ignored(val):
            return False
        return self.in_current_record(val)

Finally, we update enter() that pushes relevant variables in the current context to the stack.

In [159]:
class InputStack(InputStack):
    def enter(self, method, inputs):
        my_inputs = {k: v for k, v in inputs.items() if self.in_scope(k, v)}
        self.inputs.append(my_inputs)
        super().enter(method)

When a method returns, we also need a corresponding leave() to pop out the inputs and unwind the stack.

In [160]:
class InputStack(InputStack):
    def leave(self):
        self.inputs.pop()
        super().leave()

ScopedVars

We need to update our AssignmentVars to include information about which scope the variable was defined. We start by updating method_init().

In [161]:
class ScopedVars(AssignmentVars):
    def method_init(self):
        self.call_stack = self.create_call_stack(self.my_input)
        self.event_locations = {self.call_stack.method_id: []}

    def create_call_stack(self, i):
        return InputStack(i)

Similarly, the method_enter() now initializes the accessed_seq_var for the current method call.

In [162]:
class ScopedVars(ScopedVars):
    def method_enter(self, cxt, my_vars):
        self.current_event = 'call'
        self.call_stack.enter(cxt.method, my_vars)
        self.accessed_seq_var[self.call_stack.method_id] = {}
        self.event_locations[self.call_stack.method_id] = []
        self.register_event(cxt)
        self.update(my_vars)

The update() method now saves the context in which the value is defined. In the case of a parameter to a function, the context should be the context in which the function was called. On the other hand, a value defined during a statement execution would have the current context.

Further, we annotate on value rather than key because we do not want to duplicate variables when parameters are in context in the next line. They will have same value, but different context because they are present in a statement execution.

In [163]:
class ScopedVars(ScopedVars):
    def update(self, v):
        if self.current_event == 'call':
            context = -2
        elif self.current_event == 'line':
            context = -1
        else:
            context = -1
        for k, v in v.items():
            self._set_kv(k, (v, self.call_stack.at(context)))
        self.var_location_register(self.new_vars)
        self.new_vars = set()

We also need to save the current method invocation so as to determine which variables are in scope. This information is now incorporated in the variable name as accessed_seq_var[method_id][var].

In [164]:
class ScopedVars(ScopedVars):
    def var_name(self, var):
        return (var, self.call_stack.method_id,
                self.accessed_seq_var[self.call_stack.method_id][var])

As before, var_access simply initializes the corresponding counter, this time in the context of method_id.

In [165]:
class ScopedVars(ScopedVars):
    def var_access(self, var):
        if var not in self.accessed_seq_var[self.call_stack.method_id]:
            self.accessed_seq_var[self.call_stack.method_id][var] = 0
        return self.var_name(var)

During a variable reassignment, we update the accessed_seq_var to reflect the new count.

In [166]:
class ScopedVars(ScopedVars):
    def var_assign(self, var):
        self.accessed_seq_var[self.call_stack.method_id][var] += 1
        self.new_vars.add(self.var_name(var))
        return self.var_name(var)

We now update defined_vars() to account for the new information.

In [167]:
class ScopedVars(ScopedVars):
    def defined_vars(self, formatted=True):
        def fmt(k):
            method, i = k[1]
            v = (method, i, k[0], self.var_def_lines[k])
            return "%s[%d]:%s@%s" % v if formatted else v

        return [(fmt(k), v) for k, v in self.defs.items()]

Updating seq_vars() to account for new information.

In [168]:
class ScopedVars(ScopedVars):
    def seq_vars(self, formatted=True):
        def fmt(k):
            method, i = k[1]
            v = (method, i, k[0], self.var_def_lines[k], k[2])
            return "%s[%d]:%s@%s:%s" % v if formatted else v

        return {fmt(k): v for k, v in self.defs.items()}

Scope Tracker

With the InputStack and Vars defined, we can now define the ScopeTracker. The ScopeTracker only saves variables if the value is present in the current scope.

In [169]:
class ScopeTracker(AssignmentTracker):
    def __init__(self, my_input, trace, **kwargs):
        self.current_event = None
        super().__init__(my_input, trace, **kwargs)

    def create_assignments(self, *args):
        return ScopedVars(*args)

We define a wrapper for checking whether a variable is present in the scope.

In [170]:
class ScopeTracker(ScopeTracker):
    def is_input_fragment(self, var, value):
        return self.my_assignments.call_stack.in_scope(var, value)

We can use the ScopeTracker as follows

In [171]:
vehicle_traces = []
with Tracer(INVENTORY) as tracer:
    process_inventory(tracer.my_input)
sm = ScopeTracker(tracer.my_input, tracer.trace)
vehicle_traces.append((tracer.my_input, sm))
for k, v in sm.my_assignments.seq_vars().items():
    print(k, '=', repr(v))
process_inventory[1]:inventory@1:1 = ('1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture', ('<start>', 0))
process_inventory[1]:inventory@1:2 = ('1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture', ('process_inventory', 1))
process_inventory[1]:vehicle@3:1 = ('1997,van,Ford,E350', ('process_inventory', 1))
process_vehicle[2]:vehicle@1:1 = ('1997,van,Ford,E350', ('process_inventory', 1))
process_vehicle[2]:vehicle@1:2 = ('1997,van,Ford,E350', ('process_vehicle', 2))
process_vehicle[2]:model@2:1 = ('E350', ('process_vehicle', 2))
process_vehicle[2]:company@2:1 = ('Ford', ('process_vehicle', 2))
process_vehicle[2]:kind@2:1 = ('van', ('process_vehicle', 2))
process_vehicle[2]:year@2:1 = ('1997', ('process_vehicle', 2))
process_van[3]:model@1:1 = ('E350', ('process_vehicle', 2))
process_van[3]:company@1:1 = ('Ford', ('process_vehicle', 2))
process_van[3]:year@1:1 = ('1997', ('process_vehicle', 2))
process_van[3]:model@1:2 = ('E350', ('process_van', 3))
process_van[3]:company@1:2 = ('Ford', ('process_van', 3))
process_van[3]:year@1:2 = ('1997', ('process_van', 3))
process_inventory[1]:vehicle@3:2 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))
process_vehicle[4]:vehicle@1:1 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))
process_vehicle[4]:vehicle@1:2 = ('2000,car,Mercury,Cougar', ('process_vehicle', 4))
process_vehicle[4]:model@2:1 = ('Cougar', ('process_vehicle', 4))
process_vehicle[4]:company@2:1 = ('Mercury', ('process_vehicle', 4))
process_vehicle[4]:kind@2:1 = ('car', ('process_vehicle', 4))
process_vehicle[4]:year@2:1 = ('2000', ('process_vehicle', 4))
process_car[5]:model@1:1 = ('Cougar', ('process_vehicle', 4))
process_car[5]:company@1:1 = ('Mercury', ('process_vehicle', 4))
process_car[5]:year@1:1 = ('2000', ('process_vehicle', 4))
process_car[5]:model@1:2 = ('Cougar', ('process_car', 5))
process_car[5]:company@1:2 = ('Mercury', ('process_car', 5))
process_car[5]:year@1:2 = ('2000', ('process_car', 5))
process_inventory[1]:vehicle@3:3 = ('1999,car,Chevy,Venture', ('process_inventory', 1))
process_vehicle[6]:vehicle@1:1 = ('1999,car,Chevy,Venture', ('process_inventory', 1))
process_vehicle[6]:vehicle@1:2 = ('1999,car,Chevy,Venture', ('process_vehicle', 6))
process_vehicle[6]:model@2:1 = ('Venture', ('process_vehicle', 6))
process_vehicle[6]:company@2:1 = ('Chevy', ('process_vehicle', 6))
process_vehicle[6]:kind@2:1 = ('car', ('process_vehicle', 6))
process_vehicle[6]:year@2:1 = ('1999', ('process_vehicle', 6))
process_car[7]:model@1:1 = ('Venture', ('process_vehicle', 6))
process_car[7]:company@1:1 = ('Chevy', ('process_vehicle', 6))
process_car[7]:year@1:1 = ('1999', ('process_vehicle', 6))
process_car[7]:model@1:2 = ('Venture', ('process_car', 7))
process_car[7]:company@1:2 = ('Chevy', ('process_car', 7))
process_car[7]:year@1:2 = ('1999', ('process_car', 7))

Recovering a Derivation Tree

The main difference in apply_new_definition() is that we add a second condition that checks for scope. In particular, variables are only allowed to replace portions of string fragments that were in scope. The variable scope is indicated by scope. However, merely accounting for scope is not sufficient. For example, consider the fragment below.

def my_fn(stringval):
    partA, partB = stringval.split('/')
    return partA, partB

svalue = ...
v1, v2 = my_fn(svalue)

Here, v1 and v2 get their values from a previous function call. Not from their current context. That is, we have to provide an exception for cases where an internal child method call may have generated a large fragment as we showed above. To account for that, we define mseq() that retrieves the method call sequence. In the above case, the mseq() of the internal child method call would be larger than the current mseq(). If so, we allow the replacement to proceed.

In [172]:
class ScopeTreeMiner(TreeMiner):
    def mseq(self, key):
        method, seq, var, lno = key
        return seq

The nt_var() method needs to take the tuple and generate a non-terminal symbol out of it. We skip the method sequence because it is not relevant for the grammar.

In [173]:
class ScopeTreeMiner(ScopeTreeMiner):
    def nt_var(self, key):
        method, seq, var, lno = key
        return to_nonterminal("%s@%d:%s" % (method, lno, var))

We now redefine the apply_new_definition() to account for context and scope. In particular, a variable is allowed to replace a part of a value only if the variable is in scope -- that is, it's scope (method sequence number of either its calling context in case it is a parameter or the current context in case it is a statement) is same as that of the value's method sequence number. An exception is made when the value's method sequence number is greater than the variable's method sequence number. In that case, the value may have come from an internal call. We allow the replacement to proceed in that case.

In [174]:
class ScopeTreeMiner(ScopeTreeMiner):
    def insert_into_tree(self, my_tree, pair):
        var, values, my_scope = my_tree
        (nt_var, nt_seq), (v, v_scope) = pair
        applied = False
        for i, value_ in enumerate(values):
            key, arr, scope = value_
            self.log(2, "-> [%d] %s" % (i, repr(value_)))
            if is_nonterminal(key):
                applied = self.insert_into_tree(value_, pair)
                if applied:
                    break
            else:
                if v_scope != scope:
                    if nt_seq > scope:
                        continue
                if not v or v not in key:
                    continue
                prefix_k_suffix = [
                    (nt_var, [(v, [], nt_seq)],
                     scope) if i == 1 else (e, [], scope)
                    for i, e in enumerate(key.partition(v))
                    if e
                ]
                del values[i]
                for j, rep in enumerate(prefix_k_suffix):
                    values.insert(j + i, rep)

                applied = True
                self.log(2, " > %s" % (repr([i[0] for i in prefix_k_suffix])))
                break
        return applied

The apply_new_definition() is now modified to carry additional contextual information mseq.

In [175]:
class ScopeTreeMiner(ScopeTreeMiner):
    def apply_new_definition(self, tree, var, value_):
        nt_var = self.nt_var(var)
        seq = self.mseq(var)
        val, (smethod, mseq) = value_
        return self.insert_into_tree(tree, ((nt_var, seq), (val, mseq)))

We also modify get_derivation_tree() so that the initial node carries the context.

In [176]:
class ScopeTreeMiner(ScopeTreeMiner):
    def get_derivation_tree(self):
        tree = (START_SYMBOL, [(self.my_input, [], 0)], 0)
        for var, value in self.my_assignments:
            self.log(0, "%s=%s" % (var, repr(value)))
            self.apply_new_definition(tree, var, value)
        return tree

Example 1: Recovering URL Parse Tree

We verify that our URL parse tree recovery still works as expected.

In [177]:
url_dts = []
for inputstr in URLS_X:
    clear_cache()
    with Tracer(inputstr, files=['urllib/parse.py']) as tracer:
        urlparse(tracer.my_input)
    sm = ScopeTracker(tracer.my_input, tracer.trace)
    for k, v in sm.my_assignments.defined_vars(formatted=False):
        print(k, '=', repr(v))
    dt = ScopeTreeMiner(
        tracer.my_input,
        sm.my_assignments.defined_vars(
            formatted=False))
    display_tree(dt.tree, graph_attr=lr_graph)
    url_dts.append(dt)
('urlparse', 1, 'url', 361) = ('http://user:pass@www.google.com:80/?q=path#ref', ('<start>', 0))
('urlparse', 1, 'url', 361) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))
('urlsplit', 3, 'scheme', 412) = ('http', ('urlsplit', 3))
('urlsplit', 3, 'url', 413) = ('//user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))
('_splitnetloc', 5, 'url', 386) = ('//user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))
('_splitnetloc', 5, 'url', 386) = ('//user:pass@www.google.com:80/?q=path#ref', ('_splitnetloc', 5))
('urlsplit', 3, 'url', 415) = ('/?q=path#ref', ('urlsplit', 3))
('urlsplit', 3, 'netloc', 415) = ('user:pass@www.google.com:80', ('urlsplit', 3))
('urlsplit', 3, 'url', 420) = ('/?q=path', ('urlsplit', 3))
('urlsplit', 3, 'fragment', 420) = ('ref', ('urlsplit', 3))
('urlsplit', 3, 'query', 422) = ('q=path', ('urlsplit', 3))
('urlparse', 1, 'scheme', 369) = ('http', ('urlparse', 1))
('urlparse', 1, 'fragment', 369) = ('ref', ('urlparse', 1))
('urlparse', 1, 'query', 369) = ('q=path', ('urlparse', 1))
('urlparse', 1, 'netloc', 369) = ('user:pass@www.google.com:80', ('urlparse', 1))
('urlparse', 1, 'url', 361) = ('https://www.cispa.saarland:80/', ('<start>', 0))
('urlparse', 1, 'url', 361) = ('https://www.cispa.saarland:80/', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('https://www.cispa.saarland:80/', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('https://www.cispa.saarland:80/', ('urlsplit', 3))
('urlsplit', 3, 'rest', 432) = ('//www.cispa.saarland:80/', ('urlsplit', 3))
('urlsplit', 3, 'scheme', 435) = ('https', ('urlsplit', 3))
('urlsplit', 3, 'url', 435) = ('//www.cispa.saarland:80/', ('urlsplit', 3))
('_splitnetloc', 7, 'url', 386) = ('//www.cispa.saarland:80/', ('urlsplit', 3))
('_splitnetloc', 7, 'url', 386) = ('//www.cispa.saarland:80/', ('_splitnetloc', 7))
('urlsplit', 3, 'netloc', 438) = ('www.cispa.saarland:80', ('urlsplit', 3))
('urlparse', 1, 'scheme', 369) = ('https', ('urlparse', 1))
('urlparse', 1, 'netloc', 369) = ('www.cispa.saarland:80', ('urlparse', 1))
('urlparse', 1, 'url', 361) = ('http://www.fuzzingbook.org/#News', ('<start>', 0))
('urlparse', 1, 'url', 361) = ('http://www.fuzzingbook.org/#News', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('http://www.fuzzingbook.org/#News', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('http://www.fuzzingbook.org/#News', ('urlsplit', 3))
('urlsplit', 3, 'scheme', 412) = ('http', ('urlsplit', 3))
('urlsplit', 3, 'url', 413) = ('//www.fuzzingbook.org/#News', ('urlsplit', 3))
('_splitnetloc', 5, 'url', 386) = ('//www.fuzzingbook.org/#News', ('urlsplit', 3))
('_splitnetloc', 5, 'url', 386) = ('//www.fuzzingbook.org/#News', ('_splitnetloc', 5))
('urlsplit', 3, 'url', 415) = ('/#News', ('urlsplit', 3))
('urlsplit', 3, 'netloc', 415) = ('www.fuzzingbook.org', ('urlsplit', 3))
('urlsplit', 3, 'fragment', 420) = ('News', ('urlsplit', 3))
('urlparse', 1, 'scheme', 369) = ('http', ('urlparse', 1))
('urlparse', 1, 'fragment', 369) = ('News', ('urlparse', 1))
('urlparse', 1, 'netloc', 369) = ('www.fuzzingbook.org', ('urlparse', 1))
('urlparse', 1, 'url', 361) = ('ftp://freebsd.org/releases/5.8', ('<start>', 0))
('urlparse', 1, 'url', 361) = ('ftp://freebsd.org/releases/5.8', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('ftp://freebsd.org/releases/5.8', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('ftp://freebsd.org/releases/5.8', ('urlsplit', 3))
('urlsplit', 3, 'rest', 432) = ('//freebsd.org/releases/5.8', ('urlsplit', 3))
('urlsplit', 3, 'scheme', 435) = ('ftp', ('urlsplit', 3))
('urlsplit', 3, 'url', 435) = ('//freebsd.org/releases/5.8', ('urlsplit', 3))
('_splitnetloc', 7, 'url', 386) = ('//freebsd.org/releases/5.8', ('urlsplit', 3))
('_splitnetloc', 7, 'url', 386) = ('//freebsd.org/releases/5.8', ('_splitnetloc', 7))
('urlsplit', 3, 'url', 438) = ('/releases/5.8', ('urlsplit', 3))
('urlsplit', 3, 'netloc', 438) = ('freebsd.org', ('urlsplit', 3))
('urlparse', 1, 'scheme', 369) = ('ftp', ('urlparse', 1))
('urlparse', 1, 'url', 369) = ('/releases/5.8', ('urlparse', 1))
('urlparse', 1, 'netloc', 369) = ('freebsd.org', ('urlparse', 1))

Example 2: Recovering Inventory Parse Tree

Next, we look at recovering the parse tree from process_inventory() which failed last time.

In [178]:
with Tracer(INVENTORY) as tracer:
    process_inventory(tracer.my_input)

sm = ScopeTracker(tracer.my_input, tracer.trace)
for k, v in sm.my_assignments.defined_vars():
    print(k, '=', repr(v))
inventory_dt = ScopeTreeMiner(
    tracer.my_input,
    sm.my_assignments.defined_vars(
        formatted=False))
display_tree(inventory_dt.tree, graph_attr=lr_graph)
process_inventory[1]:inventory@1 = ('1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture', ('<start>', 0))
process_inventory[1]:inventory@1 = ('1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture', ('process_inventory', 1))
process_inventory[1]:vehicle@3 = ('1997,van,Ford,E350', ('process_inventory', 1))
process_vehicle[2]:vehicle@1 = ('1997,van,Ford,E350', ('process_inventory', 1))
process_vehicle[2]:vehicle@1 = ('1997,van,Ford,E350', ('process_vehicle', 2))
process_vehicle[2]:model@2 = ('E350', ('process_vehicle', 2))
process_vehicle[2]:company@2 = ('Ford', ('process_vehicle', 2))
process_vehicle[2]:kind@2 = ('van', ('process_vehicle', 2))
process_vehicle[2]:year@2 = ('1997', ('process_vehicle', 2))
process_van[3]:model@1 = ('E350', ('process_vehicle', 2))
process_van[3]:company@1 = ('Ford', ('process_vehicle', 2))
process_van[3]:year@1 = ('1997', ('process_vehicle', 2))
process_van[3]:model@1 = ('E350', ('process_van', 3))
process_van[3]:company@1 = ('Ford', ('process_van', 3))
process_van[3]:year@1 = ('1997', ('process_van', 3))
process_inventory[1]:vehicle@3 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))
process_vehicle[4]:vehicle@1 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))
process_vehicle[4]:vehicle@1 = ('2000,car,Mercury,Cougar', ('process_vehicle', 4))
process_vehicle[4]:model@2 = ('Cougar', ('process_vehicle', 4))
process_vehicle[4]:company@2 = ('Mercury', ('process_vehicle', 4))
process_vehicle[4]:kind@2 = ('car', ('process_vehicle', 4))
process_vehicle[4]:year@2 = ('2000', ('process_vehicle', 4))
process_car[5]:model@1 = ('Cougar', ('process_vehicle', 4))
process_car[5]:company@1 = ('Mercury', ('process_vehicle', 4))
process_car[5]:year@1 = ('2000', ('process_vehicle', 4))
process_car[5]:model@1 = ('Cougar', ('process_car', 5))
process_car[5]:company@1 = ('Mercury', ('process_car', 5))
process_car[5]:year@1 = ('2000', ('process_car', 5))
process_inventory[1]:vehicle@3 = ('1999,car,Chevy,Venture', ('process_inventory', 1))
process_vehicle[6]:vehicle@1 = ('1999,car,Chevy,Venture', ('process_inventory', 1))
process_vehicle[6]:vehicle@1 = ('1999,car,Chevy,Venture', ('process_vehicle', 6))
process_vehicle[6]:model@2 = ('Venture', ('process_vehicle', 6))
process_vehicle[6]:company@2 = ('Chevy', ('process_vehicle', 6))
process_vehicle[6]:kind@2 = ('car', ('process_vehicle', 6))
process_vehicle[6]:year@2 = ('1999', ('process_vehicle', 6))
process_car[7]:model@1 = ('Venture', ('process_vehicle', 6))
process_car[7]:company@1 = ('Chevy', ('process_vehicle', 6))
process_car[7]:year@1 = ('1999', ('process_vehicle', 6))
process_car[7]:model@1 = ('Venture', ('process_car', 7))
process_car[7]:company@1 = ('Chevy', ('process_car', 7))
process_car[7]:year@1 = ('1999', ('process_car', 7))
Out[178]:
%3 0 <start> 1 <process_inventory@1:inventory> 0->1 2 <process_inventory@1:inventory> 1->2 3 <process_inventory@3:vehicle> 2->3 23 2->23 24 <process_inventory@3:vehicle> 2->24 44 2->44 45 <process_inventory@3:vehicle> 2->45 4 <process_vehicle@1:vehicle> 3->4 5 <process_vehicle@1:vehicle> 4->5 6 <process_vehicle@2:year> 5->6 10 , 5->10 11 <process_vehicle@2:kind> 5->11 13 , 5->13 14 <process_vehicle@2:company> 5->14 18 , 5->18 19 <process_vehicle@2:model> 5->19 7 <process_van@1:year> 6->7 8 <process_van@1:year> 7->8 9 1997 8->9 12 van 11->12 15 <process_van@1:company> 14->15 16 <process_van@1:company> 15->16 17 Ford 16->17 20 <process_van@1:model> 19->20 21 <process_van@1:model> 20->21 22 E350 21->22 25 <process_vehicle@1:vehicle> 24->25 26 <process_vehicle@1:vehicle> 25->26 27 <process_vehicle@2:year> 26->27 31 , 26->31 32 <process_vehicle@2:kind> 26->32 34 , 26->34 35 <process_vehicle@2:company> 26->35 39 , 26->39 40 <process_vehicle@2:model> 26->40 28 <process_car@1:year> 27->28 29 <process_car@1:year> 28->29 30 2000 29->30 33 car 32->33 36 <process_car@1:company> 35->36 37 <process_car@1:company> 36->37 38 Mercury 37->38 41 <process_car@1:model> 40->41 42 <process_car@1:model> 41->42 43 Cougar 42->43 46 <process_vehicle@1:vehicle> 45->46 47 <process_vehicle@1:vehicle> 46->47 48 <process_vehicle@2:year> 47->48 52 , 47->52 53 <process_vehicle@2:kind> 47->53 55 , 47->55 56 <process_vehicle@2:company> 47->56 60 , 47->60 61 <process_vehicle@2:model> 47->61 49 <process_car@1:year> 48->49 50 <process_car@1:year> 49->50 51 1999 50->51 54 car 53->54 57 <process_car@1:company> 56->57 58 <process_car@1:company> 57->58 59 Chevy 58->59 62 <process_car@1:model> 61->62 63 <process_car@1:model> 62->63 64 Venture 63->64

The recovered parse tree seems reasonable.

One of the things that one might notice from our Example (2) is that the three subtrees -- vehicle[2:1], vehicle[4:1] and vehicle[6:1] are quite alike. We will examine how this can be exploited to generate a grammar directly, next.

Grammar Mining

The tree_to_grammar() is now redefined as follows, to account for the extra scope in nodes.

In [179]:
class ScopedGrammarMiner(GrammarMiner):
    def tree_to_grammar(self, tree):
        key, children, scope = tree
        one_alt = [ckey for ckey, gchildren, cscope in children if ckey != key]
        hsh = {key: [one_alt] if one_alt else []}
        for child in children:
            (ckey, _gc, _cscope) = child
            if not is_nonterminal(ckey):
                continue
            chsh = self.tree_to_grammar(child)
            for k in chsh:
                if k not in hsh:
                    hsh[k] = chsh[k]
                else:
                    hsh[k].extend(chsh[k])
        return hsh

The grammar is in canonical form, which needs to be massaged to display. First, the recovered grammar for inventory.

In [180]:
si = ScopedGrammarMiner()
si.add_tree(inventory_dt)
syntax_diagram(readable(si.grammar))
start
process_inventory@1:inventory
process_inventory@1:inventory
process_inventory@3:vehicle process_inventory@3:vehicle process_inventory@3:vehicle
process_inventory@3:vehicle
process_vehicle@1:vehicle
process_vehicle@1:vehicle
process_vehicle@2:year , process_vehicle@2:kind , process_vehicle@2:company , process_vehicle@2:model
process_vehicle@2:year
process_car@1:year process_van@1:year
process_van@1:year
1997
process_vehicle@2:kind
van car
process_vehicle@2:company
process_van@1:company process_car@1:company
process_van@1:company
Ford
process_vehicle@2:model
process_van@1:model process_car@1:model
process_van@1:model
E350
process_car@1:year
2000 1999
process_car@1:company
Chevy Mercury
process_car@1:model
Cougar Venture

The recovered grammar for URLs.

In [181]:
su = ScopedGrammarMiner()
for url_dt in url_dts:
    su.add_tree(url_dt)
syntax_diagram(readable(su.grammar))
start
urlparse@361:url
urlparse@361:url
urlsplit@394:url
urlsplit@394:url
urlsplit@435:scheme : urlsplit@432:rest urlsplit@412:scheme : urlsplit@413:url
urlsplit@412:scheme
urlparse@369:scheme
urlparse@369:scheme
ftp https http
urlsplit@413:url
_splitnetloc@386:url
_splitnetloc@386:url
// urlsplit@438:netloc / // urlsplit@438:netloc urlsplit@438:url // urlsplit@415:netloc urlsplit@415:url
urlsplit@415:netloc
urlparse@369:netloc
urlparse@369:netloc
www.cispa.saarland:80 freebsd.org user:pass@www.google.com:80 www.fuzzingbook.org
urlsplit@415:url
/# urlsplit@420:fragment urlsplit@420:url # urlsplit@420:fragment
urlsplit@420:url
/? urlsplit@422:query
urlsplit@422:query
urlparse@369:query
urlparse@369:query
q=path
urlsplit@420:fragment
urlparse@369:fragment
urlparse@369:fragment
News ref
urlsplit@435:scheme
urlparse@369:scheme
urlsplit@432:rest
urlsplit@435:url
urlsplit@435:url
_splitnetloc@386:url
urlsplit@438:netloc
urlparse@369:netloc
urlsplit@438:url
urlparse@369:url
urlparse@369:url
/releases/5.8

One might notice that the grammar is not entirely human readable, with a number of single token definitions.

Hence, the last piece of the puzzle is the cleanup method clean_grammar(), which cleans up such definitions. The idea is to look for single token definitions such that a key is defined exactly by another key (single alternative, single token, nonterminal).

In [182]:
class ScopedGrammarMiner(ScopedGrammarMiner):
    def get_replacements(self, grammar):
        replacements = {}
        for k in grammar:
            if k == START_SYMBOL:
                continue
            alts = grammar[k]
            if len(set([str(i) for i in alts])) != 1:
                continue
            rule = alts[0]
            if len(rule) != 1:
                continue
            tok = rule[0]
            if not is_nonterminal(tok):
                continue
            replacements[k] = tok
        return replacements

Once we have such a list, iteratively replace the original key where ever it is used with the token we found earlier. Repeat until none is left.

In [183]:
class ScopedGrammarMiner(ScopedGrammarMiner):
    def clean_grammar(self):
        replacements = self.get_replacements(self.grammar)

        while True:
            changed = set()
            for k in self.grammar:
                if k in replacements:
                    continue
                new_alts = []
                for alt in self.grammar[k]:
                    new_alt = []
                    for t in alt:
                        if t in replacements:
                            new_alt.append(replacements[t])
                            changed.add(t)
                        else:
                            new_alt.append(t)
                    new_alts.append(new_alt)
                self.grammar[k] = new_alts
            if not changed:
                break
            for k in changed:
                self.grammar.pop(k, None)
        return readable(self.grammar)

The clean_grammar() is used as follows:

In [184]:
si = ScopedGrammarMiner()
si.add_tree(inventory_dt)
syntax_diagram(readable(si.clean_grammar()))
start
process_inventory@1:inventory
process_inventory@1:inventory
process_vehicle@1:vehicle process_vehicle@1:vehicle process_vehicle@1:vehicle
process_vehicle@1:vehicle
process_vehicle@2:year , process_vehicle@2:kind , process_vehicle@2:company , process_vehicle@2:model
process_vehicle@2:year
process_car@1:year process_van@1:year
process_van@1:year
1997
process_vehicle@2:kind
van car
process_vehicle@2:company
process_van@1:company process_car@1:company
process_van@1:company
Ford
process_vehicle@2:model
process_van@1:model process_car@1:model
process_van@1:model
E350
process_car@1:year
2000 1999
process_car@1:company
Chevy Mercury
process_car@1:model
Cougar Venture

We update the update_grammar() to use the right tracker and miner.

In [185]:
class ScopedGrammarMiner(ScopedGrammarMiner):
    def update_grammar(self, inputstr, trace):
        at = self.create_tracker(inputstr, trace)
        dt = self.create_tree_miner(
            inputstr, at.my_assignments.defined_vars(
                formatted=False))
        self.add_tree(dt)
        return self.grammar

    def create_tracker(self, *args):
        return ScopeTracker(*args)

    def create_tree_miner(self, *args):
        return ScopeTreeMiner(*args)

The recover_grammar() uses the right miner, and returns a cleaned grammar.

In [186]:
def recover_grammar(fn, inputs, **kwargs):
    miner = ScopedGrammarMiner()
    for inputstr in inputs:
        with Tracer(inputstr, **kwargs) as tracer:
            fn(tracer.my_input)
        miner.update_grammar(tracer.my_input, tracer.trace)
    return readable(miner.clean_grammar())
In [187]:
url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])
In [188]:
syntax_diagram(url_grammar)
start
urlsplit@394:url
urlsplit@394:url
urlparse@369:scheme : _splitnetloc@386:url
urlparse@369:scheme
ftp https http
_splitnetloc@386:url
// urlparse@369:netloc urlsplit@415:url // urlparse@369:netloc urlparse@369:url // urlparse@369:netloc /
urlparse@369:netloc
www.cispa.saarland:80 freebsd.org user:pass@www.google.com:80 www.fuzzingbook.org
urlsplit@415:url
/# urlparse@369:fragment urlsplit@420:url # urlparse@369:fragment
urlsplit@420:url
/? urlparse@369:query
urlparse@369:query
q=path
urlparse@369:fragment
News ref
urlparse@369:url
/releases/5.8
In [189]:
f = GrammarFuzzer(url_grammar)
for _ in range(10):
    print(f.fuzz())
ftp://user:pass@www.google.com:80/?q=path#ref
http://freebsd.org/
ftp://freebsd.org/#News
http://www.cispa.saarland:80/?q=path#ref
http://user:pass@www.google.com:80/#ref
https://freebsd.org/
ftp://freebsd.org/#ref
ftp://freebsd.org/
ftp://freebsd.org/releases/5.8
ftp://user:pass@www.google.com:80/#News
In [190]:
inventory_grammar = recover_grammar(process_inventory, [INVENTORY])
In [191]:
syntax_diagram(inventory_grammar)
start
process_inventory@1:inventory
process_inventory@1:inventory
process_vehicle@1:vehicle process_vehicle@1:vehicle process_vehicle@1:vehicle
process_vehicle@1:vehicle
process_vehicle@2:year , process_vehicle@2:kind , process_vehicle@2:company , process_vehicle@2:model
process_vehicle@2:year
process_car@1:year process_van@1:year
process_van@1:year
1997
process_vehicle@2:kind
van car
process_vehicle@2:company
process_van@1:company process_car@1:company
process_van@1:company
Ford
process_vehicle@2:model
process_van@1:model process_car@1:model
process_van@1:model
E350
process_car@1:year
2000 1999
process_car@1:company
Chevy Mercury
process_car@1:model
Cougar Venture
In [192]:
f = GrammarFuzzer(inventory_grammar)
for _ in range(10):
    print(f.fuzz())
1997,car,Chevy,E350
1997,van,Ford,Cougar
2000,car,Chevy,Venture
1999,car,Ford,E350
1997,car,Mercury,E350
2000,van,Mercury,E350
1997,van,Mercury,Venture
2000,van,Mercury,E350
1997,van,Ford,E350
1997,car,Mercury,Venture
1997,car,Mercury,E350
1997,car,Mercury,E350
1997,van,Chevy,E350
1997,van,Mercury,E350
1997,van,Chevy,E350
2000,car,Ford,E350
2000,van,Chevy,E350
1997,van,Mercury,E350
1999,car,Chevy,E350
2000,van,Mercury,E350
2000,van,Ford,E350
2000,car,Mercury,E350
1999,car,Ford,E350
2000,van,Chevy,Cougar
1997,car,Ford,E350
1997,car,Mercury,E350
1997,car,Chevy,Venture
1997,car,Ford,Venture
1997,car,Mercury,E350
1997,van,Mercury,E350

We see how tracking scope helps us to extract an even more precise grammar.

Notice that we use String inclusion testing as a way of determining whether a particular string fragment came from the original input string. While this may seem rather error-prone compared to dynamic tainting, we note that numerous tracing tools such as dtrace() and ptrace() allow one to obtain the information we seek from execution of binaries directly in different platforms. However, methods for obtaining dynamic taints almost always involve instrumenting the binaries before they can be used. Hence, this method of string inclusion can be more generally applied than dynamic tainting approaches. Further, dynamic taints are often lost due to implicit transmission, or at the boundary between Python and C code. String inclusion