# Fuzzing APIs¶

So far, we have always generated system input, i.e. data that the program as a whole obtains via its input channels. However, we can also generate inputs that go directly into individual functions, gaining flexibility and speed in the process. In this chapter, we explore the use of grammars to synthesize code for function calls, which allows you to generate program code that very efficiently invokes functions directly.

from bookutils import YouTubeVideo
YouTubeVideo('CC1VvOGkzm8')


Prerequisites

## Synopsis¶

>>> from fuzzingbook.APIFuzzer import <identifier>


and then make use of the following features.

This chapter provides grammar constructors that are useful for generating function calls.

The grammars are probabilistic and make use of generators, so use ProbabilisticGeneratorGrammarFuzzer as a producer.

>>> from GeneratorGrammarFuzzer import ProbabilisticGeneratorGrammarFuzzer


INT_GRAMMAR, FLOAT_GRAMMAR, ASCII_STRING_GRAMMAR produce integers, floats, and strings, respectively:

>>> fuzzer = ProbabilisticGeneratorGrammarFuzzer(INT_GRAMMAR)
>>> [fuzzer.fuzz() for i in range(10)]
['-51', '9', '0', '0', '0', '0', '32', '0', '0', '0']
>>> fuzzer = ProbabilisticGeneratorGrammarFuzzer(FLOAT_GRAMMAR)
>>> [fuzzer.fuzz() for i in range(10)]
['0e0',
'-9.43e34',
'-7.3282e0',
'-9.5e-9',
'0',
'-30.840386e-5',
'3',
'-4.1e0',
'-9.7',
'413']
>>> fuzzer = ProbabilisticGeneratorGrammarFuzzer(ASCII_STRING_GRAMMAR)
>>> [fuzzer.fuzz() for i in range(10)]
['"#vYV*t@I%KNTT[q~}&-v+[zAzj[X-z|RzC$(g$Br]1tC\':5<F-"',
'""',
'"^S/"',
'"y)QDs_9"',
'")dY~?WYqMh,bwn3\\"A!02Pkgx"',
'"01n|(dd$-d.sx\\"83\\"h/]qx)d9LPNdrk$}$4t3zhC.%3VY@AZZ0wCs2 N"', '"D\\6\\xgw#TQ}$\'3"',
'"LaM{"',


## Synthesizing Composite Data¶

From basic data, as discussed above, we can also produce composite data in data structures such as sets or lists. We illustrate such generation on lists.

### Lists¶

LIST_EBNF_GRAMMAR: Grammar = {
"<start>": ["<list>"],
"<list>": [
("[]", opts(prob=0.05)),
"[<list-objects>]"
],
"<list-objects>": [
("<list-object>", opts(prob=0.2)),
"<list-object>, <list-objects>"
],
"<list-object>": ["0"],
}

assert is_valid_grammar(LIST_EBNF_GRAMMAR)

LIST_GRAMMAR = convert_ebnf_grammar(LIST_EBNF_GRAMMAR)


Our list generator takes a grammar that produces objects; it then instantiates a list grammar with the objects from these grammars.

def list_grammar(object_grammar, list_object_symbol=None):
obj_list_grammar = extend_grammar(LIST_GRAMMAR)
if list_object_symbol is None:
# Default: Use the first expansion of <start> as list symbol
list_object_symbol = object_grammar[START_SYMBOL][0]

obj_list_grammar.update(object_grammar)
obj_list_grammar[START_SYMBOL] = ["<list>"]
obj_list_grammar["<list-object>"] = [list_object_symbol]

assert is_valid_grammar(obj_list_grammar)

return obj_list_grammar

int_list_fuzzer = ProbabilisticGrammarFuzzer(list_grammar(INT_GRAMMAR))
[int_list_fuzzer.fuzz() for i in range(10)]

['[0, -4, 23, 0, 0, 9, 0, -6067681]',
'[-1, -1, 0, -7]',
'[-5, 0]',
'[1, 0, -628088, -6, -811, 0, 99, 0]',
'[-35, -10, 0, 67]',
'[-3, 0, -2, 0, 0]',
'[0, -267, -78, -733, 0, 0, 0, 0]',
'[0, -6, 71, -9]',
'[-72, 76, 0, 2]',
'[0, 9, 0, 0, -572, 29, 8, 8, 0]']

string_list_fuzzer = ProbabilisticGrammarFuzzer(
list_grammar(ASCII_STRING_GRAMMAR))
[string_list_fuzzer.fuzz() for i in range(10)]

['["gn-A$j>", "SPX;", "", "", ""]', '["_", "Qp"]', '["M", "5\\"X744", "b+5fyM!", "gR"]', '["^h", "8$u", "", "", ""]',
'["6X;", "", "T1wp%\'t"]',
'["-?Kk", "@B", "}", "", ""]',
'["FD<mqK", ")Y4NI3M.&@1/2.p", "]C#c1}z#+5{7ERA[|", "EOFM])BEMFcGM.~k&RMj*,:m8^!5*:vv%ci"]',
'["", "*B.pKI\\"L", "O)#<Y", "\\", "", "", ""]',
'["g"]',
'["", "\\JS;~t", "h)", "k", "", ""]']

float_list_fuzzer = ProbabilisticGeneratorGrammarFuzzer(list_grammar(
float_grammar_with_range(900.0, 900.9)))
[float_list_fuzzer.fuzz() for i in range(10)]

['[900.558064701869, 900.6079527708223, 900.1985188111297, 900.5159940886509, 900.1881413629061, 900.4074809145482, 900.8279453113845, 900.1531931708976, 900.2651056125504, inf, 900.828295978669]',
'[900.4956935906264, 900.8166792417645, 900.2044872129637]',
'[900.6177668624133, 900.793129850367, 900.5024769009476, 900.5874531663001, inf, 900.3476216137291, 900.5680329060473, 900.1524624203945, 900.1157565249836, 900.0943774301732, 900.1589468212459, 900.8563415304703, 900.2871041191156, 900.2469765832253, 900.408183791468]',
'[NaN, 900.1152482126347, 900.1139109179966, NaN, 900.0634308730662, 900.1918596242257]',
'[900.49418992478]',
'[900.6566851795975, NaN, 900.5585085641878, 900.8678799526169, 900.5580757140183]',
'[900.6265067760952]',
'[900.5271187218734, 900.3413004135587, 900.0362652510535, 900.2938223153569, 900.6584186055829, 900.5394909707123, 900.5119630230411, 900.2024669591465]',
'[900.5068304562362, 900.5173419618334, 900.5268996804168, 900.5247314889621, 900.1082421801126, 900.761200730868, 900.100950598924, 900.1424140649187, inf, inf, 900.4546924838603, 900.7025508468811, 900.5147250716594, 900.4943696257178, 900.814107878577, 900.3540228715348, 900.6165673939341, 900.121833279104, 900.8337503512706, 900.0607374037857, 900.2746253938637, 900.2491844866619, 900.7325728031923]',
'[900.6962790125643, 900.6055198052603, 900.0950691946015, 900.6283670716376, NaN, 900.112869956762]']


Generators for dictionaries, sets, etc. can be defined in a similar fashion. By plugging together grammar generators, we can produce data structures with arbitrary elements.

## Lessons Learned¶

• To fuzz individual functions, one can easily set up grammars that produce function calls.
• Fuzzing at the API level can be much faster than fuzzing at the system level, but brings the risk of false alarms by violating implicit preconditions.

## Next Steps¶

This chapter was all about manually writing test and controlling which data gets generated. In the next chapter, we will introduce a much higher level of automation:

• Carving automatically records function calls and arguments from program executions.
• We can turn these into grammars, allowing to test these functions with various combinations of recorded values.

With these techniques, we automatically obtain grammars that already invoke functions in application contexts, making our work of specifying them much easier.

## Background¶

The idea of using generator functions to generate input structures was first explored in QuickCheck [Claessen et al, 2000]. A very nice implementation for Python is the hypothesis package which allows to write and combine data structure generators for testing APIs.

## Exercises¶

The exercises for this chapter combine the above techniques with fuzzing techniques introduced earlier.

### Exercise 1: Deep Arguments¶

In the example generating oracles for urlparse(), important elements such as authority or port are not checked. Enrich URLPARSE_ORACLE_GRAMMAR` with post-expansion functions that store the generated elements in a symbol table, such that they can be accessed when generating the assertions.

### Exercise 2: Covering Argument Combinations¶

In the chapter on configuration testing, we also discussed combinatorial testing – that is, systematic coverage of sets of configuration elements. Implement a scheme that by changing the grammar, allows all pairs of argument values to be covered.

### Exercise 3: Mutating Arguments¶

To widen the range of arguments to be used during testing, apply the mutation schemes introduced in mutation fuzzing – for instance, flip individual bytes or delete characters from strings. Apply this either during grammar inference or as a separate step when invoking functions.

The content of this project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The source code that is part of the content, as well as the source code used to format and display that content is licensed under the MIT License. Last change: 2022-05-17 18:02:38+02:00CiteImprint