Testing
DSL-based integration tests
My favorite kind of test.
Some examples:
Property-based testing
- Really stupid properties:
- Is the output nontrivial (e.g. nonempty)?
- Does the program not crash (i.e., fuzzing)?
- Does the program satisfy its assertions/invariants/contracts?
- Property-preserving transformations: reversing a list doesn’t change the max.
- Be inspired by math:
- Test commutativity/intertwining of functions/methods
- Test inverse relationships/functions
- Test idempotency
- Test structurally inductive properties
- Differential testing: equivalence with a simplified implementation or model
See:
Obscure types of testing
Fault injection
Fault injection is a software testing technique that involves inducing failures (“faults”) in the functions called by a program. If the callee has failed to perform proper error checking and handling, these faults can result in unreliable application behavior or exploitable vulnerabilities.
https://github.com/trailofbits/krf
Metamorphic testing
Metamorphic testing relates multiple executions or input/output pairs of the same code. For example, doubling all the elements of a list also doubles the mode. It is useful in combination with property testing.
Mutation testing
Making automated changes to the code to see if the test suite will catch them. “Testing tests” as it were.
Particular types of system under test
REST
Metamorphic Testing of RESTful Web APIs
Abstract
Web Application Programming Interfaces (APIs) allow systems to interact with each other over the network. Modern Web APIs often adhere to the REST architectural style, being referred to as RESTful Web APIs. RESTful Web APIs are decomposed into multiple resources (e.g., a video in the YouTube API) that clients can manipulate through HTTP interactions. Testing Web APIs is critical but challenging due to the difficulty to assess the correctness of API responses, i.e., the oracle problem. Metamorphic testing alleviates the oracle problem by exploiting relations (so-called metamorphic relations) among multiple executions of the program under test. In this paper, we present a metamorphic testing approach for the detection of faults in RESTful Web APIs. We first propose six abstract relations that capture the shape of many of the metamorphic relations found in RESTful Web APIs, we call these Metamorphic Relation Output Patterns (MROPs). Each MROP can then be instantiated into one or more concrete metamorphic relations. The approach was evaluated using both automatically seeded and real faults in six subject Web APIs. Among other results, we identified 60 metamorphic relations (instances of the proposed MROPs) in the Web APIs of Spotify and YouTube. Each metamorphic relation was implemented using both random and manual test data, running over 4.7K automated tests. As a result, 11 issues were detected (3 in Spotify and 8 in YouTube), 10 of them confirmed by the API developers or reproduced by other users, supporting the effectiveness of the approach.