Callbacks
Callbacks allow users to execute arbitrary Python code during test runtime to perform operations unavailable via the user interface. Callbacks can be extremely useful in several ways, such as:
- Implement advanced response processing rules.
- Incorporating arbitrary health checks into the test flow.
- Restoring the target to a known good state before delivering the next test case.
- Utilizing binary instrumentation frameworks (e.g., QDBI and Frida). For example, instrument client applications to connect to GUARDARA when GUARDARA simulates a server.
- Perform other external automation during the test flow.
Creating Callbacks
You can assign Callbacks to the Callback actions within Test Flow Templates. Similarly to other assets, Callbacks are managed via the Test Assets page. To create a new Callback, click File > New Callback on the main menu of the Test Assets page.
A Callback is a Python function. The Callback skeleton that users can extend is shown below.
def callback(context):
pass
Context
The context argument of a Callback is an object that exposes several internal methods of the Engine. These are discussed below.
Runtime
| Method | Description |
|---|---|
| context.runtime.is_analysis() | Returns True if the Callback is executed during the automated issue analysis or the baseline collection phases of the testing process. It returns False otherwise. |
| context.runtime.log(severity, message) | The method allows writing messages to the test’s Activity Log. The severity supports the following values: info, warning and error. The message is expected to be a string (Python str). |
| context.runtime.performance() | Returns a dictionary (Python dict) containing performance metrics about the test. |
The performance() method returns a dictionary that looks like the one below.
{
'average_iteration_time': 0.091961,
'maximum_iteration_time': 0.203699,
'average_test_cases_per_second': 10.0,
'maximum_test_cases_per_second': 10.0,
'analytics_time': 0.18,
'current_speed': 10.0,
'remaining': 3321.6
}
The above performance statistic, for example, tells us that, in this case, we were executing an average of 10 test cases a second and, with the current speed, it may take approximately an hour for the test run to complete.
You can find an example demonstrating the use of the runtime methods exposed on GitLab.
Session
| Method | Description |
|---|---|
| context.session.get(variable_id) | Obtain the raw and rendered value of a Session Variable as a dictionary (dict) based on the variable's ID. |
| context.session.set(variable_id, type, value) | Set the value of an existing Session Variable or create a new one. The type can be either raw or rendered. |
Driver
The driver methods allow sending and receiving data from the target. The following table summarizes the driver methods exposed.
| Method | Description |
|---|---|
| context.driver.connect(props) | Allows connecting to the target using the configured driver. |
| context.driver.disconnect(props) | Allows disconnecting from the target using the configured driver. |
| context.driver.send(data, props) | Allows sending data to the target via the configured driver. The data must be Python bytes, for example: b"hello". |
| context.driver.recv(props) | Allows receiving data from the target via the configured driver. The received data is returned as Python bytes. |
The props argument is optional and allows passing configuration options to the driver method. These are the same options generated by the Flow Designer for actions.
History
The history contains information about messages previously sent to the target. In addition, the mutator instance of the previously executed and the following action is also made accessible. The following table summarizes the history methods exposed via the callback context.
The accessible history methods are summarized below.
| Method | Description |
|---|---|
| context.history.fetch() | Returns all items from the history. |
| context.history.get_last() | Returns the details of the last send action from the history. |
| context.history.get_last_mutator() | Returns the last mutator instance. |
| context.history.get_next_mutator() | Returns the next mutator instance. |
Fetch Method
The fetch method returns information about the last 20 send and receive actions. The following callback example prints the action history to the console.
def callback(context):
print(context.history.fetch())
An example, redacted output of the fetch method is shown below.
[
{
'iteration': 570,
'timestamp': 1657915040527,
'action_id': 'a4b5f0c5-58ca-4d49-bae7-eb5eff53406d',
'action_type': 'send',
'message': {
'id': '37aae2ed-1f5d-40b4-b380-4f7d4ee23d00',
'name': 'Login'
},
'length': 1644,
'data': [80, 79, 83, 84, 32, 47, 97, 112, ...],
'history': {
'index': 570,
'current_iteration_index': 570,
'mutator_id': 'a4b5f0c5-58ca-4d49-bae7-eb5eff53406d'
},
'fields': ['headers / accept / accept.value']
}, {
'iteration': 570,
'timestamp': 1657915040539,
'action_id': 'd87f71fa-3cca-46c9-a3a8-af5111ccbcaa',
'action_type': 'receive',
'message': {
'id': None,
'name': None
},
'length': 795,
'data': [72, 84, 84, 80, 47, 49, 46, 49, 32, 52, 48, 48, 32, 66, 97, 100, 32, ...]
},
...
]
The table below summarizes the common properties of the actions shown above.
| Property | Description |
|---|---|
| iteration | The value of the iteration property is identical for both actions, meaning they were performed as part of the same test case. You can also think about the iteration property as the test case number. |
| timestamp | The timestamp (in milliseconds) when the action was performed. |
| action_id | The unique ID of the action. |
| action_type | The type of the action. In the example shown above we see the details of two actions performed: a send and a receive action. |
| message | Identifier of the Message Templates involved in the execution of the action. As can be seen in the example, this information is only available for the send action as that is the only action operating using Message templates. |
| length | In case the action_type is send, it represents the length of the data sent. If the action_type is receive, it is the length of the data received. |
| data | A list of bytes as decimal values. It is the data sent or received, based on the action_type. |
The fields property of the send action is a list of strings. These are the full path of the Fields within the assigned Message template that were tested during the iteration.
The history property of the send action should be ignored as it is for internal use only, and its format and contents may change in the future without notice.
Get Last Send
The get_last method returns the last send action from the history. The following callback example prints the action details to the console.
def callback(context):
print(context.history.get_last())
An example, redacted output of the get_last method is shown below.
{
'iteration': 0,
'timestamp': 1657946903302,
'action_id': 'a4b5f0c5-58ca-4d49-bae7-eb5eff53406d',
'action_type': 'send',
'message': {
'id': '37aae2ed-1f5d-40b4-b380-4f7d4ee23d00',
'name': 'Login'
},
'length': 1422,
'data': [80, 79, 83, 84, 32, ...],
'history': {
'index': 0,
'current_iteration_index': 0,
'mutator_id': 'a4b5f0c5-58ca-4d49-bae7-eb5eff53406d'
},
'fields': []
}
The properties of the action is identical to those documented under the Fetch method.
Mutator
The instance of the test case generation engine (mutator) obtained via get_last_mutator() and get_next_mutator() expose the following methods.
| Method | Description |
|---|---|
| completed() | Returns whether all mutations completed (True) or not (False). |
| next() | Forces the currently tested Field to complete and move on to testing the next Field. |
| mutate() | Triggers one mutation of the current Field(s). This method does not return anything. |
| render() | Returns the rendered Message data. The data returned is a list of rendered Group fields on the root level of the Message. Each item in the list is a string. To get the final data to be sent, the list has to be joined together, e.g.: b"".join(list_items). |
As mentioned earlier, the get_next_mutator method allows accessing the test case generation engine of the last send action before the Callback in the Test Flow.
The Callback example below demonstrates how to get the data sent to the target by the most recently executed send action.
def callback(context):
next_mutator = context.history.get_last_mutator()
output = b"".join(next_mutator.render(peek=True))
print("LAST SENT", output)
If we wanted to do the same with the test case generation engine of the following send action, the only change required to the above example is to replace get_next_mutator with get_next_mutator.
The test case generation engine instance exposes the Root group via the message property, for example:
mutator = context.get_last_mutator()
root_group = mutator.message
The root_group in the example above exposes the following methods.
| Method | Description |
|---|---|
| mutating() | Returns the instances of the Fields currently tested as a list. |
| next() | Forces the completion of the currently mutating Fields and moves on to the next Field. |
| mutate() | Triggers one mutation of the current Field. This method does not return anything. |
| render() | Returns the rendered Message data. The data returned is a list of rendered Group fields on the root level of the Message. Each item in the list is of Python bytes. To get the final data to be sent, the list has to be concatenated, e.g.: b"".join(list_items). |
| search(name) | Get Field instance by the name of the Field. |
| search_by_id(id) | Get Field instance by the unique ID of the Field. |
An example of getting the rendered value of a specific Field (in this example called "test") handled by the mutator of the last send action is shown below.
def callback(context):
mutator = context.history.get_last_mutator()
field = mutator.message.search("test")
print(field.render(peek=True))
Exceptions
Callbacks can raise exceptions just like any Python code. There is a set of Engine-specific exceptions available via the guardara.sdk.exceptions module that Callbacks can use; these are listed below. Please note that the Engine handles any exception not listed in the table raised as an unexpected error, resulting in the termination of the test.
| Exception | Description |
|---|---|
| NextIterationException | When raised, the Engine ignores any remaining actions and continue from the next iteration. |
| TerminateException | When raised, the Engine terminates the test. The exception message is added to the test’s Activity Log. |
| PatternMatchFailConditionException | Raised when the response from the target matches a user-provided regular expression. When raised, The Engine stops test execution. |
| PatternMatchFailConditionWithReportException | Same as above but results in the generation of an issue entry within the test report. |
| PatternMatchNextConditionException | Raised when the response from the target matches a user-provided regular expression. When raised, the Engine ignore any remaining actions and moves to the next Field. |
| PatternMatchNextConditionWithReportException | Same as above but results in the generation of an issue entry within the test report. |
| PatternMatchSkipConditionException | Raised when the response from the target matches a user-provided regular expression. When raised, the engine ignores any remaining actions and moves to the next iteration. |
| PatternMatchSkipConditionWithReportException | Same as above but results in the generation of an issue entry within the test report. |
| PatternMatchPauseConditionException | Raised when the response from the target matches a user-provided regular expression. When raised, the Engine pauses test execution. |
| PatternMatchPauseConditionWithReportException | Same as above but results in the generation of an issue entry within the test report. |
| PatternMatchReportConditionException | Raised when the response from the target matches a user-provided regular expression. When raised, the Engine creates an issue entry within the report that includes the exception message. |
For example, the NextIterationException exception can be imported as shown below.
from guardara.sdk.exceptions import NextIterationException
Pattern Match Exceptions
The exceptions whose name starts with PatternMatch should be instantiated with a JSON object, for example:
PatternMatchFailConditionException({
"rule_name": "",
"message": ""
})
The rule_name allows setting an arbitrary text to identify the source of the exception. For example, when using standard response processing rules in GUARDARA, the rule_name is set to the name of the response processing rule that raised the exception.
The message is an arbitrary text to include as the Observation of the finding within the report.